Update repo stats app and leaderboard (#287)

* update repo stats app and leaderboard

* delete old functions

* fix actions

* fix spelling

* fix

* fix requirements

* fix requirements

* fix scraper

* fix gitignore

* fix something

* update requirements

* update scraper

* update leaderboard

* fix deployment setup

.github/workflows/repo_scraper.yml

@@ -1,35 +1,36 @@
 name: Run repo scraper
 
 on:
-  #push:
-  #  branches: [main]
   #schedule:
   #  - cron: "0 0 * * *"  # Run at the end of every day
   workflow_dispatch: {}  # manual executions
 
 jobs:
   scrape:
-    name: Run repo scraper
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4
+      - name: Checkout code
+        uses: actions/checkout@v4
+
       - name: Set up Python
         uses: actions/setup-python@v5
         with:
           python-version: 3.11
+          cache: 'pip'
+          cache-dependency-path: setup.py
+
+      - name: Auth with GCP
+        uses: google-github-actions/auth@v2
+        with:
+          credentials_json: ${{ secrets.GCP_SA_KEY }}
 
       - name: Install dependencies
         run: |
-          python -m pip install --upgrade pip
-          pip install -r tools/repo_scraper/requirements.txt
+          pip install -r tools/repo_stats/requirements.txt
           pip list
 
       - name: Run repo scraper
         env:
-          DROPBOX_TOKEN: ${{ secrets.DROPBOX_TOKEN }}
-          DROPBOX_APP_KEY: ${{ secrets.DROPBOX_APP_KEY }}
-          DROPBOX_APP_SECRET: ${{ secrets.DROPBOX_APP_SECRET }}
-          DROPBOX_REFRESH_TOKEN: ${{ secrets.DROPBOX_REFRESH_TOKEN }}
-          GH_TOKEN: ${{ secrets.GH_TOKEN }}
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: |
-          python tools/repo_scraper/repo_scraper.py
+          python tools/repo_stats/scraper.py

.gitignore

@@ -156,10 +156,13 @@ node_modules/
 latest_info.csv
 latest_repo_data.csv
 repo_data.csv
+group_info.csv
+repo_stats*.json
 student_repos/
 reviews.csv
 bert_sentiment_model.pt
 prediction*.json
+**/service_account_key.json
 
 # vscode
 .vscode/

.pre-commit-config.yaml

@@ -20,7 +20,7 @@ repos:
     hooks:
       # try to fix what is possible
       - id: ruff
-        args: ["--fix"]
+        args: ["--fix", "--unsafe-fixes"]
       # perform formatting updates
       - id: ruff-format
       # validate if all is fine with preview mode

reports/report.py

@@ -19,7 +19,7 @@ def no_constraints(answer, index) -> None:
 
 
 def length_constraints(answer, index, min_length, max_length) -> None:
-    """Either min or maximum length contrains for question."""
+    """Either min or maximum length constrains for question."""
     answer = answer.split()
     if not (min_length <= len(answer) <= max_length):
         warnings.warn(

s3_reproducibility/docker.md

@@ -158,6 +158,14 @@ beneficial for you to download.
     docker rm <container_id>
     ```
 
+    In general we recommend to use the `--rm` flag when running a container e.g.
+
+    ```bash
+    docker run --rm <image>
+    ```
+
+    which will automatically remove the container after it has finished running.
+
 9. Let's now move on to trying to construct a Dockerfile ourselves for our MNIST project. Create a file called
     `trainer.dockerfile`. The intention is that we want to develop one Dockerfile for running our training script and
     one for doing predictions.
@@ -303,7 +311,7 @@ beneficial for you to download.
     to start the image in interactive mode:
 
     ```bash
-    docker run -it --entrypoint sh {image_name}:{image_name}
+    docker run --rm -it --entrypoint sh {image_name}:{image_tag}
     ```
 
 16. When your training has completed you will notice that any files that are created when running your training script