GitHub actions collection

Upload to S3

You can use GitHub Actions to automatically download the latest release asset from your GitHub repository and then upload the extracted file to an S3 bucket.

First, you'll need to add your AWS credentials as GitHub secrets to allow GitHub Actions to upload to your S3 bucket. Navigate to your GitHub repository, then go to "Settings" > "Secrets" and add:

• AWS_ACCESS_KEY_ID: Your AWS access key ID.
• AWS_SECRET_ACCESS_KEY: Your AWS secret access key.

Here's an example GitHub Action configuration, saved in a .yml file under .github/workflows in your repository:

name: Download Latest Release and Upload to S3

on:
  schedule:
    - cron: '0 * * * *' # Run this job every hour

jobs:
  upload-to-s3:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout code
      uses: actions/checkout@v3

    - name: Get the latest release asset URL
      id: latest_release
      run: |
        RELEASE_URL=$(curl -s -H "Authorization: token ${{ secrets.GITHUB_TOKEN }}" \
            "https://api.github.com/repos/fabriziosalmi/blacklists/releases/latest" | \
            jq -r '.assets[].browser_download_url | select(contains("all.fqdn.blacklist.tar.gz"))')
        echo "RELEASE_URL=$RELEASE_URL" >> $GITHUB_ENV

    - name: Download the latest release asset
      run: |
        curl -L ${{ env.RELEASE_URL }} -o all.fqdn.blacklist.tar.gz

    - name: Extract the archive
      run: |
        tar xzf all.fqdn.blacklist.tar.gz blacklist.txt

    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: us-east-1  # Change this to your desired AWS region

    - name: Upload to S3
      run: |
        aws s3 cp blacklist.txt s3://your-bucket-name/blacklist.txt --acl public-read

In this example:

• The action is triggered every hour by the schedule event. You can adjust this frequency as needed.
• The job consists of several steps:
  • Checkout the code (required for some of the following steps)
  • Get the latest release asset URL of all.fqdn.blacklist.tar.gz from the GitHub API.
  • Download the latest release asset.
  • Extract blacklist.txt from the downloaded tar.gz file.
  • Configure AWS credentials.
  • Upload blacklist.txt to your S3 bucket and set it to public read access.

Note that this example uses jq to parse the JSON response from GitHub's API. GitHub's Ubuntu runners include jq by default.

Replace your-bucket-name with the name of your S3 bucket.

You can read more about each of the actions used in this workflow in their respective repositories:

• actions/checkout
• aws-actions/configure-aws-credentials

This should give you a GitHub Action that downloads the latest all.fqdn.blacklist.tar.gz, extracts blacklist.txt, and uploads it to an S3 bucket.