Skip to content

Documentation

Jump to bottom
fab edited this page Mar 5, 2024 · 9 revisions

Index

Generating the blacklist

I leverage the power of ChangeDetection (selfhosted) to monitor and incorporate updates from selected blacklists. These blacklists are downloaded hourly and aggregated into a single file using GitHub Actions.

In addition to that I run periodic in-depth reviews against the source blacklists and whitelist updates to ensure high quality informations.

Downloading the Blacklist

The blacklist file is automatically updated every hour and available for downloading at the following urls:

https://github.com/fabriziosalmi/blacklists/releases/download/latest/blacklist.txt

Implementing the Blacklist

PiHole

  1. Go to Adlists
  2. Add new adlist
https://github.com/fabriziosalmi/blacklists/releases/download/latest/blacklist.txt
  1. Go to Tools > Update Gravity and click the Update button

AdGuard Home

  1. Go to Filters
  2. Click on Add Blacklist
  3. Add custom blacklist and save
https://github.com/fabriziosalmi/blacklists/releases/download/latest/blacklist.txt

Squid

Squid configuration for blacklist

  1. Download and copy the blacklist to the squid folder
wget -O blacklist.txt https://github.com/fabriziosalmi/blacklists/releases/download/latest/blacklist.txt
cp blacklist.txt /etc/squid/conf.d/blacklist.txt
  1. /etc/squid/squid.conf additional configuration
acl blacklist dstdomain /etc/squid/conf.d/blacklist.txt     # <= blacklist location
http_access deny blacklist                                  # <= deny acl
  1. Restart squid
  2. Setup a cronjob to automatically update the blacklist

Squid configuration for block direct ip requests

If you're using Squid as an outgoing proxy and want to block direct IP requests (both HTTP and HTTPS) while only allowing client requests with host headers, you can achieve this by adding specific access control lists (ACLs) and http_access rules in your Squid configuration.

Here are the steps to configure Squid to achieve this:

  1. Edit the Squid Configuration File:

Open the Squid configuration file (squid.conf) in a text editor:

sudo nano /etc/squid/squid.conf
  1. Define ACLs for Requests with Host Headers:

Define an ACL for requests that have host headers:

acl with_host_header dstdomain . # Matches requests with a domain name
acl ip_request dstdom_regex ^\d+\.\d+\.\d+\.\d+$ # Matches requests with IP addresses
  1. Block Direct IP Requests:

Now, allow requests with host headers while denying those with direct IP addresses:

http_access deny ip_request
http_access allow with_host_header
  1. Other Required Access Controls:

You'll probably have other http_access lines in your configuration for various rules. Make sure that the order of these rules does not conflict with the rules you just added. In Squid, the first matching rule wins, so more specific rules should come before more general ones.

  1. Save and Restart Squid:

After making these changes, save the configuration file and restart Squid to apply the changes:

sudo systemctl restart squid

With these changes, Squid will deny requests made directly to IP addresses and will only allow requests with host headers. Ensure you test the configuration after applying the changes to make sure it works as intended and to identify if there are any other conflicting rules.

Linux, Windows, OSX and any device with a browser able to install the uBlock Origin extension

  1. Open the browser and go to the uBlock Origin dashboard by clicking on the extension icon > settings icon
  2. Go to the end of the page and in the Import form paste this url
https://github.com/fabriziosalmi/blacklists/releases/download/latest/blacklist.txt
  1. Click on the Apply Changes button in the top of the page
  2. You will find the blacklist in the Custom list at the end of the page, before the Import form
  3. You can force blacklist refresh from the same page when needed

Linux (nftables)

!! WARNING: use at your own risk !!

  1. Download this script to your Linux box
  2. Give permission and execute
chmod +x nft_blacklist_fqdn.sh && ./nft_blacklist_fqdn.sh

Integrate your whitelist

You can open an issue to whitelist specific domains.

If you want to keep your whitelist secret you can use this script which use a whitelist.txt file to filter out domains from blacklist.txt file:

#!/bin/bash

# Remove comments and empty lines, then trim whitespace
clean_file() {
    grep -v '^#' "$1" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//' | grep -v '^$'
}

# Use temporary files to store cleaned contents
clean_file blacklist.txt > /tmp/cleaned_blacklist.txt
clean_file whitelist.txt > /tmp/cleaned_whitelist.txt

# Remove domains from blacklist that are present in the whitelist
grep -vFxf /tmp/cleaned_whitelist.txt /tmp/cleaned_blacklist.txt > cleaned_blacklist.txt

# Cleanup
rm /tmp/cleaned_blacklist.txt /tmp/cleaned_whitelist.txt

Steps:

  1. The clean_file function processes the input files to remove comments (lines starting with #), empty lines, and trims whitespace.
  2. The grep -vFxf command is used to remove exact lines from the blacklist that match any line in the whitelist.
  3. Temporary files are cleaned up at the end.

To use this script:

  1. Save it as, say, clean_blacklist.sh.
  2. Make it executable: chmod +x clean_blacklist.sh.
  3. Run it: ./clean_blacklist.sh.

After execution, you'll have a file named cleaned_blacklist.txt with the domains from the blacklist.txt after removing any that appear in whitelist.txt.

Privoxy

#!/bin/bash

# URL of your remote blacklist.txt
BLACKLIST_URL="https://github.com/fabriziosalmi/blacklists/releases/download/latest/blacklist.txt"

# Path to Privoxy's user.action file
USER_ACTION_FILE="/etc/privoxy/user.action"

# Path to the log file
LOG_FILE="/var/log/privoxy_blacklist_update.log"

# Fetch the latest blacklist and format it for inclusion in user.action
echo "$(date): Updating Privoxy blacklist from $BLACKLIST_URL..." >> "$LOG_FILE"

# Fetching blacklist
formatted_blacklist=$(curl -s "$BLACKLIST_URL" | sed 's/^/.&/' | sed 's/$/& /')

# Check if curl command failed, or if the fetched blacklist is empty
if [[ $? -ne 0 || -z "$formatted_blacklist" ]]; then
  echo "$(date): Failed to update blacklist. Either fetch failed or the fetched list is empty." >> "$LOG_FILE"
  exit 1
fi

# Create a backup of the current user.action file
if ! cp "$USER_ACTION_FILE" "$USER_ACTION_FILE.bak"; then
  echo "$(date): Failed to create a backup of user.action file." >> "$LOG_FILE"
  exit 1
fi

# Generate a new user.action file
awk -v bl="$formatted_blacklist" '/{ +block{Blacklisted domain} }/{print;print bl;next}1' "$USER_ACTION_FILE.bak" > "$USER_ACTION_FILE.tmp"

# Check if awk command failed
if [[ $? -ne 0 ]]; then
  echo "$(date): Failed to generate new user.action file." >> "$LOG_FILE"
  exit 1
fi

# Replace old user.action file with the new one
if ! mv "$USER_ACTION_FILE.tmp" "$USER_ACTION_FILE"; then
  echo "$(date): Failed to replace old user.action file with the new one." >> "$LOG_FILE"
  exit 1
fi

# Restart Privoxy to apply changes
if ! service privoxy restart; then
  echo "$(date): Failed to restart Privoxy service." >> "$LOG_FILE"
  exit 1
fi

echo "$(date): Privoxy blacklist update completed successfully." >> "$LOG_FILE"

Important Notes:

  1. Permissions and Paths:

    • Make sure the script has the proper permissions and is executable.
    • Ensure the script is run by a user with sufficient privileges to read and write the Privoxy configuration and restart the service.
    • The log file path (/var/log/privoxy_blacklist_update.log) should be writable by the user running the script.

  2. Testing:

    • Test this script in a safe environment before deploying it in production to avoid any service disruption.
    • Check the log file for any errors during the execution of the script.

  3. Scheduling:

    • Schedule the script to run at the desired frequency using cron or another task scheduler. The provided crontab example runs it hourly.

  4. Securing:

    • Secure the log file by setting appropriate file permissions to prevent unauthorized access, as it might contain sensitive information.

CoreDNS

  1. Setup Environment for CoreDNS Plugin Development:

    • Make sure you have Go installed on your machine.
    • Clone the CoreDNS repository: git clone https://github.com/coredns/coredns

  2. Create Your Plugin:

    • Navigate to the plugin directory of the cloned repository.
    • Create a new directory for your plugin, e.g., cd plugin && mkdir blacklist.
    • In this directory, create a basic structure of files for your plugin, such as setup.go (for setup logic) and blacklist.go (for core logic).

  3. Develop the Plugin:

    • Your plugin needs to periodically download the blacklist, parse it, and then use the FQDNs to block queries.
    // blacklist.go

package blacklist

import (
    "github.com/coredns/coredns/request"
    "github.com/miekg/dns"
    "net/http"
    "time"
)

type Blacklist struct {
    Next       plugin.Handler
    Blacklist  map[string]bool
    URL        string
}

func (b *Blacklist) ServeDNS(ctx context.Context, w dns.ResponseWriter, r *dns.Msg) (int, error) {
    state := request.Request{W: w, Req: r}
    domain := state.Name()

    if _, found := b.Blacklist[domain]; found {
        m := new(dns.Msg)
        m.SetRcode(r, dns.RcodeNameError)
        w.WriteMsg(m)
        return dns.RcodeNameError, nil
    }

    return plugin.NextOrFailure(b.Name(), b.Next, ctx, w, r)
}

func (b *Blacklist) Name() string {
    return "blacklist"
}

func (b *Blacklist) UpdateBlacklist() {
    for {
        resp, err := http.Get(b.URL)
        if err != nil {
            // Handle error, maybe with some logging.
            continue
        }

        newBlacklist := make(map[string]bool)
        scanner := bufio.NewScanner(resp.Body)
        for scanner.Scan() {
            line := scanner.Text()
            newBlacklist[line] = true
        }
        b.Blacklist = newBlacklist

        resp.Body.Close()
        time.Sleep(24 * time.Hour)  // Update every 24 hours.
    }
}
    // setup.go

package blacklist

import (
    "github.com/coredns/coredns/core/dnsserver"
    "github.com/coredns/coredns/plugin"
    clog "github.com/coredns/coredns/plugin/pkg/log"
    "github.com/mholt/caddy"
)

var log = clog.NewWithPlugin("blacklist")

func init() {
    plugin.Register("blacklist", setup)
}

func setup(c *caddy.Controller) error {
    b := &Blacklist{
        URL: "https://github.com/fabriziosalmi/blacklists/releases/download/latest/blacklist.txt",
    }

    go b.UpdateBlacklist()

    dnsserver.GetConfig(c).AddPlugin(func(next plugin.Handler) plugin.Handler {
        b.Next = next
        return b
    })

    return nil
}

  4. Update CoreDNS to Recognize Your Plugin:

    • Add your plugin to plugin.cfg in the root of the CoreDNS repo: blacklist:github.com/coredns/coredns/plugin/blacklist.

  5. Compile CoreDNS with Your Plugin:

    • From the root of the CoreDNS repository, run: make

  6. Configuration:

    • In your Corefile, simply use blacklist.

  7. Run Your Custom CoreDNS Build:

    • Use the built binary to start CoreDNS.

Docker

A Docker image is available to run a blacklist mirror on your own environment.

  • docker pull fabriziosalmi/blacklists:latest
  • docker run -p 80:80 fabriziosalmi/blacklists
  • The blacklist will be available at http://$DOCKER_IP/blacklist.txt
  • To update the blacklist restart the container
Clone this wiki locally