Battling bots

I've already blocked entire countries to combat scrapers, I update my robots.txt for well-behaved bots (and 403 any included in the list that access anything other than my robots.txt). In addition to these steps, I've started blocking traffic from IPs contained in Spamhaus' DROP and DROPv6 lists.

The script I use to handle these lists and countries is as follows.

The script creates temporary ipsets, fetches fresh data from Spamhaus and IPdeny, then atomically swaps the sets so the live rules are never empty during an update.

#!/bin/bash

# Directories
BLOCK_DIR="/etc/firewall/blocklists"
mkdir -p "$BLOCK_DIR"
# Countries to block
COUNTRIES="cn ru ir kp vn"
echo "=== Starting firewall blocklist update ==="
# Create temporary ipsets
ipset create -exist spamhaus-drop-ipv4-temp hash:net family inet
ipset create -exist spamhaus-drop-ipv6-temp hash:net family inet6
ipset create -exist country-block-ipv4-temp hash:net family inet
# Download and process Spamhaus DROP lists (JSON format)
echo "Downloading Spamhaus DROP lists..."
echo "  - DROP IPv4"
curl -s "https://www.spamhaus.org/drop/drop_v4.json" -o "$BLOCK_DIR/drop_v4.json"
grep -o '"cidr":"[^"]*"' "$BLOCK_DIR/drop_v4.json" | cut -d'"' -f4 | while read -r ip; do
    ipset add -exist spamhaus-drop-ipv4-temp "$ip"
done
echo "  - DROP IPv6"
curl -s "https://www.spamhaus.org/drop/drop_v6.json" -o "$BLOCK_DIR/drop_v6.json"
grep -o '"cidr":"[^"]*"' "$BLOCK_DIR/drop_v6.json" | cut -d'"' -f4 | while read -r ip; do
    ipset add -exist spamhaus-drop-ipv6-temp "$ip"
done
# Download and process country blocks (IPv4 only)
echo "Downloading country blocklists (IPv4)..."
for country in $COUNTRIES; do
    echo "  - $country"
    curl -s "https://www.ipdeny.com/ipblocks/data/aggregated/${country}-aggregated.zone" -o "$BLOCK_DIR/${country}.zone"
    cat "$BLOCK_DIR/${country}.zone" | while read -r ip; do
        ipset add -exist country-block-ipv4-temp "$ip"
    done
done
# Create permanent sets if they don't exist
ipset create -exist spamhaus-drop-ipv4 hash:net family inet
ipset create -exist spamhaus-drop-ipv6 hash:net family inet6
ipset create -exist country-block-ipv4 hash:net family inet
# Swap temp sets with live ones atomically
ipset swap spamhaus-drop-ipv4-temp spamhaus-drop-ipv4
ipset swap spamhaus-drop-ipv6-temp spamhaus-drop-ipv6
ipset swap country-block-ipv4-temp country-block-ipv4
# Destroy temp sets
ipset destroy spamhaus-drop-ipv4-temp
ipset destroy spamhaus-drop-ipv6-temp
ipset destroy country-block-ipv4-temp
# Apply iptables rules if they don't exist
echo "Applying iptables rules..."
# Spamhaus DROP - IPv4
if ! iptables -C INPUT -m set --match-set spamhaus-drop-ipv4 src -j DROP 2>/dev/null; then
    iptables -I INPUT 1 -m set --match-set spamhaus-drop-ipv4 src -j DROP
fi
# Spamhaus DROP - IPv6
if ! ip6tables -C INPUT -m set --match-set spamhaus-drop-ipv6 src -j DROP 2>/dev/null; then
    ip6tables -I INPUT 1 -m set --match-set spamhaus-drop-ipv6 src -j DROP
fi
# Country blocking - IPv4
if ! iptables -C INPUT -m set --match-set country-block-ipv4 src -j DROP 2>/dev/null; then
    iptables -I INPUT 2 -m set --match-set country-block-ipv4 src -j DROP
fi
# Save iptables rules
netfilter-persistent save
# Report stats
SPAMHAUS_V4=$(ipset list spamhaus-drop-ipv4 | grep 'Number of entries' | awk '{print $4}')
SPAMHAUS_V6=$(ipset list spamhaus-drop-ipv6 | grep 'Number of entries' | awk '{print $4}')
COUNTRY_V4=$(ipset list country-block-ipv4 | grep 'Number of entries' | awk '{print $4}')
echo "=== Update complete ==="
echo "Spamhaus DROP IPv4: $SPAMHAUS_V4"
echo "Spamhaus DROP IPv6: $SPAMHAUS_V6"
echo "Country block IPv4: $COUNTRY_V4"
echo "Total blocked networks: $((SPAMHAUS_V4 + SPAMHAUS_V6 + COUNTRY_V4))"

With user agent, country and DROP mitigations in place, I've been able to keep the server this site sits on performant and accessible. The script above runs daily to keep my block lists up to date.