Website Link Checker

A Python CLI tool that crawls websites and checks for broken links. It respects same-domain constraints, implements rate limiting to be respectful to servers, and provides detailed reporting of link status.

This project was created to test olicorne.org with assistance from aider.chat.

• Comprehensive link checking: Crawls all internal links starting from a given URL
• Same-domain constraint: Stays within the target website by default
• External link checking: Optionally check external links without crawling them
• Rate limiting: Implements 1-second delay between requests to the same domain
• Pattern-based filtering: Ignore URLs matching regex patterns
• Retry logic: Automatically retries failed URLs and handles rate limiting (403/429)
• Detailed logging: Both console output and rotating log files
• Parent tracking: Shows which pages contain broken links for easier fixes
• PEP 723 compatible: Can be run directly with uv run

Installation

This script uses PEP 723 inline script metadata, so you can run it directly with uv:

uv run website_link_checker.py --url https://example.com

Alternatively, install dependencies manually:

pip install beautifulsoup4 requests click loguru

Usage

Basic usage:

uv run website_link_checker.py --url https://example.com

Options

• --url URL (required): Starting URL to check
• --ignore-regex PATTERN: Regex patterns to ignore URLs (can be specified multiple times)
• --allow-external: Also check external links (but don't crawl them)
• --user-agent STRING: Custom User-Agent string (default: Firefox 122 on Linux)

Examples

Check a website and ignore admin pages:

uv run website_link_checker.py \
  --url https://example.com \
  --ignore-regex ".*\/admin\/.*"

Check a website including external links:

uv run website_link_checker.py \
  --url https://example.com \
  --allow-external

Ignore multiple URL patterns:

uv run website_link_checker.py \
  --url https://example.com \
  --ignore-regex ".*\.pdf$" \
  --ignore-regex ".*\/private\/.*"

Output

The script provides:

1. Real-time console output: Shows each URL as it's checked with its status code
2. Detailed log file: website_link_checker.log with full debug information (rotates at 10 MB)
3. Summary report: Groups URLs by status code and shows all pages where broken links appear

Example output:

✓ [200] https://example.com/about
✓ [200] https://example.com/contact
✗ [404] https://example.com/old-page (found on https://example.com/about)
============================================================
CRAWL SUMMARY
============================================================
Total URLs checked: 42
Status 200: 38 URLs
Status 404: 4 URLs
  - https://example.com/old-page
    Found on 2 page(s):
      * https://example.com/about
      * https://example.com/sitemap

How It Works

1. Breadth-first crawl: Starts from the given URL and follows all internal links
2. Rate limiting: Enforces 1-second delay between requests to the same domain
3. URL normalization: Removes fragments (#section) to avoid duplicate checks
4. Error handling: Retries failed URLs with appropriate delays for rate limiting
5. Parent tracking: Maintains a record of which pages link to each URL for better error reporting

License

This project is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0).

Credits

Created with assistance from aider.chat, an AI pair programming tool.