Bug report

Bug description:

urllib.robotparser.RobotFileParser.parse() raises a bare ValueError and aborts parsing of the entire robots.txt file when a Crawl-delay or Request-rate directive has a value written with a non-decimal Unicode digit.

from urllib.robotparser import RobotFileParser
rp = RobotFileParser()
rp.parse(["User-agent: *", "Crawl-delay: ²"])   # U+00B2 SUPERSCRIPT TWO

Actual:

ValueError: invalid literal for int() with base 10: '²'

Request-rate: ²/5 fails the same way.

Expected: the malformed directive is silently ignored, like every other malformed line in a robots.txt file. Allow and Disallow already do try/except ValueError: pass, and an invalid value such as Crawl-delay: pears or Request-rate: 9/banana is already ignored (there are existing tests for those). One bad value should not abort parsing of the whole file.

Root cause: parse() guards the int() conversion with str.isdigit(), but str.isdigit() returns True for non-decimal Unicode digits (superscripts, subscripts, circled digits, ...) that int() then rejects. str.isdecimal() is the predicate that matches the decimal digits int() accepts.

Tested on 3.13, 3.14, and main.

CPython versions tested on:

3.13, 3.14, CPython main branch

Operating systems tested on:

macOS

Linked PRs

• gh-153404: Silently ignore non-decimal digits in robots.txt Crawl-delay and Request-rate #153405
• [3.15] gh-153404: Silently ignore non-decimal digits in robots.txt Crawl-delay and Request-rate (GH-153405) #153830
• [3.14] gh-153404: Silently ignore non-decimal digits in robots.txt Crawl-delay and Request-rate (GH-153405) #153831
• [3.13] gh-153404: Silently ignore non-decimal digits in robots.txt Crawl-delay and Request-rate (GH-153405) #153832