lospino.so

Abrade | Jul 9, 2026

Abrade After the Happy Path

Abrade v0.3.0 refreshes a 2017 URL-pattern crawler around transport success, content truth, and operational failure.

A restrained technical diagram of Abrade generating URL candidates across network boundaries.

In 2017, Abrade had a narrow premise: many web systems expose resources through patterned URLs, and a small command-line program can make those patterns visible. It was not a theory of the web. It was a tool for one kind of pressure: generate candidate paths, request them, and record which ones appear to exist.

That premise still holds, but it is less clean than it first looks. A patterned URL is only the beginning of the interface. Between a generated path and a useful record sit DNS names, TLS handshakes, virtual hosts, HTTP methods, redirects, status codes, shell pages, application defaults, cancellation pages, empty records, rate limits, transport errors, and operator mistakes.

A crawler that treats all of those as one question - did this URL work? - will eventually lie.

Abrade v0.3.0, published on July 4, 2026, is a refresh around that boundary. It is still a C++23 command-line crawler for probing URL patterns quickly. It still generates candidate resource paths and performs concurrent HTTP requests. By default, it uses HEAD. It can fetch bodies with GET. It supports TLS, SOCKS5 proxies, local Tor proxy routing, stdin-fed paths, body filters, conservative redirects, and dry-run generation.

The important change is not that the option list became longer. The important change is that the tool now treats the crawler as a set of separable claims. Transport success is not content truth. Content truth is not operational success. Operational failure should not be hidden inside a plausible-looking result set.

That is the rule behind the refresh.

The old premise met newer infrastructure

The original Abrade post assumed that the interesting part was path discovery. Given a URL shape, the tool could vary a token, request candidates, and keep the successful ones. That works best when the interface is simple: one host, one scheme, one status convention, and one application behavior.

Modern web infrastructure is not usually that simple. The first failure can arrive before HTTP exists.

One authorized diagnostic sample used a by-event URL shape with a numeric identifier range. The sample matters here not because of anything specific about that site’s records, but because it exposed two crawler mistakes that generalize.

First, HTTPS attempts failed before Abrade reached a meaningful application response. An openssl s_client check with -servername succeeded, which isolated the issue: the connection needed Server Name Indication. The failure was not evidence that the target was unreachable. It was evidence that the client had not presented the DNS host name during the TLS handshake, and the virtual host rejected or mishandled the connection without it.

Second, once SNI was added and transport succeeded, many 200 responses were still not useful records. A successful HTTP status said the server returned an HTTP response. It did not say the body contained the domain object the operator was looking for.

Those are different boundaries. The first is a transport boundary. The second is a content boundary. A crawler that collapses them produces clean output for the wrong reason.

TLS success is a protocol claim

Abrade v0.3.0 sends SNI when operating in TLS mode against DNS host names. That detail matters because HTTPS is not only encryption around HTTP. It is also a routing interface. Shared infrastructure often uses the name presented by the client to choose the certificate and virtual host behavior. Without SNI, the server may return a default certificate, reject the handshake, or route the request somewhere other than the intended application.

That kind of failure can look like a discovery failure from the outside. A generated path produces no useful response. The easy story is that the candidate did not exist. The actual mechanism may be that the crawler never spoke to the intended HTTPS endpoint.

The refresh also separates encrypted transport from verified identity. TLS mode can establish an encrypted connection. --verify adds platform CA-chain validation and DNS host-name verification. Those are related claims, but they are not the same claim. Encryption says the bytes are protected on the connection as established. Verification says the peer’s certificate chains to a trusted authority and matches the DNS host being requested.

That distinction is mundane, but crawler output depends on mundane distinctions. If a diagnostic run is meant to tell an operator which candidate paths exist on a known service, the operator needs to know whether failures came from path absence, handshake behavior, certificate validation, name mismatch, routing, or application response. Treating all of those as “not found” makes the result easier to read and harder to trust.

Abrade does not solve every TLS deployment oddity. It does not turn a crawler into a browser, a compliance scanner, or a certificate investigation suite. The supported claim is narrower: in TLS mode, DNS host names are sent as SNI; when --verify is requested, platform CA-chain and DNS host-name verification are added. That is enough to remove one observed false-negative class, and to make verification an explicit operator choice rather than an accidental side effect.

A status code is a transport signal

The older crawler shape encouraged a simple interpretation: 2xx means found. That is sometimes adequate. Static files often behave this way. A generated path either maps to a file or it does not. HEAD is cheap, the status code is informative, and storing the successful candidates is useful.

Applications are not static file systems. A web app can return 200 for an empty search result, a placeholder page, a no-event page, a canceled entry, a shell page hydrated by client-side JavaScript, a login wall, or a route whose header and footer render even when the domain object is missing. In those cases, 2xx is not false. It is just answering a different question.

Abrade v0.3.0 keeps the default HEAD behavior because it remains useful for a first pass. The change is in how the tool lets a run move beyond transport status. --contents writes body-only files rather than serialized HTTP response headers. That matters because body filters should operate on application content, not on a mixed artifact where protocol headers can accidentally satisfy or mask a pattern.

The filtering options make the second question explicit. --require and --require-regex keep bodies containing required evidence. --reject and --reject-regex remove bodies containing evidence of absence, shell output, or other known non-record states. --screen remains an alias for --reject, preserving the older term while making the mechanism clearer.

A compact diagnostic pass can start without touching the network:

abrade example.com '/items/{1:5}' --test

That asks only whether the generated candidates are shaped as expected. It does not claim that the resources exist. It does not test TLS. It does not inspect content. It verifies the generation side of the interface before the run creates traffic.

A content-oriented pass should make the application evidence explicit:

abrade example.com '/items/{1:1200}' \
  --contents \
  --out ./out/items \
  --require 'Item ID' \
  --reject 'No item found' \
  --verify \
  --init 10

The strings above are examples, not universal predicates. On a real system, the required and rejected terms should come from authorized knowledge of that application’s response bodies. The point is not that every crawler run needs those exact filters. The point is that a useful discovery result often needs a body-level predicate in addition to a status-level predicate.

A 200 response can be evidence that a candidate path reached an application route. It is not, by itself, evidence that the route contains the domain record under investigation.

Redirects are part of the interface

Redirect handling has the same problem. A crawler can follow redirects aggressively and appear more browser-like, but the result may stop describing the candidate path. It may describe a login page, canonical landing page, locale redirect, CDN default, or error route that every candidate reaches.

Abrade v0.3.0 keeps redirects conservative: they are opt-in and limited to the same scheme and same authority. That boundary is intentionally narrow. It allows a diagnostic run to accommodate ordinary in-place canonicalization without silently turning a resource probe into a cross-site walk.

This is not a moral claim about redirects. It is an evidence claim. If the question is whether https://example.com/items/0042 has a meaningful record, then a redirect to a different authority changes the question. Even a same-authority redirect should be treated as part of the observed mechanism, not as an invisible transport convenience.

Discovery tools are most useful when they preserve the shape of what happened. A redirect policy that is too permissive can increase apparent success while lowering interpretability.

Operational failure needs its own accounting

Crawler output is often read after the run is complete, which means the summary must carry enough information to challenge the result. Abrade v0.3.0 prints attempted requests, 2xx responses, non-2xx responses, filtered bodies, runtime errors, bytes written, elapsed seconds, requests per second, and MiB per second.

Those numbers are not decoration. They are a guard against overreading. If a run attempted 1,200 candidates and recorded 37 body files, that count means something different depending on whether there were zero runtime errors or 400 runtime errors. If filters removed most 2xx responses, that supports the claim that status alone was too broad. If bytes written are unexpectedly small, the content archive may not contain what the operator thinks it contains. If throughput collapses, the bottleneck may be network behavior, remote throttling, proxy behavior, local I/O, or configuration.

The exit statuses follow the same separation. Exit status 0 is used for help, --test, or a completed run without transport or runtime errors. Exit status 1 is used for a completed run with transport or runtime errors, or for an unexpected runtime failure. Exit status 2 is used for parser or option validation failure before the scrape begins.

That split makes automation less ambiguous. A completed run with some transport failures is not the same as an invalid command line. An invalid command line is not the same as a valid run whose application responses failed a content predicate. Scripts need those differences because repeated diagnostics tend to become scheduled diagnostics, and scheduled diagnostics tend to fail at the edges.

The build refresh is part of the interface

A maintenance release is not only about runtime behavior. For a C++ tool, the build and release surface is also an interface under stress.

Abrade v0.3.0 moves the project to a C++23 baseline and a target-based CMake shape with presets. Dependencies are described through a vcpkg manifest. Tests use Catch2 v3 and CTest. Abrade targets build with strict warnings and warnings-as-errors. Formatting, static analysis, coverage presets, and quality gates are part of the repository’s normal path rather than a separate memory exercise. The source layout now separates src/abrade core code from src/cli orchestration.

Those details are easy to file under housekeeping, but they change what can be trusted about future edits. A crawler sits at the boundary between local assumptions and remote behavior. Its own project structure should not add more ambiguity than necessary. The core library and the command-line orchestration should be separable. Build configuration should describe targets rather than rely on incidental global state. Tests should run through a standard harness. Release archives should be produced in a repeatable way and paired with checksums.

The v0.3.0 release is tag-based and publishes archives for Linux x64, macOS arm64, and Windows x64, along with SHA256SUMS:

That does not prove the tool is correct. Checksums do not prove the crawler’s model. A modern CMake layout does not prove its TLS behavior. Static analysis does not prove a body filter captures domain truth. The supported claim is more limited: the project now has a clearer build contract, a clearer release contract, and a repository layout that makes the runtime boundaries easier to inspect and maintain.

This matters because release mechanics can otherwise blur into product claims. A downloadable archive is evidence that something was built for a platform. It is not evidence that a probe is authorized, that a target can tolerate traffic, that a status code means a record exists, or that a content filter is semantically correct.

Documentation as boundary-setting

The refreshed documentation lives in the Abrade docs hub. It covers build setup, command-line use, networking behavior, responsible use, API and internal contracts, project history, layout, and releasing. Documentation cannot remove the need for judgment. It can, however, state which questions the tool is designed to answer.

Abrade is suitable for authorized resource discovery, migration checks, and repeatable diagnostics against systems you own or are allowed to test. It should not be used as encouragement for broad third-party scraping.

That boundary is not a legal flourish appended to an otherwise neutral mechanism. It is part of the mechanism. High-concurrency URL probing consumes someone’s infrastructure and can reveal information that was not intended to be enumerated. Even when the implementation is modest, the operational shape matters.

The right use case is a bounded question against an authorized system. Did a migration preserve expected paths? Which generated resources still return a body containing the expected marker? Are known absence pages being confused with records? Does a service behave differently when TLS verification is enabled? Does a redirect policy mask the result?

The wrong use case is treating the public web as an unbounded inventory surface.

The rule

The 2017 version of Abrade was built around a practical observation: patterned URLs are common enough to probe. The 2026 refresh keeps that observation but adds the parts that the clean story leaves out.

A discovery tool needs to separate three forms of evidence.

Transport truth asks whether a request reached an endpoint and what the protocol reported. TLS, SNI, certificate verification, request method, status code, redirects, proxy routing, and runtime errors live here.

Content truth asks whether the response body contains evidence of the domain object being sought. Body-only output, required markers, rejection markers, and regex filters live here.

Operational failure asks whether the run itself was complete enough to interpret. Attempt counts, filtered counts, runtime errors, throughput, bytes written, elapsed time, and exit status live here.

None of those layers replaces the others. A verified TLS connection can return a useless 200. A body filter can be well chosen but applied to an incomplete run. A clean exit can still encode a bad assumption about what the application body means. A release archive can be reproducible without making any claim about whether a target should be probed.

Abrade v0.3.0 is available from the release page, and the source remains in the Abrade repository. The documentation hub is the better starting point for build, CLI, networking, release, and responsible-use details. Use it for authorized discovery, migration checks, and repeatable diagnostics on systems you own or are allowed to test.

Related