Bypassing automated traffic detection

This should have been a fifteen-minute job. I needed to pull some data from an API regularly, so I figured I’d script it. Devtools showed me the exact request the page was making — payload structure, authentication headers, everything — and it all replayed cleanly in the browser. I exported it as a cURL command, pasted it into my terminal, and ran it.

That’s where it fell apart. Instead of the JSON I expected, I got a bare Cloudflare page with Ray ID and 403 status code. The requests were identical down to the last header, yet the browser got through and curl didn’t. I tried scripting the whole thing in Ruby instead, and got the exact same 403.

What happened is that Cloudflare had decided my request didn’t come from a human, and it made that call before my request ever reached the API. In this article, we will walk through how that detection works — from the naive checks to the genuinely clever ones — why standard HTTP libraries give themselves away instantly, and one way to look like a real browser at the layer that matters most.

The problem

Every request you send carries dozens of small signals about who sent it — the headers you set, the order they’re in, even the way your connection is encrypted. Bot detection is the art of reading those signals, and it’s built in layers: a few cheap checks anyone can dodge, sitting on top of one or two that are genuinely hard to fake. It’s worth taking them from the bottom up.

User-Agent checks

The oldest trick is to read the User-Agent header. A request that announces itself as curl/8.4.0 or python-requests/2.31.0 is obviously not Chrome, so the server rejects it. This stops nobody, because the header is just a string the client chooses to send:

curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 \
  (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" https://example.com

One flag and you’re “Chrome.” Because this is so trivial to defeat, no serious anti-bot system relies on it alone — but it’s still the first gate, and it filters out the laziest scrapers.

The next step up is to look at all the headers, not just one. Real browsers send a consistent, ordered set of headers — Accept, Accept-Language, Accept-Encoding, sec-ch-ua, sec-fetch-*, and so on — in a particular order, with particular values. A scripted client usually sends a much smaller set, in a different order, and often omits the sec-* hints entirely.

You can fix this too. It’s tedious, but you can copy a real browser’s headers verbatim and replay them in the right order. Plenty of scraping libraries do exactly this. So the arms race continues upward.

TLS fingerprinting — the check you can’t see

Here’s the one that catches people off guard. Every check above happens after a connection is established — they inspect headers, cookies, or page behavior. TLS fingerprinting happens during the TLS handshake, before a single byte of your HTTP request is sent.

When any client opens an HTTPS connection, the very first thing it sends is a ClientHello message describing how it wants to negotiate encryption: which TLS version, which cipher suites, which extensions, which elliptic curves, in which order. This message is sent in the clear, and — crucially — its exact shape is determined by the TLS library the client was built with, not by anything you set in your code.

This is the trap:

• curl is built on OpenSSL.
• Python’s requests, Ruby’s Net::HTTP, Node.js’s https module, Go’s net/http — each is built on its own TLS stack, and each produces a ClientHello that looks nothing like a browser’s.
• Chrome uses BoringSSL with its own very specific configuration.

So an OpenSSL ClientHello is trivially identifiable as non-browser traffic. You can set a perfect User-Agent, replicate every header in the right order, send the right cookies — and it doesn’t matter, because the fingerprint that gives you away was already on the wire before any of that.

This is the magic behind Cloudflare’s bot detection (and Akamai’s, and DataDome’s). When you get blocked by Cloudflare, you’ll often notice you receive a Cloudflare error page, not a response from the origin server. That’s the tell: your request never reached the application at all. Cloudflare sits in front of it, read your TLS handshake, decided “this is OpenSSL, not a browser,” and answered on the origin’s behalf. The API you were trying to reach has no idea you ever knocked.

And unlike the User-Agent, you can’t fix this from your code. The fingerprint is a property of the binary you’re running. To change it, you have to change the TLS stack itself.

How TLS fingerprinting works

To understand why the fingerprint is so stable — and so revealing — it helps to look at what’s actually in a ClientHello.

When a TLS handshake begins, the client sends a structured message containing, among other things:

• The TLS version it supports
• A list of cipher suites, in preference order
• A list of extensions (SNI, ALPN, supported groups, signature algorithms, etc.), in a specific order
• The elliptic curves / supported groups it offers
• The EC point formats

None of these are secret, and none of them are random. The TLS library picks them, and it picks them the same way every time. So if you take these fields, concatenate them in order, and hash the result, you get a short string that acts as a signature for “what kind of client is this.” That’s the core idea behind JA3 and its successor JA4, the two most common TLS fingerprinting schemes.

A JA3 fingerprint, for example, is an MD5 hash of a comma-joined list of the TLS version, cipher suites, extensions, elliptic curves, and point formats:

Hash that, and you get something like cd08e31494f9531f560d64c695473da9 — a fingerprint that’s identical across millions of real Chrome installs and completely different for curl, requests, or Net::HTTP.

This is what makes it such a powerful filter for defenders:

• It’s invisible to the client. There’s no header to inspect or response to read — the fingerprint is computed from a handshake your code doesn’t control.
• It’s expensive to fake. Spoofing it means reimplementing the handshake to match a browser byte-for-byte, which most HTTP libraries simply can’t do because their TLS stack isn’t built that way.
• It’s stable and well-known. Anti-bot vendors maintain allowlists of known browser fingerprints. If yours isn’t on the list, you’re presumed to be a bot.

JA4 refines JA3 — it’s more granular and harder to evade — but the principle is the same: your code never chooses this fingerprint, your TLS library does — and a library that isn’t a browser’s gives itself away every time.

Seeing it for yourself

You can use tls.peet.ws service to see TLS fingerprints parsed from your connection. Hit it from the command line like that:

curl -s https://tls.peet.ws/api/all | jq '.tls | {ja3_hash, ja4}'

Then open https://tls.peet.ws/api/all in your browser and compare the ja3_hash and ja4 values. They won’t match. Same machine, same network, same destination — but the handshake curl sends is unmistakably not the one Chrome sends.

The solution

If your TLS library is what gives you away, the solution is to swap it for one that handshakes like Chrome. That’s why I made coorl — a curl-compatible HTTP client that mimics Chrome’s TLS fingerprint instead of advertising the standard OpenSSL signature.

The approach is deliberately narrow. coorl doesn’t run a headless browser, doesn’t execute JavaScript, and doesn’t carry the weight of Chromium. It’s a small command-line client that does one thing the other tools can’t: it performs the TLS handshake the way Chrome does, so the ClientHello it puts on the wire produces a Chrome JA3/JA4 fingerprint. To Cloudflare and friends, the connection looks like it’s coming from a real browser at exactly the layer they inspect first.

Everything else stays familiar. coorl mirrors curl’s flags, so adopting it is mostly a matter of swapping the command name:

# Basic GET
coorl https://httpbin.org/get

# POST with JSON
coorl -X POST -H "Content-Type: application/json" \
  -d '{"key":"value"}' https://httpbin.org/post

# Send cookies
coorl -b "session=abc123; theme=dark" https://httpbin.org/cookies

That said, coorl is not a full drop-in replacement for curl. curl has accumulated decades of features and supports dozens of protocols and hundreds of flags; coorl deliberately implements only the most-used subset — the handful of flags over HTTP and HTTPS that cover the vast majority of everyday requests.

Also, it’s worth being clear about what this does and doesn’t solve. coorl gets you past TLS-layer detection — the JA3/JA4 check that blocks requests before they reach the origin. It is not a CAPTCHA solver and it doesn’t execute JavaScript challenges, so a site that gates content behind a full browser challenge will still need a real browser. But a surprising amount of “you have been blocked” comes down to nothing more than the TLS fingerprint, and for that class of problem a tool that simply sounds like Chrome is all you need.

Closing thought

If there’s a single idea to take away, it’s that the most powerful checks are the ones you can’t see or control from code. Headers, user agents, cookies — all of it can be forged, so detection keeps sinking to layers the client doesn’t control, and right now that’s the TLS handshake. Whether you’re scraping or defending, that’s where the real signal is.