← pybrws

Making Python impersonate Firefox

Part one: a 403 my headers could not explain, and what it took to work out why.

Ernesto Pérez Costa · October 2026 · ~11 min

I write scrapers for a living, which in practice has meant reaching for whichever library is currently winning. A target starts refusing you, you swap requests for something with a browser mode, it works again, you move on. That worked for years and I never once asked what the library was doing differently.

What finally bothered me was how easy the refusals are to reproduce. Take a URL that a plain Python client gets a 403 on. Open the same URL in Firefox on the same machine over the same connection: it loads. Copy the browser's headers out of devtools, all of them, in order, lower-cased, until a field-by-field diff of the two requests comes back empty.

Still 403.

Which is the interesting part, because it means the server is not reacting to anything in the request. It decided earlier. So what did I send before the request that gave me away?

Opening Wireshark

I had never looked at a TLS handshake before. The filter is tls.handshake.type == 1. Load a page in Firefox and there it is: the very first thing the browser sends, before anything else, 1,901 bytes of it.

There is no HTTP in that packet. No URL, no User-Agent, no cookies. What is in it is a list of cipher suites the browser supports, a list of extensions in a particular order, and a lot of crypto key material:

total                       1,901 bytes
  fixed header                 76
  15 cipher suites, in order
  17 extensions, in order     232   (all of them except the two below)
    key_share               1,327   70% of the whole message
    encrypted_client_hello    282

Then I ran the same filter against my Python script, and the two messages did not look remotely alike. Different ciphers, a different number of them, different extensions, a different order.

This turns out to be well known to everyone except me. There is a service, tls.peet.ws, that tells you what it saw. Both of these are measured, not copied from anywhere:

Firefox 157   t13d1517h2_8daaf6152771_3cbfd9057e0d
httpx         t13d1712h1_ab0a1bf427ad_8e6e362c5eac
              │   │ │ │
              │   │ │ └ first and last char of the first ALPN entry
              │   │ └ extension count   (17 vs 12)
              │   └ cipher count       (15 vs 17)
              └ TLS 1.3, SNI present

That is a JA4 fingerprint. There is an older one called JA3 that does the same job with an MD5. Either way the server computes it from the handshake, looks it up, and can decide everything before you have asked for anything.

So my scraper was introducing itself as a Python script in the first packet it ever sent, and no amount of header copying was ever going to help.

First idea: just configure Python's ssl

My first thought was that this must be a settings problem. Python lets you set the cipher list, so I set it to Firefox's. Wireshark confirmed the ciphers now matched.

Still 403. And the fingerprint still did not match.

The extension count was wrong, the extension order was wrong, and there is no API for either one. OpenSSL emits its extensions in the order OpenSSL emits them. It is not trying to be Firefox and there is no flag that asks it to.

I spent a while looking for a way around this before accepting there isn't one from Python. You can patch OpenSSL — that is roughly what curl-impersonate does, and if you just want this to work, go use it or curl_cffi and stop reading. I wanted to know how it worked, so I took the other route and wrote the message myself.

Getting a capture I could trust

To copy a ClientHello you need one, and you need it not to have been touched on the way. So: a tiny TLS server on localhost with a self-signed certificate, open it in Firefox, click through the warning, and dump the raw first packet to a hex file.

That sounds like overkill when Wireshark is right there, but having the bytes in a file is what makes the rest of this possible — every claim below is a diff against that file, and the tests still run against it today.

Which bytes am I allowed to change?

Before you can copy something you have to know which parts of it are supposed to vary. Five captures in a row, same conditions, all 1,901 bytes:

A fixed extension order was the best news in this project. It means "identical to the capture" is a target you can actually hit, rather than a distribution you have to imitate. Chrome is not like this at all, which is the next post.

The 1,327-byte extension

Seventy percent of the message is one field, key_share, and my first attempt at it was to paste the captured bytes into my client. Why not? They were valid bytes from a real browser.

The handshake ran all the way to the end and then failed, with no error saying which part was wrong.

This took me an embarrassingly long time to understand, so here it is plainly. key_share is the client's half of a key exchange. The server takes it, combines it with its own half, and both sides end up with the same secret without ever sending it. If I paste a capture, I am sending a public key whose private half I do not have. The server does its side correctly. I cannot do mine. Everything after that is encrypted with keys that do not match, and TLS does not tell you which byte was wrong — it just fails the final check.

Firefox sends three of these:

The big one is post-quantum, and 1,216 is 1184 + 32: an ML-KEM-768 key first, an X25519 key second. I got that order wrong on the first try and it is not guessable, you just compare against the capture until the lengths line up. Both keypairs have to be generated fresh on every connection:

self._x25519[group] = x25519.X25519PrivateKey.generate()
self._mlkem[group]  = mlkem.MLKEM768PrivateKey.generate()

# the share is just the two public halves, in that order
return (self._mlkem[group].public_key().public_bytes_raw()
        + self._x25519[group].public_key().public_bytes(*_RAW))

One more thing that confused me: Firefox advertises five groups but only sends shares for three. That is not a bug. The server is allowed to come back and say "none of those, use this other one you mentioned", so you have to be able to produce a share for all five even though you normally send three. The asymmetry itself is part of the fingerprint.

The extension I assumed was noise

Extension 65037 is 282 bytes that change completely between captures, and for a day I treated it as random padding. It is not. It is encrypted_client_hello, and decoding it across the five captures gives this:

ECH exists because the SNI — the field naming which site you want — is the last part of TLS still sent in the clear. ECH encrypts an inner hello carrying the real name, against a key the server publishes in DNS.

But if only ECH users sent the extension, sending it would itself be the signal. So browsers send a decoy of exactly the right shape whether or not they are doing real ECH. Firefox does this by default. This is the decoy.

One assumption I did check rather than assume: a real ECH payload contains the inner hello, which contains the real hostname, so a longer hostname might make it longer. Tried it with a 66-character name. The hello grew by exactly the 43 bytes the name added and the ECH payload stayed at 240. So the constant is safe for any host.

Then my own browser started lying to me

Partway through, I re-checked Firefox against tls.peet.ws, expecting the reference I had been building against. I got this:

stored reference  t13d1517h2_8daaf6152771_3cbfd9057e0d
my client now     t13d1517h2_8daaf6152771_3cbfd9057e0d
the browser now   t13d2713h2_bd9791dd9fd6_ecd0401ec68b

My client matched the reference. The browser did not match either. Which made no sense for about ten minutes, until I looked at what the browser was supposedly sending: twenty-seven ciphers instead of fifteen, every one of them listed twice, which Firefox's TLS library never does. encrypt_then_mac, which is an OpenSSL thing. renegotiation_info before server_name, the reverse of Firefox's order. And none of the five extensions I had been using all along to tell Firefox's stack from Chrome's.

The HTTP/2 layer was rebuilt too, different stream id, different window sizes. Something was terminating both layers and re-originating them.

It was NordVPN's Threat Protection. It installs a local proxy and a root certificate to inspect HTTPS, and it runs while the VPN is disconnected. Having the app open was enough.

I still find this one a bit much. It is ordinary consumer software, sold as a security feature, and what it does is indistinguishable from a malicious proxy: it replaces the TLS and HTTP fingerprint of every request you make. Nothing broke. No warning appeared. Every page rendered normally. The only reason I saw it at all is that this project's whole point is asking a server what it actually received.

If you ever capture reference fingerprints, check this first. On a machine running one of these you are not measuring your browser, you are measuring your antivirus. The captures I already had predated it and still carry the real markers, so I kept them and did not re-capture anything until the certificate on an ordinary site was issued by a public CA again.

Where it got to

ours   1901 bytes
real   1901 bytes

version ........... match
cipher suites ..... match   15, same order
extensions ........ match   17, same order
payloads .......... match   byte for byte

random ............ differs must (32 random bytes)
session id ........ differs must (32 random bytes)
key_share ......... differs must (fresh keypair)
ECH payload ....... differs must (drawn per connection)

Same ja4 as the capture. Which is a weaker sentence than it sounds, and it took me a while to admit why: the capture is mine, the parser that reads it is mine, and the server that reports what arrived is also mine. Three instruments, one author. Writing a thing and then grading it with a rig you built yourself is how you convince yourself and nobody else.

Asking someone else

So, two outside sources. The first is thumbprint.me, which reflects the caller's own fingerprints back as JSON at /api/v1/probe — someone else's edge, someone else's parser, across a real network path. It computes:

t13d1517h2_8daaf6152771_3cbfd9057e0d

Character for character what my own server reported. That is the number going external, and it is the single most useful thing in this post.

The better part is a thing I did not know the service did. It keeps captures of real browsers, taken on its own machines, with client, version and OS recorded. The probe tells you how many of those recorded sessions share your fingerprint:

TLS ClientHello ..... 29 sessions
HTTP/2 settings ..... 11 sessions
request headers ..... 10 sessions

So the handshake does not merely match my Firefox. It collides with other people's Firefoxes, on other machines, and it does so at three layers independently.

Their catalogue is published, so the collision can be looked at rather than trusted. 849 sessions, 133 of them Firefox across 32 versions. This client's ja4 appears in thirty of them:

firefox 157.0     linux
firefox 156.0.1   windows
firefox 156.0     windows, linux
firefox 155.0.1   macos, windows
firefox 155.0     linux
firefox 154.0.1   macos, linux
firefox 154.0     windows, linux

Including a 157, which is the version this profile claims to be. And the same hash on Windows, macOS and Linux — so the ClientHello carries no trace of the operating system. The thing that does carry it is the TCP handshake underneath, which no library writes and which reported Windows about me correctly without being asked. If you run a Windows profile from a Linux box, that is a contradiction nothing above the kernel can fix.

Four versions, three operating systems, one fingerprint. Firefox's hello does not move per release, which is the part that makes modelling it worth the trouble at all.

The second source is the only one that actually pays rent: servers that did not ask to be measured. Ten public sites serve this client and refuse a mature Python HTTP/2 client, same address, same headers, same rounds, interleaved so that neither arm gets a quiet minute to itself:

zillow  ticketmaster  indeed  yelp  homedepot
macys  sephora  stubhub  quora  seatgeek

Three to four rounds each, no variance in any cell, across Cloudflare, Akamai, DataDome and CloudFront. crunchbase.com rides along in every round as a control: it serves both arms, so a run where this client passes everything would show up as a broken experiment rather than a triumph.

And the four I threw out, which is the part that makes the other ten worth anything. redfin.com answered 405 Method Not Allowed, which my own helper scored as a pass because it only tested for 403, 429 and 503. expedia, vrbo and wayfair had the plain client at 429, which is rate limiting and not a judgement about the client — and quite possibly caused by my own scan. Not evidence, not kept.

What this does not do

Now the limit, which is not a detail.

A JavaScript challenge — the "checking your browser" interstitial — does not care about your handshake. It challenges a real Chrome too. A real Chrome gets through by running the script, which is a capability, not a disguise. No ClientHello on earth skips that step.

What a fingerprint decides is one layer earlier: whether you get the page or get sent to the challenge at all. That is a real difference and it is worth the work, but if someone tells you their TLS library defeats bot protection, that is the part they are not telling you.

There is also a site that simply wins. g2.com refuses this client on /, on /categories/crm, on /products/slack/reviews, with or without a query string. The one URL it does serve, /?x=, it serves to the plain Python client too — so that gap is not earned and I have no idea what it is. Why one path slips through is not observable from outside and I am not going to guess at it in public.

Also: I got the ClientHello right, and the ClientHello is one message. The rest of the handshake has its own behaviour, HTTP/2 has an entirely separate fingerprint, and the same browser does something different again over QUIC.