I have been seeing a lot of benchmarking stuff on the internet for everything, starting from hosting to hair dryers. BIG thanks to Claude and Codex.
But how good is that whole concept of Benchmarking? Especially in our scraping world.
My thought:
Benchmarking is more like a photograph, not a TRUE measurement
Not sure, if that statement applies to everything under the sun, but in a few industries I am familiar with benchmarking is more like a photograph with a timestamp.
It can be factually correct at THAT time.
Especially, things related to servers, networks and computers . Industries, products or services working on these things usually are multi-variate equations. And often new variables show up depending on your work.
How or Why I am SAYING this?
I am in the middle of building an opensource web scraping api that anyone can set up with one terminal command and start scraping HTML of websites open on the internet
As a part of testing my opensource web scraping api, I started with a very simple test: Scraping Wikipedia. the easiest scraphing task by design.
I started with this to just to test the curl, httpx and log them compare them to other sophisticated engines like Camoufox or Playwright.
I Know its a dead easy one. It was easy for me to with almost every engine I had in my scraper.
But the logs I had for these tests, the numbers, they were full of surprises. Atleast for my preconcieved ideas of scraping and scraping Wikipedia.
Let me paint the picture of how I did it it:
Date: 2026-11-03 — first run, n=3 (three fetches in under a second per engine). Target: en.wikipedia.org/wiki/Web_scraping
Date: 2026-11-04 — second run, n=5. Same setup: same IP, same router, almost the same internet speed.
And here are the results:
Wikipedia scrape benchmark: 7 engines, 3 runs over two days (en.wikipedia.org/wiki/Web_scraping)
Three things jumped out at me.
Run 2 and run 3 should look like run 1. But they don't.
httpx took 1345ms, then 232ms, then 317ms.
Same site, same laptop, two days apart. That's a 5x swing in a number most benchmarks present as a fact.
My first instinct was "my internet was slow on run 1." It wasn't, really.
The gap between run 2 and run 3 is only ~85ms. That's noise.
But, the 1345ms on R1 which is 5x bigger and the possible reasons could be different Wikipedia edge node, TLS negotiation, or something else was in the way.
I haven't decomposed latency into DNS/TCP/TLS/TTFB chunks yet, httpx exposes that timing and it would tell us exactly which hop ate the 1345ms. That's a planned run.
Here's the uncomfortable part for anyone publishing benchmarks: if I'd run the benchmark once and shipped it, my article would say 'httpx is 1.3 seconds on Wikipedia.' Run it on another day and it says '232 milliseconds.' Both are true, neither is the whole story.
The browsers are the calm ones, and for a reason.
camoufox: 16070, 16272, 15871ms. cookie_replay: 20168, 16745, 16540ms. Under 1% variance across all three runs.
That pattern is the thesis of this whole article in one number: what's expensive in a browser engine is the stuff the internet isn't involved in. Booting Firefox, loading the anti-detect profile, the fingerprint model, rendering the page, waiting for JS. That work takes ~16 seconds regardless of whether you're on fiber or a bad hotel connection. Bandwidth is a rounding error next to it.
Patchright tells the same story from the other side — it's a browser that got fast (~930ms) because it skips most of that boot work, and it still delivers the rendered page (258KB, like camoufox).
So the fast engines swing with the network. The slow engines are stable because they barely care about the network. That's the observation I want to keep, but it comes with a hedge: I only ran this on Wikipedia, and only 3 to 5 samples per engine.
Pass rate is the honest column. The medians hide the failures.
Look down the primp row: 67%, 40%, 60%. The medians look fast — 383ms, 230ms. But the pass rate tells you what's actually happening: Wikipedia rate-limited us on the fast engines. 429s, three of five requests in run 2. When HTTP engines fire in a burst, the site notices, and the burst is exactly what a benchmark does.
nodriver flips the same way: 100%, 40%, 100%. Its run-2 failures were transport errors — the CDP connection dying mid-scrape — and its byte count jumped from 234KB to 417KB across runs, which I've seen happen with CDP before. The pass rate column is where every one of those scars shows up. Median latency alone would have told you nodriver is fine.
So: for a benchmark, the smallest n that still shows you thumbs down when things fail is worth more than the prettiest median.
What this means if you're scraping Wikipedia at scale
Concrete advice from 35 logged requests:
-
Use httpx, expect 429s. It passed 100% every run, dropped to sub-second on warm days. But Wikipedia rate-limits bursts, and a benchmark is a burst. At 100 pages you'll hit it; at 1,000 pages you'll hit it a lot. Sleep between requests or pace your RPM.
-
Don't trust your first latency number. Budget with the slow day, not the fast one. The 1345ms run and the 232ms run are both Wikipedia.
-
The browsers are predictable in the worst way. camoufox at 16s/page is a dead certainty. If you need rendered HTML and not the 258KB, patchright gave us nearly the same bytes at ~40x less time.
-
Every failure reason is logged — 429s, transport errors, empty responses — so the pass column isn't guesswork. That's the column to watch.
Note: I haven't run this at 100, 500, or 1,000 pages yet. That's the stress test this article is heading toward — throughput, concurrency, where the knee is.
This article is a series now
I'm treating this as a living benchmark, not a one-off. Each run in this folder (articles/data/tier-tests/) is dated and numbered. This page gets a new row when I run it again, so the trend replaces the snapshot. That's the honest shape for this kind of data — logging it, not just reporting it.
If you're scraping Wikipedia or any rate-limited site and your numbers disagree with one of mine, or you've seen the 429 behavior flip — I want to hear it. That's half the point of printing this publicly.
Next run: stress test at 100 → 500 → 1,000 pages. It drops on this page when it ships.
Updates log
- Initial publish. Three runs of a Wikipedia scrape benchmark: 7 engines, pass rates, latency variance, and what single-run benchmarks get wrong.
- Just made the presentation better with very minor edits. No substnatial changes on numbers or facts