Build in Public · Open Web Scraping API · November 2026

A Benchmark Is a Photograph, Not a Measurement

This is a just a thought, a byproduct of my testing procedures for my Opensource Web Scraping API . I tested 7 scraping engines, and on 2 different days with simple wikipedia page and I think one time benchmarking numbers are not reliable

Published November 5 2026

I have been seeing a lot of benchmarking stuff on the internet for everything, starting from hosting to hair dryers. BIG thanks to Claude and Codex.

But how good is that whole concept of Benchmarking? Especially in our scraping world.

My thought:

Benchmarking is more like a photograph, not a TRUE measurement

Not sure, if that statement applies to everything under the sun, but in a few industries I am familiar with benchmarking is more like a photograph with a timestamp.

It can be factually correct at THAT time.

Especially, things related to servers, networks and computers . Industries, products or services working on these things usually are multi-variate equations. And often new variables show up depending on your work.

How or Why I am SAYING this?

I am in the middle of building an opensource web scraping api that anyone can set up with one terminal command and start scraping HTML of websites open on the internet

As a part of testing my opensource web scraping api, I started with a very simple test: Scraping Wikipedia. the easiest scraphing task by design.

I started with this to just to test the curl, httpx and log them compare them to other sophisticated engines like Camoufox or Playwright.

I Know its a dead easy one. It was easy for me to with almost every engine I had in my scraper.

But the logs I had for these tests, the numbers, they were full of surprises. Atleast for my preconcieved ideas of scraping and scraping Wikipedia.

Let me paint the picture of how I did it it:

Note

Date: 2026-11-03 — first run, n=3 (three fetches in under a second per engine). Target: en.wikipedia.org/wiki/Web_scraping

Note

Date: 2026-11-04 — second run, n=5. Same setup: same IP, same router, almost the same internet speed.

And here are the results:

Engine
Run 1 (n=3) — pass / median
Run 2 (n=5) — pass / median
Run 3 (n=5) — pass / median
httpx
100% / 1345ms
100% / 232ms
100% / 317ms
curl_cffi
100% / 920ms
100% / 241ms
100% / 315ms
primp
67% / 677ms
40% / 383ms
60% / 230ms
camoufox
100% / 16070ms
100% / 16272ms
100% / 15871ms
patchright
100% / 1352ms
100% / 859ms
100% / 930ms
nodriver
100% / 3214ms
40% / 7557ms
100% / 2467ms
cookie_replay
100% / 20168ms
100% / 16745ms
100% / 16540ms

Wikipedia scrape benchmark: 7 engines, 3 runs over two days (en.wikipedia.org/wiki/Web_scraping)

Three things jumped out at me.

Run 2 and run 3 should look like run 1. But they don't.

httpx took 1345ms, then 232ms, then 317ms.

Same site, same laptop, two days apart. That's a 5x swing in a number most benchmarks present as a fact.

My first instinct was "my internet was slow on run 1." It wasn't, really.

The gap between run 2 and run 3 is only ~85ms. That's noise.

But, the 1345ms on R1 which is 5x bigger and the possible reasons could be different Wikipedia edge node, TLS negotiation, or something else was in the way.

Note

I haven't decomposed latency into DNS/TCP/TLS/TTFB chunks yet, httpx exposes that timing and it would tell us exactly which hop ate the 1345ms. That's a planned run.

Here's the uncomfortable part for anyone publishing benchmarks: if I'd run the benchmark once and shipped it, my article would say 'httpx is 1.3 seconds on Wikipedia.' Run it on another day and it says '232 milliseconds.' Both are true, neither is the whole story.

The browsers are the calm ones, and for a reason.

camoufox: 16070, 16272, 15871ms. cookie_replay: 20168, 16745, 16540ms. Under 1% variance across all three runs.

That pattern is the thesis of this whole article in one number: what's expensive in a browser engine is the stuff the internet isn't involved in. Booting Firefox, loading the anti-detect profile, the fingerprint model, rendering the page, waiting for JS. That work takes ~16 seconds regardless of whether you're on fiber or a bad hotel connection. Bandwidth is a rounding error next to it.

Patchright tells the same story from the other side — it's a browser that got fast (~930ms) because it skips most of that boot work, and it still delivers the rendered page (258KB, like camoufox).

So the fast engines swing with the network. The slow engines are stable because they barely care about the network. That's the observation I want to keep, but it comes with a hedge: I only ran this on Wikipedia, and only 3 to 5 samples per engine.

Pass rate is the honest column. The medians hide the failures.

Look down the primp row: 67%, 40%, 60%. The medians look fast — 383ms, 230ms. But the pass rate tells you what's actually happening: Wikipedia rate-limited us on the fast engines. 429s, three of five requests in run 2. When HTTP engines fire in a burst, the site notices, and the burst is exactly what a benchmark does.

nodriver flips the same way: 100%, 40%, 100%. Its run-2 failures were transport errors — the CDP connection dying mid-scrape — and its byte count jumped from 234KB to 417KB across runs, which I've seen happen with CDP before. The pass rate column is where every one of those scars shows up. Median latency alone would have told you nodriver is fine.

So: for a benchmark, the smallest n that still shows you thumbs down when things fail is worth more than the prettiest median.

What this means if you're scraping Wikipedia at scale

Concrete advice from 35 logged requests:

  • Use httpx, expect 429s. It passed 100% every run, dropped to sub-second on warm days. But Wikipedia rate-limits bursts, and a benchmark is a burst. At 100 pages you'll hit it; at 1,000 pages you'll hit it a lot. Sleep between requests or pace your RPM.

  • Don't trust your first latency number. Budget with the slow day, not the fast one. The 1345ms run and the 232ms run are both Wikipedia.

  • The browsers are predictable in the worst way. camoufox at 16s/page is a dead certainty. If you need rendered HTML and not the 258KB, patchright gave us nearly the same bytes at ~40x less time.

  • Every failure reason is logged — 429s, transport errors, empty responses — so the pass column isn't guesswork. That's the column to watch.

Note: I haven't run this at 100, 500, or 1,000 pages yet. That's the stress test this article is heading toward — throughput, concurrency, where the knee is.

This article is a series now

I'm treating this as a living benchmark, not a one-off. Each run in this folder (articles/data/tier-tests/) is dated and numbered. This page gets a new row when I run it again, so the trend replaces the snapshot. That's the honest shape for this kind of data — logging it, not just reporting it.

If you're scraping Wikipedia or any rate-limited site and your numbers disagree with one of mine, or you've seen the 429 behavior flip — I want to hear it. That's half the point of printing this publicly.

Next run: stress test at 100 → 500 → 1,000 pages. It drops on this page when it ships.


Updates log

  • Initial publish. Three runs of a Wikipedia scrape benchmark: 7 engines, pass rates, latency variance, and what single-run benchmarks get wrong.
  • Just made the presentation better with very minor edits. No substnatial changes on numbers or facts