The real reasons sites detect scrapers — browser fingerprints, request patterns, TLS signatures — and the practical fixes that actually work.
This is the single most common message I get. Someone builds a scraper with Requests or Selenium, it runs beautifully for a few days, and then it starts returning CAPTCHAs, empty pages, or 403 errors.
The site did not necessarily change. Your scraper simply accumulated enough signals for the detection system to make up its mind.
Here is what those signals actually are, in rough order of how often they cause the problem.
Humans browse irregularly. They read a page for eleven seconds, then four, then ninety. A default scraper fires requests every 0.4 seconds, forever, with machine precision.
The fix: randomised delays with a realistic distribution — not "sleep(1)" but "sleep(uniform(2, 7))", plus occasional longer pauses. Slower scraping that finishes is infinitely faster than aggressive scraping that gets banned on page 300.
Modern anti-bot systems run JavaScript in your browser and collect dozens of properties: canvas rendering output, WebGL vendor strings, installed fonts, screen dimensions, timezone, hardware concurrency, and audio context behaviour.
Plain Selenium leaks its identity immediately — most notably through the "navigator.webdriver" flag, which is set to "true" by default and is the first thing every detection script checks.
The fix: stealth-patched browsers. "undetected-chromedriver" for Python, or "playwright-stealth", patch the most obvious tells. They are not magic, but they clear the low bar that catches 80% of naive scrapers.
Before a single byte of HTTP is exchanged, your client performs a TLS handshake — and the exact order of cipher suites and extensions it offers forms a fingerprint (often called a JA3 hash).
Python's "requests" library has a JA3 signature that looks nothing like Chrome's. You can set a perfect Chrome User-Agent header and still be identified as a Python script before the server even reads your headers.
The fix: use "curl_cffi" or a similar library that impersonates real browser TLS stacks, or drive an actual browser.
Datacenter IP ranges — AWS, DigitalOcean, Hetzner — are catalogued and scored. Some sites block them outright regardless of behaviour.
The fix, in ascending order of cost:
Rotate sensibly. A single IP making 10,000 requests is obvious; so is an IP that changes on every single request while carrying the same session cookie.
Real browsers send a specific, ordered set of headers: "Accept", "Accept-Language", "Accept-Encoding", "Sec-Fetch-*", and "sec-ch-ua" client hints. They are internally consistent — a Chrome 120 User-Agent comes with Chrome 120's exact header set.
Scrapers frequently send a Chrome User-Agent with three headers and no client hints. That mismatch is trivially detectable.
The fix: copy the complete header set from real browser DevTools, and keep it consistent with the User-Agent you claim.
Advanced systems watch what happens *after* the page loads. Did the mouse move? Was there any scrolling? Did focus events fire? Did the visitor request the CSS and images, or only the HTML?
The fix: for the hardest targets, simulate movement and scroll before interacting, and let the page load its resources rather than blocking them all for speed.
Solving services exist and cost roughly $1–$3 per thousand solves. They work.
But a CAPTCHA is a symptom, not the disease. If you are solving thousands of them, your scraper has already been flagged and you are paying a tax on bad fingerprinting. Fix the signals above and the CAPTCHAs largely stop appearing.
Start cheap and only climb when you have to:
1. Plain HTTP requests with realistic headers and rate limiting
2. Add "curl_cffi" for TLS impersonation
3. Add rotating residential proxies
4. Switch to a stealth-patched real browser
5. Add behavioural simulation
6. Add CAPTCHA solving as a last resort
Most projects never need to go past step three. Teams get into trouble by starting at step four, burning money on browser infrastructure for a site that would have accepted plain requests with a two-second delay.
Anti-bot work is not a one-time fix — protections update, and a scraper that is unmaintained is a scraper that will eventually break. Every system I build includes failure alerting, so you find out from an email rather than from a silently empty spreadsheet.
Contact me at sam@autosmartcode.com and tell me which site is blocking you. I'll tell you what it will take to get through it reliably.