New format
How to protect a site from scraping: captcha, limits, honeypot, and common sense
Scrapers harvest prices, copy, reviews, and catalogs. You can’t fully “close” a site against a motivated collector, but you can make collection much costlier and cut damage to the server and SEO.
Below: practical defense layers and common mistakes. What scraping is and where ethical collection ends live in a neighboring piece; here — the site owner’s side. No instructions on defeating protection.
Why protect
Aggressive scraping copies content, pulls prices for dumping, floods forms, and loads the server. Sometimes the goal is competitor analytics (prices), sometimes auto-filling clones.
Priority: site availability for people and search engines, data integrity, less theft of unique content.
Captcha and behavioral scoring
Classic captcha on every step annoys. More modern — risk scoring (bot/human) and a challenge only when suspicious. Cookies/session reduce repeated checks for returning users.
Remember: paid captcha-solving services exist — one captcha alone won’t stop a determined collector. Combine with request limits.
Honeypot, IP, and limits
Honeypot: a hidden element a bot clicks or fills. The event is a log signal and a reason to tighten rules for that IP or session.
IP signals (hosting vs consumer ISP, PTR for known crawlers) help but break with proxies. Rate limit is more reliable: many URLs/sec from one address → throttle or a temporary ban. Separately tell a traffic spike from a DDoS.
When to cut access:
- anomalous RPS from one IP/subnet
- traffic that skips typical human patterns
- mass catalog crawl with no referrer / odd UA
- attacks on forms and the admin area
Services and the legal layer
CDN/WAF with bot management (Cloudflare and peers) take part of the load: limits, JS challenge, geo/ASN rules. Paid antibot tools make scraping costlier but don’t give absolute guarantees — names and plans change; pick for traffic and budget.
In the terms of use, ban automated collection. That doesn’t replace tech, but supports claims when content is copied. Also protect unique copy and photos with clone monitoring.
Practical minimum:
- rate limit on catalog and API
- monitoring 5xx and log anomalies
- honeypot on forms
- captcha/challenge by risk
- don’t cut search crawlers
- backups and catalog integrity checks
FAQ
Is captcha on every page a good plan?
Usually no: it hits UX and conversion. Better risk scoring and a challenge only on suspicious behavior.
Can I block all bots?
No. You need “good” search crawlers and preview services. Cut anomalous traffic, not all robotic traffic.
Does robots.txt help?
For well-behaved robots — yes. A malicious scraper ignores it; it’s not the only defense.
What is a honeypot?
A hidden bait (link/field) a human doesn’t see but a dumb bot touches. Helps detect — not a silver bullet.
Are CDN and WAF required?
For high load and frequent attacks — useful (limits, bot management). A small site often needs rate limit + log monitoring.
Does 100% protection exist?
No. The goal is to lower damage and cost for the attacker, plus legal and contractual measures for content.
Captcha on every page — and scrapers still hammer the catalog?
We’ll set rate limits, risk challenges, and a crawler allowlist — raise scrape cost without killing SEO.
Discuss the task
