Anti-Web Scraping Measures
Discussions center on techniques to detect, block, and deter web scrapers and bots, especially aggressive AI crawlers, including rate limiting, Cloudflare protections, behavioral detection, and challenges in implementation.
Activity Over Time
Top Contributors
Keywords
Sample Comments
Did they imagine it would slow down scrapers/bots?
Maybe it’s to block ai scrapers
You mean like overloading their servers with needless scraping?
I am not well versed in this problem but can't the web servers rate limit by known IP addresses of these crawler/scrapers?
Does Cloudflare works for preventing scrapers?
Sounds like they have identified you as a crawler / scraper and don't want another Cambridge analytics incident.
Malicious how? No one is going to stop bots scraping the site so why penalize humans?
Wouldn't it be easy for e.g. Facebook to train a classifier on the behavior of your scrapers, and from there block you?
There is a multi-million dollar industry around blocking scrapers, it's not "just add rate limiting".
Did they just make us all human web scrapers?