Bot User-Agent Spoofing
The cluster centers on discussions about web crawlers and scrapers impersonating legitimate bots like Googlebot via user agents to evade detection, along with methods to identify and block such bots.
Activity Over Time
Top Contributors
Keywords
Sample Comments
Isn’t it somewhat likely that a lot of shady crawlers pretend to be a google bot with their user agent?
Did you try a Googlebot or Baidubot user-agent?
This seems like a reasonable thing to do to detect bots.
Yes. Bots do it all the time. GoogleBot, AhrefsBot, etc.
Meh, maybe they are catering to their main user base, the bot scrapers?
A huge amount of the web is only crawlable with a googlebot user-agent and specific source IPs.
Yeah I think something somewhere is saying 'damn, that's a suspect User Agent, must be a bot'.. like really?
How can you tell it works? How do you know the bot owners aren't returning to crawl those parts with User-Agent strings that impersonate ordinary users?
This sounds to me like someone is simply using GoogleBot in their request headers whilst probing.
So if the bots use a google useragent it avoids the links?