robots.txt Compliance
The cluster focuses on discussions about whether web crawlers, especially Google, respect and honor robots.txt files, their legal enforceability, and alternatives for controlling site scraping.
Activity Over Time
Top Contributors
Keywords
Sample Comments
curious, will the robots.txt be really honored? maybe legal issue if not?
Most robots don't honour robots.txt anyways...
Does Google have a track record of not respecting robots.txt? Otherwise why is it a problem?
In practice robots.txt is just a suggestion; even Google doesn't fully respect it.
Just because you put something in robots.txt doesn't mean they will adhere to it.
Naive question, do people no longer respect robots.txt?
you know about robots.txt right?
You'd prefer they didn't honor robots.txt at all?
Respecting robots.txt is probably the best plan.
Google respects robots.txt files from webmasters who don't want Google to scrape their content