IA have increasingly ignored robots.txt since the mid-2010s, going on at least e... | Hacker News

Hacker News .hnnew | past | comments | ask | show | jobs | submit

dredmorbius on Jan 31, 2024 | parent | context | favorite | on: Google search drops cache link from search results

IA have increasingly ignored robots.txt since the mid-2010s, going on at least eight years now:

<https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...>

<https://blog.archive.org/2016/12/17/robots-txt-gov-mil-websi...>

JoshTriplett on Jan 31, 2024 [–]

That's excellent news, thank you!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact