HN2new | past | comments | ask | show | jobs | submitlogin

I believe the intent was that if robots.txt claims to disallow a section of a website, then French law would say that section is off-limits to visitors and visiting disallowed sections is punishable by law.

I doubt that this is a valid interpretation of the law. The robots.txt file simply mentions parts of the site that shouldn't be indexed, not which parts of a site shouldn't be viewed by the general public. You use a robots.txt file to prevent a search engine from following links that would enumerate all the possible dynamically-generated content on your site to conserve resources and to prevent junk results appearing for a search user.

Tangentially relevant: when you do have something you want indexed, it should probably be a static page that lives at a permanent URL. But never attempt to use a robots.txt file to "hide" sensitive data.



Off limits to robots, hence the name.

edit: I think you edited to clarify since I commented. Thanks.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: