HN2new | past | comments | ask | show | jobs | submitlogin

With what kind of sites do you see this as as a problem and what level of traffic do you get from these scrapers? There was a discussion [0] 11 days ago about this concerning a search engine, but for many other websites you can just pre-generate almost all of the content and use microcaching to cover anything that has to be dynamic to drive the per-request cost to almost 0.

[0] https://hackernews.hn/item?id=32339314



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: