Indeed, I've got a site which gets a lot of bot traffic and google bot is pretty... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

remus 35 days ago | parent | context | favorite | on: Cloudflare's new AI traffic options for customers

Indeed, I've got a site which gets a lot of bot traffic and google bot is pretty sensible compared to a lot of other mainstream bots.

weird-eye-issue 35 days ago [–]

Yes, it really does not make that many requests. In fact lots of site owners struggle with having it not crawl and index their site enough

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact