Files
PocketVeto/frontend
Jack Levy 1afe8601ed chore: expand robots.txt — block AI scrapers and SEO bots
Add blocks for AI training crawlers (GPTBot, CCBot, Bytespider, anthropic-ai,
Google-Extended, PerplexityBot, YouBot, cohere-ai), SEO tool bots (AhrefsBot,
SemrushBot, DotBot, MJ12bot, BLEXBot), and /_next/ static chunks. Add
Crawl-delay: 10 for well-behaved bots.

Authored by: Jack Levy
2026-03-15 21:49:05 -04:00
..
2026-02-28 21:08:19 -05:00
2026-02-28 21:08:19 -05:00
2026-02-28 21:08:19 -05:00
2026-02-28 21:08:19 -05:00
2026-02-28 21:08:19 -05:00