Jack Levy 1afe8601ed chore: expand robots.txt — block AI scrapers and SEO bots ...

Add blocks for AI training crawlers (GPTBot, CCBot, Bytespider, anthropic-ai,
Google-Extended, PerplexityBot, YouBot, cohere-ai), SEO tool bots (AhrefsBot,
SemrushBot, DotBot, MJ12bot, BLEXBot), and /_next/ static chunks. Add
Crawl-delay: 10 for well-behaved bots.

Authored by: Jack Levy

2026-03-15 21:49:05 -04:00

831 B

Raw Blame History

View Raw