chore: expand robots.txt — block AI scrapers and SEO bots

Add blocks for AI training crawlers (GPTBot, CCBot, Bytespider, anthropic-ai,
Google-Extended, PerplexityBot, YouBot, cohere-ai), SEO tool bots (AhrefsBot,
SemrushBot, DotBot, MJ12bot, BLEXBot), and /_next/ static chunks. Add
Crawl-delay: 10 for well-behaved bots.

Authored by: Jack Levy

frontend/public/robots.txt

@@ -16,3 +16,47 @@ Disallow: /following
 Disallow: /collections
 Disallow: /alignment
 Disallow: /api/
+Disallow: /_next/
+
+Crawl-delay: 10
+
+# AI training crawlers
+User-agent: GPTBot
+Disallow: /
+
+User-agent: CCBot
+Disallow: /
+
+User-agent: Bytespider
+Disallow: /
+
+User-agent: anthropic-ai
+Disallow: /
+
+User-agent: Google-Extended
+Disallow: /
+
+User-agent: PerplexityBot
+Disallow: /
+
+User-agent: YouBot
+Disallow: /
+
+User-agent: cohere-ai
+Disallow: /
+
+# SEO tool crawlers
+User-agent: AhrefsBot
+Disallow: /
+
+User-agent: SemrushBot
+Disallow: /
+
+User-agent: DotBot
+Disallow: /
+
+User-agent: MJ12bot
+Disallow: /
+
+User-agent: BLEXBot
+Disallow: /