# Statlign — robots.txt # LLM/AI guidance: https://statlign.com/llms.txt # Agent API (MCP) docs: https://statlign.com/docs/mcp # All content is public and crawlable. Predictions are visible to search # engines, news crawlers, and AI/LLM agents to attract new users. # Private/account-only routes below are intentionally excluded from indexing. # --- Search engines --- User-agent: Googlebot Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /auth Disallow: /*/auth Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation User-agent: Googlebot-News Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /auth Disallow: /*/auth Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation User-agent: Googlebot-Image Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /auth Disallow: /*/auth Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation User-agent: Google-InspectionTool Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /auth Disallow: /*/auth Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation User-agent: Storebot-Google Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /auth Disallow: /*/auth Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation # --- Google AdSense / Ad crawlers --- # Mediapartners-Google crawls pages to select relevant ads; AdsBot checks # landing-page quality. Both need full access to every public page, and both # must skip account-only routes so ad crawl budget isn't wasted on gated views. User-agent: Mediapartners-Google Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation Disallow: /auth Disallow: /*/auth User-agent: AdsBot-Google Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation Disallow: /auth Disallow: /*/auth User-agent: AdsBot-Google-Mobile Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation Disallow: /auth Disallow: /*/auth User-agent: AdsBot-Google-Mobile-Apps Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation Disallow: /auth Disallow: /*/auth # Crawl-delay is only honoured by the engines below. Google ignores it (and can # flag it in validation), so Googlebot / AdsBot are deliberately left without # one — their crawl rate stays managed by Google. User-agent: Bingbot Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /auth Disallow: /*/auth Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation Crawl-delay: 1 User-agent: Slurp Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /auth Disallow: /*/auth Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation Crawl-delay: 1 User-agent: DuckDuckBot Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /auth Disallow: /*/auth Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation User-agent: Baiduspider Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /auth Disallow: /*/auth Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation Crawl-delay: 2 User-agent: YandexBot Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /auth Disallow: /*/auth Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation Crawl-delay: 2 # --- Social / link preview crawlers --- User-agent: Twitterbot Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /auth Disallow: /*/auth Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation User-agent: facebookexternalhit Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /auth Disallow: /*/auth Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation User-agent: LinkedInBot Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /auth Disallow: /*/auth Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation # --- AI / LLM agents (allowed so predictions surface in AI search) --- User-agent: GPTBot Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /auth Disallow: /*/auth Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation User-agent: OAI-SearchBot Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /auth Disallow: /*/auth Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation User-agent: ChatGPT-User Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /auth Disallow: /*/auth Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation User-agent: PerplexityBot Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /auth Disallow: /*/auth Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation User-agent: ClaudeBot Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /auth Disallow: /*/auth Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation User-agent: Claude-Web Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /auth Disallow: /*/auth Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation User-agent: Google-Extended Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /auth Disallow: /*/auth Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation User-agent: CCBot Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /auth Disallow: /*/auth Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation User-agent: Applebot Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /auth Disallow: /*/auth Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation User-agent: Applebot-Extended Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /auth Disallow: /*/auth Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation # --- Everyone else --- User-agent: * Allow: / Allow: /ads.txt Allow: /app-ads.txt Allow: /llms.txt Allow: /robots.txt Allow: /sitemap-index.xml Allow: /3b50eb7b51eb46dea369f989aff1e903.txt Disallow: /auth Disallow: /*/auth Disallow: /reset-password Disallow: /*/reset-password Disallow: /settings Disallow: /*/settings Disallow: /wallet Disallow: /*/wallet Disallow: /upload Disallow: /*/upload Disallow: /admin Disallow: /*/admin Disallow: /match-curation Disallow: /*/match-curation Crawl-delay: 1 # --- Sitemaps --- # sitemap-index.xml is the authoritative entry point: it lists every generated # shard (all locale sitemaps, all per-sport/per-league segment sitemaps and # every Google News shard). The files below are also listed directly so # crawlers that ignore sitemap indexes still reach the main feeds and the # locale shards for the Canada / Africa growth markets (English Canada/Nigeria/ # South Africa/Kenya/Ghana routes live in the main sitemap; French Canada is # served by the fr shard). Sitemap: https://statlign.com/sitemap-index.xml Sitemap: https://statlign.com/sitemap.xml Sitemap: https://statlign.com/news-sitemap.xml Sitemap: https://statlign.com/sitemap-fr.xml # Plain-text assets crawlers fetch directly (ads.txt, llms.txt, verification, robots). Sitemap: https://statlign.com/sitemap-files.xml