# See https://www.robotstxt.org/robotstxt.html for documentation # Allow all crawlers. # # Content-Signal grants the three uses explicitly rather than leaving them # unstated. Under contentsignals.org an omitted signal "neither grants nor # restricts", and some crawlers treat that ambiguity conservatively — so silence # is not the same as yes. We want the opposite of conservative: the compliance # guide and the resource articles exist to be found, and AI crawlers already read # this site around three times more than search engines do. # # search building a search index # ai-input feeding content into a model to answer a live question # ai-train training or fine-tuning models # use=full summarise and reproduce, the most permissive of the use extension # # Do NOT turn on Cloudflare's managed robots.txt to "tidy" this. Its default is # search=yes, ai-train=no, which would silently reverse the decision below. User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=yes, use=full Allow: / # Disallow authenticated/private areas Disallow: /my-stuff/ Disallow: /settings/ Disallow: /onboarding/ Disallow: /api/ Disallow: /admin/ # Sitemap location Sitemap: https://prism.guide/sitemap.xml # Structured summary for AI agents and LLM crawlers # https://llmstxt.org — AI crawlers read this site roughly 3x more than search # engines do, and nothing pointed them at it before. LLM-Content: https://prism.guide/llms.txt