# Machine-readable content for AI agents: # index: https://revelraw.com/llms.txt # full site, one file: https://revelraw.com/llms-full.txt # any page as Markdown: replace .html with .md (or append .md to extensionless URLs) # Standard crawlers — unnamed bots (incl. future AI crawlers) keep full access User-agent: * Allow: / Disallow: /auth/ Disallow: /samples/ Disallow: /promo/ Disallow: /free-year/ Disallow: /black-friday/ # Search engines only: also blocked from the .md mirrors + llms-full.txt — # near-duplicates of the HTML pages, built for AI agents, never meant to rank # (~340 duplicate URLs; suspected factor in the Aug-17 re-rank). Only these # named engines are affected; every other bot falls under * above. User-agent: Googlebot Allow: / Disallow: /auth/ Disallow: /samples/ Disallow: /promo/ Disallow: /free-year/ Disallow: /black-friday/ Disallow: /*.md$ Disallow: /llms-full.txt User-agent: Bingbot Allow: / Disallow: /auth/ Disallow: /samples/ Disallow: /promo/ Disallow: /free-year/ Disallow: /black-friday/ Disallow: /*.md$ Disallow: /llms-full.txt User-agent: YandexBot Allow: / Disallow: /auth/ Disallow: /samples/ Disallow: /promo/ Disallow: /free-year/ Disallow: /black-friday/ Disallow: /*.md$ Disallow: /llms-full.txt # AI / LLM crawlers — explicitly allowed for GEO discoverability User-agent: GPTBot Allow: / Disallow: /auth/ Disallow: /samples/ Disallow: /promo/ Disallow: /free-year/ Disallow: /black-friday/ User-agent: ChatGPT-User Allow: / Disallow: /auth/ Disallow: /samples/ Disallow: /promo/ Disallow: /free-year/ Disallow: /black-friday/ User-agent: OAI-SearchBot Allow: / Disallow: /auth/ Disallow: /samples/ Disallow: /promo/ Disallow: /free-year/ Disallow: /black-friday/ User-agent: ClaudeBot Allow: / Disallow: /auth/ Disallow: /samples/ Disallow: /promo/ Disallow: /free-year/ Disallow: /black-friday/ User-agent: Claude-Web Allow: / Disallow: /auth/ Disallow: /samples/ Disallow: /promo/ Disallow: /free-year/ Disallow: /black-friday/ User-agent: anthropic-ai Allow: / Disallow: /auth/ Disallow: /samples/ Disallow: /promo/ Disallow: /free-year/ Disallow: /black-friday/ User-agent: PerplexityBot Allow: / Disallow: /auth/ Disallow: /samples/ Disallow: /promo/ Disallow: /free-year/ Disallow: /black-friday/ User-agent: Perplexity-User Allow: / Disallow: /auth/ Disallow: /samples/ Disallow: /promo/ Disallow: /free-year/ Disallow: /black-friday/ User-agent: Google-Extended Allow: / Disallow: /auth/ Disallow: /samples/ Disallow: /promo/ Disallow: /free-year/ Disallow: /black-friday/ User-agent: Applebot Allow: / Disallow: /auth/ Disallow: /samples/ Disallow: /promo/ Disallow: /free-year/ Disallow: /black-friday/ User-agent: Applebot-Extended Allow: / Disallow: /auth/ Disallow: /samples/ Disallow: /promo/ Disallow: /free-year/ Disallow: /black-friday/ User-agent: Bytespider Allow: / Disallow: /auth/ Disallow: /samples/ Disallow: /promo/ Disallow: /free-year/ Disallow: /black-friday/ User-agent: CCBot Allow: / Disallow: /auth/ Disallow: /samples/ Disallow: /promo/ Disallow: /free-year/ Disallow: /black-friday/ User-agent: meta-externalagent Allow: / Disallow: /auth/ Disallow: /samples/ Disallow: /promo/ Disallow: /free-year/ Disallow: /black-friday/ User-agent: Amazonbot Allow: / Disallow: /auth/ Disallow: /samples/ Disallow: /promo/ Disallow: /free-year/ Disallow: /black-friday/ User-agent: cohere-ai Allow: / Disallow: /auth/ Disallow: /samples/ Disallow: /promo/ Disallow: /free-year/ Disallow: /black-friday/ Sitemap: https://revelraw.com/sitemap.xml