# robots.txt for markgrid.ai # # Policy: fully open to search and AI-answer crawlers. MarkGrid's problem is # under-exposure, not over-exposure. Nothing here should be tightened until # the site is actually being cited. # # Note on Google-Extended and CCBot in particular: blocking either is the # single most common self-inflicted GEO wound. Google-Extended gates Gemini # grounding, and Common Crawl feeds a large share of every open training # corpus. Both stay allowed. # --------------------------------------------------------------------------- # Default # --------------------------------------------------------------------------- # Only the three paths that are not content stay closed. Everything a human # can read, a crawler can read. # # The tracking-parameter rules that used to live here (utm_, fbclid, gclid) # were removed deliberately. Blocking ?utm_source=... in robots.txt does not # prevent duplicate indexing — it prevents the crawl that would have found # the pointing back at the clean URL. So a shared or # linked campaign URL became uncrawlable, and its ranking signals were # stranded instead of consolidated onto the canonical. Every page here ships # a self-referencing canonical, which is the mechanism that actually handles # parameter duplicates. User-agent: * Allow: / Disallow: /api/ Disallow: /admin/ Disallow: /studio/ # --------------------------------------------------------------------------- # OpenAI # GPTBot -> model training corpus # OAI-SearchBot -> ChatGPT Search index (this is the one that drives citations) # ChatGPT-User -> live fetch when a user's prompt triggers browsing # --------------------------------------------------------------------------- User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / # --------------------------------------------------------------------------- # Anthropic # --------------------------------------------------------------------------- User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: anthropic-ai Allow: / # --------------------------------------------------------------------------- # Google # --------------------------------------------------------------------------- User-agent: Googlebot Allow: / User-agent: Googlebot-Image Allow: / User-agent: Google-Extended Allow: / User-agent: GoogleOther Allow: / User-agent: Google-CloudVertexBot Allow: / # --------------------------------------------------------------------------- # Perplexity — fastest citation feedback loop, highest priority # --------------------------------------------------------------------------- User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # --------------------------------------------------------------------------- # Microsoft / Bing — also powers Copilot # --------------------------------------------------------------------------- User-agent: bingbot Allow: / User-agent: BingPreview Allow: / User-agent: msnbot Allow: / # --------------------------------------------------------------------------- # Meta, Amazon, Apple, ByteDance, Mistral, You.com, Common Crawl, others # --------------------------------------------------------------------------- User-agent: meta-externalagent Allow: / User-agent: Meta-ExternalFetcher Allow: / User-agent: FacebookBot Allow: / User-agent: Amazonbot Allow: / User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / User-agent: Bytespider Allow: / User-agent: MistralAI-User Allow: / User-agent: YouBot Allow: / User-agent: CCBot Allow: / User-agent: Diffbot Allow: / User-agent: cohere-ai Allow: / User-agent: Timpibot Allow: / User-agent: omgili Allow: / User-agent: AI2Bot Allow: / User-agent: ImagesiftBot Allow: / # --------------------------------------------------------------------------- # Other search engines # --------------------------------------------------------------------------- User-agent: Slurp Allow: / User-agent: DuckDuckBot Allow: / User-agent: DuckAssistBot Allow: / User-agent: Kagibot Allow: / User-agent: PetalBot Allow: / User-agent: Baiduspider Allow: / User-agent: YandexBot Allow: / # --------------------------------------------------------------------------- # Social preview crawlers # --------------------------------------------------------------------------- User-agent: Twitterbot Allow: / User-agent: facebookexternalhit Allow: / User-agent: LinkedInBot Allow: / User-agent: WhatsApp Allow: / User-agent: Slackbot Allow: / User-agent: Slackbot-LinkExpanding Allow: / # --------------------------------------------------------------------------- # Sitemaps. Regenerated on every build by scripts/generate-seo-files.mjs. # sitemap.xml is an index; the section files below are listed too, because # some agents read robots.txt and fetch one section rather than following an # index. sitemap-all.xml is the same inventory as one flat file. # --------------------------------------------------------------------------- Sitemap: https://markgrid.ai/sitemap.xml Sitemap: https://markgrid.ai/sitemap-pages.xml Sitemap: https://markgrid.ai/sitemap-modules.xml Sitemap: https://markgrid.ai/sitemap-solutions.xml Sitemap: https://markgrid.ai/sitemap-glossary.xml Sitemap: https://markgrid.ai/sitemap-articles.xml # --------------------------------------------------------------------------- # RSL 1.0 — machine-readable licensing terms for AI crawlers and training # pipelines. Permissive with an attribution condition: the goal is to be # trained on and cited, not to be quoted anonymously. # # Kept as a comment, not a `License:` line. That field is an RSL extension and # not part of the robots.txt grammar, so validators (Lighthouse among them) # report the whole file as invalid because of it — which costs more than the # directive gains. The licence is still discoverable: it is linked from every # page's and listed on /sitemap. # https://markgrid.ai/rsl.xml # --------------------------------------------------------------------------- # LLM-friendly content index — https://llmstxt.org/ # https://markgrid.ai/llms.txt # # Human- and agent-readable site index, every URL with a description: # https://markgrid.ai/sitemap