# Cedarpool — robots + AI-crawler policy
#
# TDM RIGHTS RESERVATION (machine-readable): Cedarpool expressly reserves all
# rights in its works and its database (including the sui generis database
# right under Directive 96/9/EC) against text-and-data-mining within the
# meaning of Article 4(3) of Directive (EU) 2019/790. Automated extraction,
# mining, or use of this site's content for AI/ML training is prohibited.
# See /ai.txt and the Terms of Service. Licensed, entitlement-gated AI access
# for subscribers is available — contact us.
#
# SCOPE — THE BLOG IS OPEN (RJ ruling 2026-10-17). /dash/v2/blog is MARKETING
# copy, written to be read and cited, so it is deliberately OPEN to every
# crawler below — including the AI training crawlers. The reservation above is
# UNCHANGED for everything else: the reconciled data, the dashboards and the API
# stay reserved, and licensed AI access is still the commercial product.
#
# robots.txt resolves by LONGEST MATCH, so the `Allow: /dash/v2/blog` line in
# each block below wins over that block's blanket `Disallow: /` for the blog
# path only. Keep this file in step with:
#   /ai.txt                                  (same scope, documented)
#   deploy/Caddyfile.public + .private       (X-Robots-Tag must exclude the path)
#   deploy/cedarpool.com/_headers            (Cloudflare equivalent)

# --- AI / LLM crawlers: reserved everywhere EXCEPT the public blog ----------
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-Web
User-agent: Claude-User
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: PerplexityBot
User-agent: Bytespider
User-agent: meta-externalagent
User-agent: FacebookBot
User-agent: cohere-ai
User-agent: Diffbot
User-agent: omgili
User-agent: TimpiBot
User-agent: ImagesiftBot
Disallow: /
Allow: /dash/v2/blog

# --- ordinary search engines: allowed on the public (aggregate-only) site ----
# The internal pages are named here as well as blocked at the gate (they answer 403):
# robots.txt is the courtesy that stops a crawler asking at all. The list came from the
# copy this box served before 2026-10-23, when the policy amendment reached the
# deploy/static and deploy/cedarpool.com copies but never the one that is actually SERVED
# (dashboard/root/) — so the two were merged rather than one replacing the other.
User-agent: *
Disallow: /dash/v2/ops/
Disallow: /dash/ops/
Disallow: /dash/v2/admin.html
Disallow: /dash/v2/audit_report.html
Disallow: /dash/v2/owed.html
Disallow: /dash/v2/signoff.html
Disallow: /dash/v2/outstanding.html
Disallow: /dash/v2/grant_gaps.html
Disallow: /dash/v2/issuer_inconsistencies.html
Disallow: /dash/v2/nonaccrual_review.html
Disallow: /dash/status.html
Disallow: /-/

# The sitemaps. A sitemap a crawler cannot find is a file nobody reads, and
# neither of these was named here. The v2 one is generated by
# scripts/build_sitemap.py, which lists what is actually public by REMOVING
# everything the edge blocks, everything the ACCESS GATE refuses an anonymous
# crawler (policies/feature_visibility.json), and everything the navigation does
# not link to — so a page switched off or unlinked tomorrow leaves the sitemap
# tomorrow, rather than being advertised to crawlers as a URL that answers 403
# or 404. The host is www: the apex 301s, and a sitemap of redirects is a map of
# places the content is not.
Sitemap: https://www.cedarpool.com/sitemap.xml
Sitemap: https://www.cedarpool.com/dash/v2/sitemap.xml