# RedHen Labs robots.txt
# Last updated: 2026-09-17
#
# POLICY: ALLOW ALL, ATTRIBUTION REQUESTED.
# Two exceptions at the bottom, both blocked on BEHAVIOUR, never policy.
#
# Read, index, quote, summarize, cite — including for model training.
# The terms, and the attribution we ask for, are at
# https://rrw-ads.com/license and are published machine-readably as
# `creditText` / `license` / `usageInfo` in the schema.org JSON-LD on
# every article. We are not monetizing the writing; it exists to be
# found, so reach is worth more to us than restriction.
#
# ⚠️ Nothing here is enforcement. robots.txt states a preference and only
# well-behaved operators honour it. The attribution ask lives in the
# license + schema, not in this file.
#
# Authenticated + transactional routes (login, dashboard, settings) are
# kept out of indexes via X-Robots-Tag: noindex (Render header) AND
# <meta name="robots" noindex> (RobotsMeta in AppRouter.jsx).
# NOT via Disallow — Disallow blocks the crawl entirely, so the crawler
# never sees the noindex signal and the URL can still get indexed from
# links elsewhere. Allow the crawl; let noindex do the work.

Sitemap: https://rrw-ads.com/sitemap.xml

# ===================
# Search Engines
# ===================
User-agent: Googlebot
Allow: /
Crawl-delay: 1

User-agent: Bingbot
Allow: /
Crawl-delay: 2

# ===================
# AI crawlers — answer-time retrieval (these produce citations)
# (ChatGPT, Claude, Perplexity, Meta AI, Cohere)
# These fetch a page to answer a question and can link back, which is
# the use our license is written for.
# ===================
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Claude-Web
User-agent: PerplexityBot
User-agent: FacebookBot
User-agent: cohere-ai
Allow: /
Crawl-delay: 5

# ===================
# Model-training crawlers — ALLOWED, deliberately
# ===================
# Google-Extended and Applebot-Extended are TRAINING opt-out tokens, not
# search tokens: Googlebot and Applebot serve search and AI Overviews
# regardless of what these say. Allowing them permits Gemini / Apple
# Intelligence training.
#
# ⚠️ Understood trade: training cannot honour our attribution condition.
# A model that has absorbed this text into its weights has no source
# record and nothing to link. We accept that in exchange for presence in
# answers given without browsing. Already-ingested content cannot be
# withdrawn later; only future crawls are affected by a change here.
User-agent: Google-Extended
User-agent: Applebot-Extended
Allow: /
Crawl-delay: 5

# ===================
# Blocked — on BEHAVIOUR, not on policy
# ===================
# 🔴 THESE TWO ARE BLOCKED FOR WHAT THEY REQUEST, NOT FOR WHAT THEY DO
# WITH IT. Measured in Cloudflare AI Crawl Control on 2026-09-17, the
# traffic arriving under these two user agents asked for:
#
#     /ssl/server.key                      a TLS private key
#     /config/anthropic.json               an API key file
#     /@fs/home/ec2-user/.aws/config       AWS creds, via Vite's @fs path
#     /media../.azure/credentials          Azure creds, with .. traversal
#     /azure.json                          Azure service principal
#     /api/v1/config                       config dump
#
# Common Crawl does not request TLS private keys. These user agents are
# camouflage — they get allowlisted or ignored by nearly everyone, which
# is exactly what makes them useful cover. Same actor as the
# /proc/self/environ probes and the ~130 spoofed-Chrome auth failures
# from rotating Alibaba/Tencent /24s the same day.
#
# ⚠️ CCBot WAS UNBLOCKED EARLIER ON 2026-09-17 AND IS BLOCKED AGAIN.
# The policy reasoning for allowing it was sound — Common Crawl feeds
# training corpora, and we want the reach — but the traffic actually
# wearing that UA on this site is credential probing. Policy says allow;
# behaviour says no. Behaviour wins.
#
# 🔴 AND THIS FILE IS NOT WHAT STOPS THEM. Cloudflare's violations report
# shows both of these requesting paths already covered by `Disallow: /`
# (including on api.rrw-ads.com, whose robots.txt disallows everything).
# robots.txt is a declaration of intent; the enforcement is the Block
# Crawler toggle in Cloudflare → AI Crawl Control → Security. Keep this
# file honest anyway, so the intent is on the record.
User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

# ===================
# Default (all other crawlers) + CONTENT SIGNALS
# ===================
# `Content-Signal` is the machine-readable form of the policy at the top
# of this file, and the three keys are independent:
#
#   search    = may appear in search results
#   ai-input  = may be used as AI input for inference and grounding
#               ⭐ THIS is the one that produces citations with a link
#   ai-train  = may be used to train or fine-tune a model
#               ⭐ this one cannot cite by construction — see the note
#                 in the training group above; we allow it anyway
#
# All three are `yes` deliberately. Amzi 2026-09-17: he wants the
# exposure, the writing is not monetised anywhere, and the license at
# /license already grants commercial quoting and summarising in return
# for a named source and a linked canonical URL.
#
# ⚠️ WRITTEN HERE BY HAND ON PURPOSE. Cloudflare's Bot Preference Sync is
# switched ON for this zone and its dashboard reports Content Signals as
# "Declared" on all four hostnames — but nothing was being prepended:
# verified 2026-09-17 by fetching /robots.txt as a plain client AND as
# GPTBot, zero `Content-Signal` lines either way, and Cloudflare's own
# Agent Readiness scan agreed by marking the check INCOMPLETE. Two
# independent reads beat a toggle. → feedback_amazon_answer_is_the_truth
# If Sync ever does start prepending, expect a SECOND Content-Signal line
# above this one and reconcile them — do not assume this file is the only
# source.
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
Crawl-delay: 5