# robots.txt for allatsea.net
# Last updated: 2026-08-28 — TEC path blocks retired (MP-043); they had begun
#                blocking our own events layer and the teardown's 301s.
#
# Search engines and citation/answer engines: full access (default wildcard)
# AI training crawlers: opt out (except GPTBot + ClaudeBot — see 2026-08-25 note below)
#
# Most training-crawler blocks below are duplicated at the Cloudflare edge
# (AI Crawl Control). These directives provide defense in depth and cover
# bots that aren't on Cloudflare's list.

User-agent: *
Disallow:

# === Events Calendar query-string blocks (added 2026-05-07) ===
#
# The Events Calendar plugin generated infinite-permutation archive URLs via
# AJAX filter parameters. ⚠ TEC ITSELF IS GONE (deleted 2026-08-28, MP-043) and
# these params are no longer produced by anything we run — the rules stay because
# the legacy URLs are still being PROBED, and a query string bypasses the nginx
# FastCGI cache, so each probe would otherwise reach PHP.
#
# The old note here said the canonical /events/, /venue/ and /organizer/ paths were
# handled by Yoast noindex. That is no longer the mechanism: those paths now 301 to
# /events/ via Redirection, and they are deliberately CRAWLABLE so the redirects can
# actually be followed. See the two retired blocks below.
#
# These Disallow rules cover the query-string variants that should never
# have been crawl targets in the first place. They were responsible for ~84%
# of GSC's "Crawled - currently not indexed" bucket (97,749 URLs as of
# 2026-05-03) and ~97% of "Duplicate without user-selected canonical".
#
# Pattern note: /*?*PARAM= means "any path, then '?', then any chars, then
# 'PARAM='." This catches the parameter whether it's first, second, or later
# in the query string. Without the second '*' we'd miss URLs like
#   /events/list/?tribe-bar-date=...&ical=1
# because 'ical=' isn't the first param after '?'.

Disallow: /*?*ical=
Disallow: /*?*outlook-ical=
Disallow: /*?*eventDisplay=
Disallow: /*?*tribe-bar-date=
Disallow: /*?*tribe_event_display=
Disallow: /*?*eventDate=
# === TEC path-based virtual archives — RETIRED 2026-08-28 (MP-043) ===
# WAS: Disallow: /events/  +  Allow: /events/$  +  Allow: /events/list/
#
# Written 2026-06-09 against The Events Calendar's infinite day/tag/category virtual
# archives. TEC is now DELETED — plugin files gone, tables dropped, and the whole
# permutation space answers a cheap 301 to /events/ from Redirection instead of an
# expensive uncached query. There is nothing left to trap.
#
# ⚠ AND IT HAD STARTED BLOCKING US. We built the new events engine on the SAME PATH,
# so `Disallow: /events/` was blocking all three basin landings —
# /events/caribbean/, /events/mediterranean/, /events/north-america/ — while those
# pages were meta-indexable AND listed in aas-events-calendar.xml. A sitemap that
# advertises URLs robots.txt forbids is a bug, not a policy. `Allow: /events/$`
# rescued only the hub root, which is exactly why it went unnoticed: the page anyone
# would spot-check was the one page still allowed.
#
# Nothing replaces these rules. The 301s are the correct signal for a retired URL —
# a Disallow prevents the redirect from ever being FETCHED, so the old URLs could
# never consolidate onto the hub. The query-string blocks above still cover the
# legacy ?ical= / ?tribe-bar-date= probes that keep arriving.

# === TEC organizer/venue archives — UNBLOCKED 2026-08-28 (MP-043 teardown) ===
# WAS: Disallow: /organizer/  +  Disallow: /venue/
#
# The 2026-08-03 block was right at the time and for a real reason: 1,732 requests in
# three days, 969 of them paginated, crawlers walking to /page/343/, each page an
# uncached expensive TEC query — 21x 504 + 8x 500 while doing it. SemrushBot was 856
# of those hits, then bingbot 88, Googlebot 49.
#
# That cost is gone with TEC. There are no venue or organizer archives left to
# paginate; both paths now answer a single cheap 301 to /events/.
#
# ⚠ THE BLOCK HAD TO GO, because it was silently defeating the teardown. A crawler
# that may not FETCH a URL never sees its redirect, so 116 URLs carrying ~1,245
# impressions/90d could never consolidate onto the hub — they would simply linger as
# stale entries. Disallow retires nothing; it only hides the instruction that would.
#
# ⚠ If deep-pagination abuse ever returns here, the answer is a Cloudflare rule
# (verified-bot-exempt), NOT a Disallow — the same lesson as the 410-vs-444 shield:
# block the abuser, never the instruction.

# === REST API: post-views-counter endpoint — added 2026-07-27 ===
# /wp-json/post-views-counter/view-post/<id> is a POST-only route. Bingbot
# discovers the URL from the plugin's inline JS and crawls it with GET, which
# returns rest_no_route 404s (613 hits / 341 distinct post IDs in 9 days).
# Nothing here is a crawl target; the posts themselves index normally.
Disallow: /wp-json/post-views-counter/


# === AI training crawlers — opt out ===

# GPTBot and ClaudeBot: ALLOWED as of 2026-08-25 (owner ruling) so ChatGPT and
# Claude can read and cite the site. Their citation/user-fetch siblings
# (ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot) were always allowed.
# PerplexityBot is allowed too (never blocked here). Note these two also feed
# model training at their vendors; the ruling accepts that trade.

# Older / alternate Anthropic UA
User-agent: anthropic-ai
Disallow: /

# Google-Extended ALLOWED 2026-09-05 (owner ruling, Chris): token governs Gemini grounding/training only, never Search or AI Overviews; training accepted as the price of citation, as for GPTBot/ClaudeBot (MP-074).

# Apple Intelligence training (Applebot for Search remains allowed)
User-agent: Applebot-Extended
Disallow: /

# Amazon Alexa / Q training
User-agent: Amazonbot
Disallow: /

# ByteDance / TikTok training
User-agent: Bytespider
Disallow: /

User-agent: TikTokSpider
Disallow: /

# Meta training (Meta-ExternalFetcher remains allowed)
User-agent: Meta-ExternalAgent
Disallow: /

# Common Crawl — feeds many AI training datasets
User-agent: CCBot
Disallow: /

# Huawei training
User-agent: PetalBot
Disallow: /

# Cohere training
User-agent: cohere-ai
Disallow: /

User-agent: cohere-training-data-crawler
Disallow: /

# Diffbot
User-agent: Diffbot
Disallow: /

# Webz.io / Omgili — feeds many AI training datasets
User-agent: omgili
Disallow: /

User-agent: omgilibot
Disallow: /

# Google Cloud Vertex (training)
User-agent: Google-CloudVertexBot
Disallow: /

# Timpi (training)
User-agent: Timpibot
Disallow: /

# Anchor Browser
User-agent: AnchorBrowser
Disallow: /

# Novellum AI training
User-agent: Novellum
Disallow: /

# ProRata.ai
User-agent: ProRataInc
Disallow: /

Sitemap: https://www.allatsea.net/sitemap_index.xml