Every Agent-Readiness Check Passed on Six Sites. Cloudflare Still 403'd the Real Calls.

Date: 2026-11 | Sites: 6 | Scanner score: 5/5 on all six | Real agent calls on one zone: 403

I shipped agent-discovery endpoints on six content sites: /.well-known API catalog and agent cards, a /mcp JSON-RPC endpoint, an /a2a endpoint, and a WebMCP bridge. The readiness scanner gave every site a perfect score. Then a real agent POSTed to /mcp on one of them and got a 403 from Cloudflare.

The scanner was not lying. It just never made the call it was scoring.

The blind spot in readiness scanners

The scanner I used fetches the discovery cards — /.well-known/api-catalog, the A2A agent card, the MCP server card, the skills manifest — and checks they exist, parse, and point at plausible endpoints. Six checks per site: apiCatalog, a2aAgentCard, agentSkills, mcpServerCard, webMcp, contentSignals. All six sites passed all six. Level 5, the top rating.

None of those checks POST anything. The scanner reads the card that advertises /mcp and moves on. Whether /mcp actually accepts a JSON-RPC request from the open internet is a different question — and on one zone the answer was no.

The fix for the process gap came later and is the boring part: a weekly monitor that actually POSTs to /mcp and /a2a through Cloudflare from an external client, diffs against last week, and alerts on regression. That is the check that matters. The badge is marketing.

Veto 1: Bot Fight Mode is zone-wide and unskippable

The 403 came from Cloudflare's free Bot Fight Mode. Two properties make it nasty for agent endpoints:

1. It is zone-wide. On the free plan there is no skip rule, no exception list, no path bypass. If it is on, it challenges every request it classifies as automated, everywhere on the zone.
2. A non-browser POST to a JSON-RPC endpoint is exactly what it exists to stop. Every external agent calling /mcp or /a2a got challenged, and a JSON-RPC client does not solve JS challenges. Result: 403.

The endpoints are read-only — search and page fetch, same data the public search page already serves. The thing BFM was "protecting" was a public API. Meanwhile the actual attack surface — /login, /admin, /register — needed protection, not a blanket block on machine clients.

So BFM went off, replaced with one WAF custom rule that challenges only the admin surface, and only from outside the two countries the site actually serves:

(http.request.uri.path eq "/login"
 or http.request.uri.path eq "/register"
 or starts_with(http.request.uri.path, "/admin"))
and not ip.geoip.country in {"CZ" "SK"}
→ Managed Challenge

Verified with the Cloudflare request tracer before touching anything else:

The other five zones already had BFM off — which is why five sites passed the real-call test and one did not. The stack was identical; the zone dashboard was not. Rollback, if ever needed, is one API call:

curl -X PUT "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/bot_management" \
  -H "Authorization: Bearer $CF_TOKEN" \
  -d '{"fight_mode": true}'

Veto 2: AI Crawl Control rewrites your robots.txt

Same day, different zone, different silent override. Cloudflare's AI Crawl Control (the bot_management API fields ai_training, ai_user, ai_agents) does not just block crawlers — it rewrites the served robots.txt, injecting its own Disallow blocks for GPTBot, ClaudeBot, CCBot and friends, plus its own Content-Signal header.

The application had an explicit, deliberate robots policy: Content-Signal: ai-train=yes, search=yes, ai-input=yes. The zone was contradicting it at the edge. Anyone fetching robots.txt saw Cloudflare's opinion, not the site's. The app never knew.

One zone had it enabled — possibly pre-enabled, because a new zone can come with these toggles on. The fix is three fields:

curl "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/bot_management" \
  -H "Authorization: Bearer $CF_TOKEN" | jq '.result | {ai_training, ai_user, ai_agents}'

All three must be disabled. This is now on the new-zone checklist, next to "BFM off" and "managed robots.txt off" — the third zone-level feature that overrides an app-level policy file. Cloudflare's managed robots.txt had the same problem on that zone: it injected ai-train=no over the app's own signal.

Veto 3: nginx deny all on dotfiles also blocks /.well-known

Not Cloudflare, but the same shape of bug — a broad deny rule eating a path you just added. Two of the six sites had the classic hardening line:

location ~ /\. { deny all; }

/.well-known/api-catalog matches \. too. 403, again, this time from nginx. The fix is a negative lookahead:

location ~ /\.(?!well-known) { deny all; }

.env and .git stay blocked, /.well-known/* gets through. Three independent layers — WAF, edge robots rewriting, origin nginx — each with a rule that was correct when written and wrong the day the endpoints shipped.

What I found

The failure pattern across all three vetoes is identical: a default-on protective layer sits between your application and the internet, and it does not know your application grew new public endpoints. BFM was right in 2023. AI Crawl Control's defaults are reasonable for a site that wants nothing to do with AI. The nginx dotfile deny is standard hardening advice. Every one of them silently vetoed a policy decision made later, in application code, that they could not see.

The diagnostic order that worked: test through the edge, not around it. Local checks against the origin passed the whole time — the endpoints were fine. Only curl against the public hostname, from a client that looks like an agent (no browser TLS fingerprint, no cookies), reproduced the 403. The request tracer then named the exact rule responsible in one query, which beat an hour of reading WAF event logs.

What I'd change

Every synthetic check now has to make the real call it claims to verify. The readiness scanner stays — it is useful for card validity — but the green badge is no longer the done criterion. The done criterion is an external POST returning 200, run weekly, alerting on diff. And every new Cloudflare zone gets the three-line audit before anything ships: BFM off, AI Crawl Control fields disabled, managed robots.txt off. Defaults are decisions someone else made for you.