The First Sync Batch Returned 0 Updates. The Next Five Held 21.

Endpoint: 1 (batched Search Console sync)  |  Sites: 4 (297–636 pages)  |  Runs: ~40, Jul–Oct 2026  |  Status: loop rules settled

On July 18 the scheduled sync for three of my four sites opened with a batch of 50 pages and updated exactly zero of them. The loop I had at the time treated that as "no fresh data" and moved on. The next batches on those same sites returned +9, +4, +1, +3, +4. Stopping at the first empty batch was silently under-syncing 15–25 pages per site, every run, while reporting success.

The endpoint is fine. The bug was my reading of an honest answer.

The endpoint

Stripped to shape:

POST /api/admin/sync/auto            # 50 pages per call, ?limit= up to 200
→ {"success":true,"done":false,"processed":50,"updated":0,"noData":50,"failed":0}

Three properties drove every mistake that followed:

• The cursor lives server-side, in a settings row. You just re-POST; there's no cursor param to thread or lose.
• done stays false until the walk covers every page. A single call is never the sync.
• Pages are processed in a fixed order, and fresh Search Console data is not uniformly distributed across that order.

That last one is the whole trap. updated: 0 means "no fresh data in these 50 pages". It does not mean "no fresh data".

The evidence

Same day, same endpoint, per-batch update counts:

A fourth site returned 0 across all five of its batches that day — genuinely no fresh data. Which is why the first fix was a heuristic, not an always-loop: declare a site empty only after 2+ consecutive empty batches or done:true.

The heuristic wasn't enough either

On August 4 I let the loop run to done:true with no early stop. The largest site (636 published pages, 13 batches) got walked for 626 pages versus ~450 under the 2-empty rule, and the run found 91 updates — that site's highest total ever. Early stopping wasn't just risky, it was leaving real data behind every single day, because two sparse batches in a row still preceded a dense one often enough to matter.

The final rules, each paid for:

1. Loop to `done:true`. No data-driven early stop.
2. Cap iterations above `ceil(published / limit)`. A 10-call cap silently truncated two sites mid-cycle in an October run — they needed 13 and 12 batches. At limit=50 that's a ceiling of ~20; or pass limit=200 and cut 6–13 batches down to 2–4 with no rate-limit fallout.
3. Reset the cursor when you need a verifiable pass. Without ?reset=1, a loop finishes only the remainder of whatever cycle was already mid-walk. One September run's first pass looked "done" after one batch per site — it had synced the tail of a stale cycle. A server-side cursor is convenient until you can't tell whose walk you're finishing.
4. Verify with an invariant, not the response. After the run, the sum of processed per site must equal the published-page count, checked straight against the database (636 / 313 / 297 / 543 that week). success:true says the call worked. Only the arithmetic says the walk completed.

Zero cuts both ways

On October 12 one site returned 0 updates across all 313 of its pages, immediately after three syncs of 42, 19 and 42. Same number as July's trap, opposite meaning: the API was healthy — another site updated 36 pages in the same run — and the site simply had an impression-less day. An empty batch means "stop" only when done:true closed the walk and the invariant holds. The count alone can't distinguish "no data yet" from "no data", and those two demand opposite reactions: keep going, or don't.

What the loop did to my tooling

Three consumer-side bugs only surfaced because this job runs daily:

• A jq filter read $done — an undefined jq variable — instead of the .done field. Undefined variables fail quietly, so every batch got classified "non-JSON" and the run misclassified its own results. Cost one bogus "0 synced, 6 errors" alert before anyone reread the filter.
• A shell hook that intercepts and reformats command output mangled JSON bodies in a pipe, making a delivered Telegram alert look failed — so the sender retried and delivered it twice. API responses get parsed from a file now, never from a pipe.
• A background run that looked dead — empty results file, nothing in ps — was alive across the sandbox boundary, and the premature "resume" re-walked 639 pages on top of the still-running cycle. Poll the side-effect files before re-running a job that's idempotent but not free.

The loop that survived

sync_site() {
  post "?reset=1"                        # fresh, verifiable cycle
  for i in $(seq 1 "$CAP"); do           # CAP >= pages/limit + slack
    post "?limit=${LIMIT}" -o batch.json
    [[ $(jq -r '.done' batch.json) == true ]] && break
  done
  cat batches/*.json | jq -s 'map(.processed) | add'   # must equal published count
}

Four sites, serially. 1,770 pages, 37 batches, 105 seconds end to end. The July version of this job covered the same distance, fetched a fraction of the data, and told me everything was fine.

What I'd change

The endpoint should return remaining next to done, and walk pages in freshness order instead of a fixed one — then an empty batch would actually mean the tank was empty. On my side, per-batch deltas belong in the daily alert, not just the total; a zero-first-batch hides too easily behind a healthy-looking sum.