My Paged Sync Script Lied Three Ways in Three Weeks. Each Time Every Counter Said Success.

A sync endpoint that returns 50 records per call and done:false until the cycle ends is trivial to call once and surprisingly easy to call wrong. Over three weeks my orchestration script misreported the same job three different ways, and the server never returned a single error.

The job: pull Search Console performance data into four sites' databases. The endpoint processes pages in batches of 50 (configurable up to 200), keeps the cursor server-side, and answers with processed, updated, noData, failed and done. Page counts per site: 636, 313, 297 and 553. That is 13, 7, 6 and 12 batches at the default size.

# one call = one batch; re-POST until done:true
curl -s -X POST "https://site-a.example/api/admin/sync/auto?reset=1"   # start fresh cycle
curl -s -X POST "https://site-a.example/api/admin/sync/auto"           # advance, repeat

Lie 1: the iteration cap

First version of the loop had for i in range(10). Ten felt generous. Two sites needed 13 and 12 batches.

The script exited cleanly, wrote its report, and showed failed: 0. The two big sites were simply cut off mid-cycle: 136 and 53 pages never synced (10 batches x 50 = 500 pages covered), with no signal anywhere because done:false is a normal state, not an error.

Fix is arithmetic, not retry logic:

cap = ceil(published_pages / batch_size) + 3   # 636 / 50 -> 13 + 3 = 16

And a reconciliation that does not trust the loop at all: the sum of processed across batches must equal the published page count read directly from the database. 636 / 313 / 297 / 553 matched. Any other number is a truncated run.

Lie 2: the last batch is not the run

Later run, new script. Report showed noData of 13, 11, 34 and 42 for the four sites and nothing updated. That pattern looked exactly like an upstream quota throttle I had already been bitten by, where errors get swallowed as "no data".

I started the throttle playbook: paced re-sync, 15 seconds between calls. Before that, one solo diagnostic call. It returned updated: 7 in a single batch. Data was flowing.

The bug was in my script. It overwrote the counters on every iteration instead of accumulating them, so the report held only the final batch's numbers, and the final batch of a cycle is the short remainder. Accumulated totals for the same run: 164 pages updated (74 / 43 / 34 / 13), 0 failed.

# wrong: totals = resp          (last batch wins)
for k in ("processed", "updated", "noData", "failed"):
    totals[k] += resp[k]        # right: accumulate per batch

Rule I wrote down: before invoking a scary hypothesis (throttling), probe one solo call. It costs one request and rules out your own script.

Lie 3: the tool timeout was shorter than the job

Pacing is the correct response to a real throttle: 15 s between batches is about 240 upstream queries a minute, and a full four-site pass takes around 13 minutes. The shell tool running the script had a 600 s default limit.

The script was killed at 596 s. The arithmetic, from the log:

Fourth site never started. Exit looked like an ordinary termination, and the partial report listed three healthy sites. The cursor lives server-side, so recovery was cheap: run the missing site alone, no reset=1. Final tally for the run: 108 pages updated, 0 failed.

What the three have in common

Each failure produced output that looked like a healthy run. A truncated loop, a last-batch snapshot and a killed process all end with plausible numbers and no exception. Paged APIs make this worse because "not done yet" and "done" differ by one boolean.

The checks that now run at the end of every sync:

1. last response had done:true          (loop actually finished)
2. sum(processed) == published count     (nothing skipped, nothing doubled)
3. all sites present in the report       (a killed run drops the tail)
4. counters accumulated, not overwritten

Check 2 alone would have caught lies 1 and 3. Check 3 is the cheapest and catches the timeout. Also: a paced job that takes 13 minutes gets an explicit timeout of 15+ minutes, or runs per site.

What I'd change

The loop belongs server-side. A run-all endpoint that owns the cursor, the pacing and the accounting would remove the client from the failure path entirely. Until then the reconciliation line against the database stays, because it is the only check that does not share a bug with the script it audits.