My API Returned 20 of 55 Records, No Error. The Dedup Check Created 8 Duplicates.

Model: n/a | Date: 2026-08 → 2026-10 | Status: worked around, not fixed

A catalog API returned 20 tools. The database held 55 active ones. No error, no totalItems, no next link. Every agent that "checked for duplicates first" was checking against 36% of the data.

What broke

The setup is a Symfony app exposing a JSON-LD collection endpoint (API Platform style). Agents use it to answer one question before creating a record: does this already exist?

curl -s https://site-a.example/api/ai_tools | jq '.member | length'
# 20

SELECT COUNT(*) FROM ai_tool WHERE is_active = 1;
-- 55

Pagination did not recover the rest. page=2 and itemsPerPage=100 both returned the same 20. The response had no hydra:totalItems and no hydra:next, so nothing in the payload said "there is more".

The dedup logic was correct. The input was truncated.

The damage

Eight empty stub records existed when I finally ran a GROUP BY name. Each was a tool an agent had "discovered" and created, because the 20-item window did not contain the original.

The listing was not a random sample. It was whatever the default query returned first, so whole categories could be partly or fully invisible to the agents.

It was the same bug twice

The content endpoint on the same app had a sibling problem, found while refreshing old articles:

• itemsPerPage silently capped at 20 regardless of the value sent
• ~228 published items meant 12 pages
• ?slug=my-article was ignored: the endpoint returned the unfiltered first page

That last one is worse than truncation. A filter that is ignored returns valid-looking data. Code that does result[0] after ?slug= gets the wrong record and proceeds happily.

# wrong: ?slug= is ignored, [0] is just "the newest item"
item = get(f"/api/content_items?slug={slug}")["member"][0]

# right: walk every page, build the map locally
ids, page = {}, 1
while True:
    r = get(f"/api/content_items?page={page}")
    ids.update({m["slug"]: m["id"] for m in r["member"]})
    if "next" not in r["view"]:
        break
    page += 1

Note next, not hydra:next. The reference doc the agents were given said hydra:member and hydra:next. The live response used member and next. Code written to the doc found zero items and zero further pages, which also looks like success.

A related silent pass

Same app, different day: an edit inserted a section into a markdown body by string replacement. The inserted block began with a code fence; the anchor ended with one. Fence count went odd, and every heading after that point rendered inside a <pre> block and vanished from the page.

The verification step was new_text in html. It passed. The text was there, inside a code block.

# before PATCH: parity must not change
assert len(re.findall(r'^```', body, re.M)) == fences_before

# after publish: compare structure, not substrings
assert h2_h3_list(live_html) == h2_h3_list(expected_html)

Presence checks answer "is it somewhere". They do not answer "is it where a reader sees it".

The red herring

An uncommitted diff in the repository layer was sitting in the working tree: a JSON_SEARCH to LIKE rewrite for MariaDB compatibility. It looked like the obvious culprit for a listing that returned too few rows. It only touched search queries. The default collection query was unaffected, and the real cap lives in the collection provider's default behavior, which I have not fixed.

Comparing the API count against COUNT(*) ruled it out faster than reading the diff.

What I changed

• Dedup reads the database directly. The API is not an inventory source until it reports a total.
• Any list read must compare len(items) against an independent count. Mismatch aborts the run.
• Filters are verified once: ask for a slug that cannot exist and assert the result is empty. If it returns 20 items, the filter is a no-op.
• Post-edit verification checks headings, not strings.

The independent-count rule alone would have caught the 20-vs-55 gap on day one. It costs one COUNT(*).

What I'd change

I would make the first call in any new integration a deliberate nonsense query. ?slug=zzzz-does-not-exist returning anything but an empty list tells you the whole filter layer is decorative, before you build a dedup step on top of it. I trusted an endpoint because it returned 200 and well-formed JSON.