My Freshness Alert Could Never Fire: 'Fresh' While 62% of 171,454 Offers Were a Week Stale

Model: n/a | Stale offers: 106,531 / 171,454 | Feeds failing: 26 of 28 | Status: fixed, metric replaced

The health endpoint returned {"status":"OK","offers":"fresh"} while 62% of the active catalog had not been refreshed in over a week. The Telegram alert wired to that field was structurally incapable of firing.

I run a price-comparison backend that ingests vendor feeds into one offers table. Staleness is the core failure mode: a price nobody refreshed is a wrong price. So I built a monitor for it. The monitor read a field that could not detect it.

The check that passes by construction

The freshness check was one aggregate:

-- "is the catalog fresh?"
SELECT MAX(last_seen) FROM offers WHERE status = 'active';
-- compare to now() - 48h

MAX asks whether any offer was seen recently. With multiple vendors, one working feed answers yes forever. The other vendors can rot indefinitely.

Measured on the day I audited it:

Every stale-price problem I found came from a manual audit, never from the monitor. That is the tell: if every incident is found by a human, the alert is decorative.

Replacement 1: the ratio

The fix for the first problem is a ratio, not an extremum:

SELECT
  count(*) FILTER (WHERE last_seen < now() - interval '7 days') AS stale,
  count(*)                                                      AS total
FROM offers WHERE status = 'active';

Alert on stale / total, not on the newest row. A percentile works too. Any aggregate that a single healthy row can satisfy is a liveness check wearing a freshness label.

I also added a per-vendor endpoint that separates attempted but failing (recent last_attempt, old last_success) from genuinely unchanged feed. Offers last_seen alone cannot tell those apart. On the first run it reported 1 healthy, 5 stale, 1 dead out of 7 active vendors, and showed that feeds were attempted constantly and failed 6 of 7 times.

Replacement 2 failed the same way

Three months later the new endpoint missed an outage. A dead SOCKS proxy variable had leaked into the worker's process environment, and every feed fetch through curl exited with code 28 (timeout). Direct fetches returned 200. One vendor survived because its client opted out of the proxy.

Daily feed success rate during the outage:

The per-vendor endpoint should have screamed. It floored the "days since last success" figure to an integer and treated anything at or below 2 days as healthy. An outage shorter than the rounding window is invisible by design. The first full day of failure always reads as green.

The check was correct about the data and wrong about the threshold. A feed that ran daily and missed two runs is not healthy; it is 100% failed for its cadence.

The part nobody asked for: 28 remediation runs

Failing feeds triggered the automatic repair path. Each failing vendor spawned an agent run to diagnose and fix its feed. That produced 28 runs in total.

All 28 failed. Not because the agents were bad: the LLM backend returned HTTP 429 with a quota reset several days out. The only thing standing between 28 unattended agents and a shared root cause was a rate limit.

The root cause was one environment variable. Per-vendor remediation fans out one agent per symptom. When 26 of 28 feeds fail at once, that is one infrastructure fault, and 26 parallel fixers will each "repair" something different.

What changed

# first check when many vendors fail together: the worker's real env
cat /proc/$(pm2 pid worker)/environ | tr '\0' '\n' | grep -i proxy

# restart without inheriting shell state
env -u ALL_PROXY -u all_proxy pm2 restart worker --update-env

• Freshness alerts use ratios (stale share of active offers), never MAX.
• Per-vendor health compares days since last success to the feed's own cadence, in hours, unrounded.
• If more than a few vendors fail on the same day, the remediation path stops and raises one infrastructure alert instead of spawning one agent per vendor.

What I'd change

Test every alert by making it fire. I never forced the freshness monitor to alarm; a single synthetic stale vendor would have shown in one minute that it could not. Both failures were found by audit, because both monitors were written against the happy path and verified against nothing.