Model: none | Date: 2026-11-21 | Limit: MemoryHigh 3 GB | Status: admission gate in
MemAvailable read about 4 GB. Load was 150. The agent API was not answering, and the process manager still reported the process online.
The limit that does not kill
Four days earlier an OOM had rebooted the box, and the protection that landed afterward put the process manager and every agent it spawns into one systemd cgroup: MemoryHigh=3G, MemoryMax=4G, swap capped at 1 GB. A separate user slice sat higher, at 6 GB high and 8 GB max. The OOM itself had been a bad ranking: a browser with oom_score_adj 300 and about 20 MB RSS was the preferred victim, while a Node dev server at about 1.4 GB was killed twice and another runtime sat near 450 MB. A janitor with a 2-hour age limit, checking every 30 minutes, never caught the pile-up in time.
memory.max is the killer. memory.high is the throttle. Cross the high line and the kernel parks those tasks in mem_cgroup_handle_over_high. The process stays in the supervisor's process list. It stops making progress, and so does every sibling in the cgroup, including the HTTP API that would have let me cancel the work.
What filled it
Six content jobs were started by hand through the HTTP "run now" route. That route did not consult the scheduler, whose cap is one concurrent scheduled run. Each agent was about 300 MB, plus its own set of tool servers at about 70-90 MB each. The cgroup reached 3.2 GB.
| Reading | Value |
| MemoryHigh / MemoryMax | 3 GB / 4 GB |
| Cgroup at the freeze | 3.2 GB (about 107% of high) |
| PSI some / full | ~97% / ~60% |
| Load, incident note | 150 |
| Load, same-day decision | 116 |
| MemAvailable | ~4 GB |
| Process manager | online |
PSI some near 97% means almost the whole window had at least one task stalled on memory reclaim. full near 60% means every runnable task was stalled for most of that window. Host free memory is a different counter: the cgroup was over its throttle line, and the host still had free pages. MemAvailable still read about 4 GB.
The two load numbers are both written down for this burst. The incident note says 150. The decision that closed the bypass says 116.
3.2 GB is over memory.high and under memory.max. The kernel throttled the cgroup instead of OOM-killing it. The tasks were busy in reclaim. free and the process manager's "online" state both still looked healthy.
The hole
The scheduler already refused a second scheduled job. The manual endpoint did not. Six requests, six agent processes, one shared cgroup, and the API that could have stopped them lived inside the throttle.
A health check on MemAvailable, or on the process manager's "online" bit, stays green through this. The checks that go red:
cat /proc/pressure/memory
# some avg10=... full avg10=...
# full avg10 in the tens means the whole cgroup is stalling
# cgroup that owns the process manager
cat /sys/fs/cgroup/<manager-slice>/memory.current
cat /sys/fs/cgroup/<manager-slice>/memory.high memory.current above memory.high, with PSI full in the tens of percent, is the API timeout. The stack trace will not say so. The tasks sit in mem_cgroup_handle_over_high and the accept loop simply does not run.
The gate
New runs now wait when the cgroup is at or above 80% of memory.high, or when PSI full avg10 is at least 30. A query flag forces one run through. A browser janitor enters a pressure mode at PSI full 20 and starts dropping stale browser processes. The admission decision is a pure function of the counters, cached for 2 seconds:
// high wins over max, max wins over host MemTotal.
// psi is the "full avg10" field from /proc/pressure/memory.
function decide({ current, high, max, total, psi }) {
const limit = number(high) || number(max) || number(total);
const ratio = limit && current != null ? current / limit : 0;
const reasons = [];
if (ratio >= 0.8) reasons.push(`cgroup memory ${Math.round(ratio * 100)}% of its limit`);
if (psi >= 30) reasons.push(`memory stall ${psi.toFixed(0)}% (PSI full avg10)`);
return { busy: reasons.length > 0, reason: reasons.join(", ") || null };
} At 3.2 GB against a 3 GB high line, that function is busy at 107% before it reads PSI. PSI full at 60 would have blocked on its own, even with a missing cgroup file. The manual route was changed the same day so a non-forced click obeys the concurrency cap and parks the extra runs on the scheduler. The force flag is still the way to rebuild this incident on purpose.
Falling through to host MemTotal when the cgroup files are unreadable is the failure mode to watch. A gate that cannot see memory.high will compare the agent against all of RAM, decide the box is fine, and admit the run that tips the real cgroup over.
It stalled again the same day
The gate refuses new work. It does not shrink work already inside the cgroup. A later note from 2026-11-21: 8 agent sessions, 2.3 GB of them, pushed the same cgroup over memory.high again. One API process in that cgroup sat in D state. Its /health timed out.
2.3 GB across 8 sessions is 288 MB each, in line with the ~300 MB figure from the freeze. The sessions alone were under the 80% admission line: 2.3 / 3.0 is 77%. The rest of the process manager, the API and the other daemons in that cgroup, is what carried memory.current across memory.high. An admission check on new runs cannot see that until the line is already crossed, and by then the API is the thing being throttled. Holding the seventh run does not unstick the listener.
The ceiling was raised the same day, MemoryHigh from 3G to 5G and MemoryMax from 4G to 6G, as a runtime change that dies on reboot. The note records the durable fix next to it: start the agent sessions outside the process manager's cgroup, in their own scope.
What I'd change
Let the 5 GB raise expire. A higher memory.high moves the cliff and leaves the API in the same cgroup as the agents. Run the sessions in their own scope. Keep the 80% and PSI-30 gate on the cgroup they actually land in. Point the health check at PSI full and memory.current, and take MemAvailable and "process is online" off that panel. The manual run button stays behind the same cap as the scheduler.