← Back to lab

My Ledger Said +4.07 Units. The Season Was −8.67.

Two disk-full runs truncated a JSON bet ledger mid-write. The dashboard showed +4.07 units. Six-hour cron logs put the season at −8.67 units, ROI −10.2%.

My Ledger Said +4.07 Units. The Season Was −8.67.
Wipes: 2026-10-03 and 2026-10-28  |  Picks lost: 71  |  Live P&L: +4.07u  |  Season: −8.67u on 85  |  ROI: −10.2%

The dashboard for a paper football book I run said +4.07 units. On 2026-11-17 the season, once the deleted history was added back, was −8.67 units on 85 graded picks, ROI −10.2%. Two cron runs had truncated the JSON ledger when the disk filled, and the next run treated the empty file as a new season.

The ledger is one file. A job rewrites it every six hours: freeze new picks, settle results, dump the list. Flat 1-unit stakes. That file was the only track record.

open(path, "w") deletes first

This is the whole bug:

with open(path, "w") as f:      # file is now zero bytes
    json.dump(ledger, f)        # ENOSPC here leaves the zero-byte file

open with "w" truncates before json.dump runs. If the dump dies on ENOSPC, the previous ledger is already gone. The next run did this:

if not os.path.exists(path):
    ledger = []                 # first run, fine
else:
    ledger = json.loads(open(path).read() or "[]")  # empty file → no history

An empty file and a missing file took the same branch. The book restarted from zero. Nothing logged a wipe. The UI kept rendering whatever the live file held, and that file was only the picks made after the restart.

The recovery script, written afterwards, flags a wipe as the settled count dropping from above zero to zero between two log lines. Nothing checked that on the day.

Two segments, exact totals, no picks

The per-pick rows are gone. Every six-hour run did log one line: pick count, settled count, open count, P&L. Those lines are enough to rebuild the totals and a six-hour equity curve. They are not enough to rebuild a selection or a price, and I did not invent either. The archive says so in its own note.

SegmentPicksSettledOpen at wipeP&L
2026-07-07 → 2026-10-03 06:1548470−6.91u
2026-10-09 → 2026-10-28 12:0023221−5.83u
Archived71691−12.74u

The incident note I wrote that week said "71 graded picks." The log lines are tighter. 71 is the pick count (48 + 23). Settled is 69. The second wipe still had 1 open pick. The first segment's log line is 48 picks, 47 settled, 0 open — one row is in neither bucket, and the log does not say what it was. I left it as logged.

The equity curve in the archive is 39 points, one per six-hour line that survived. That is the resolution. Anything between two lines is gone.

The +4.07 was sixteen bets

Recorded on 2026-11-17, after both wipes and after the archive existed:

BookGradedP&LROI at 1u
Live file only (what the UI showed)16+4.07u+25.4%
Two archived segments69−12.74u−18.5%
Season (live + archive)85−8.67u−10.2%

The arithmetic is the check. −6.91 + −5.83 = −12.74. −12.74 + 4.07 = −8.67. −8.67 / 85 = −10.2%, which is the ROI written down that day. The 16 is 85 − 69. The live file was a 16-bet sample with a fat green number sitting on top of a −12.74u hole.

That is the part that changes a decision. A +25% window on 16 bets is the kind of number that gets a staking rule "fixed." Pick selection was not changed. A separate, pre-registered test the same day — drop one market — moved overall ROI by +0.95 percentage points against a frozen bar of +1.0, and was rejected. The model was not the thing that had broken.

What replaced open(..., "w")

Every ledger write now goes through one module. The rules are small and all of them are aimed at this failure:

MIN_FREE_MB = 200

def write_json(path, obj):
    if free_mb(path) < MIN_FREE_MB:
        raise LedgerError(f"only {free_mb(path):.0f} MB free — refusing")
    tmp = f"{path}.tmp.{os.getpid()}"
    with open(tmp, "w") as f:
        f.write(json.dumps(obj))
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, path)          # old file stays until this returns

def read_json(path, default):
    if not os.path.exists(path):
        return default             # missing: first run
    text = open(path).read()
    if not text.strip():
        raise LedgerError("exists but is EMPTY — refusing to treat it as no history")
    return json.loads(text)       # invalid JSON raises too; file left untouched

os.replace is atomic on the same filesystem. A crash or a full disk leaves the previous file byte-identical. The temp file is unlinked on the way out.

The read side is the half I had wrong. A missing file may be a default. A file that exists and does not parse is an error: the script exits, the file stays, an alert fires. Fail closed. The old code failed open, into an empty list, and then wrote that empty list back on the next successful dump. One truncated write was enough to make the following run permanent.

Three more guards, because a correct write can still delete history on purpose:

  • A no-loss check refuses a write that drops a key present on disk, or that clears result on a pick that was already settled.
  • One flock covers every read-modify-write of the shared files. Timeout is 900 seconds, then the run errors instead of starting a second writer.
  • An append-only JSONL journal records each freeze and each settle. A daily snapshot keeps 60 days. The journal is the copy a ledger can be rebuilt from. It did not exist on 3 October or on 28 October, so those 71 picks cannot come back as rows.

The headline the UI and the morning digest print is season = live + archive. The live file alone is not a result.

The free-disk floor is a backstop, not the fix. It refuses to create the temp file under 200 MB free. It would not have saved 3 October: that writer was still open(path, "w"), and the truncation happens before any dump, free-space check or not. Order of operations is the bug. The threshold only helps the new writer.

What I'd change

I would have shipped the journal before the first wipe, not after the second. Totals at six-hour resolution are enough to stop a false +4.07u from rewriting the season. They are not enough to audit a single pick, and that audit is gone. The detection rule the recovery script uses — settled count fell to zero — should have been a page on the day, not a reconstruction two weeks later.