max_tokens 4000 Cut 32 of 56 Generated Pages Mid-Table. My HTML Validator Only Checked Table Tags.

Model: deepseek-v4-flash | Date: 2026-11 | Status: fixed

38 of 56 published pages on a new travel-guide site were structurally broken. 32 of them were simply cut off by max_tokens: 4000. The generation pipeline had an HTML validator. It passed all of them.

What broke

The site generates long guides through OpenRouter. Each page is one completion, stored as an HTML fragment. A page-health script I run over stored pages flagged 38 of 56 as severe: lists ending mid-word, tables with no closing tag, headings as the last element.

The validator on the write path looked like this:

// the whole check, before the fix
export function assertWellFormedHtml(html: string): void {
  // table rows that lost their angle brackets
  if (/^\s*\/?(?:tr|td|th|table|tbody|thead)\s*$/im.test(html)) throw ...
  // unbalanced table / tr / td / th counts
  ...
}

It was written after one specific incident: a model emitting tdParkovánítd instead of <td>Parkování</td>. Narrow on purpose, to avoid rejecting valid content. It did that job. It checked nothing else.

Three blind spots

1. finish_reason was never read. The API says exactly when output was cut: finish_reason: "length". The response was parsed, the string was stored, the field was dropped. A guide that hit 4,000 tokens ended in the middle of a <li> and went live.

2. The cap was too low for the content. Long guides routinely needed more than 4,000 output tokens. 32 of 56 pages hit the ceiling. Raising it to 8,000 gave long pages room to finish. It did not replace the check: a cap is a guess, finish_reason is a fact.

3. Table-only validation misses everything else. Once I looked at the 38 pages, the defects were not just truncation:

Table-tag balance catches none of these.

The fix

The validator now takes the finish reason and rejects on 12 classes. A rejected completion is retried and never stored.

export function assertWellFormedHtml(html: string, finishReason?: string | null) {
  if (finishReason === 'length') throw new Error('output truncated (finish_reason=length)');
  // fences, doctype, spaced tags, markdown headings, "<h<>" ...
  // then a stack walk over a whitelist of known tags:
  //   unknown tag      -> reject
  //   stray close tag  -> reject (unless the end tag is optional: p, li, td ...)
  //   unclosed <strong> before </li> -> reject
  // last: ends on a heading, or mid-sentence inside p/li -> reject
}

Two details that matter. HTML allows omitting </p> and </li>, so the stack walk treats those as optional or it rejects valid output. And a trailing closed section is fine; a trailing heading is not.

Testing it against what already existed

A validator rewrite is a liability if it rejects good pages. I ran it over all 626 published pages across the sites that share this code:

The 4 disagreements were markdown bold (**text**), which the save path converts to <strong> anyway. So the write-path check and the audit script now flag the same defects at two points: before storage and after. 12 unit tests pin the cases, plus two that must pass: a complete page with tables, FAQ and a trailing closed section, and implicit </p>/</li>.

Cleaning up the 38

Regeneration was off the table: a 60-day rule on live pages blocks rewriting anything that recently changed. The owner waived it once for a minimal structural repair: cut the garbled tail, keep title, meta, H1 and URL.

Zero traffic is why a repair was safe. Six pages were then regenerated with facts restricted to what the old content already said: the stub went from 121 to 800 words, the other five grew 47–110%. Unsourced figures (a parking price, a distance with no source) were dropped rather than kept.

What I'd change

The validator should have started from the API contract, not from the last incident. Every LLM write path gets three checks on day one: finish_reason, a structure parse, and a "does this end like a document" test.

A narrow validator written after one failure feels safe because it never produces false positives. It also produces no signal on the next failure. The stored-page audit was the only thing that found this, and only after the pages were already live.