A Daily Check Crashed for 3 Weeks on a Cache Permission. Two Scheduled Jobs Never Ran for a Month.

Model: none | Date: 2026-11 | Status: fixed

A daily quality:check command logged check_error every morning for roughly 3 weeks. Two other scheduled jobs, a daily interlinking pass and a weekly article refresh, had not executed for about a month. Both failures looked unrelated. Both came from one deploy shortcut: running a Laravel app as two different OS users.

Context

The app is a small Laravel catalog site. php-fpm serves it as www-data. A cron-driven agent loop runs artisan as a normal login user. CACHE_STORE=file. That is the whole setup, and it is common on a single VPS.

Failure 1: the cache directory only one user can write

The symptom, copied from the log:

file_put_contents(storage/framework/cache/data/3f/a1/…): Failed to open stream: Permission denied

The cause took one ls -ld. When a web request is the first to cache a key, php-fpm creates the hashed subdirectory as www-data with mode 2755 (umask 022, setgid inherited from the parent). The cron user can read it, not write it. The next CLI call that hashes into that directory crashes inside Cache::put.

Which directory a key lands in depends on its hash, so the failure is intermittent per key and permanent per directory. Every daily run hit at least one.

The fix is not a chmod. A one-time chmod -R 775 is undone the next time www-data creates a directory. What holds:

# own what exists, make dirs group-writable, files rw
chown -R cronuser:cronuser storage/framework/{cache,views,sessions}
find storage/framework -type d -exec chmod 2775 {} +
find storage/framework -type f -exec chmod 664 {} +

# ACL for both users on existing entries...
setfacl -R -m u:www-data:rwX,u:cronuser:rwX storage/framework
# ...AND a default ACL, so directories created later inherit it
setfacl -R -d -m u:www-data:rwX,u:cronuser:rwX storage/framework

The second setfacl with -d is the part people skip. Without it the bug returns the first time fpm writes a key into a new bucket.

One red herring cost time: php artisan tinker run as www-data fails on a psysh config directory that www-data cannot create. That error has nothing to do with the cache. The faster test is a direct file_put_contents plus mkdir as each user.

Failure 2: the job that was never scheduled

While reading the quality dashboard I noticed two heartbeat alerts that never cleared: job_stale, severity high, for the interlinking job and the refresh job. Neither had run in about a month.

The scheduler entry looked correct:

$schedule->command('interlinking:run')
    ->dailyAt('09:30')
    ->when(fn () => !empty(config('services.openrouter.api_key')));

The when() gate exists so the job does not run without an LLM key. It worked as designed. The project .env contained OPENROUTER_API_KEY= with an empty value. The real key lived in a shell-level env file outside the project, which interactive shells source and artisan does not read.

So the gate returned false every day, schedule:run skipped the job, nothing was logged, and exit code stayed 0. The heartbeat monitor, written separately, still expected a run. That mismatch is the only reason the problem surfaced at all.

What the two bugs share

The first bug was loud and ignored. The second was silent and found by accident. Findings from the failed check also piled up: a crashed check cannot auto-resolve old findings, so the dashboard showed "weeks old" issues that were really stale state from a check that had stopped working.

The audit I now run on every new site

Four commands, all read-only:

# 1. who owns the cache tree, and is there a default ACL?
getfacl storage/framework/cache | head -20

# 2. is every key a gated job needs actually non-empty in the file artisan reads?
grep -E '^(OPENROUTER_API_KEY)=' .env | sed 's/=.*/=<set>/;t;s/$/ EMPTY/'

# 3. which scheduled jobs does the app think are due?
php artisan schedule:list

# 4. rebuild the config cache after touching .env, then re-check
php artisan config:clear && php artisan config:cache && php artisan schedule:list

schedule:list is the useful one. A job whose when() gate is false still shows a "Next Due" time, so it does not prove the job will run. The extra check: run the gate expression through artisan tinker as the cron user and confirm it returns true.

What I found

• Both bugs predate any monitoring that could have named them. The cache crash was only visible because the check wrote its own error row; the gated jobs only because a separate heartbeat existed.
• Every other site built from the same starter shares the setup, so both traps apply there too. Cache ACLs and a non-empty key are now a deploy-time checklist item.
• After the key went in and the config cache was rebuilt, schedule:list showed both jobs with a fresh Next Due time.

What I'd change

Stop sharing a file cache across two users at all. A Redis or database cache removes the entire permission class. And a gated job must log when its gate is false, one line per skip. A when() that returns false in silence is indistinguishable from a job that is broken, and a month is a long time to find out which.