Production incident · Nginx · Linux permissions
The permission race behind intermittent Nginx cache 500s
How a recursive ownership race made Nginx lose access to FastCGI cache files, and how directory ACLs delivered a safer, faster fix.
01 · The incident
The failures looked random. The filesystem was being precise.
The sites were healthy, the application code had not changed, and most cached requests were fast. Then, under traffic, a small number returned HTTP 500.
The useful clue was in the Nginx error log: permission denied while opening a FastCGI cache file. Nginx had created the file, successfully served from the same cache moments earlier, and then somehow lost the ability to read it.
A recovery task appeared to fix the problem. Running it again could also bring the problem back. That contradiction narrowed the investigation: the repair process was not merely observing corrupted state. It was participating in it.
open() "/cache/example/…" failed (13: Permission denied)
upstream: fastcgi://unix:/run/php/php-fpm.sock
result: cached request returns HTTP 50002 · The ownership model
One process wrote the cache. Another needed to purge it.
Nginx naturally created cache files as its own worker user. A WordPress site user also needed to purge cached entries through a helper plugin. The original solution made the site user own the cache tree, then widened permissions so Nginx could continue reading it.
That sounds reasonable until the cache becomes active. Nginx is continuously creating and replacing files. A recursive ownership repair is walking the same tree at the same time. The system has two writers and no atomic boundary between them.
Cache worker
Needs stable read and write access
It creates cache files, opens them on later requests, and expects its ownership model to remain valid.
Site process
Needs permission to remove entries
It does not need to read or own cached response bodies. It only needs to remove names from cache directories.
03 · The race condition
Two recursive commands created a gap large enough for production traffic.
The repair used one recursive pass to change ownership and another to change permissions. Shell syntax can place those commands on the same line; the filesystem does not therefore perform them as one transaction. A busy cache keeps changing between the passes.
The result was a narrow but repeatable state: the cache file belonged to the site user while retaining a restrictive mode inherited from its creation. The site user owned it; Nginx could not read it. The next cache hit became a 500.
Recursive work also scaled with the number of cached paths. More sites and more objects meant a wider race window, a slower repair, and more opportunities to revisit files that did not need changing. The repair was working quite hard—unfortunately, on the wrong abstraction.
04 · The ACL insight
Deleting a file is permission on its parent directory.
On Unix-like systems, removing a directory entry is governed primarily by write and execute permission on the parent directory—not ownership of the file being removed.
That changed the design. Nginx could remain the owner of every cache file, preserving reliable read access. The site user could receive a narrow directory ACL that allowed cache entries to be unlinked without taking ownership of their contents.
Before
Transfer ownership repeatedly
Walk files and directories, change owners, change modes, then repeat after configuration or cache events.
After
Keep ownership stable
Nginx owns its files. The directory grants the site account only the access needed to purge names.
directory owner → cache worker
directory group → cache service group
named ACL user → site account: write + traverse
file owner → cache worker, unchangedA default ACL on new directories preserved the rule as the cache grew. Set-group-ID on directories kept group inheritance predictable. Most importantly, no request-time helper needed to chase new files and repair them after creation.
05 · Safe recovery
A root-level repair tool must distrust the tree it repairs.
Changing the ownership model solved the race, but migration and repair still required privileged filesystem work. That introduced a separate question: what can the repair command safely believe?
A marker placed inside a site-writable directory is not authoritative. It can become stale, be replaced, or be redirected through a symbolic link. A privileged command that trusts it may change permissions outside the intended cache tree.
Guardrail
Trust external identity
Resolve the site account from a trusted system mapping, never from a marker stored inside a writable cache directory.
Guardrail
Replace, do not accumulate
Write an authoritative ACL so stale named-user entries disappear instead of surviving forever.
Guardrail
Refuse symlink surprises
Validate structural directories before privileged commands traverse them. A repair tool must not follow a user-controlled detour.
Guardrail
Scope every mutation
Touch only directories that can be mapped to a real site. Skip orphaned or reserved paths deliberately.
The final repair path rebuilt canonical state from trusted mappings, removed unexpected structural symlinks, skipped directories that could not be tied to a real site, and replaced ACLs authoritatively. Re-running it produced the same result. That is what recovery should feel like: boring, bounded, and unsurprising.
06 · The performance pass
Once the model stopped drifting, routine work became constant-time.
The old process assumed ownership could drift after every cache write or configuration regeneration, so it repeatedly walked large directory trees. The ACL model removed the source of that drift.
Routine regeneration could now inspect the top-level directory—owner, mode, and expected ACL—and take a fast path when the state was canonical. Recursive enforcement remained available for explicit migration or repair, where paying the full cost was justified.
Routine path
Inspect, then leave it alone
A small canonical-state check replaces repeated recursive ownership and permission passes.
Repair path
Force full enforcement deliberately
Migration and operator-triggered recovery retain a comprehensive, idempotent repair mode.
07 · Lessons carried forward
Model the capability, then choose the permission.
The incident disappeared when the design stopped asking, “Who should own these files?” and started asking, “What is each process actually allowed to do?”
- Ownership is not a convenient synonym for access. Grant the narrow capability a process needs.
- Sequential shell commands are not atomic. On a live tree, the gap between them is part of the design.
- Repair paths are security boundaries. Treat writable markers, symlinks, and orphaned paths as hostile inputs.
- Authoritative state beats additive repair. Replace the permission model instead of layering new exceptions over old ones.
- The best optimization often removes work. Stable invariants make expensive reconciliation unnecessary.
Intermittent failures often tempt us toward more logging, more retries, and more aggressive repair. Here, the reliable answer was smaller: let Nginx own what it creates, let the site user remove only what it must, and stop moving the ground beneath both of them.