The permission race behind intermittent Nginx cache 500s

01 · A difficult production bug

Working on xCloud keeps me on the edge of my seat.

Almost every week I end up working on something I have never touched before—deep Linux internals, web server behaviour, filesystem quirks, or performance problems that only appear under real traffic.

Recently, while working on xCloud, I fixed one of the trickiest production issues I have faced so far. It had been giving our team headaches because it appeared only under very specific conditions and was incredibly difficult to reproduce consistently.

The sites looked healthy. The application code had not changed. Most cached requests were extremely fast. Then, under traffic, a small number of visitors received HTTP 500 responses.

So let us walk through what happened, how we tracked it down, and why the final solution had almost nothing to do with PHP or WordPress.

02 · A little context

How FastCGI full-page caching removes most of the work.

At xCloud, we are building a hosting platform that automates much of the heavy lifting behind running WordPress and PHP applications. One of the features we provide is FastCGI full-page caching through Nginx.

Without a page cache, a normal request travels through the complete application stack. Even if the page has not changed, PHP and the database still repeat the work.

Visitor
   ↓
Nginx
   ↓
PHP → WordPress → Database

With FastCGI cache, the first request still generates the page. Nginx stores the final HTML on disk, and the next visitor receives that file directly.

Visitor
   ↓
Nginx
   ↓
Cached HTML file

03 · Cache invalidation

WordPress still needs a way to remove stale pages.

Imagine publishing a new blog post. If Nginx keeps serving yesterday's cached HTML, visitors will never see the update. A fast cache that serves old content is merely a very efficient liar.

WordPress plugins such as nginx-helper solve this by purging the relevant cache files whenever content changes. The plugin deletes the cached entry, and Nginx regenerates it on the next request.

That sounds simple enough. It is simple enough—until Linux permissions become involved.

04 · The filesystem state

How Nginx creates its cache files.

Whenever Nginx generates a cache file, it creates that file for its own worker user with restrictive permissions. In simplified form, the state looks like this:

Owner:       xcloud
Group:       xcloud
Permissions: 0600

If you are unfamiliar with Linux permission modes, 0600 means only the file owner can read or write it. The group gets nothing. Everyone else gets nothing.

That is actually a sensible default. Nginx is the process serving the files, so Nginx owning them is exactly what we want.

05 · Where things went wrong

We made the website user own the cache.

The WordPress purge plugin needed permission to remove cache files. Our original answer was straightforward: after Nginx created the files, change the cache tree so the website user owned it.

chown -R site_user:xcloud cache/
chmod -R 770 cache/

After those commands, WordPress could access the cache directory and remove files when content changed. Everything looked correct in testing.

Then production traffic arrived.

06 · The race condition nobody wanted

The commands were reasonable. Their timing was not.

Imagine Nginx actively serving thousands of requests. While it is creating new cache files, the recursive ownership command is walking through the same directory tree. One command changes ownership. Another command changes permissions. Nginx continues creating files between both passes.

Some files received both updates. Some received neither. The dangerous case was a file whose ownership changed but whose restrictive mode remained untouched:

Owner:       site_user
Group:       xcloud
Permissions: 0600

Remember what 0600 means: only the owner can read the file. The owner is now the website user. Nginx still runs as xcloud. When Nginx later tries to serve that cached response, Linux refuses.

open(cache-file)
→ Permission denied
→ HTTP 500

07 · Thinking about the actual problem

Did WordPress really need to own the files?

After staring at the problem for quite a while, I stopped asking, “How do we let WordPress own cache files?”

I asked a smaller question instead: “Does WordPress actually need to own them?”

The answer was no. WordPress never reads the cached response body. It never writes to that file. The purge plugin only deletes the entry when content changes.

That small realization changed the entire design.

08 · One of Linux's coolest permission rules

Deleting a file is permission on its parent directory.

Deleting a file in Linux is surprisingly unintuitive. You do not generally need permission on the file itself. You need write and execute permission on the directory containing it.

That is because deleting a file is really removing the filename entry from its parent directory. Linux asks whether the process can write to and traverse that directory. File ownership is not the deciding factor.

09 · Enter ACLs

Grant the exact access without changing ownership.

Traditional Linux permissions give us three broad buckets: owner, group, and others. Access Control Lists extend that model by allowing permissions for a specific additional user.

Directory owner: xcloud
Directory group: xcloud
Additional ACL:  site_user → write + traverse

The website user can now remove cache entries without becoming the owner of cache files. Even better, the ACL belongs on directories—the exact place Linux checks for deletion.

We stopped touching cache files entirely. They remain xcloud:xcloud 0600. Nginx always owns its own cache and can always read it. The site user can purge entries through the directory ACL but still cannot read cached response contents.

No recursive ownership changes. No race condition. And the permission model now follows least privilege instead of working around it.

10 · Why this also made everything faster

The fastest recursive operation is the one you remove.

The old implementation repeatedly ran recursive chown and chmod operations across large cache trees. On servers with many sites, that meant walking thousands of files again and again.

After moving to the ACL model, the cache tree became stable. Ownership no longer drifted by design. Routine work could verify the directory state without rewriting every cached file.

The result was not merely a correctness fix. It removed unnecessary filesystem work and made routine operations considerably faster on larger servers.