nginx -t Passed. All 30 Sites Returned 521 for an Hour.

Model: none | Incident: 2 in 7 days | Blast radius: ~30 vhosts | Status: workaround, not fixed

nginx -t said syntax is ok, test is successful. nginx was in failed state, and every site behind Cloudflare was returning 521.

The setup

One VPS, about 30 vhosts, all behind Cloudflare. nginx listens on the wildcard:

listen 443 ssl;
listen [::]:443 ssl;

A private dashboard on the same box was exposed over the tailnet with tailscale serve, which binds HTTPS on the tailnet addresses, port 443, specific IPs:

100.x.y.z:443
[fd7a:115c:a1e0::...]:443

Why it breaks

On Linux, a wildcard bind (0.0.0.0:443) cannot coexist with a specific-address bind on the same port when the second socket got there first. Whoever starts second loses with EADDRINUSE.

nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)

That's a runtime failure, not a config error. nginx -t parses the config and checks syntax and file references. It does not prove the ports are free. So the one command everybody runs before a reload gave a false all-clear.

Incident timeline

The second time I was diagnosing from a different host with no direct route to the origin, so I could confirm the 521s and the proxy path but not touch the box. Alerting worked; fixing didn't. 521 from a healthy edge with fine neighbors (other sites through the same proxy returned 301) is the signature of an origin-side failure, not a proxy block.

Diagnosing it in 30 seconds

Skip the config. Ask the kernel who owns :443:

systemctl is-active nginx                  # failed
journalctl -u nginx -n 20 --no-pager       # bind() ... (98: Address already in use)
sudo ss -ltnp 'sport = :443'               # shows the tailscaled listeners

If ss lists tailscaled on the tailnet addresses and nothing from nginx, that's the whole story.

Why 521 and not 502

502 means the origin answered badly. 521 means the origin refused the connection outright. Cloudflare serving 521 for every hostname at once, with real cf-ray edge IDs, points at one dead listener on the origin rather than a bad deploy of any single app. Thirty unrelated sites failing in the same minute is a shared-listener problem by definition.

Two details made the second incident look stranger than it was. Requests to unrelated third-party hosts through the same outbound proxy returned 301 normally, which ruled out the proxy path. And nginx -t stayed green, which is exactly what you'd expect if you believe -t validates the runtime.

The fix that works

sudo tailscale serve reset        # frees :443
sudo systemctl start nginx
curl -ks -H 'Host: site-a.example' https://127.0.0.1/ -o /dev/null -w '%{http_code}\n'

Check systemctl is-active nginx and a real request against loopback with a Host header. Do not check nginx -t.

The dashboard stays reachable over the tailnet on its plain HTTP port, so only the :443 front door is lost.

Why it recurs

Re-enabling tailscale serve on 443 re-creates the conflict. Three permanent options:

Pick the second or third. I left it as a workaround for two incidents because the first option looked like the "proper" fix and was too invasive to do casually. That was the mistake: the cheap options were always available.

What I'd change

Add a post-reload check that hits a vhost over loopback and alerts on non-2xx/3xx, instead of trusting nginx -t. Both detections here came from noticing downstream 521s, not from the service itself.