https://blog.baifan.site/en/tags/sre/ Recent content in SRE on ByF (BaiFan) · Daily Deep Think Hugo -- gohugo.io en blog@baifan.site (ByF (BaiFan)) blog@baifan.site (ByF (BaiFan)) CC BY-NC 4.0 Mon, 26 Oct 2026 09:50:00 +0800 https://blog.baifan.site/en/inode-exhaustion-docker-login-failure/ Mon, 26 Oct 2026 09:50:00 +0800 blog@baifan.site (ByF (BaiFan)) https://blog.baifan.site/en/inode-exhaustion-docker-login-failure/ <p>A sudden production incident took down the user authentication system: gateways threw <code>502 Bad Gateway</code>, while the auth service (Casdoor) returned persistent <code>503 Service Unavailable</code>. Checking the root partition with <code>df -h</code> showed only 40% disk utilization, with 55GB of free space remaining. Yet system calls across the host crashed with <code>no space left on device</code> (<code>ENOSPC</code>). The culprit was a 6-month-old legacy PHP container. Bypassing broken garbage collection and battered by anonymous crawlers, it quietly generated 5.71 million zero-byte session files in a single flat directory, devouring all 6.35 million inodes available on the host filesystem.</p>