Fix Docker: No Space Left on Device

1. Introduction

no space left on device is the error that takes down build agents, breaks deploys, and wedges databases at 2am. Docker is a frequent culprit because /var/lib/docker quietly accumulates dangling images, stopped containers, unused volumes, and — the biggest hidden offender — build cache. The disk fills, writes fail, and everything that touches the filesystem starts erroring.

This guide shows you how to find what is consuming the disk (bytes vs inodes — they fail the same way), reclaim space safely without nuking data you need, and stop it from recurring. Commands are Linux-host oriented, which covers CI runners, EC2 instances, and Kubernetes nodes.

2. What "No Space Left" Really Means

There are two independent ways a filesystem runs out: blocks (actual bytes) and inodes (the count of files). Millions of tiny files can exhaust inodes while df -h still shows free space — so always check both.

df -h            # block/byte usage per mount
df -i            # inode usage per mount  (look for IUse% at 100%)

• Docker stores images, containers, volumes, and build cache under /var/lib/docker (often on / or a dedicated mount)
• The overlay2 driver keeps every image layer; deleted-but-dangling layers still occupy space until pruned
• Container logs (json-file driver) grow unbounded by default and can fill the disk on their own

3. Common Causes

• Accumulated dangling images and stopped containers from repeated builds/deploys
• BuildKit/legacy build cache growing to many GB (the most common surprise)
• Unused named/anonymous volumes never cleaned up (databases, caches)
• Unbounded container logs under /var/lib/docker/containers/*/*-json.log
• Inode exhaustion from millions of small files (node_modules layers, caches)
• A single huge image or a runaway container writing into its writable layer instead of a volume
• On Kubernetes nodes: image/log growth triggering DiskPressure and pod eviction

4. Step-by-Step Diagnosis and Fix

Step 1: Measure — bytes and inodes, then where

df -h ; df -i                         # blocks and inodes
sudo du -xh /var/lib/docker --max-depth=1 | sort -rh | head

# Docker's own accounting of reclaimable space
docker system df
docker system df -v                   # per-image / per-volume / cache detail

Step 2: Reclaim the safe, obvious wins first

# Remove stopped containers, dangling images, unused networks, build cache.
# This does NOT remove named volumes or images still in use.
docker system prune

# Also drop the build cache explicitly (often the biggest chunk)
docker builder prune                  # add -a to remove ALL cache

Step 3: Prune images and containers deliberately

# Dangling images only (safe)
docker image prune

# ALL images not used by a container (aggressive — will re-pull later)
docker image prune -a

# Remove exited containers
docker container prune

# Time-box it: remove objects older than 24h
docker system prune -a --filter "until=24h"

Step 4: Handle volumes carefully (data lives here)

# List volumes and see which are dangling (not attached to a container)
docker volume ls
docker volume ls -f dangling=true

# Remove ONLY unused volumes — never blind-prune a DB volume
docker volume prune

# Inspect a volume's mountpoint/size before deleting
docker volume inspect <volume-name>

Step 5: Cap container logs (a frequent silent filler)

# Find the biggest container logs
sudo sh -c 'du -sh /var/lib/docker/containers/*/*-json.log | sort -rh | head'

# Fix globally in /etc/docker/daemon.json, then restart Docker:
{
  "log-driver": "json-file",
  "log-opts": { "max-size": "10m", "max-file": "3" }
}
# sudo systemctl restart docker  (recreate containers to apply)

Step 6: Fix inode exhaustion

# If df -i shows IUse% ~100% but df -h has space, you're out of inodes.
# Find directories with the most files:
sudo find /var/lib/docker -xdev -type f | cut -d/ -f1-6 | sort | uniq -c | sort -rh | head

# Pruning images/build cache usually frees millions of small-file inodes:
docker system prune -a
docker builder prune -a

5. Verification Steps

# Space and inodes recovered
df -h ; df -i
docker system df

# Prove writes work again
docker run --rm alpine sh -c 'dd if=/dev/zero of=/tmp/t bs=1M count=50 && echo OK'

# On a Kubernetes node, confirm DiskPressure cleared
kubectl describe node <node> | grep -i pressure

6. Common Mistakes

• Only checking df -h and missing an inode (df -i) exhaustion
• Running docker system prune -a --volumes in a panic and deleting a database volume
• Freeing space but never capping logs, so the disk fills again within days
• Assuming the pressure is Docker without measuring — it may be app data or logs elsewhere
• Pruning on a Kubernetes node by hand instead of fixing image/log limits so kubelet GC handles it

7. Prevention Tips

• Set max-size/max-file log limits in daemon.json on every host and CI runner
• Add a scheduled docker system prune -af --filter "until=168h" on build agents
• Give /var/lib/docker its own volume so Docker growth can't take down the OS disk
• Use multi-stage builds and .dockerignore to keep images and cache small — see Docker build failures in CI/CD
• On Kubernetes, tune kubelet image GC thresholds; disk pressure is what triggers pod eviction

8. FAQ

9. Summary