Firdooze

Fixing "No Space Left on Device" Despite Free Space on Linux

You run a command, and Linux throws back "No space left on device" — but df -h clearly shows gigabytes free. This contradiction is one of the more disorienting errors in Linux administration, and it has nothing to do with your disk being full in the traditional sense. The real culprit is almost always one of three things: inode exhaustion, a full tmpfs partition, or reserved block space on an ext4 filesystem. This guide walks through each cause in diagnostic order, from most common to least.

Why Linux Reports "No Space Left" When Disk Space Is Available

Linux can refuse to write new data even with free disk blocks because block space is only one of several resources required to create a file. A filesystem needs both free data blocks (where file content lives) and a free inode (a metadata slot that records the file's existence, ownership, and permissions). Run out of either, and you get the same error.

The three most common root causes are:

  • Inode exhaustion — the filesystem has used every available inode slot, typically caused by millions of tiny cache files or session files accumulating over time.
  • Full tmpfs / /tmp — a RAM-backed filesystem mounted at /tmp or /run has hit its size limit, which is separate from your physical disk.
  • Reserved blocks on ext4 — by default, ext4 reserves 5% of total capacity for the root user. On a 2 TB drive, that's 100 GB of space that appears "used" to normal processes.

The diagnostic approach matters here. Jumping straight to deleting files without knowing which resource is exhausted can waste time and occasionally cause unintended data loss. Start with identification, then remediation.

Step 1: Check Inode Usage with df -i

Run df -i to check inode usage across all mounted filesystems. A filesystem at 100% inode utilization will show IUse% at 100% even if df -h shows plenty of free space.

df -i

The output columns are: Filesystem, Inodes (total), IUsed, IFree, and IUse%. Look for any partition where IUse% is at or near 100%. A typical culprit looks like this:

Filesystem      Inodes  IUsed   IFree IUse% Mounted on
/dev/sda1      3276800 3276800       0  100% /

If you see that, inode exhaustion is your problem. XFS dynamically allocates inodes and rarely hits this ceiling, but ext4 has a fixed inode count set at format time — once those slots fill up, no new files can be created regardless of available block space.

How to Free Up Inodes (Finding and Removing Small Files)

The fastest way to free inodes is to find and delete the directories containing massive numbers of small files. Inode exhaustion almost always comes from accumulated session files, PHP opcache fragments, mail queue entries, or application log shards.

Use this command to find which directories hold the most files:

find / -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head -20

This prints the parent directory of every file, counts occurrences, and sorts by frequency. The top entries are your targets. Common offenders include /var/spool/postfix, /var/lib/php/sessions, and application-specific cache directories.

Once you've identified the directory, verify its contents before deleting:

ls /var/lib/php/sessions | wc -l

To safely remove old session files (older than 24 hours in this example):

find /var/lib/php/sessions -type f -mtime +1 -delete

Avoid running rm -rf blindly on directories you haven't inspected. A few seconds of verification prevents accidental removal of active session data or mail queue items.

Step 2: Check for a Full tmpfs or /tmp Partition

If inode usage looks normal, check whether /tmp or another tmpfs mount is full. tmpfs is a RAM-based filesystem — it doesn't consume physical disk space, but it has a size cap (usually 50% of RAM by default on systemd systems) and can fill independently.

df -h | grep tmp

If /tmp shows 100% usage, the fix is straightforward. Clear it manually (after verifying nothing critical is running):

find /tmp -type f -atime +1 -delete

To resize a tmpfs mount without rebooting:

mount -o remount,size=4G /tmp

For a permanent change, edit /etc/fstab and update the size option on the tmpfs line for /tmp. Keep in mind that enlarging tmpfs reduces available RAM for running processes — on memory-constrained servers, this trade-off matters.

Step 3: Check Reserved Block Space on ext4

On ext4 filesystems, 5% of total capacity is reserved by default for the root user, which means non-root processes see that space as unavailable. On large drives, this reservation can silently consume enormous amounts of usable space.

Check the current reserved block percentage with:

tune2fs -l /dev/sda1 | grep -i reserved

You'll see output like Reserved block count: 1638400 and Block size: 4096. Multiply those to get reserved bytes. On a 2 TB partition, 5% equals roughly 100 GB sitting idle.

For non-root partitions (data drives, secondary volumes), reducing this to 1% is generally safe:

tune2fs -m 1 /dev/sda1

Don't reduce the reserved percentage to zero on your root partition. That 5% buffer exists so the system can still write logs and run administrative commands even when the disk is nearly full. Removing it entirely on / can make a full-disk situation unrecoverable without a live boot environment.

Other Less Common Causes (Deleted Files Still Held Open, /proc, ulimit)

If none of the above resolves the error, you're dealing with an edge case. Three worth checking:

Deleted Files Still Held Open by Processes

When a process deletes a file but still holds the file descriptor open, the kernel keeps the data blocks allocated until the process releases the handle. The file disappears from the directory listing, but the space isn't freed. This is a common source of confusion after log rotation.

Find these zombie file handles with:

lsof | grep deleted

The fix is to restart the process holding the handle. For a web server: systemctl restart nginx. No data loss occurs — the process will reopen its log files fresh.

Pseudo-Filesystem Quirks (/proc, /dev)

Writes to /proc or /dev can occasionally trigger the error when a kernel interface rejects the operation for reasons unrelated to disk space. If the error appears only when writing to paths under these directories, the issue is a kernel or driver configuration problem, not a storage problem.

File Descriptor Limits (ulimit)

Processes have a per-user and per-process limit on open file descriptors set by ulimit. Hitting this ceiling produces the same error in some contexts. Check the current limits:

ulimit -n
cat /proc/sys/fs/file-max

If a high-traffic application is consistently bumping against this limit, raise it in /etc/security/limits.conf or via the service's systemd unit file using LimitNOFILE.

Preventing "No Space Left on Device" Errors in the Future

Prevention comes down to monitoring the right metrics before they hit 100%, not just watching raw disk usage.

  • Monitor inodes alongside blocks. Tools like Prometheus with the node_exporter expose inode metrics. Set alerts at 80% inode utilization, not just disk utilization.
  • Configure log rotation properly. logrotate with aggressive retention policies (e.g., rotate 7, compress, delaycompress) prevents log directories from accumulating thousands of files.
  • Set tmpfs size explicitly in /etc/fstab rather than relying on system defaults. Knowing exactly how much RAM is allocated to /tmp removes one variable from future incidents.
  • Choose filesystem parameters carefully at format time. When formatting ext4 partitions intended for large numbers of small files (mail servers, cache stores), increase the inode ratio with mkfs.ext4 -i 4096 /dev/sdX to allocate more inodes per block. This can't be changed after formatting without a reformat.
  • Use XFS for workloads with unpredictable file counts. XFS allocates inodes dynamically from free space, so inode exhaustion is effectively impossible unless you've also run out of disk blocks.

A simple cron job can catch problems before they become outages:

df -i | awk 'NR>1 && $5+0 >= 80 {print "INODE WARNING: " $0}' | mail -s "Inode Alert" [email protected]

Frequently Asked Questions

What is the fastest command to diagnose "No Space Left on Device" on Linux?

Run df -h && df -i back to back. The first command checks block usage; the second checks inode usage. If either shows a partition at 100%, you've found the cause in under ten seconds.

Can inode exhaustion happen on XFS or only on ext4?

XFS dynamically allocates inodes from free block space, so it doesn't have a fixed inode ceiling. Inode exhaustion is effectively an ext4 (and ext3) problem. On XFS, running out of inodes means you've also run out of disk space.

How do I find which directory is consuming the most inodes?

Use find / -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head -20. This lists the top 20 directories by file count without crossing filesystem boundaries.

Is it safe to reduce the reserved block percentage with tune2fs?

Yes, on non-root data partitions. Reducing from 5% to 1% on a secondary drive carries minimal risk. On the root filesystem, keep at least 1-2% reserved so the system can function during disk-full events.

How do I check if a deleted file is still holding disk space open?

Run lsof | grep deleted. Any file listed there has been unlinked from the directory tree but is still consuming blocks. Restarting the process that holds the file descriptor frees the space immediately.

{{HOMEPAGE_LINKS}}