Inodes used 100% while filesystem is not full

Problem(Abstract)

This document addresses what steps to take when a filesystem has 100% inodes used, but the filesystem itself is not 100% used. This document applies to AIX version 5.

Symptom

Unable to create any new files in filesystem.

Cause

100% inodes used in the filesystem.

Environment

Diagnosing the problem

Resolving the problem



JFS
1) When inode % reach 100%, even if the filesystem has free space, new
files/links/directories can not be created.
To have more available inodes, delete the un-necessary files/links/
directories.
Alternatively, enlarge the JFS or backup the JFS, recreate the JFS
with smaller nbpi value and restore data from the backup to the
newly created JFS. (if its not a default rootvg filesystem)

JFS2
2) JFS2 does not have the nbpi parameter. JFS2 manages the inode creation
dynamically. In theory, the number of inodes could be unlimited,
given that the filesystem is large enough.
Of course when the J2 filesystem is fully used, no inodes can be
created.

Increasing the size of a JFS2 filesystem increases the number of blocks
allocated to the filesystem but does not change anything about
the number of inodes allocated or in use.

When a new file or directory is created in a JFS2 filesystem, the JFS2 code determines
if it should allocate more space for inodes based on some allocation policies selected
to help filesystem access performance. If the filesystem determines it does not need to
allocate more space for inodes yet, then the %Iused will go up since more inodes are being
used by the filesystem but no more space has been allocated to inodes yet. If the
filesystem determines it does need to allocate more space for inodes, then the %Iused value
could go down since more space for inodes was allocated which lowers the percent being used.


Jfs2 should allocate inodes as needed in 16k chunks. However it
is possible if filesystem is fragmented to have a lot of space available
but not have contiguous 16k chunks to use for a new group of inodes.
Because of this, a new feature was added with a new parameter for chfs.
This parameter, vix which stands for "variable inode extent", allows the
size of an inode extent (an inode extent describes where inodes are)
to have a size less then 16k. This allows the operating system to fit
new inodes into regions less then 16k and takes care of problem.

APAR IY68174 (on AIX 5.3) was
applied, which support the JFS2 variable inode extent size.

Try the following:
#chfs -a vix=yes <filesystem name>
when you run
#lsfs -q <filesystem name>
you should see VIX=YES

See if this solves the problem.

Other causes of 100% inode usage:

1). There are files within the filesystems that are opened by applications but have been removed from
the filesystem tree. This behavior is documented in the unlink() system call as follows.

When all links to a file are removed and no process has the file open, all resources associated with
the file are reclaimed, and the file is no longer accessible. If one or more processes have the file
open when the last link is removed, the directory entry disappears.

However, the removal of the file contents is postponed until all references to the file are closed.

You can use the fuser command with the -dV flag on the full path to the device on which the filesystem
resides. This will display files that have been removed but are still open. It will also report the
inode number and size of such files. Using the process ID returned for these files, you can instruct
the source application to close these files, or you can exit the application. Once this has occurred,
and fuser no longer shows this deleted file, the space will be returned to the filesystem for general use.

If the filesystem had a shared library that was deleted and the process that used the library is no
longer active, the library will still be open on the loader list. fuser will not detect these situations,
but they can be remedied by running the slibclean command. This will flush any shared libraries from
the loader list that are no longer active, and if they were deleted, the space will then be reclaimed.

2). The presence of many tiny files will (in jfs) cause the inode usage to be consumed, but
the filesystem will not be full.


Use this command to locate small files. Run the command for the appropriate filesystem.
/ (root) is used in this example.

find / -xdev -size -2048 -ls |sort -r +6 >> smallfiles.out

Then vi the smallfiles.out file to identify if the system has an abundance of these.

One example:

Many files in the /dev directory of the type __pv##.

__pv* files are temporary and created when the physical device is opened
for I/O. They should be removed when the I/O completes. IF they are not:

Varyoff the vgs, after unmounting the filesystems, so that no I/O is being done.
Then you can safely remove the __pv## files. You can alternatively set the vgs to not
autovaryon, and reboot the system so that only rootvg is active upon boot time, and then
proceed with removing the __pv## files in the /dev directory.

Any other tiny files, when there are many of them, will consume the inodes. Auditing,
homegrown scripting, anything that is generating a number of small files. These can be
identified using the above find command.