Directory index full!

Today I encountered with short system hang in my 2 node RAC database.When I investigate the cause of the problem a saw “Kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!” warning in the /var/log/messages file.

Jan 22 14:45:09 mydb1 kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!
Jan 22 14:45:09 mydb1 kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!
Jan 22 14:45:09 mydb1 kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!
Jan 22 14:45:16 mydb1 kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!
Jan 22 14:45:17 mydb1 kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!
Jan 22 14:45:17 mydb1 kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!
Jan 22 14:45:17 mydb1 kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!
Jan 22 14:45:17 mydb1 kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!

The “Directory index full!” warning means that there is a problem with inode’s .
To check inode’s

[oracle@esddb1 ~]$ df -i

Here we saw that in root directory percentage of the used idnode’s is 22 %.
Let’s look what  says RedHat knowledgebase about  this warning:

• The ‘directory index full’ error will be seen if there are lots of files/directories in the filesystem so that the tree reaches its indexing limits and cannot keep track further.
• There is a limit in ext4 of the directory structure, a directory on ext4 can have at most 64000 subdirectories.

I found that there are 14 million *.aud files was generated under  /u01/app/12.1.0.2/grid/rdbms/audit/ directory.

# for dir in `ls -1`; do echo $dir; find ./$dir -type f|wc -l; done

The cause of this problem is audit files which created for each connection which connects as sys user. This files needed for security compliance reasons and we can delete old files. Old *.aud files not needed by any Oracle ASM process and can be deleted without any impact to the system.

You can clean old *.aud files like below:

[oracle@mydb1 ~]$ cd /u01/app/12.1.0.2/grid/rdbms/audit
[oracle@mydb1 audit]$ find /u01/app/grid/11.2.0/grid/rdbms/audit -maxdepth 1 -name '*.aud' -mtime +10 -delete -print

After cleanup, we can saw that percentage of free inodes increased and there are no WARNING messages in /var/log/messages file:

[oracle@mydb1 ~]$ df -i 
[oracle@mydb1 ~]$ less /var/log/messages