Directory index full!

Today I encountered with short system hang in my 2 node RAC database.When I investigate the cause of the problem a saw “Kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!” warning in the /var/log/messages file.

Jan 22 14:45:09 mydb1 kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!
Jan 22 14:45:09 mydb1 kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!
Jan 22 14:45:09 mydb1 kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!
Jan 22 14:45:16 mydb1 kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!
Jan 22 14:45:17 mydb1 kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!
Jan 22 14:45:17 mydb1 kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!
Jan 22 14:45:17 mydb1 kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!
Jan 22 14:45:17 mydb1 kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!

The “Directory index full!” warning means that there is a problem with inode’s .
To check inode’s

[oracle@esddb1 ~]$ df -i

Here we saw that in root directory percentage of the used idnode’s is 22 %.
Let’s look what  says RedHat knowledgebase about  this warning:

• The ‘directory index full’ error will be seen if there are lots of files/directories in the filesystem so that the tree reaches its indexing limits and cannot keep track further.
• There is a limit in ext4 of the directory structure, a directory on ext4 can have at most 64000 subdirectories.

I found that there are 14 million *.aud files was generated under  /u01/app/12.1.0.2/grid/rdbms/audit/ directory.

# for dir in `ls -1`; do echo $dir; find ./$dir -type f|wc -l; done

The cause of this problem is audit files which created for each connection which connects as sys user. This files needed for security compliance reasons and we can delete old files. Old *.aud files not needed by any Oracle ASM process and can be deleted without any impact to the system.

You can clean old *.aud files like below:

[oracle@mydb1 ~]$ cd /u01/app/12.1.0.2/grid/rdbms/audit
[oracle@mydb1 audit]$ find /u01/app/grid/11.2.0/grid/rdbms/audit -maxdepth 1 -name '*.aud' -mtime +10 -delete -print

After cleanup, we can saw that percentage of free inodes increased and there are no WARNING messages in /var/log/messages file:

[oracle@mydb1 ~]$ df -i 
[oracle@mydb1 ~]$ less /var/log/messages

 

How to automatically copy archivelogs from ASM to local disc

This script will create folder on local disk, copy archivelogs of yesterday to this folder and compress folder with zip

#!/bin/bash
. /home/oracle/.bash_profile
cd /home/oracle/backup/rman/archivelogs/
mkdir `date -d "1 day ago" "+%Y_%m_%d"`
. /home/oracle/grid_env
for i in $(asmcmd ls +FRA/TESTDB/ARCHIVELOG/`date -d "1 day ago" "+%Y_%m_%d"`);
do asmcmd cp +FRA/TESTDB/ARCHIVELOG/`date -d "1 day ago" "+%Y_%m_%d"`/$i /home/oracle/backup/rman/archivelogs/`date -d "1 day ago" "+%Y_%m_%d"`;
done
zip -r `date -d "1 day ago" "+%Y_%m_%d"`.zip `date -d "1 day ago" "+%Y_%m_%d"`
rm -r `date -d "1 day ago" "+%Y_%m_%d"`

OCR üçün mirror diskgroup-un yaradılması, OCR backupın alınması

İlk olaraq Virtualbox-da 2 ədəd disk yaradırıq. Bunun üçün VirtualBoxManager-i açıb diski əlavə edəcəyimiz nod-lardan birini seçirik, daha sonra Settings->Storage->Create new disk seçirik: