Directory index full!


Today I encountered with short system hang in my 2 node RAC database.When I investigate the cause of the problem a saw “Kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!” warning in the /var/log/messages file.

Jan 22 14:45:09 mydb1 kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!
Jan 22 14:45:09 mydb1 kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!
Jan 22 14:45:09 mydb1 kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!
Jan 22 14:45:16 mydb1 kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!
Jan 22 14:45:17 mydb1 kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!
Jan 22 14:45:17 mydb1 kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!
Jan 22 14:45:17 mydb1 kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!
Jan 22 14:45:17 mydb1 kernel: EXT4-fs warning (device sda1): ext4_dx_add_entry:2021: Directory index full!

The “Directory index full!” warning means that there is a problem with inode’s .
To check inode’s

[oracle@esddb1 ~]$ df -i

i556^cimgpsh_orig

Here we saw that in root directory percentage of the used idnode’s is 22 %.
Let’s look what  says RedHat knowledgebase about  this warning:

  • The ‘directory index full’ error will be seen if there are lots of files/directories in the filesystem so that the tree reaches its indexing limits and cannot keep track further.
  • There is a limit in ext4 of the directory structure, a directory on ext4 can have at most 64000 subdirectories.

I found that there are 14 million *.aud files was generated under  /u01/app/12.1.0.2/grid/rdbms/audit/ directory.

# for dir in `ls -1`; do echo $dir; find ./$dir -type f|wc -l; done

The cause of this problem is audit files which created for each connection which connects as sys user. This files needed for security compliance reasons and we can delete old files. Old *.aud files not needed by any Oracle ASM process and can be deleted without any impact to the system.

You can clean old *.aud files like below:

[oracle@mydb1 ~]$ cd /u01/app/12.1.0.2/grid/rdbms/audit
[oracle@mydb1 audit]$ find /u01/app/grid/11.2.0/grid/rdbms/audit -maxdepth 1 -name '*.aud' -mtime +10 -delete -print

After cleanup, we can saw that percentage of free inodes increased and there are no WARNING messages in /var/log/messages file:

[oracle@mydb1 ~]$ df -i 
[oracle@mydb1 ~]$ less /var/log/messages

 

Data Pump job fails with error – ORA-31634: job already exists


Today when I want to export schema with data pump, I encountered with an error ORA-31634: job already exists. I queried dba_datapump_jobs table and saw that it has 99 jobs with the NOT RUNNING state.

SELECT owner_name,
job_name,
operation,
job_mode,
state
FROM dba_datapump_jobs;

When we are not using job name for data pump job, Oracle generates default name to the job and in Oracle data pump can generate up to 99 unique jobs. When job name already exists or you are running many expdp jobs at the same time ( more than 99 jobs), then data pump cannot generate a unique name and you get this error. Another reason why this problem occurs is when jobs are aborted, or when KEEP_MASTER=y used for the data pump the records stay there.

There are two solutions to this problem:

The first solution is dropping this orphaned tables, use result of the query below to drop tables

SELECT 'DROP table ' || owner_name || '.' || job_name || ';'
FROM DBA_DATAPUMP_JOBS
WHERE STATE = 'NOT RUNNING';
DROP TABLE DP_USER.SYS_EXPORT_SCHEMA_01 PURGE;
…
...
DROP TABLE DP_USER.SYS_EXPORT_SCHEMA_99 PURGE; 

2) The second solution is to use unique job name in data pump jobs like below:

expdp dp_user/password directory=DP_DIR dumpfile=backup.dmp logfile=logfile.log job_name=your_unique_job_name schemas=schema_name

Now you can run data pump jobs without any problem