Some time ago I wrote a short blog about dependencies between the number of enabled CPUs and how many databases you could build. Today we got another error when we were trying to create a new database. Here is the screenshot of the error.
If you can’t read it on a small screen it says “Create Database operation failed due to an unknown error. Refer to work request ID 2580d3ff-064e-4e6f-ab06-1327fd02f40e when opening a Service Request at My Oracle Support.” and provide an error code which is “Error“
I don’t want to reiterate how helpful such a message is and how much valuable such information is in the troubleshooting process. But having some experience with the ExaCC I went straight to the last created log in the “/var/opt/oracle/log//dbaasapi/db/createdb” directory . In the log the reason was stated absolutely clear:
_TESTDB.cfg matching passwdINFO: freespace in RECOC1 diskgroup is : 11473 GBINFO: freespace in DATA diskgroup is : 0 GB[WARNING] [DBAAS-11000] - The following location does not have enough space: DATA. CAUSE: The minimum required free space is 50 GB. ACTION: Verify the disk space usage and free an additional space of 0 GB and try again.
So it says “The following location does not have enough space: DATA. CAUSE: The minimum required free space is 50 GB.”. Let’s have a look at the diskgroup DATA and find out how much free space I have.
SQL> SELECT name,total_mb/1024/3 total_gb, free_mb/1024/3 free_gb, usable_file_mb FROM v$asm_diskgroup; NAME TOTAL_GB FREE_GB USABLE_FILE_MB ------------------------------ ---------- ---------- -------------- DATAC1 91956 2217.47266 -2960582 RECOC1 30648 14953.1133 13568457 SPRC1 306480 296081.004 285751641 SQL>
It appears I have enough space available but my usable space is negative. Is it possible that the ExaCC tool checks usable space instead of free space? I checked the tools code and in the Perl module rac.pm found that indeed Oracle had conditions for the usable space in the diskgroup.
0;\n" if ($get_raw_free =~ m/no/i);
print FS "col name format a30 \nselect name,free_mb from v\$asm_diskgroup where usable_file_mb > 0;\n" if ($get_raw_free !~ m/no/i);
...
redacted
...
It seems it returns correct value for free space only if the usable_file_mb value is positive.
A short summary. When you monitor free space in the diskgroup on your ExaCC please include the usable_file_mb to the monitoring. If you exhaust the usable free space (which is different from the mirrored free space) it will not affect any of your existing databases but prevent you from creating a new one.