Oracle OCI Database service storage allocation.

Today I would like to discuss the block storage allocation in a VM based Oracle DBCS system. Several times in different conversations it was mentioned that the block storage will be allocated with triple redundancy on the ASM level. Let’s check it out.
If we try to allocate the minimum size volume 256GB for an Oracle VM based DBCS it shows the total storage as 712GB.
Screen Shot 2020-01-06 at 10.08.39 AM.png
But why does it show the 712GB?
Screen Shot 2020-01-06 at 10.11.22 AM.png
And, if we increase the initial storage allocation to 1024GB the total allocation will grow to 1480Gb.
Screen Shot 2020-01-06 at 10.13.28 AM.png
It is not the triple allocation of storage. But where are the additional 456GB?
Let’s have a look at the actual allocation in the ASM on one of the DBCS VM.
I’ve created a DBCS VM with 256GB ASM based storage for data and here are block storage volumes presented to the system:

[grid@gleborcl ~]$ lsblk
NAME                        MAJ:MIN    RM  SIZE RO TYPE MOUNTPOINT
sda                           8:0       0   58G  0 disk
|-sda1                        8:1       0  486M  0 part /boot/efi
|-sda2                        8:2       0  1.4G  0 part /boot
`-sda3                        8:3       0 52.2G  0 part
|-VolGroupSys4-LogVolRoot 249:0       0   35G  0 lvm  /
`-VolGroupSys4-LogVolSwap 249:1       0   16G  0 lvm  [SWAP]
sdb                           8:16      0   64G  0 disk
sdc                           8:32      0   64G  0 disk
sdd                           8:48      0   64G  0 disk
sde                           8:64      0   64G  0 disk
sdf                           8:80      0   64G  0 disk
sdg                           8:96      0   64G  0 disk
sdh                           8:112     0   64G  0 disk
sdi                           8:128     0   64G  0 disk
sdj                           8:144     0  200G  0 disk /u01
asm!commonstore-330         248:168961  0    5G  0 disk /opt/oracle/dcs/commonstore

We have eight 64GB disks attached to the system as volumes and 200GB as a volume for Oracle binaries. It gives us exactly 712GB in total. And here we can see how the eight volumes are used.

SQL> SELECT name,path,total_mb FROM v$asm_disk ORDER BY 1;
 
NAME			       PATH				TOTAL_MB
------------------------------ ------------------------------ ----------
DATA_0000		       /dev/DATADISK3			   65536
DATA_0001		       /dev/DATADISK2			   65536
DATA_0002		       /dev/DATADISK1			   65536
DATA_0003		       /dev/DATADISK4			   65536
RECODISK1		       /dev/RECODISK1			   65536
RECODISK2		       /dev/RECODISK2			   65536
RECODISK3		       /dev/RECODISK3			   65536
RECODISK4		       /dev/RECODISK4			   65536
 
8 ROWS selected.
 
SQL> SELECT name,TYPE,total_mb FROM v$asm_diskgroup;
 
NAME			       TYPE	TOTAL_MB
------------------------------ ------ ----------
DATA			       EXTERN	  262144
RECO			       EXTERN	  262144
 
SQL>

The data and reco disk groups are created with external redundancy and have four 64GB disks which give us 256GB usable space for each group. And, by the way, if you are wondering whether Oracle uses ASMLib or AFD here is how Oracle provides disks names and permissions.

[grid@gleborcl ~]$ cat /etc/udev/rules.d/70-names.rules
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360e2e6804e814b04bf647bbd60c92978", SYMLINK+="DATADISK1",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="3600984cc51b945ae9142bb8a6890c444", SYMLINK+="DATADISK2",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360e7ee9971f5452caaff44c6f0b0ea2f", SYMLINK+="DATADISK3",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36007a37863594be48f687a92b73b6ba8", SYMLINK+="DATADISK4",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360446c5789094176991e3773b7503877", SYMLINK+="RECODISK1",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360a67d4641874d9ca8a2a22f8719ca56", SYMLINK+="RECODISK2",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360a4a4df343041e3b45b88ebe39d67cd", SYMLINK+="RECODISK3",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36019ec16293445d19fe8a3734792ec19", SYMLINK+="RECODISK4",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36093853c644b4e619a2f7ead2b8f38ee", SYMLINK+="localdisk",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
[grid@gleborcl ~]$

It is provided by the UDEV rules. The ASMlib and AFD are not used in the Oracle cloud.
What if we scale storage up to 1024GB?
Screen Shot 2020-01-06 at 12.19.16 PM.png

Oracle attaches new 256GB volumes to the system

[grid@gleborcl ~]$ lsblk
NAME                        MAJ:MIN    RM  SIZE RO TYPE MOUNTPOINT
sda                           8:0       0   58G  0 disk
|-sda1                        8:1       0  486M  0 part /boot/efi
|-sda2                        8:2       0  1.4G  0 part /boot
`-sda3                        8:3       0 52.2G  0 part
|-VolGroupSys4-LogVolRoot 249:0       0   35G  0 lvm  /
`-VolGroupSys4-LogVolSwap 249:1       0   16G  0 lvm  [SWAP]
sdb                           8:16      0   64G  0 disk
sdc                           8:32      0   64G  0 disk
sdd                           8:48      0   64G  0 disk
sde                           8:64      0   64G  0 disk
sdf                           8:80      0   64G  0 disk
sdg                           8:96      0   64G  0 disk
sdh                           8:112     0   64G  0 disk
sdi                           8:128     0   64G  0 disk
sdj                           8:144     0  200G  0 disk /u01
sdk                           8:160     0  256G  0 disk
sdl                           8:176     0  256G  0 disk
sdm                           8:192     0  256G  0 disk
sdn                           8:208     0  256G  0 disk
asm!commonstore-330         248:168961  0    5G  0 disk /opt/oracle/dcs/commonstore
[grid@gleborcl ~]$
SQL> SELECT name,TYPE,total_mb FROM v$asm_diskgroup;
 
NAME			       TYPE	TOTAL_MB
------------------------------ ------ ----------
DATA			       EXTERN	 1048576
RECO			       EXTERN	  262144
 
SQL>

And after rebalancing operation, the old four 64GB disks for the data disk group are deleted and detached from the system.

[grid@gleborcl ~]$ lsblk
NAME                        MAJ:MIN    RM  SIZE RO TYPE MOUNTPOINT
sda                           8:0       0   58G  0 disk
|-sda1                        8:1       0  486M  0 part /boot/efi
|-sda2                        8:2       0  1.4G  0 part /boot
`-sda3                        8:3       0 52.2G  0 part
|-VolGroupSys4-LogVolRoot 249:0       0   35G  0 lvm  /
`-VolGroupSys4-LogVolSwap 249:1       0   16G  0 lvm  [SWAP]
sdf                           8:80      0   64G  0 disk
sdg                           8:96      0   64G  0 disk
sdh                           8:112     0   64G  0 disk
sdi                           8:128     0   64G  0 disk
sdj                           8:144     0  200G  0 disk /u01
sdk                           8:160     0  256G  0 disk
sdl                           8:176     0  256G  0 disk
sdm                           8:192     0  256G  0 disk
sdn                           8:208     0  256G  0 disk
asm!commonstore-330         248:168961  0    5G  0 disk /opt/oracle/dcs/commonstore
[grid@gleborcl ~]$

Now we have four 64GB disks for the reco disk group, four 256GB disks for data and the same 200GB for binaries totaling 1480GB. The binaries and the reco disks allocations are the same for data from 256 to 1024GB. For a 2048GB data disk group, it is different with 408GB for the reco disk group (4x102GB). With the bigger data storage allocation, the size of the reco disk group is increasing.

So, Oracle is using external redundancy ASM disk groups relying on the storage layer as a safety net. Planning the storage allocation is a bit tricky and we need to verify how much block storage is going to be allocated for each size of the data disk group. I tried to find it in the documentation but was not able to locate it. Probably the easiest way is to go the OCI console and push the button to scale storage for any DBCS system or the button to create a new DBCS. It will show how much storage in total is going to be allocated. I hope it might help to properly plan and estimate the cost of the resources on OCI.