Today I would like to discuss the block storage allocation in a VM based Oracle DBCS system. Several times in different conversations it was mentioned that the block storage will be allocated with triple redundancy on the ASM level. Let’s check it out.
If we try to allocate the minimum size volume 256GB for an Oracle VM based DBCS it shows the total storage as 712GB.
But why does it show the 712GB?
And, if we increase the initial storage allocation to 1024GB the total allocation will grow to 1480Gb.
It is not the triple allocation of storage. But where are the additional 456GB?
Let’s have a look at the actual allocation in the ASM on one of the DBCS VM.
I’ve created a DBCS VM with 256GB ASM based storage for data and here are block storage volumes presented to the system:
[grid@gleborcl ~]$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 58G 0 disk |-sda1 8:1 0 486M 0 part /boot/efi |-sda2 8:2 0 1.4G 0 part /boot `-sda3 8:3 0 52.2G 0 part |-VolGroupSys4-LogVolRoot 249:0 0 35G 0 lvm / `-VolGroupSys4-LogVolSwap 249:1 0 16G 0 lvm [SWAP] sdb 8:16 0 64G 0 disk sdc 8:32 0 64G 0 disk sdd 8:48 0 64G 0 disk sde 8:64 0 64G 0 disk sdf 8:80 0 64G 0 disk sdg 8:96 0 64G 0 disk sdh 8:112 0 64G 0 disk sdi 8:128 0 64G 0 disk sdj 8:144 0 200G 0 disk /u01 asm!commonstore-330 248:168961 0 5G 0 disk /opt/oracle/dcs/commonstore
We have eight 64GB disks attached to the system as volumes and 200GB as a volume for Oracle binaries. It gives us exactly 712GB in total. And here we can see how the eight volumes are used.
SQL> SELECT name,path,total_mb FROM v$asm_disk ORDER BY 1; NAME PATH TOTAL_MB ------------------------------ ------------------------------ ---------- DATA_0000 /dev/DATADISK3 65536 DATA_0001 /dev/DATADISK2 65536 DATA_0002 /dev/DATADISK1 65536 DATA_0003 /dev/DATADISK4 65536 RECODISK1 /dev/RECODISK1 65536 RECODISK2 /dev/RECODISK2 65536 RECODISK3 /dev/RECODISK3 65536 RECODISK4 /dev/RECODISK4 65536 8 ROWS selected. SQL> SELECT name,TYPE,total_mb FROM v$asm_diskgroup; NAME TYPE TOTAL_MB ------------------------------ ------ ---------- DATA EXTERN 262144 RECO EXTERN 262144 SQL>
The data and reco disk groups are created with external redundancy and have four 64GB disks which give us 256GB usable space for each group. And, by the way, if you are wondering whether Oracle uses ASMLib or AFD here is how Oracle provides disks names and permissions.
[grid@gleborcl ~]$ cat /etc/udev/rules.d/70-names.rules KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360e2e6804e814b04bf647bbd60c92978", SYMLINK+="DATADISK1", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="3600984cc51b945ae9142bb8a6890c444", SYMLINK+="DATADISK2", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360e7ee9971f5452caaff44c6f0b0ea2f", SYMLINK+="DATADISK3", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36007a37863594be48f687a92b73b6ba8", SYMLINK+="DATADISK4", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360446c5789094176991e3773b7503877", SYMLINK+="RECODISK1", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360a67d4641874d9ca8a2a22f8719ca56", SYMLINK+="RECODISK2", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360a4a4df343041e3b45b88ebe39d67cd", SYMLINK+="RECODISK3", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36019ec16293445d19fe8a3734792ec19", SYMLINK+="RECODISK4", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36093853c644b4e619a2f7ead2b8f38ee", SYMLINK+="localdisk", OWNER="grid", GROUP="asmadmin", MODE="0660" [grid@gleborcl ~]$
It is provided by the UDEV rules. The ASMlib and AFD are not used in the Oracle cloud.
What if we scale storage up to 1024GB?
Oracle attaches new 256GB volumes to the system
[grid@gleborcl ~]$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 58G 0 disk |-sda1 8:1 0 486M 0 part /boot/efi |-sda2 8:2 0 1.4G 0 part /boot `-sda3 8:3 0 52.2G 0 part |-VolGroupSys4-LogVolRoot 249:0 0 35G 0 lvm / `-VolGroupSys4-LogVolSwap 249:1 0 16G 0 lvm [SWAP] sdb 8:16 0 64G 0 disk sdc 8:32 0 64G 0 disk sdd 8:48 0 64G 0 disk sde 8:64 0 64G 0 disk sdf 8:80 0 64G 0 disk sdg 8:96 0 64G 0 disk sdh 8:112 0 64G 0 disk sdi 8:128 0 64G 0 disk sdj 8:144 0 200G 0 disk /u01 sdk 8:160 0 256G 0 disk sdl 8:176 0 256G 0 disk sdm 8:192 0 256G 0 disk sdn 8:208 0 256G 0 disk asm!commonstore-330 248:168961 0 5G 0 disk /opt/oracle/dcs/commonstore [grid@gleborcl ~]$
SQL> SELECT name,TYPE,total_mb FROM v$asm_diskgroup; NAME TYPE TOTAL_MB ------------------------------ ------ ---------- DATA EXTERN 1048576 RECO EXTERN 262144 SQL>
And after rebalancing operation, the old four 64GB disks for the data disk group are deleted and detached from the system.
[grid@gleborcl ~]$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 58G 0 disk |-sda1 8:1 0 486M 0 part /boot/efi |-sda2 8:2 0 1.4G 0 part /boot `-sda3 8:3 0 52.2G 0 part |-VolGroupSys4-LogVolRoot 249:0 0 35G 0 lvm / `-VolGroupSys4-LogVolSwap 249:1 0 16G 0 lvm [SWAP] sdf 8:80 0 64G 0 disk sdg 8:96 0 64G 0 disk sdh 8:112 0 64G 0 disk sdi 8:128 0 64G 0 disk sdj 8:144 0 200G 0 disk /u01 sdk 8:160 0 256G 0 disk sdl 8:176 0 256G 0 disk sdm 8:192 0 256G 0 disk sdn 8:208 0 256G 0 disk asm!commonstore-330 248:168961 0 5G 0 disk /opt/oracle/dcs/commonstore [grid@gleborcl ~]$
Now we have four 64GB disks for the reco disk group, four 256GB disks for data and the same 200GB for binaries totaling 1480GB. The binaries and the reco disks allocations are the same for data from 256 to 1024GB. For a 2048GB data disk group, it is different with 408GB for the reco disk group (4x102GB). With the bigger data storage allocation, the size of the reco disk group is increasing.
So, Oracle is using external redundancy ASM disk groups relying on the storage layer as a safety net. Planning the storage allocation is a bit tricky and we need to verify how much block storage is going to be allocated for each size of the data disk group. I tried to find it in the documentation but was not able to locate it. Probably the easiest way is to go the OCI console and push the button to scale storage for any DBCS system or the button to create a new DBCS. It will show how much storage in total is going to be allocated. I hope it might help to properly plan and estimate the cost of the resources on OCI.