Oracle 12cR1 12.1.0.1 2-node RAC on CentOS 6.4 on VMware Workstation 9 – Part VII

Time Required: 60 minutes

Class Materials:

  • 20GB of disk space
  • an ASCII text editor

Next we are going to add shared disk for the RAC cluster to use.

In earlier releases of Oracle RAC, a mimimal RAC could be stood up on less than 5GB of ASM disk. With Oracle 12c that is no longer possible.

For this demonstration I am going to create a 20GB shared disk. If you are tight on space you just about shoe horn it into a 10GB disk.

Make sure both VMs are shut down. If you want to make a clone of your progress then now is your last chance to do so without the complication of shared external disk.

We will add a new disk to Ruggero first, and then add the same disk into Magda.

Using the VMware Workstation menu, right click on Ruggero and select Settings.

This will bring up the Virtual Machine Settings menu. Click on the Add button.

Add Hardwre Wizard

At the Add Hardware Wizard menu, select Hard Disk and click Next.

Select a Disk

Select Create a new virtual disk and click Next.

Select a Disk Type

When the Select a Disk Type menu appears, select SCSI (Recommended) and for mode check Indepenent and the radio button Persistent.

Specify Disk Capacity

When the Specify Disk Capacity menu appear, set the new disk size at 20GB. You can scrape by with 10GB if you disable archive-log mode and dispense with the sample schemas, but I want to keep these.

Check the All all disk space now. button and Store virtual disk as a single file

Specify Disk Capacity

NOTE: These last two options are very important for our RAC. Trying to create a shared disk with VMware Workstation using thin-provisioning will result in disk corruption.

Next VMware Workstation will ask where you want to store your new disk and what you want to call it. By default it will want to place the virtual disk in the same directory as the rest of the Ruggero VM. However this new disk is to be shared, so I prefer to place it in a separate directory.

VMware Workstation stores its VMs by default in the user’s Documents directory under a sub-directory called Virtual Machines. In this directory you will find sub-directories for both our VMs Ruggero and Magda.

I have created a new sub-directory called Shared Disk and then added a further sub-directory for each RAC cluster I have set up under VMware Workstation.

Since I am naming this cluster La Rondine, I am storing the shared virtual disk in the Shared Disk/La Rondine sub-directory with the name asmdisk01.vmdk.

VMware will ask you confirm your selection. Press Finish to confirm.

Now we need to move our new disk to a different virtual SCSI device within the VM. In order to avoid all manner of cluster disk issues, we need to place the shared disks on a different SCSI controller to the local disks.

Highlight the Ruggero VM in the VMware Workstation menu and select Settings. When the Virtual Machine Settings menu appears, highlight Hard Disk 2 and select the Advanced button on the right hand side.

The Hard Disk Advanced Settings menu allows to modify the Virtual device node. By default this will be set to SCSI 0:1, but we need to change this to SCSI 1:0. This will move our new disk to SCSI controller 1, device 0.

Next we will have to resort to a little hacking to make our shared disk really sharable. We are now leveraging VM functionality that is not really supported in VMware Workstation 9, but does appear to work, at least for demo purposes.

Navigate to the directory where the Ruggero VM files are stored, in most cases this will be Documents/Virtual Machines/Ruggero. In there you will find a file called Ruggero.vmx

The VMX file contains the hardware profile of the VMs we have created. Before we do anything else, make a copy of it so we have something to go back to if we make a mess.

Open the VMX file with an ASCII editor. Make sure you have an editor able to open and save in ASCII format. Word Processors are a poor choice here as they are more likely to add symbols that will corrupt the file.

Open the VMX file and add the following directives:

disk.EnableUUID = "TRUE"
disk.locking = "FALSE"
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize = "0" 
diskLib.dataCacheMinReadAheadSize = "0" 
diskLib.dataCachePageSize = "4096" 
diskLib.maxUnsyncedWrites = "0" 
scsi1.sharedBus = "virtual"

 
These changes force the VMs to not try to buffer reads and writes to the disk, which would result in block corruption to the shared database. The new directives can be added anywhere in the VMX file, although I tend to add them immediately above the SCSI settings.

The following is a sub-section taken from the VMX file of my Ruggero VM. I have not dumped the entire file for the sake of clarity.

replay.supported = "FALSE"
disk.EnableUUID = "TRUE"
disk.locking = "FALSE"
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize = "0" 
diskLib.dataCacheMinReadAheadSize = "0" 
diskLib.dataCachePageSize = "4096" 
diskLib.maxUnsyncedWrites = "0" 
scsi1.sharedBus = "virtual"
scsi0.pciSlotNumber = "16"
scsi0.present = "TRUE"
scsi0.virtualDev = "lsilogic"
scsi0:0.fileName = "Ruggero-000004.vmdk"
scsi0:0.present = "TRUE"
scsi0:0.redo = ""
scsi0:1.present = "FALSE"
scsi1.present = "TRUE"
scsi1.virtualDev = "lsilogic"
scsi1:0.fileName = "D:\Virtual Machines\Shared Disk\La Rondine\asmdisk01.vmdk"
scsi1:0.mode = "independent-persistent"
scsi1:0.present = "TRUE"
serial0.fileType = "thinprint"
serial0.present = "TRUE"
sound.autodetect = "TRUE"

 
Now navigate to the VMX file for Magda and add the same directives for disk, diskLib and scsi1. You do NOT need to use the VMware interface to make these changes. Editing the VMX file directly is the better option.

The following is taken from the VMX file of my Magda VM. Again I have not dumped the entire file for the sake of clarity. As you can see I have copied the entries for disk, diskLib and scsi1 from the Ruggero VMX file.

replay.supported = "FALSE"
disk.EnableUUID = "TRUE"
disk.locking = "FALSE"
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize = "0" 
diskLib.dataCacheMinReadAheadSize = "0" 
diskLib.dataCachePageSize = "4096" 
diskLib.maxUnsyncedWrites = "0" 
scsi1.sharedBus = "virtual"
scsi1.present = "TRUE"
scsi1.virtualDev = "lsilogic"
scsi1:0.fileName = "D:\Virtual Machines\Shared Disk\La Rondine\asmdisk01.vmdk"
scsi1:0.mode = "independent-persistent"
scsi1:0.present = "TRUE"
scsi0.pciSlotNumber = "16"
scsi0.present = "TRUE"
scsi0.virtualDev = "lsilogic"
scsi0:0.fileName = "Ruggero-cl1.vmdk"
scsi0:0.present = "TRUE"
scsi0:0.redo = ""
serial0.fileType = "thinprint"
serial0.present = "TRUE"
sound.autodetect = "TRUE"

 
Now you can start up both Ruggero and Magda. As you do so, you might notice a small warning in the bottom right hand corner of the VMware window that states:

Clustering is not supported for WMware Workstation – this setting will be ignored

Don’t worry about this. This is expected behavior.

When Ruggero is full booted up, we should be able to create a partition table on the new shared disk, which will be presented to Linux as /dev/sdb.

In the following example we create a partition at the 1MB offset.

[root@ruggero ~]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xd5b605e3.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): u
Changing display/entry units to sectors

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First sector (63-41943039, default 63): 2048
Last sector, +sectors or +size{K,M,G} (2048-41943039, default 41943039): 
Using default value 41943039

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

 
We have created a partition on the new shared disk and now we have to create a UDEV rule for the new disk.

If we have Oracle Enterprise Linux then we could use the formerly free ASMLib tool to stamp and manage the disks across our cluster. However Oracle no longer provides that option for non Oracle branded Linux, so we are going to use UDEV rules instead.

First we need to know the SCSI ID of the new disk. The SCSI ID is a unique identifier that is constant across all machines that access a physical disk. It allows us to know exactly which disk we are talking to regardless of which machine we might be accessing it from.

[root@ruggero ~]# scsi_id -g -u -d /dev/sdb
36000c2991392fe6ed8ed10c219bf3bb3

 
Next we are going to create a UDEV rule to manage this disk. Create a file called /etc/udev/rules.d/99-oracleasm.rules and add the following entries to the file as a single line:

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", 
RESULT=="36000c2991392fe6ed8ed10c219bf3bb3", NAME="oracleasm/asm-disk1", 
OWNER="oracle", GROUP="asmadmin", MODE="0660"

 
In the above example, I have instructed the UDEV rules processor to scan for devices with the name sd?1 meaning it will pick up disk partitions on SCSI disks.

If the SCSI ID matches the one shown, the disk will be presented as /dev/oracleasm/asm-disk1 and will be owned by oracle with a group ownership of asmadmin. This will allow ASM to manage the disk.

Now we can start the UDEV rules processor and determine if the disk is shown correctly:

[root@ruggero ~]# /sbin/start_udev
Starting udev: [  OK  ]
 
[root@ruggero Desktop]# ls -al /dev/ora*
total 0
drwxr-xr-x.  2 root   root        60 Jul  7 21:38 .
drwxr-xr-x. 19 root   root      3840 Jul  7 21:38 ..
brw-rw----.  1 oracle asmadmin 8, 17 Jul  7 21:38 asm-disk1

 
As you can see the new device is listed correctly with the expected ownership.

Now we need to add the shared disk to Magda. We already edited the VMX file and restarted the VM, so now log into Magda as root and run the fdisk command again:

[root@magda /]# fdisk /dev/sdb

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): u
Changing display/entry units to sectors

 
This time if we just print the partition table, we can see the partition created over on Ruggero.

Command (m for help): p

Disk /dev/sdb: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xd5b605e3

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048    41943039    20970496   83  Linux

 
Now type w to write the partition table and exit fdisk.

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

 
Now we need to add UDEV rules to Magda. To ensure that I get the right ones, I am simply going to copy the existing file from Ruggero using scp.

If you are doing this for the first time you will need to confirm connecting Ruggero. You will also need to enter the root password to complete the copy.

[root@magda ~]# cd /etc/udev/rules.d
[root@magda rules.d]# scp root@ruggero:/etc/udev/rules.d/99-ora* .
root@ruggero's password: 
99-oracleasm.rules                            100%  196
     0.2KB/s   00:00 

 
Now we start the UDEV rules on Magda and check to see if the disk is visible:

[root@magda rules.d]# start_udev
Starting udev:                                             [  OK  ]
[root@magda rules.d]# ls -al /dev/ora*
total 0
drwxr-xr-x.  2 root   root        60 Jul  8 22:02 .
drwxr-xr-x. 19 root   root      3840 Jul  8 22:02 ..
brw-rw----.  1 oracle asmadmin 8, 17 Jul  8 22:02 asm-disk1

 
Now that we have shared disk in our cluster, we are now ready to install our 12c RAC database!

Article Quick Navigation
Previous Step Main Index Next Step
Advertisements

9 thoughts on “Oracle 12cR1 12.1.0.1 2-node RAC on CentOS 6.4 on VMware Workstation 9 – Part VII

  1. Hello,
    I have problem with string
    “We will add a new disk to Ruggero first, and then add the same disk into Magda.”
    You described first disk Ruggero, but regarding Magda is not clear.
    I have error, if repeat same steps like for Ruggero.
    Could you describe steps for Magda?
    Thank you,
    Alexej

    • Alexej,

      Where are you having a problem? Magda’s shared disk is added when you edit the VMX file.

      After the reboot Magda will have an /dev/sdb device, and you can use fdisk to create a partition table as shown later in the same blog post.

      UDEV rules should be identical on both Ruggero and Magda.

      HTH.

      Gruff

  2. Hi,
    First of all thank you very much for posting a very helpful document. I’m having trouble at part Vii. I added disk on Ruggero and did fdisk /de/sdb successfully. However, when I try to do the same on Magda, I’m getting “unable to open /dev/sdb.” error. I’ve copied all the parameters you mentioned in Magda’s .vmx file. Appreciate your help.

    Thanks
    Rob

  3. Hi, I have problem with this part

    [root@eurydice rules.d]# scp root@ruggero:/etc/udev/rules.d/99-ora* .
    ssh: connect to host ruggero port 22: No route to host

    thanks in advance

  4. Hello

    This worked for me on CentOS 7

    KERNEL==”sd?1″, SUBSYSTEM==”block”, PROGRAM==”/usr/lib/udev/scsi_id -g -u -d /dev/$parent”, RESULT==”36000c2947d950bbbcf8d2653a4f60bbb”, SYMLINK+=”oracleasm/asm-disk1″, OWNER=”oracle”, GROUP=”asmadmin”, MODE=”0660″

    Thanks for the awesome guide!

    • The 2048 sector offset means that with 512-byte sectors, the partition will be 1MB. This ensures fully aligned I/O against the underlying disks.

  5. hi

    I did a reboot after the last step of Part VII and i always cant start up the centOS. It always appear a black screen..

    may I know why? Or should I finish the guide from VII to XI?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s