Oracle 11gR2 2-node RAC on VMware Workstation 8 – Part VIII

Time Required: 60 minutes

Class Materials:

• 5GB of disk space
• an ASCII text editor

Next we are going to add some shared disk to our new freshly minted VMs

VMware Workstation makes the allocation of shared disk to VMs very simple. Shared disk has been the biggest obstacle to create Oracle RAC clusters at home, but now VMware gives us a reliable and portable solution that does not require fire-wire hacks, your own NFS server or a SCSI disk array.

Best of all this whole solution can exist on a single laptop, so you can take your RAC on the road.

Make sure both VMs are shut down. If you want to make a clone of your progress then now is your last chance to do so without the complication of shared external disk.

We will add a new disk to Orpheus first, and then add the same disk into Eurydice.

Using the VMware Workstation menu, right click on Orpheus and select Settings.

This will bring up the Virtual Machine Settings menu. Click on the Add button.

At the Add Hardware Wizard menu, select Hard Disk and click Next.

Select Create a new virtual disk and click Next.

When the Select a Disk Type menu appears, select SCSI (Recommended) and for mode check Indepenent and the radio button Persistent.

When the Specify Disk Capacity menu appear, set the new disk size at 5GB. This should be just enough to support our demo RAC database. If you have an abundance of disk space then by all means set the disk size much larger.

Check the All all disk space now. button and Store virtual disk as a single file

NOTE: These last two options are very important for our RAC. Trying to create a shared disk with VMware Workstation using thin-provisioning will result in disk corruption.

Next VMware Workstation will ask where you want to store your new disk and what you want to call it. By default it will want to place the virtual disk in the same directory as the rest of the Orpheus VM. However this new disk is to be shared, so I prefer to place it in a separate directory.

VMware Workstation stores its VMs by default in the user’s Documents directory under a sub-directory called Virtual Machines. In this directory you will find sub-directories for both our VMs Orpheus and Eurydice. I suggest creating a new sub-directory here called Shared Disk and storing our new virtual disk in the Shared Disk directory with the name asm disk 1.vmdk.

VMware will ask you confirm your selection. Press Finish to confirm.

Now we need to move our new disk to a different virtual SCSI device within the VM. In order to avoid all manner of cluster disk issues, we need to place the shared disks on a different SCSI controller to the local disks.

Highlight the Orpheus VM in the VMware Workstation menu and select Settings. When the Virtual Machine Settings menu appears, highlight Hard Disk 2 and select the Advanced button on the right hand side.

The Hard Disk Advanced Settings menu allows to modify the Virtual device node. By default this will be set to SCSI 0:1, but we need to change this to SCSI 1:0. This will move our new disk to SCSI controller 1, device 0.

Next we will have to resort to a little hacking to make our shared disk really sharable. We are now leveraging VM functionality that is not really supported in VMware Workstation 8, but does appear to work, at least for demo purposes.

Navigate to the directory where the Orpheus VM files are stored, in most cases this will be Documents/Virtual Machines/Orpheus. In there you will find a file called Orpehus.vmx

The VMX file contains the hardware profile of the VMs we have created. Before we do anything else, make a copy of it so we have something to go back to if we make a mess.

Open the VMX file with an ASCII editor. Make sure you have an editor able to open and save in ASCII format. Word Processors are a poor choice here as they are more likely to add symbols that will corrupt the file.

The VMX file is a unsorted mess. Why VMware doesn’t perform a simple sort on the file before saving it to disk is a mystery. Still at least it isn’t some gigantic XML monstrosity. I recommend that if your editor allows it, you perform a sort on the entire file and save it back to disk before making any changes.

This is where the magic happens, and I have to acknowledge it was Dan Norris who figured this out, not me. In order to make our new disk fully clusterable, we are going to add the following directives to the VMX file:

disk.locking = "FALSE"
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize = "0" 
diskLib.dataCacheMinReadAheadSize = "0" 
diskLib.dataCachePageSize = "4096" 
diskLib.maxUnsyncedWrites = "0" 
scsi1.sharedBus = "virtual"

 
These changes force the VMs to not try to buffer reads and writes to the disk, which would result in block corruption to the shared database. The new directives can be added anywhere in the VMX file, although I tend to add them immediately above the SCSI settings.

The following is a sub-section taken from the VMX file of my Orpheus VM. I have not dumped the entire file for the sake of clarity.

replay.supported = "FALSE"
disk.locking = "FALSE"
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize = "0" 
diskLib.dataCacheMinReadAheadSize = "0" 
diskLib.dataCachePageSize = "4096" 
diskLib.maxUnsyncedWrites = "0" 
scsi1.sharedBus = "virtual"
scsi0.pciSlotNumber = "16"
scsi0.present = "TRUE"
scsi0.virtualDev = "lsilogic"
scsi0:0.fileName = "local disk 1-cl2.vmdk"
scsi0:0.present = "TRUE"
scsi0:0.redo = ""
scsi0:1.present = "FALSE"
scsi1.present = "TRUE"
scsi1.virtualDev = "lsilogic"
scsi1:0.fileName = "C:\Users\thorng1.CORP\Documents\Virtual Machines\Shared Disk\asm disk 1.vmdk"
scsi1:0.mode = "independent-persistent"
scsi1:0.present = "TRUE"
serial0.fileType = "thinprint"
serial0.present = "TRUE"
sound.autodetect = "TRUE"

 
Now navigate to the VMX file for Eurydice and add the same directives for diskLib and scsi1. You do NOT need to use the VMware interface to make these changes. Editing the VMX file directly is the better option.

The following is taken from the VMX file of my Eurydice VM. Again I have not dumped the entire file for the sake of clarity. As you can see it should match Orpheus exactly.

replay.supported = "FALSE"
disk.locking = "FALSE"
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize = "0" 
diskLib.dataCacheMinReadAheadSize = "0" 
diskLib.dataCachePageSize = "4096" 
diskLib.maxUnsyncedWrites = "0" 
scsi1.sharedBus = "virtual"
scsi0.pciSlotNumber = "16"
scsi0.present = "TRUE"
scsi0.virtualDev = "lsilogic"
scsi0:0.fileName = "local disk 1-cl2.vmdk"
scsi0:0.present = "TRUE"
scsi0:0.redo = ""
scsi1.present = "TRUE"
scsi1.virtualDev = "lsilogic"
scsi1:0.fileName = "C:\Users\thorng1.CORP\Documents\Virtual Machines\Shared Disk\asm disk 1.vmdk"
scsi1:0.mode = "independent-persistent"
scsi1:0.present = "TRUE"
serial0.fileType = "thinprint"
serial0.present = "TRUE"
sound.autodetect = "TRUE"

 
Now you can start up both Orpheus and Eurydice. As you do so, you might notice a small warning in the bottom right hand corner of the VMware window that states:

Clustering is not supported for WMware Workstation – this setting will be ignored

Don’t worry about this. This is expected behavior.

When Orpheus is full booted up, we should be able to create a partition table on the new shared disk, which will be presented to Linux as /dev/sdb.

In the following example we create a partition at the 1MB offset.

[root@orpheus ~]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): u
Changing display/entry units to sectors

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First sector (63-10485759, default 63): 2048
Last sector or +size or +sizeM or +sizeK (2048-10485759, default 10485759): 
Using default value 10485759

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

 
Okay cool. We have created a partition on the new shared disk. Now we can stamp it for use by ASM by using the ASMLib we installed back in Part V.

[root@orpheus ~]# oracleasm createdisk DATA /dev/sdb1
Writing disk header: done
Instantiating disk: done

To verify the operation worked as expected, we can list the ASM disks as follows:

[root@orpheus ~]# oracleasm listdisks
DATA

 
Now we need to see if the changes we just did are visible to Eurydice. Log into Eurydice as root and start the fdisk command. Now if we list the partition table we should see that one already exists, as we just created it on Orpheus.

[root@eurydice ~]# fdisk /dev/sdb

The number of cylinders for this disk is set to 1448.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sdb: 5368 MB, 5368709120 bytes
181 heads, 40 sectors/track, 1448 cylinders
Units = cylinders of 7240 * 512 = 3706880 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1        1449     5241856   83  Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

 
Note: It is important to re-write the partition header table from fdisk so that Eurydice can see the partitioned device /dev/sdb1.

Now we can use ASMLib on Eurydice to scan the devices for any ASM stamped disks:

[root@eurydice ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...

Finally if we check to see what was found, we should see that the ASM disk for DATA has been discovered!

[root@eurydice ~]# oracleasm listdisks
DATA

 
The shared disk is visible to both Orpheus and Eurydice. We are now ready to install our RAC database!

Article Quick Navigation
Previous Step	Main Index	Next Step