Oracle 12cR1 12.1.0.1 2-node RAC on CentOS 6.4 on VMware Workstation 9 – Part IV

Time Required: 60 minutes

Class Materials:

  • none

Next we are going to create two networks for our RAC system to talk on.

The Oracle RAC system requires at least two network connections between cluster nodes. One network will be the public IP and the second will be a private IP reserved for inter-cluster traffic.

VMware Workstation makes this setup very simple.

VMware Workstation allows us to create virtual networks for both tasks. We will also leave the current Bridged network adapter in place, allowing the 12c RAC to communicate with the outside world to download patches and such.

We will need a total of eight IP addresses to support our completed 12c RAC cluster. We will need static IP address on the public subnet for each host, plus a virtual IP for each host. In addition we need at least two SCAN IP addresses, plus two static IP addresses on the private subnet for cluster communication.

It is a good idea to make a note of these before you start the Oracle software install so that you are ready with the addresses you need.

In my example I am going to use the following:

Ruggero Magda
Public IP 10.10.1.110 10.10.1.120
Private IP 10.10.2.110 10.10.2.120
Virtual IP 10.10.1.111 10.10.1.121
SCAN IP 10.10.1.112 10.10.1.122

Shut down your CentOS 6.4 VM and use the VMware interface to access the the network editor using: Edit->Virtual Network Editor.

This allows us to create new virtual networks that allow our guest VMs to talk amongst themselves, to the host computer and/or to the outside world.

I am going to recycle virtual networks I used for my two previous RAC blog posts, but for those who have not read that I will briefly repeat the steps here.

For this example I am going to create VMnet2 and VMnet3.

VMnet2 will be a host-only network allowing communication to other VMs and the host, but not to the outside world. We will assign a Subnet IP of 10.10.1.0 to VMnet2, and a Subnet mask of 255.255.255.0

VMnet3 will also be a host-only network allowing communication to other VMs and the host, but not to the outside world. We will assign a Subnet IP of 10.10.2.0 to VMnet3, and a Subnet mask of 255.255.255.0

For both VMnet2 and VMnet3 DHCP is enabled.

Now we have our two networks created in VMware, we will add two new NICs to our VM.

Right click on the VM in the VMware Workstation menu, and select Settings. This will bring up the Virtual Machine Settings panel.

Now click Add

The Add Hardware Wizard allows us to select Network Adapter

On the Network Adapter Type menu, we will select the Custom: Specific virtual network radio button, and use the drop down to select VMnet2.

Now repeat the above steps to add another Network Adapter, this time using VMnet3.

Our final VM hardware configuration should look like this:

Now start up the VM and log in as root.

To set the machine name we are going to edit the file /etc/sysconfig/network.

I am going to name my first VM Ruggero and re-use the domain name of hadesnet which I created for my original RAC system. You can use whatever names you like.

[root@localhost etc]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=ruggero.hadesnet

 
Next we are going to add static IP addresses for Ruggero and the yet to be created Magda which will be the second node of our 12c RAC.

Edit the file /etc/hosts and add the bolded entries:

[root@localhost etc]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

# Public network
10.10.1.110 ruggero ruggero.hadesnet
10.10.1.120 magda magda.hadesnet
# VIP network
10.10.1.111 ruggero-vip ruggero-vip.hadesnet
10.10.1.121 magda-vip magda-vip.hadesnet
# Private network
10.10.2.110 ruggero-priv ruggero-priv.hadesnet
10.10.2.120 magda-priv magda-priv.hadesnet

 
Next we are going to edit the network adapters to set those static IP addresses. You can use the Linux desktop to do this, or you can directly edit the configuration files.

To edit the configuration files, shift to the directory /etc/sysconfig/network-scripts. You should see the following file ifcfg-eth0.

We need to edit this to add the directive PEERDNS=no shown here in bolded text:

[root@localhost network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE="eth0"
BOOTPROTO="dhcp"
HWADDR="00:0C:29:78:B0:63"
IPV6INIT="yes"
NM_CONTROLLED="yes"
ONBOOT="yes"
TYPE="Ethernet"
UUID="6d4ed643-7be9-4575-aba4-2783b8d9347d"
PEERDNS=no

 
This change will prevent Linux from trying to override the DNS server we are about to create.

NOTE: As soon as you save changes to this file, the interface adapter will reset. If you are logged into your Linux VM through telnet or ssh, your session will be terminated and you will have to log back in.

Next we are going to configure the interface for the cluster public network on VMnet2. This will have a subnet of 10.10.1.x.

Use the following command to determine which interface we need:

[root@localhost network-scripts]# ifconfig -a | egrep HWaddr -A 1
eth0      Link encap:Ethernet  HWaddr 00:0C:29:78:B0:63  
          inet addr:192.168.0.8  Bcast:192.168.0.255  Mask:255.255.255.0
--
eth1      Link encap:Ethernet  HWaddr 00:0C:29:78:B0:6D  
          inet addr:10.10.1.133  Bcast:10.10.1.255  Mask:255.255.255.0
--
eth2      Link encap:Ethernet  HWaddr 00:0C:29:78:B0:77  
          inet addr:10.10.2.128  Bcast:10.10.2.255  Mask:255.255.255.0

 
Interface eth1 has an IP address of 10.10.1.133, so this is the interface we are looking for. We can see that the MAC address is 00:0C:29:78:B0:6D.

We are going to create a file called /etc/sysconfig/network-scripts/ifcfg-Public with the following settings:

[root@localhost network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-Public
BOOTPROTO=none
PREFIX=24
DEFROUTE=yes
DNS1=10.10.1.110
DNS2=75.75.75.75
DOMAIN=hadesnet
IPADDR=10.10.1.110
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME=Public
ONBOOT=yes
TYPE=Ethernet
HWADDR=00:0C:29:78:B0:6D

 
Note how we have used the MAC address we discovered in the previous step, and we have set the IP address to 10.10.1.110. We have also defined a primary and secondary DNS server, and set the domain to hadesnet.

Next we are going to configure the cluster private network on VMnet3, which has a subnet of 10.10.2.x. With the command we used earlier, we can see that eth2 has an IP of 10.10.2.128 so this is the interface we are looking for. This interface has a MAC address of 00:0C:29:78:B0:77.

We are now going to create a file called /etc/sysconfig/network-scripts/ifcfg-Private with the following settings:

BOOTPROTO=none
PREFIX=24
DEFROUTE=yes
IPADDR=10.10.2.110
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME=Private
ONBOOT=yes
TYPE=Ethernet
HWADDR=00:0C:29:78:B0:77

 
Now we need to restart the network service to make sure our changes are recognized:

[root@localhost network-scripts]# service network restart
Shutting down interface Private:  Device state: 3 (disconnected)
                                                           [  OK  ]
Shutting down interface Public:  Device state: 3 (disconnected)
                                                           [  OK  ]
Shutting down interface eth0:  Device state: 3 (disconnected)
                                                           [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                            [  OK  ]
Bringing up interface Private:  Active connection state: activated
Active connection path: /org/freedesktop/NetworkManager/ActiveConnection/4
                                                           [  OK  ]
Bringing up interface Public:  Active connection state: activated
Active connection path: /org/freedesktop/NetworkManager/ActiveConnection/5
                                                           [  OK  ]
Bringing up interface eth0:  Active connection state: activating
Active connection path: /org/freedesktop/NetworkManager/ActiveConnection/6
state: activated
Connection activated
                                                           [  OK  ]

 
The network service has restarted, let’s check what IP addresses our interfaces are now using:

[root@localhost network-scripts]# ifconfig -a | egrep HWaddr -A 1
eth0      Link encap:Ethernet  HWaddr 00:0C:29:78:B0:63  
          inet addr:192.168.0.8  Bcast:192.168.0.255  Mask:255.255.255.0
--
eth1      Link encap:Ethernet  HWaddr 00:0C:29:78:B0:6D  
          inet addr:10.10.1.110  Bcast:10.10.1.255  Mask:255.255.255.0
--
eth2      Link encap:Ethernet  HWaddr 00:0C:29:78:B0:77  
          inet addr:10.10.2.110  Bcast:10.10.2.255  Mask:255.255.255.0

 
Okay good, we have the IP addresses we wanted. Now check the /etc/resolv.conf file. The changes we made stated that the cluster public ethernet adapter has a local DNS server, with a secondary one at 75.75.75.75.

[root@localhost ~]# cat /etc/resolv.conf
# Generated by NetworkManager
search hadesnet
nameserver 10.10.1.110
nameserver 75.75.75.75

 
This configuration allows us to create a local DNS server on Linux, but forward any requests we cannot resolve to the standard DNS server provided by your network administrator or home internet provider.

So next we need to create that DNS server. Back in Part II we loaded the RPMs to configure a DNS server. In case you missed it, it is the RPM bind-9.*.el6.x86_64.rpm and it is provided on the CentOS 6.4 install disc.

Now we need to edit the /etc/named.conf file to define our hadesnet domain and set up a DNS server on the local machine.

The important changes are shown in bold, but in fact you can simply replace the existing contents of your file with what is shown here.

The listen-on directive established a DNS server on port 53 of the cluster public NIC at 10.10.1.110.

The recursion directive is set to no to prevent abuse in a reflection DDoS attack. Maybe not likely in our VMware sandbox, but best practice nonetheless.

Finally we configure our new hadesnet domain in a file called hadesnet.zone.

//
// named.conf
//
// Provided by Red Hat bind package to configure the ISC BIND named(8) DNS
// server as a caching only nameserver (as a localhost DNS resolver only).
//
// See /usr/share/doc/bind*/sample/ for example named configuration files.
//

options {
	listen-on port 53 { 127.0.0.1; 10.10.1.110; };
	listen-on-v6 port 53 { ::1; };
	directory 	"/var/named";
	dump-file 	"/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
	allow-query     { any; };
	recursion no;

	dnssec-enable yes;
	dnssec-validation yes;
	dnssec-lookaside auto;

	/* Path to ISC DLV key */
	bindkeys-file "/etc/named.iscdlv.key";

	managed-keys-directory "/var/named/dynamic";
};

logging {
        channel default_debug {
                file "data/named.run";
                severity dynamic;
        };
};

zone "." IN {
        type hint;
        file "named.ca"; 
};

zone "hadesnet" IN {
        type master;
        file "hadesnet.zone";
        allow-update { none; };
};

include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";

 
Now let’s create that hadesnet.zone file. The file needs to be located in /var/named:

[root@localhost etc]# cat /var/named/hadesnet.zone
$TTL 86400
@   IN  SOA     ruggero.hadesnet. root.hadesnet. (
        2013042201  ;Serial
        3600        ;Refresh
        1800        ;Retry
        604800      ;Expire
        86400       ;Minimum TTL
)
@	IN  NS      larondine-scan.hadesnet.
larondine-scan     IN  A    10.10.1.112 
larondine-scan     IN  A    10.10.1.122 

 
The last two lines here are where the magic happens. We have defined the DNS alias larondine-scan to resolve to both 10.10.1.112 and 10.10.1.122.

Note that Oracle 12c RAC expects a SCAN address to resolve to at least three addresses, but since we only have two nodes we are going to limit this to two addresses.

Okay, we have made a lot of changes here. So let’s reboot our VM. When it comes back up, the root prompt should have changed to ruggero.

Let’s check that our static IP addresses have stuck on eth1 and eth2:

[root@ruggero ~]# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 00:0C:29:78:B0:63  
          inet addr:192.168.0.8  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe78:b063/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:18 errors:0 dropped:0 overruns:0 frame:0
          TX packets:30 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:10702 (10.4 KiB)  TX bytes:3657 (3.5 KiB)

eth1      Link encap:Ethernet  HWaddr 00:0C:29:78:B0:6D  
          inet addr:10.10.1.110  Bcast:10.10.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe78:b06d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 b)  TX bytes:900 (900.0 b)

eth2      Link encap:Ethernet  HWaddr 00:0C:29:78:B0:77  
          inet addr:10.10.2.110  Bcast:10.10.2.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe78:b077/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 b)  TX bytes:900 (900.0 b)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:18 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1112 (1.0 KiB)  TX bytes:1112 (1.0 KiB)

 
Now let’s start up the DNS server we created right before the reboot. The first start up may take a few minutes as the rndc.key file is generated:

[root@ruggero ~]# service named start
Generating /etc/rndc.key:

[  OK  ]
Starting named: [  OK  ]

[root@ruggero ~]# chkconfig named on

 
The last command shown above sets the service to auto-start on reboot.

Now we can check to see if our DNS resolution is working:

[root@ruggero ~]# nslookup larondine-scan.hadesnet
Server:         10.10.1.110
Address:        10.10.1.110#53

Name:   larondine-scan.hadesnet
Address: 10.10.1.112
Name:   larondine-scan.hadesnet
Address: 10.10.1.122

 

The last step is to edit the NTP time server. Oracle RAC relies on keeping the time servers of each node in sync. Although there are several methods to achieve this, the 12c installer will throw a warning if the NTP time slewing option is not set.

Since we are going to edit it anyway, we might as well activate it and use it to keep the two RAC nodes in time sync.

First, let’s backup the existing NTP configuration file:

[root@ruggero etc]# cp /etc/ntp.conf /etc/ntp.conf.orig

 
Now edit the file /etc/ntp.conf and edit out the existing NTP servers, replacing them ones we are going to create locally. Since the file is very large, I am only showing the changes you need to make:

#server 0.centos.pool.ntp.org
#server 1.centos.pool.ntp.org
#server 2.centos.pool.ntp.org

server  ruggero 
server  magda
fudge   ruggero stratum 10

 
What we have basically done in the above example, is to establish each of our two RAC nodes as the time server for the other. This is somewhat crude but effective. If you are certain that your test RAC will always have a valid internet connection then you can ignore this step and leave the default CentOS time servers in place.

Now change the ownership of the /etc/ntp directory to the ntp user:

[root@ruggero etc]# chown ntp:ntp /etc/ntp

 
Finally we add the slewing option to the NTP server. This option basically prevents NTP from setting the clock backwards in the event of disparity between the local clock and the time server. Instead time slewing will slow down clock advancement until the time-server catches up. RAC cannot tolerate the system clock being set backward.

The -x option is added to the OPTIONS directive as follows:

[root@localhost etc]# cat /etc/sysconfig/ntpd
# Drop root to id 'ntp:ntp' by default.
OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid -g"

 
Now we can start the NTP time server as follows:

[root@ruggero etc]# service ntpd start
Starting ntpd: [  OK  ]

[root@ruggero etc]# chkconfig ntpd on

 
Check the operation of the NTP time server as follows:

[root@ruggero etc]#  ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 ruggero         .INIT.          16 u    -   64    0    0.000    0.000   0.000
 magda           .INIT.          16 u    -   64    0    0.000    0.000   0.000

 

Article Quick Navigation
Previous Step Main Index Next Step
Advertisements

One thought on “Oracle 12cR1 12.1.0.1 2-node RAC on CentOS 6.4 on VMware Workstation 9 – Part IV

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s