How to setup IP Multi Pathing (IPMP) public interface on Solaris for Oracle Grid Infrastructure 11.2

Using two network interfaces, vnet1 and vnet2 it is possible to create a redundant public interface that is used as one by Grid Infrastructure. If one of the interfaces (and depending on network design, links) is unavailable the public interface is still available.

Current configuration:

$ srvctl config nodeapps

Network exists: 1/165.156.22.240/255.255.255.240/vnet1, type static
VIP exists: /myserver01-vip/165.156.22.246/165.156.22.240/255.255.255.240/vnet1, hosting node myserver01
VIP exists: /myserver02-vip/165.156.22.247/165.156.22.240/255.255.255.240/vnet1, hosting node myserver02
GSD exists
ONS exists: Local port 6100, remote port 6200, EM port 2016

$ crsctl check cluster -all
**************************************************************
myserver01:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
myserver02:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************

Below are the network interfaces, you can see the VIP (.246) and SCANs (.248, .250) bound to vnet1. vnet2 has nothing running on it.

$ /usr/sbin/ifconfig -a
....
vnet1: flags=9040843 mtu 1500 index 3
        inet 165.156.22.251 netmask fffffff0 broadcast 165.156.22.255
        groupname topside
vnet1:1: flags=1000843 mtu 1500 index 3
        inet 165.156.22.244 netmask fffffff0 broadcast 165.156.22.255
vnet1:2: flags=1040843 mtu 1500 index 3
        inet 165.156.22.246 netmask fffffff0 broadcast 165.156.22.255
vnet1:4: flags=1040843 mtu 1500 index 3
        inet 165.156.22.248 netmask fffffff0 broadcast 165.156.22.255
vnet1:6: flags=1040843 mtu 1500 index 3
        inet 165.156.22.250 netmask fffffff0 broadcast 165.156.22.255
vnet2: flags=9040843 mtu 1500 index 4
        inet 165.156.22.252 netmask fffffff0 broadcast 165.156.22.255
        groupname topside
...
...

To perform the change an outage for all PUBLIC traffic is required:

$ srvctl stop listener
$ srvctl stop cvu
$ srvctl stop scan_listener
$ srvctl stop scan$ srvctl stop listener
$ srvctl stop cvu
$ srvctl stop scan_listener
$ srvctl stop scan

Check that nothing is bound to the interfaces (on all nodes in the cluster):

$ /usr/sbin/ifconfig -a

As root (SUDO setup in this case, below the SUBNET mask ends in .240, your network may be different!)

$ /opt/sfw/bin/sudo $ORACLE_HOME/bin/srvctl modify nodeapps\
         -n myserver01 -A 165.156.22.246/255.255.255.240/vnet1\|vnet2

$ srvctl config nodeapps
Network exists: 1/165.156.22.240/255.255.255.240/vnet1:vnet2, type static
VIP exists: /myserver01-vip/165.156.22.246/165.156.22.240/255.255.255.240/vnet1:vnet2, hosting node myserver01
VIP exists: /myserver02-vip/165.156.22.247/165.156.22.240/255.255.255.240/vnet1:vnet2, hosting node myserver02
GSD exists
ONS exists: Local port 6100, remote port 6200, EM port 2016

$ srvctl start nodeapps

Check the SCAN listener also:

$ srvctl config scan
SCAN name: mycluster-scan, Network: 1/165.156.22.240/255.255.255.240/vnet1:vnet2
SCAN VIP name: scan1, IP: /mycluster-scan/165.156.22.249
SCAN VIP name: scan2, IP: /mycluster-scan/165.156.22.250
SCAN VIP name: scan3, IP: /mycluster-scan/165.156.22.248
$ srvctl start scan
$ srvctl start scan_listener
$ srvctl start cvu 

11.1 Grid Agent "Some recommended prerequisites checks are failed"

Passing the -debug option to runInstaller provides more information on where a pre-requisite could be failing. The 11.1 Grid Agent no longer uses a GUI and is required to be installed command line. The following error message does not provide many clues as to what the pre-requisite fail could be:

./runInstaller -silent -responseFile /oracle/product/software/agent/linux/agent/response.rsp

Starting Oracle Universal Installer...

Checking Temp space: must be greater than 150 MB.   Actual 1083 MB    Passed

Checking swap space: must be greater than 150 MB.   Actual 16383 MB    Passed

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2012-05-10_10-36-59AM. 

Please wait ...

*** Check for updates ***

*** Select Installation Type ***

*** Check Prerequisites ***

Some recommended prerequisites checks are failed. You might get errors during installation. Please fix those prerequisites and start the installation again.

Further, the README.TXT contains the following:

For detailed information on using silent installation method to install the Management agent please refer to .Enterprise Manager Silent Installations. described in the Enterprise Manager Grid Control Basic Installation available at:

http://www.oracle.com/technology/documentation/oem.html

The URL contains the documentation for all Oracle products. 

The documentation is here:

Oracle® Enterprise Manager

Grid Control Basic Installation Guide 

11g Release 1 (11.1.0.1.0) 

E15838-09

http://docs.oracle.com/cd/E11857_01/install.111/e15838.pdf

After reading through the documentation, the -debug flag can be used for the runInstaller which gives us more detailed information as to what the problem can be:

./runInstaller -debug -silent -responseFile /oracle/product/software/agent/linux/agent/response.rsp

In Progress

Checking if Oracle software certified on the current O/S...  1

In Progress

Checking for required packages installed on the system ....  1

In Progress

Checking whether the software compatibile for current o/s...  7

In Progress

Checking for sufficient diskspace in TEMP location...  1

In Progress

Checking for sufficient diskspace in Inventory location...  1

In Progress

Checking for the Hostname...  1

This is a prerequisite condition to test whether the Oracle software is certified on the current O/S or not.Succeeded

This is a prerequisite condition to test whether the minimum required packages are available on the system.Succeeded

This is a prerequisite condition to test whether the software is compatible for this o/sFailed

This check ensures that sufficient diskspace is available in system TEMP location.Succeeded

This check ensures that sufficient diskspace is available in system Inventory location.Succeeded

This is a prerequisite condition to test whether the host name where the installation will be done, is correct or not.Succeeded

Silent Install page at  3 oracle.sysman.install.oneclick.EMGCPrereqDialog,pageTitle=Check Prerequisites,label=Check Prerequisites,selected

Some recommended prerequisites checks are failed. You might get errors during installation. Please fix those prerequisites and start the installation again.