Download Openfiler

http://www.openfiler.com/community/download/ choose: x86 – ISO Image (x86) – Installation ISO image (x86) download result: file openfiler-2.3-x86-disc1.iso size 314MB

1. Create a virtual machine for Openfiler

where

Map the CD/DVD drive using the ISO file of OpenFiler openfiler-2.3-x86-disc1.iso
Link: How to create a virtual machine

2. Install Openfiler

Steps to install Openfiler with all default:
 

..and reboot
- Openfiler present the Welcome Terminal. Pic01
- Login as root and password specified during the installaltion
- and shutdown immediate with: # shutdown -h now
- Next we want add the shared storage disk

3. Create virtual storage shared disks

For a RAC installation, we need for example:

Link: Howto create virtual shared disks on VMWare

Result: created shared disks

Directory of M:\VM\SharedDisks
06/11/2009  01:08 AM    <DIR>          .
06/11/2009  01:08 AM    <DIR>          ..
06/11/2009  01:03 AM     1,073,741,824 disk1-flat.vmdk
06/11/2009  01:03 AM               396 disk1.vmdk
06/11/2009  01:06 AM     5,368,709,120 disk2-flat.vmdk
06/11/2009  01:06 AM               397 disk2.vmdk
06/11/2009  01:08 AM     5,368,709,120 disk3-flat.vmdk
06/11/2009  01:08 AM               397 disk3.vmdk
   6 File(s) 11,811,161,254 bytes
   2 Dir(s)  23,387,648,000 bytes free

4. Add shared disks to machine openfiler

5. Configure iSSCI storage with Openfiler

Download Oracle Enterprise Linux 5.3

http://edelivery.oracle.com/linux
choice: Enterprise Linux Release 5 Update 3 Media Pack for x86 (32 bit)
download result: 5 CD or 1 DVD 2.7GB

Create a VMWare virtual machine with:

Name: RAC1
RAM: 768 MB
HD: 8GB
Network Interface 1: Bridged
Network Interface 2: Host-only
Map the CD/DVD drive using the ISO file of Oracle Enterprise Linux
Link: How to create a virtual machine

Step to install Linux

Install VMWare Tools

- Click Menu VM -> VMWare Tools..
- Open a terminal:
 

[root@rac1 ~]# ls /media/VMware\ Tools/
manifest.txt                       VMwareTools-6.5.0-118166.tar.gz
VMwareTools-6.5.0-118166.i386.rpm
[root@rac1 ~]# cd /media/VMware\ Tools/
[root@rac1 VMware Tools]# rpm -ivh VMwareTools-6.5.0-118166.i386.rpm
Preparing...                ########################################### [100%]
   1:VMwareTools            ########################################### [100%]

The installation of VMware Tools 6.5.0 for Linux completed successfully.
You can decide to remove this software from your system at any time by
invoking the following command: "rpm -e VMwareTools".

Before running VMware Tools for the first time, you need to
configure it for your running kernel by invoking the
following command: "/usr/bin/vmware-config-tools.pl".

Enjoy,

    --the VMware team

Run:

[root@rac1 VMware Tools]# /usr/bin/vmware-config-tools.pl

and follow the screen dialog..
Reboot

Preparing for Oracle database installation

Using Oracle Validated RPM to automatize the install all required RPMs for a optimal database installation

For the first time, during configuring the nodes, ocfs2console fails with message:

"o2cb_ctl: Unable to access cluster service while creating node"

TODO:
- Quit ocfs2console
- Stop the service
- Remove file /etc/ocfs2/cluster.conf
- Restart ocfs2console
- Configure the nodes again
 

[root@rac1 ~]#  /etc/init.d/ocfs2 stop
Stopping Oracle Cluster File System (OCFS2)                [  OK  ]

[root@rac1 ~]# /etc/init.d/o2cb offline ocfs2

[root@rac1 ~]# /etc/init.d/o2cb unload
Unmounting ocfs2_dlmfs filesystem: OK
Unloading module "ocfs2_dlmfs": OK
Unmounting configfs filesystem: OK
Unloading module "configfs": OK

[root@rac1 ~]# rm -f /etc/ocfs2/cluster.conf
[root@rac1 ~]# /usr/sbin/ocfs2console &

For Oracle RAC installation we need share storage for OCR, Voting Disk, ASM..

Shared Disks

1. Create a directory for shared disks:

Open a CMD. Create a directory for all shared disks:

C:\> mkdir M:\VM4\SharedDisks

Create the shared disk manual with vmware-vdiskmanager.exe

C:\> cd "C:\Program Files\VMware\VMware Workstation"
vmware-vdiskmanager.exe -c -s 1Gb -a lsilogic -t 4 M:\VM4\SharedDisks\ocfs2.vmdk
vmware-vdiskmanager.exe -c -s 3Gb -a lsilogic -t 4 M:\VM4\SharedDisks\asm1.vmdk
vmware-vdiskmanager.exe -c -s 3Gb -a lsilogic -t 4 M:\VM4\SharedDisks\asm2.vmdk
vmware-vdiskmanager.exe -c -s 3Gb -a lsilogic -t 4 M:\VM4\SharedDisks\asm3.vmdk
vmware-vdiskmanager.exe -c -s 3Gb -a lsilogic -t 4 M:\VM4\SharedDisks\asm4.vmdk

where the paraneters are:

and disk type is
 

Note:
- Only with disk type 4 can a RAC installation later successful
- Disk type 4 (preallocated ESX-type virtual disk) is NEW in
VMWare workstation 6.5 Beta and VMWare Server 2.0 Beta.
- Disk type 2 (preallocated virtual disk) dont’work !
- With disk type 2, later when install Clusterware you get the error:
..service OracleCSService in improper PENDING state, err(0)
..service OracleCSService in improper PENDING state, err(997)

2. Add disk to virtual machine

The shared disk are ready created, now we must add them to the all virtual machines functionally as RAC nodes
For example when the first RAC node name RAC1 then open file \RAC1\RAC1.vmx with notepad and add this:

# Begin Virtual VM Disks for Oracle RAC
disk.locking = "FALSE"
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize = "0"
diskLib.dataCacheMinReadAheadSize = "0"
diskLib.dataCachePageSize = "4096"
diskLib.maxUnsyncedWrites = "0"

scsi1.present = "TRUE"
scsi1.virtualDev = "lsilogic"
scsi1.sharedBus = "VIRTUAL"

scsi1:0.present = "TRUE"
scsi1:0.fileName = "M:\VM4\SharedDisks\ocfs2.vmdk"
scsi1:0.mode = "independent-persistent"
scsi1:0.deviceType = "disk"

scsi1:1.present = "TRUE"
scsi1:1.fileName = "M:\VM4\SharedDisks\asm1.vmdk"
scsi1:1.mode = "independent-persistent"
scsi1:1.deviceType = "disk"

scsi1:2.present = "TRUE"
scsi1:2.fileName = "M:\VM4\SharedDisks\asm2.vmdk"
scsi1:2.mode = "independent-persistent"
scsi1:2.deviceType = "disk"

scsi1:3.present = "TRUE"
scsi1:3.fileName = "M:\VM4\SharedDisks\asm3.vmdk"
scsi1:3.mode = "independent-persistent"
scsi1:3.deviceType = "disk"

scsi1:4.present = "TRUE"
scsi1:4.fileName = "M:\VM4\SharedDisks\asm4.vmdk"
scsi1:4.mode = "independent-persistent"
scsi1:4.deviceType = "disk"

# End Virtual VM Disks for Oracle RAC

-Power on the machine and check if the OS see all shared disks, for example in Linux

# ls -l /dev/sd*
or
# fdisk -l

Steps to create a VMWare virtual machine

Add a second NIC

Sometime we need a second NIC, for example, for a RAC installation we need a second NIC for PRIVATE network. Todo:
 

Fixed Size: Fixed SGA and other internal allocations needed by the Oracle Database instance.
Also space for background processes and else..
You can not changed it. There are no pararameter to change SGA fixed size

http://download.oracle.com/docs/cd/B19306_01/server.102/b14231/create.htm#sthref381

Variable Size: the sum of
Shared Pool  Size + Large Pool Size + Java Pool Size + Streams Pool Size + Free SGA Memory Available

Variable size:
1. When starup instance you see Variable Size and other infos about SGA, for example:

SQL> startup
ORACLE instance started.

Total System Global Area  448790528 bytes
Fixed Size                  1249536 bytes
Variable Size             251662080 bytes
Database Buffers          188743680 bytes
Redo Buffers                7135232 bytes
Database mounted.

2.

SQL> select * from v$sga;

NAME                      VALUE
-------------------- ----------
Fixed Size              1249536
Variable Size         255856384
Database Buffers      184549376
Redo Buffers            7135232

3.

SQL> select sum(bytes) "Variable size" from v$sgainfo where name in
  2       ( 'Shared Pool Size'
  3       , 'Large Pool Size'
  4       , 'Java Pool Size'
  5       , 'Streams Pool Size'
  6       , 'Free SGA Memory Available'
  7       );

Variable size
-------------
    255852544

the calculation is just roughly!

For a RAC installation you must specify a default gateway for the PUBLIC network. When crs starting, during the VIP status check, the default gateway of server is used as a ping target. Upon a ping failure, Oracle will decide that the current interface where the VIP is running has failed, and will initiate VIP failover.

Without a gateway a message is log in file $ORA_CRS_HOME/log/linux1/racg

2009-06-12 00:17:59.173: [ RACG][3068118720] [16742][3068118720][ora.linux1.vip]:
checkIf: Default gateway is not defined (host=linux1.localdomain) and eror when start a instance

[oracle@linux1 ~]$ srvctl start instance -d orcl -i orcl1
PRKP-1001 : Error starting instance orcl1 on node linux1
CRS-0215: Could not start resource 'ora.orcl.orcl1.inst'.

TODO:

1. Set a valid default gateway for PUBLIC network
Menu System -> Administration -> Network -> Edit eth0 -> Default gateway address

2. or if you don’t have a gateway, then open file in $CRS_HOME/bin/racgvip, change the value of variable

FAIL_WHEN_DEFAULTGW_NOT_FOUND=0

stop, start crs and all nodeapps again

Stop instance orcl1 of database orcl

Verify if instance running?

[oracle@rac1 ~]$ ps -ef | grep orcl1 | grep pmon
oracle   12465     1  0 23:36 ?        00:00:00 ora_pmon_orcl1

Shutdown instance orcl1

[oracle@rac1 ~]$ srvctl stop instance -d orcl -i orcl1
[oracle@rac1 ~]$ ps -ef | grep orcl1 | grep pmon
[oracle@rac1 ~]$

[oracle@rac1 ~]$ sqlplus sys/doan@orcl as sysdba
...
SQL> select instance_number, instance_name from gv$instance order by 1;

INSTANCE_NUMBER INSTANCE_NAME
--------------- ----------------
              2 orcl2

Instance orcl1 is down. Instance orcl2 is running

Start instance orcl1 again

Start it!

[oracle@rac1 ~]$ srvctl start instance -d orcl -i orcl1

Verify if instance running?

[oracle@rac1 ~]$ ps -ef | grep orcl1 | grep pmon
oracle   31185     1  0 23:50 ?        00:00:00 ora_pmon_orcl1

[oracle@rac1 ~]$ sqlplus sys/doan@orcl as sysdba
...
SQL> select instance_number, instance_name from gv$instance order by 1;

INSTANCE_NUMBER INSTANCE_NAME
--------------- ----------------
              1 orcl1
              2 orcl2

Now there are two instance on RAC!

Note: blank is No!
 

- Cannot use ASM to store Oracle Clusterware files, because these files must be accessible before any ASM instance starts

- Oracle recommends: ASM as the storage for database and recovery files.

Links: http://download.oracle.com/docs/cd/B19306_01/install.102/b14203/storage.htm#CDECEBGH
 

Install 10gR2 RAC on Oracle EL5:
At end of root.sh on the last node vipca will fail to run with the following error:

...
Running vipca(silent) for configuring nodeapps
/u01/app/oracle/product/crs/jdk/jre//bin/java: error while loading shared libraries:
libpthread.so.0: cannot open shared object file: No such file or directory
[root@rac2 crs]#

..or later when you run srvctl to check the cluster and database configuration, for example

[oracle@rac2 lib]$ srvctl status database -d orcl
/u01/app/oracle/product/10.2.0/db_1/jdk/jre/bin/java: error while loading shared libraries:
libpthread.so.0: cannot open shared object file: No such file or directory

TODO: workaround!
There are 3 files must be modified on all nodes:

  /u01/app/oracle/product/crs/bin/vipca
  /u01/app/oracle/product/crs/bin/srvctl
  /u01/app/oracle/product/10.2.0/db_1/bin/srvctl

Modified vipca

Open file /u01/app/oracle/product/crs/bin/vipca, add the line in red:
 

if [ "$arch" = "i686" -o "$arch" = "ia64" ]
then
      LD_ASSUME_KERNEL=2.4.19
      export LD_ASSUME_KERNEL
fi
unset LD_ASSUME_KERNEL
#End workaround

Modified srvctl

Open file /u01/app/oracle/product/crs/bin/srvctl, add the line in red:
 

#Remove this workaround when the bug 3937317 is fixed
LD_ASSUME_KERNEL=2.4.19
export LD_ASSUME_KERNEL
unset  LD_ASSUME_KERNEL

After working around above, vipca will fail to run with the following error
if the VIP IP’s are in a non-routable range [10.x.x.x, 172.(16-31).x.x or 192.168.x.x]:

[root@rac2 bin]# ./vipca
Error 0(Native: listNetInterfaces:[3])
  [Error 0(Native: listNetInterfaces:[3])]

SOLUTION: run this for example

[root@rac2 bin]# pwd
/u01/app/oracle/product/crs/bin
[root@rac2 bin]# ./oifcfg setif -global eth0/192.168.1.0:public
[root@rac2 bin]# ./oifcfg setif -global eth1/10.10.10.0:cluster_interconnect
[root@rac2 bin]# ./oifcfg getif
eth0  192.168.1.0  global  public
eth1  10.10.10.0  global  cluster_interconnect

NOW: Run vipca again

[root@rac2 bin]# ./vipca

and follow the screen..

for more info: see MetaLink Note 414163.1