Setup a storage server for RAC using Openfiler and iSCSI
June 25th, 2009 . by adminDownload Openfiler
 |
http://www.openfiler.com/community/download/ choose: x86 – ISO Image (x86) – Installation ISO image (x86) download result: file openfiler-2.3-x86-disc1.iso size 314MB |
1. Create a virtual machine for Openfiler
| Openfiler Server | |
|---|---|
| Name: | Openfiler23 |
| Memory: | 768 MB |
| HardDisk: | 3GB |
| Network Adapter 1 | Bridget |
| Network Adapter 2 | Host-only |
where
| NIC | IP address | Subnet | Gateway | Network connection |
|---|---|---|---|---|
| eth0 | 192.168.1.7 | 255.255.255.0 | 192.168.1.1 | Bridged |
| eth1 | 192.168.192.7 | 255.255.255.0 | Host-only |
Map the CD/DVD drive using the ISO file of OpenFiler openfiler-2.3-x86-disc1.iso
Link: How to create a virtual machine
2. Install Openfiler
Steps to install Openfiler with all default:
| Screen | Action |
|---|---|
| On VMWare right click machine Openfiler23 -> Power On to start the installation |
|
| Boot Screen | Press <ENTER> to start the graphical installation |
| Welcome to rPath Linux | Press OK or Skip the media test |
| Welcome to Openfiler NSA (32-bit PAE) | Click Next |
| Dialog Keyboard Configuration | - Choose the appropritate keyboard – Click Next |
| Partitioning Setup | Accept default automatically partition .Click Next .Click Yes when message: Would you like to initialize this drive… |
| Automatic Partitioning | Accept all default Click next |
| Disk Setup | Accept all default .Click next |
| Network Configuration | - Select eth0 – Click Button Edit: .Deselect ‘Configure using DHCP’ .Select ‘Activate on boot’ .IP Address: 192.168.1.7 .Netmask: 255.255.255.0 .Click Ok – Select eth0 – Click Button Edit: – Choose Set the hostname manually: openfiler23.localdomain – Message Error With Data: Yout have not specified the field
|
| Time Zone Selection | - Select your Location – Click Next |
| Set Root Password | - Specify password for root – Click Next |
| About to Install | - Click Next |
..and reboot
- Openfiler present the Welcome Terminal. Pic01
- Login as root and password specified during the installaltion
- and shutdown immediate with: # shutdown -h now
- Next we want add the shared storage disk
3. Create virtual storage shared disks
For a RAC installation, we need for example:
| Overview shared disks | ||
|---|---|---|
| Disk 1 | 1GB | Place for Clusterfile OCR, VotingDisk |
| Disk 2 | 5GB | Place for database file. ASM volume 1, 2, 3,4 |
| Disk 3 | 5GB | |
Link: Howto create virtual shared disks on VMWare
Result: created shared disks
Directory of M:\VM\SharedDisks 06/11/2009 01:08 AM <DIR> . 06/11/2009 01:08 AM <DIR> .. 06/11/2009 01:03 AM 1,073,741,824 disk1-flat.vmdk 06/11/2009 01:03 AM 396 disk1.vmdk 06/11/2009 01:06 AM 5,368,709,120 disk2-flat.vmdk 06/11/2009 01:06 AM 397 disk2.vmdk 06/11/2009 01:08 AM 5,368,709,120 disk3-flat.vmdk 06/11/2009 01:08 AM 397 disk3.vmdk 6 File(s) 11,811,161,254 bytes 2 Dir(s) 23,387,648,000 bytes free
4. Add shared disks to machine openfiler
| Steps | |
|---|---|
| Right click virtual machine Openfiler23 -> Setting | |
| Dialog Virtual Machine Settings | - Click Button Add |
| Dialog Add Hardware Wizard | - Choose Hard Disk – Click Next |
| Dialog Select a Disk | - Choose Use an existing virtual disk – Click Next |
| Dialog Select an Existing Disk | - Specify Disk file: M:\VM\SharedDisks\disk1.vmdk – Click Mode Independent – Click Finish – When message ‘Convert existing virtual disk to newer format?’ choose ‘Keep Existing Format’ – Click OK – Repeat the step to add shared disk2 and 3 |
| Click OK to finish adding shared disk | |
5. Configure iSSCI storage with Openfiler
| Steps to configure Openfiler iSCSI | |||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Starting | - Power on machine Openfiler23. Pic01 – From host machine open a browser with address: https://192.168.1.7:446/ – Screen: When Certificate Error: Navigation Blocked. Pic02 – Click Continue to this website (not recommended) – Login with Username: openfiler. Password=password. Pic03 |
||||||||||||||||||||||||
| Enable service iSCSI | Pic 04 – Click tab ‘Services’ – Click link ‘Enable’ at row ‘iSCSI target server’ |
||||||||||||||||||||||||
| Network Access Configuration | - Click tab ‘System’. Scroll down Specified ips from which can access Openfiler
Summary: Pic05 |
||||||||||||||||||||||||
| Create disk partition | - Click tab ‘Volumes’ – Click ‘Block Devices’ on topic box ‘Volumes section’ on the right – Click link /dev/sdb. Pic06 – Scroll down and click button ‘Create’ .Pic07 – Goback by click ‘Block Devices’ and create partitions for /dev/sdc, /dev/sdd – Summary Pic08 |
||||||||||||||||||||||||
| Create new Volume Group | - Click ‘Volume Group’ on topic box ‘Volumes section’ on the right – Specify Volume Group name, for example: vstorage – Select all disks. My volume group should contents all 3 disks – Click ‘Add volume group’ – see: Pic09 |
||||||||||||||||||||||||
| Add Volume |
- Click ‘Add Volume’ on topic box ‘Volumes section’ on the right - Repeat the steps to create 4 new volumes: asm1 bis asm4 with size 2.5GB(2560MB)
Summary: Pic11 |
||||||||||||||||||||||||
| Create and configure Target |
- Click Tab ‘Volumes’. Add an iSCSI Target: Mapping LUN Network ACL |
||||||||||||||||||||||||
service iscsi is not coming up…tried starting but no luck…though previous process completed without any issue…
[root@rac1 ~]# service iscsi status
iscsid is stopped
[root@rac1 ~]# chkconfig iscsid on
[root@rac1 ~]# chkconfig iscsi on
[root@rac1 ~]# service iscsi status
iscsid is stopped
[root@rac1 ~]# chkconfig –list | grep iscsi
iscsi 0:off 1:off 2:on 3:on 4:on 5:on 6:off
iscsid 0:off 1:off 2:on 3:on 4:on 5:on 6:off
[root@rac1 ~]#
service start working after a reboot…
though discovery is not working, above command is returning nothing…iscsiadm -m discovery -t st -p 192.168.192.7
I am able to ping on all ip’s to each other (RAC1 to openfiler23)
All previous steps completed successfully…lemme know if there is anything missed?
See my note about troubleshooting when discover iSCSI targets not work:
http://d-h-n.de/blog/?p=542#install_iscsi_initiator
- Goto the machine “iSCSI Server” Openfiler23, check the files
/etc/initiators.deny
vi file /etc/initiators.deny on the Target iSCSI Server set comment the line
# iqn.2006-01.com.openfiler:xxx.xxxxxxxxxxxx ALL
again my mistake…i did this but on rac1 didn’t realize you are asking me to do it on Openfiler23…
what an amazing documentation you have done…i am trying oracle installation on linux first time…mostly a windows guy all my life…but these steps are just so eas to followy…
for some reason its not letting me post big message..
i put it on my website:
http://www.starinnovative.com/oraRac/OracleRAC.html
please suggest…
Thx !! prakash
issue fixed by running below commands…
CVUQDISK_GRP=dba; export CVUQDISK_GRP
cd /mnt/hgfs/OSource/Clusterware_Linux/clusterware/rpm
rpm -iv cvuqdisk-1.0.1-1.rpm
Thank you for the your comments!
1. Yes, network configuaration is the most important when install RAC.
My RAC install is just for learning, not for production.
There is therefore no DHCP, DNS..
2.’OSource’ folder is the my shared folder, which contents Linux, Oracle Software etc..
I’m trying to document everything, but it is unfortunately not perfect..
Yes i forget some thing here
3. Shared folder HGFS is one of the basic things you need know when using VMWare..
I think all know about it, it is boring to write about it
4. I think you do not need to install VMWare Tools on the Openfiler machine,
because Openfiler run on Linux text mode and VMWare just supports for fast
graphics performance, shared folders, drag drop …
see http://www.vmware.com/support/ws55/doc/new_guest_tools_ws.html
5.To mount a ISO File automatically when start up Linux, open file /etc/fstab and add following:
/mnt/hgfs/OSource/OLinux/OEL53.iso /mnt/OEL53 iso9660 ro,loop,auto 0 0DNS
I do not understand your answer about DNS, why do you want change the default DNS?
CVU
Instead of using CVU 10g from Oracle Clusterware install disk, try download and install CVU 11g:
See my post on Oracle Forum:
http://forums.oracle.com/forums/thread.jspa?messageID=3615435�
..and my “perfect” (without anny eror) Cluster Verifify with CVU:
http://d-h-n.de/blog/?p=531
ERROR: Could not find a suitable set of interfaces for VIPs.
see: http://d-h-n.de/blog/?p=523
This errors can be ignored, when run CVU cluvfy 10g:
Package existence check failed for compat-gcc-7.3-2.96.128Package existence check failed for compat-gcc-c++-7.3-2.96.128
Package existence check failed for compat-libstdc++-7.3-2.96.128
Package existence check failed for compat-libstdc++-devel-7.3-2.96.128
Package existence check failed for openmotif-2.2.3
But when you run install and run CVU 11g, you will no longer see it
Hey i like your documentation very much, this is probably the only documentation using workstation 6.5 to configure 10.g (though I was origionally looking for 11g documantation)..and it helped me a lot on learning vmware/linux/oraclerac in just less than 1 week…
you set primany dsn when configuring network on rac2, as per picture, i got confised…probably its not needed…
regarding dns i just want to follow your steps as closely as possible…and i dont have static ip thus bridged will not work for me…
didn’t knew CVU error can be ignored…though good now…
somehow when you search on google for that hgfs disabled error there is no link which gives that answer you gave…i spend 3 hours atleast searching..
what do you have in tools folder in osource…i want to follow exactly te way you have configured…so that to a void any silly mistake…
once 10g done i will try my own 11g installation to see how much i understood…
btw I am sql dba and havn’t touch oracle for last 5 years…you have an expert command over vmware/oracle/linux…which is commendable…
thanks for all your help and guidence….now I am going to follow rac cluster installation….
Prakash
faced 2 issues…first @93% vmware messaged rac2.vmdk is not accisible, i tried retry and after some time rac2 automatically restarted…
here are some of the messages thrown in terminal window…(i restore old copy and tried again same error at same point, though it seems it didnot affected anything other than getting error at vip configuration later on rac2, see below…)
Caught Cluster ExceptionPRKC-1044 : Failed to check remote command execution setup for node rac1.localdomain using shells /usr/bin/ssh and /usr/bin/rsh
rac1.localdomain: Connection refused
[PRKC-1044 : Failed to check remote command execution setup for node rac1.localdomain using shells /usr/bin/ssh and /usr/bin/rsh
rac1.localdomain: Connection refused]
[PRKC-1044 : Failed to check remote command execution setup for node rac1.localdomain using shells /usr/bin/ssh and /usr/bin/rsh
rac1.localdomain: Connection refused
[PRKC-1044 : Failed to check remote command execution setup for node rac1.localdomain using shells /usr/bin/ssh and /usr/bin/rsh
rac1.localdomain: Connection refused]]
error at step when starting vip application…(tried retry dosn’t work and then ignore and it finished, hoping its ignorable error)
[root@rac2 bin]# ./vipca
[root@rac2 racg]# tail -n30 ora.rac2.vip.log
Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, 2005 Oracle. All rights reserved.
2009-07-26 11:44:45.327: [ RACG][3067807424] [14072][3067807424][ora.rac2.vip]: Interface eth0 checked failed (host=rac2.localdomain)
Invalid parameters, or failed to bring up VIP (host=rac2.localdomain)
2009-07-26 11:44:45.327: [ RACG][3067807424] [14072][3067807424][ora.rac2.vip]: clsrcexecut: env ORACLE_CONFIG_HOME=/u01/crs/oracle/product/10.2.0/crs
2009-07-26 11:44:45.327: [ RACG][3067807424] [14072][3067807424][ora.rac2.vip]: clsrcexecut: cmd = /u01/crs/oracle/product/10.2.0/crs/bin/racgeut -e _USR_ORA_DEBUG=0 54 /u01/crs/oracle/product/10.2.0/crs/bin/racgvip start rac2
2009-07-26 11:44:45.327: [ RACG][3067807424] [14072][3067807424][ora.rac2.vip]: clsrcexecut: rc = 1, time = 9.600s
2009-07-26 11:44:54.576: [ RACG][3067807424] [14072][3067807424][ora.rac2.vip]: Interface eth0 checked failed (host=rac2.localdomain)
Invalid parameters, or failed to bring up VIP (host=rac2.localdomain)
2009-07-26 11:44:54.577: [ RACG][3067807424] [14072][3067807424][ora.rac2.vip]: clsrcexecut: env ORACLE_CONFIG_HOME=/u01/crs/oracle/product/10.2.0/crs
2009-07-26 11:44:54.577: [ RACG][3067807424] [14072][3067807424][ora.rac2.vip]: clsrcexecut: cmd = /u01/crs/oracle/product/10.2.0/crs/bin/racgeut -e _USR_ORA_DEBUG=0 54 /u01/crs/oracle/product/10.2.0/crs/bin/racgvip check rac2
2009-07-26 11:44:54.577: [ RACG][3067807424] [14072][3067807424][ora.rac2.vip]: clsrcexecut: rc = 1, time = 9.250s
2009-07-26 11:44:54.577: [ RACG][3067807424] [14072][3067807424][ora.rac2.vip]: end for resource = ora.rac2.vip, action = start, status = 1, time = 19.270s
seems I am not able to ping 192.168.10.21 or 11 (vip ip’s), I dont remember i set anything other than entering into the hosts file of bpoth machines…
is there any config required to have these two vip working?
amazing…somehow after 2-3 restarts it works fine…something strange, but it worked…
Thx!!
[...] own project I used Internet resources of course, and a few of them are listed below. A useful blog to setup storage with Openfiler Build Your Own Oracle RAC 11g Cluster on Oracle Enterprise Linux [...]
[...] own project I used Internet resources of course, and a few of them are listed below. A useful blog to setup storage with Openfiler Build Your Own Oracle RAC 11g Cluster on Oracle Enterprise Linux [...]