d-h-n.de
Blog about Oracle, Linux..

Setup a storage server for RAC using Openfiler and iSCSI

June 25th, 2009 . by admin

Download Openfiler

Openfiler http://www.openfiler.com/community/download/
choose: x86 – ISO Image (x86) – Installation ISO image (x86)
download result: file openfiler-2.3-x86-disc1.iso size 314MB
 

 

1. Create a virtual machine for Openfiler

Openfiler Server
Name: Openfiler23
Memory: 768 MB
HardDisk: 3GB
Network Adapter 1 Bridget
Network Adapter 2 Host-only

where

NIC IP address Subnet Gateway Network connection
eth0 192.168.1.7 255.255.255.0 192.168.1.1 Bridged
eth1 192.168.192.7 255.255.255.0   Host-only

Map the CD/DVD drive using the ISO file of OpenFiler openfiler-2.3-x86-disc1.iso
Link: How to create a virtual machine

 

2. Install Openfiler

Steps to install Openfiler with all default:
 

Screen Action
On VMWare right click machine Openfiler23
-> Power On to start the installation
 
 
Boot Screen Press <ENTER> to start the graphical installation
Welcome to rPath Linux Press OK or Skip the media test
Welcome to Openfiler NSA (32-bit PAE) Click Next
Dialog Keyboard Configuration - Choose the appropritate keyboard
– Click Next
Partitioning Setup Accept default automatically partition
.Click Next
.Click Yes when message: Would you like to initialize this drive…
Automatic Partitioning Accept all default
Click next
Disk Setup Accept all default
.Click next
Network Configuration - Select eth0 – Click Button Edit:
  .Deselect ‘Configure using DHCP’
  .Select ‘Activate on boot’
  .IP Address: 192.168.1.7
  .Netmask: 255.255.255.0
  .Click Ok

– Select eth0 – Click Button Edit:
  .Deselect ‘Configure using DHCP’
  .Select ‘Activate on boot’
  .IP Address: 192.168.192.7
  .Netmask: 255.255.255.0
  .Click Ok

– Choose Set the hostname manually: openfiler23.localdomain
– Setting Gateway: 192.168.1.1
– Click Next

– Message Error With Data: Yout have not specified the field
"Primary DNS"..
– Click Continue

 

Time Zone Selection - Select your Location
– Click Next
Set Root Password - Specify password for root
– Click Next
About to Install - Click Next

..and reboot
- Openfiler present the Welcome Terminal. Pic01
- Login as root and password specified during the installaltion
- and shutdown immediate with: # shutdown -h now
- Next we want add the shared storage disk

 

 

3. Create virtual storage shared disks

For a RAC installation, we need for example:

Overview shared disks
Disk 1 1GB Place for Clusterfile OCR, VotingDisk
Disk 2 5GB Place for database file. ASM volume 1, 2, 3,4
Disk 3 5GB

Link: Howto create virtual shared disks on VMWare

Result: created shared disks

Directory of M:\VM\SharedDisks
06/11/2009  01:08 AM    <DIR>          .
06/11/2009  01:08 AM    <DIR>          ..
06/11/2009  01:03 AM     1,073,741,824 disk1-flat.vmdk
06/11/2009  01:03 AM               396 disk1.vmdk
06/11/2009  01:06 AM     5,368,709,120 disk2-flat.vmdk
06/11/2009  01:06 AM               397 disk2.vmdk
06/11/2009  01:08 AM     5,368,709,120 disk3-flat.vmdk
06/11/2009  01:08 AM               397 disk3.vmdk
   6 File(s) 11,811,161,254 bytes
   2 Dir(s)  23,387,648,000 bytes free

 
 

4. Add shared disks to machine openfiler

Steps
Right click virtual machine Openfiler23 -> Setting
Dialog Virtual Machine Settings - Click Button Add
Dialog Add Hardware Wizard - Choose Hard Disk
– Click Next
Dialog Select a Disk - Choose Use an existing virtual disk
– Click Next
Dialog Select an Existing Disk - Specify Disk file: M:\VM\SharedDisks\disk1.vmdk
– Click Mode Independent
– Click Finish
– When message ‘Convert existing virtual disk to newer format?’
choose ‘Keep Existing Format’
– Click OK
– Repeat the step to add shared disk2 and 3
Click OK to finish adding shared disk

 
 

5. Configure iSSCI storage with Openfiler

Steps to configure Openfiler iSCSI
Starting - Power on machine Openfiler23. Pic01
– From host machine open a browser with address:
https://192.168.1.7:446/
– Screen: When Certificate Error: Navigation Blocked. Pic02
– Click Continue to this website (not recommended)
– Login with Username: openfiler. Password=password. Pic03
Enable service iSCSI Pic 04
– Click tab ‘Services’
– Click link ‘Enable’ at row ‘iSCSI target server’
Network Access Configuration - Click tab ‘System’. Scroll down
Specified ips from which can access Openfiler

Name Network/Host Netmask Type
rac1-public 192.168.1.10 255.255.255.0 Shared
rac1-priv 192.168.192.10 255.255.255.0 Shared
rac2-public 192.168.1.11 255.255.255.0 Shared
rac2-priv 192.168.192.11 255.255.255.0 Shared

Summary: Pic05

Create disk partition - Click tab ‘Volumes’
– Click ‘Block Devices’ on topic box ‘Volumes section’ on the right
– Click link /dev/sdb. Pic06
– Scroll down and click button ‘Create’ .Pic07
– Goback by click ‘Block Devices’ and create partitions for /dev/sdc, /dev/sdd
– Summary Pic08
Create new Volume Group - Click ‘Volume Group’ on topic box ‘Volumes section’ on the right
– Specify Volume Group name, for example: vstorage
– Select all disks. My volume group should contents all 3 disks
– Click ‘Add volume group’
– see: Pic09
Add Volume

- Click ‘Add Volume’ on topic box ‘Volumes section’ on the right
– Scroll down
– Create a volume
– Volume name: ocfs2
– Volume Description: ocr and voting disk
– Required Space(MB): 1024
– Filesystem/Volume type: iSCSI
– Click Create
– see: Pic10

- Repeat the steps to create 4 new volumes: asm1 bis asm4 with size 2.5GB(2560MB)

Volume Name Volume Description Volume Size Volume type
ocfs2 ocr and voting disk 1024MB iSCSI
asm1 asm volume 1 2560MB iSCSI
asm2 asm volume 2 2560MB iSCSI
asm3 asm volume 3 2560MB iSCSI
asm4 asm volume 4 2560MB iSCSI

Summary: Pic11

Create and configure Target

- Click Tab ‘Volumes’.
– Click ‘iSCSI Targets’ on topic box ‘Volumes section’ on the right

Add an iSCSI Target:
– Sub tab: Target Configuration
– Accept generated default Target IQN. Click Add
Pic12

Mapping LUN
– Sub tab: LUN Mapping
– Click Button Map to map a volume. Pic13
– Map all other volumes!
– Summary: LUN Mapping Pic14

Network ACL
– Click subtab ‘Network ACL’
– Select access ‘Allow’ for all hosts. Click button Update.
– Summary: Pic15
 

 


14 Responses to “Setup a storage server for RAC using Openfiler and iSCSI”

  1. comment number 1 by: Prakash Heda

    service iscsi is not coming up…tried starting but no luck…though previous process completed without any issue…

    [root@rac1 ~]# service iscsi status
    iscsid is stopped
    [root@rac1 ~]# chkconfig iscsid on
    [root@rac1 ~]# chkconfig iscsi on
    [root@rac1 ~]# service iscsi status
    iscsid is stopped
    [root@rac1 ~]# chkconfig –list | grep iscsi
    iscsi 0:off 1:off 2:on 3:on 4:on 5:on 6:off
    iscsid 0:off 1:off 2:on 3:on 4:on 5:on 6:off
    [root@rac1 ~]#

  2. comment number 2 by: Prakash Heda

    service start working after a reboot…

    though discovery is not working, above command is returning nothing…iscsiadm -m discovery -t st -p 192.168.192.7

    I am able to ping on all ip’s to each other (RAC1 to openfiler23)

    All previous steps completed successfully…lemme know if there is anything missed?

  3. comment number 3 by: admin

    See my note about troubleshooting when discover iSCSI targets not work:
    http://d-h-n.de/blog/?p=542#install_iscsi_initiator

    - Goto the machine “iSCSI Server” Openfiler23, check the files
    /etc/initiators.deny

    vi file /etc/initiators.deny on the Target iSCSI Server set comment the line
    # iqn.2006-01.com.openfiler:xxx.xxxxxxxxxxxx ALL

  4. comment number 4 by: Prakash Heda

    again my mistake…i did this but on rac1 didn’t realize you are asking me to do it on Openfiler23…

    what an amazing documentation you have done…i am trying oracle installation on linux first time…mostly a windows guy all my life…but these steps are just so eas to followy…

  5. comment number 5 by: Prakash Heda

    for some reason its not letting me post big message..

    i put it on my website:

    http://www.starinnovative.com/oraRac/OracleRAC.html

    please suggest…

    Thx !! prakash

  6. comment number 6 by: Prakash Heda

    issue fixed by running below commands…

    CVUQDISK_GRP=dba; export CVUQDISK_GRP
    cd /mnt/hgfs/OSource/Clusterware_Linux/clusterware/rpm
    rpm -iv cvuqdisk-1.0.1-1.rpm

  7. comment number 7 by: admin

    Thank you for the your comments!

    1. Yes, network configuaration is the most important when install RAC.
    My RAC install is just for learning, not for production.
    There is therefore no DHCP, DNS..

    2.’OSource’ folder is the my shared folder, which contents Linux, Oracle Software etc..
    I’m trying to document everything, but it is unfortunately not perfect..
    Yes i forget some thing here

    3. Shared folder HGFS is one of the basic things you need know when using VMWare..
    I think all know about it, it is boring to write about it ;)

    4. I think you do not need to install VMWare Tools on the Openfiler machine,
    because Openfiler run on Linux text mode and VMWare just supports for fast
    graphics performance, shared folders, drag drop …
    see http://www.vmware.com/support/ws55/doc/new_guest_tools_ws.html

    5.To mount a ISO File automatically when start up Linux, open file /etc/fstab and add following:
    /mnt/hgfs/OSource/OLinux/OEL53.iso /mnt/OEL53 iso9660 ro,loop,auto 0 0

    DNS
    I do not understand your answer about DNS, why do you want change the default DNS?

    CVU
    Instead of using CVU 10g from Oracle Clusterware install disk, try download and install CVU 11g:

    See my post on Oracle Forum:
    http://forums.oracle.com/forums/thread.jspa?messageID=3615435&#3615435

    ..and my “perfect” (without anny eror) Cluster Verifify with CVU:
    http://d-h-n.de/blog/?p=531

    ERROR: Could not find a suitable set of interfaces for VIPs.
    see: http://d-h-n.de/blog/?p=523

    This errors can be ignored, when run CVU cluvfy 10g:
    Package existence check failed for compat-gcc-7.3-2.96.128
    Package existence check failed for compat-gcc-c++-7.3-2.96.128
    Package existence check failed for compat-libstdc++-7.3-2.96.128
    Package existence check failed for compat-libstdc++-devel-7.3-2.96.128
    Package existence check failed for openmotif-2.2.3

    But when you run install and run CVU 11g, you will no longer see it

  8. comment number 8 by: Prakash Heda

    Hey i like your documentation very much, this is probably the only documentation using workstation 6.5 to configure 10.g (though I was origionally looking for 11g documantation)..and it helped me a lot on learning vmware/linux/oraclerac in just less than 1 week…

    you set primany dsn when configuring network on rac2, as per picture, i got confised…probably its not needed…

    regarding dns i just want to follow your steps as closely as possible…and i dont have static ip thus bridged will not work for me…

    didn’t knew CVU error can be ignored…though good now…

    somehow when you search on google for that hgfs disabled error there is no link which gives that answer you gave…i spend 3 hours atleast searching..

    what do you have in tools folder in osource…i want to follow exactly te way you have configured…so that to a void any silly mistake…

    once 10g done i will try my own 11g installation to see how much i understood…

    btw I am sql dba and havn’t touch oracle for last 5 years…you have an expert command over vmware/oracle/linux…which is commendable…

    thanks for all your help and guidence….now I am going to follow rac cluster installation….

    Prakash

  9. comment number 9 by: Prakash Heda

    faced 2 issues…first @93% vmware messaged rac2.vmdk is not accisible, i tried retry and after some time rac2 automatically restarted…

    here are some of the messages thrown in terminal window…(i restore old copy and tried again same error at same point, though it seems it didnot affected anything other than getting error at vip configuration later on rac2, see below…)
    Caught Cluster ExceptionPRKC-1044 : Failed to check remote command execution setup for node rac1.localdomain using shells /usr/bin/ssh and /usr/bin/rsh
    rac1.localdomain: Connection refused
    [PRKC-1044 : Failed to check remote command execution setup for node rac1.localdomain using shells /usr/bin/ssh and /usr/bin/rsh
    rac1.localdomain: Connection refused]
    [PRKC-1044 : Failed to check remote command execution setup for node rac1.localdomain using shells /usr/bin/ssh and /usr/bin/rsh
    rac1.localdomain: Connection refused
    [PRKC-1044 : Failed to check remote command execution setup for node rac1.localdomain using shells /usr/bin/ssh and /usr/bin/rsh
    rac1.localdomain: Connection refused]]

  10. comment number 10 by: Prakash Heda

    error at step when starting vip application…(tried retry dosn’t work and then ignore and it finished, hoping its ignorable error)
    [root@rac2 bin]# ./vipca

    [root@rac2 racg]# tail -n30 ora.rac2.vip.log
    Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, 2005 Oracle. All rights reserved.
    2009-07-26 11:44:45.327: [ RACG][3067807424] [14072][3067807424][ora.rac2.vip]: Interface eth0 checked failed (host=rac2.localdomain)
    Invalid parameters, or failed to bring up VIP (host=rac2.localdomain)

    2009-07-26 11:44:45.327: [ RACG][3067807424] [14072][3067807424][ora.rac2.vip]: clsrcexecut: env ORACLE_CONFIG_HOME=/u01/crs/oracle/product/10.2.0/crs

    2009-07-26 11:44:45.327: [ RACG][3067807424] [14072][3067807424][ora.rac2.vip]: clsrcexecut: cmd = /u01/crs/oracle/product/10.2.0/crs/bin/racgeut -e _USR_ORA_DEBUG=0 54 /u01/crs/oracle/product/10.2.0/crs/bin/racgvip start rac2

    2009-07-26 11:44:45.327: [ RACG][3067807424] [14072][3067807424][ora.rac2.vip]: clsrcexecut: rc = 1, time = 9.600s

    2009-07-26 11:44:54.576: [ RACG][3067807424] [14072][3067807424][ora.rac2.vip]: Interface eth0 checked failed (host=rac2.localdomain)
    Invalid parameters, or failed to bring up VIP (host=rac2.localdomain)

    2009-07-26 11:44:54.577: [ RACG][3067807424] [14072][3067807424][ora.rac2.vip]: clsrcexecut: env ORACLE_CONFIG_HOME=/u01/crs/oracle/product/10.2.0/crs

    2009-07-26 11:44:54.577: [ RACG][3067807424] [14072][3067807424][ora.rac2.vip]: clsrcexecut: cmd = /u01/crs/oracle/product/10.2.0/crs/bin/racgeut -e _USR_ORA_DEBUG=0 54 /u01/crs/oracle/product/10.2.0/crs/bin/racgvip check rac2

    2009-07-26 11:44:54.577: [ RACG][3067807424] [14072][3067807424][ora.rac2.vip]: clsrcexecut: rc = 1, time = 9.250s

    2009-07-26 11:44:54.577: [ RACG][3067807424] [14072][3067807424][ora.rac2.vip]: end for resource = ora.rac2.vip, action = start, status = 1, time = 19.270s

  11. comment number 11 by: Prakash Heda

    seems I am not able to ping 192.168.10.21 or 11 (vip ip’s), I dont remember i set anything other than entering into the hosts file of bpoth machines…

    is there any config required to have these two vip working?

  12. comment number 12 by: Prakash Heda

    amazing…somehow after 2-3 restarts it works fine…something strange, but it worked…

    Thx!!


  13. [...] own project I used  Internet resources of course, and a few of them are listed  below. A useful blog to setup storage with Openfiler Build Your Own Oracle RAC 11g Cluster on Oracle Enterprise Linux [...]


  14. [...] own project I used  Internet resources of course, and a few of them are listed  below. A useful blog to setup storage with Openfiler Build Your Own Oracle RAC 11g Cluster on Oracle Enterprise Linux [...]

Leave a Reply

Name

Mail (never published)

Website