When you run Linux on VMWare, you see this message by booting or by run udevtest

[root@rac1 ~]# udevtest /block/sda/sda1
add_to_rules: unknown key 'SUBSYSTEMS'
add_to_rules: unknown key 'ATTRS{vendor}'
add_to_rules: unknown key 'ATTRS{model}'
add_to_rules: unknown key 'SUBSYSTEMS'
add_to_rules: unknown key 'ATTRS{vendor}'
add_to_rules: unknown key 'ATTRS{model}'
main: looking at device '/block/sda/sda1' from subsystem 'block'
udev_rules_get_name: add symlink 'disk/by-id/scsi-36000c29e2bdf945e78262600987a25de-part1'
udev_rules_get_name: add symlink 'disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0-part1'
....

 
 
TODO:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1023898
 
 
File /etc/udev/rules.d/99-vmware-scsi-udev.rules before:

#
# VMware SCSI devices Timeout adjustment
#
# Modify the timeout value for VMware SCSI devices so that
# in the event of a failover, we don't time out.
# See Bug 271286 for more information.
#
# Note: The Udev systems vary from distro to distro.  Hence all of the
#       extra entries.

# Redhat systems
ACTION=="add", BUS=="scsi", SYSFS{vendor}=="VMware, " , SYSFS{model}=="VMware Virtual S",
RUN+="/bin/sh -c 'echo 180 >/sys$DEVPATH/device/timeout'"

# Debian systems
ACTION=="add", SUBSYSTEMS=="scsi", ATTRS{vendor}=="VMware  " , ATTRS{model}=="Virtual disk    ",
RUN+="/bin/sh -c 'echo 180 >/sys$DEVPATH/device/timeout'"

# SuSE / Ubuntu systems
ACTION=="add", SUBSYSTEMS=="scsi", ATTRS{vendor}=="VMware, " , ATTRS{model}=="VMware Virtual S",
RUN+="/bin/sh -c 'echo 180 >/sys$DEVPATH/device/timeout'"

 
 
In my case, i use OracleLinux or Redhat
File /etc/udev/rules.d/99-vmware-scsi-udev.rules after:

#
# VMware SCSI devices Timeout adjustment
#
# Modify the timeout value for VMware SCSI devices so that
# in the event of a failover, we don't time out.
# See Bug 271286 for more information.
#
# Note: The Udev systems vary from distro to distro.  Hence all of the
#       extra entries.

# Redhat systems
ACTION=="add", BUS=="scsi", SYSFS{vendor}=="VMware " , SYSFS{model}=="Virtual disk ",
RUN+="/bin/sh -c 'echo 180 >/sys$DEVPATH/device/timeout'"

 
 
Test:

Run udevtest again:

[root@rac1 ~]# udevtest /block/sda/sda1
main: looking at device '/block/sda/sda1' from subsystem 'block'
udev_rules_get_name: add symlink 'disk/by-id/scsi-36000c29e2bdf945e78262600987a25de-part1'
udev_rules_get_name: add symlink 'disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0-part1'
run_program: '/lib/udev/vol_id --export /dev/.tmp-8-1'
....

The question is exm: How can i create a new ASM diskgroup with 2 disks using raw device?
 
 
Add 3 new virtual disks
 

Create partition

Create 1 new primary partition for each disk:

 fdisk /dev/sdf

“n” : new partition
“p” : primary partition
“1″ : 1. partittion
Enters: accept all default
“w” : write

 fdisk /dev/sdg

 fdisk /dev/sdh

Binding raw devices

Edit file /etc/sysconfig/rawdevices

# raw device bindings
# format:    
#           
# example: /dev/raw/raw1 /dev/sda1
#          /dev/raw/raw2 8 5
/dev/raw/raw1 /dev/sdf1
/dev/raw/raw2 /dev/sdg1
/dev/raw/raw3 /dev/sdh1

Restart service

service rawdevices restart

Change owner, permissions

add this to /etc/rc.local

chown grid:oinstall /dev/raw/raw1
chown grid:oinstall /dev/raw/raw2
chmod 664 /dev/raw/raw1
chmod 664 /dev/raw/raw2

Note: the file permission should be 664 but not 600, when you use job role separation (user grid and oracle).
With permission 600 you get privilege problem when create database (with user oracle) on this raw device.
 
 

List raw devices

[root@oel57 ~]# raw -a
/dev/raw/raw1:  bound to major 8, minor 81
/dev/raw/raw2:  bound to major 8, minor 97
/dev/raw/raw3:  bound to major 8, minor 113

[root@oel57 ~]# ls -l /dev/raw
total 0
crw-rw-r-- 1 grid oinstall 162, 1 Apr 25 14:41 raw1
crw-rw-r-- 1 grid oinstall 162, 2 Apr 25 14:41 raw2
crw-rw-r-- 1 grid oinstall 162, 3 Apr 25 14:41 raw3

Test

create diskgroup:

[grid@oel57 ~]$ sqlplus / as sysasm
SQL> CREATE DISKGROUP raw_dg EXTERNAL REDUNDANCY DISK
'/dev/raw/raw1' name raw_disk1,
'/dev/raw/raw2' name raw_disk2,
'/dev/raw/raw3' name raw_disk3;

SQL> select a.name, b.name, b.path
   from v$asm_diskgroup a, v$asm_disk b
   where a.name = 'RAW_DG'
   and a.group_number=b.group_number;
NAME                 NAME                 PATH
-------------------- -------------------- ------------------------------
RAW_DG               RAW_DISK2            /dev/raw/raw2
RAW_DG               RAW_DISK3            /dev/raw/raw3
RAW_DG               RAW_DISK1            /dev/raw/raw1

now you can exm. run dbca to create a new database storing in this ASM diskgroup

Create for example a VM machine named mydns and install following:

Need RPMs

[root@mydns ~]# rpm -qa | grep bind
ypbind-1.19-12.el5_6.1
bind-9.3.6-16.P1.el5
bind-utils-9.3.6-16.P1.el5
bind-libs-9.3.6-16.P1.el5

/etc/named.conf

#  minimal DNS conf file for RAC 11g

options {
    //i Forward any name this DNS can't resolve to my router.
    forwarders { 192.168.80.2; };

    // Directory where named will look for zone files.
    directory "/var/named";
};

# Forward Zone
zone "mydom.com" IN {
    type master;
    file "mydom.com.zone";
    allow-update { none; };
};

# Reverse Zone
zone "80.168.192.in-addr.arpa" IN {
    type master;
    file "80.168.192.zone";
    allow-update { none; };
};

Overview

Zone file /etc/named/mydom.com.zone

$ORIGIN mydom.com.
$TTL 1D          ; time-to-live   - (1 day)

@  IN SOA  mydns.mydom. master.mydom.com. (
        201011021   ; serial number  - (yyyymmdd+s)
        1d          ; refresh        - (1 day)
        1h          ; retry          - (1 hour)
        1w          ; expire         - (1 week)
        60          ; minimum        - (1 minute)
)
;
@       NS mydns        ; mydns.mydom.com is the name server

localhost   A        127.0.0.1

; RAC nodes
rac1        A        192.168.80.151
rac2        A        192.168.80.152
rac1-vip    A        192.168.80.171
rac2-vip    A        192.168.80.172
rac1-priv   A        192.168.136.151
rac2-priv   A        192.168.136.152

; DNS
mydns       A        192.168.80.140

; SCAN - Single Client Access Name
rac-scan    A        192.168.80.201
rac-scan    A        192.168.80.202
rac-scan    A        192.168.80.203

Reverse zone file /etc/named/80.168.192.zone

$ORIGIN 80.168.192.in-addr.arpa.
$TTL 1D          ; time-to-live   - (1 day)

@  IN SOA  mydns.mydom. master.mydom.com. (
        201011021   ; serial number  - (yyyymmdd+s)
        1d          ; refresh        - (1 day)
        1h          ; retry          - (1 hour)
        1w          ; expire         - (1 week)
        60          ; minimum        - (1 minute)
)
;
@  NS mydns     ; mydns.mydom.com is the name server

; RAC nodes
151                     PTR     rac1.mydom.com.
152                     PTR     rac2.mydom.com.

; RAC-VIP nodes
171                     PTR     rac1-vip.mydom.com.
172                     PTR     rac1-vip.mydom.com.

; DNS server
40                      PTR     mydns.mydom.com. 

; SCAN - Single Client Access Name
201                     PTR     rac-scan.mydom.com.
202                     PTR     rac-scan.mydom.com.
203                     PTR     rac-scan.mydom.com.

/etc/resolv.conf (in DNS machine)

[root@mydns ~]# cat /etc/resolv.conf
nameserver 127.0.0.1
search mydom.com

Check config file

[root@mydns ~]# named-checkconf /etc/named.conf 

[root@mydns ~]# named-checkzone mydom.com /var/named/mydom.com.zone
zone mydom.com/IN: loaded serial 201011021
OK

[root@mydns ~]# named-checkzone 80.168.192.in-addr.arpa  /var/named/80.168.192.zone
zone 80.168.192.in-addr.arpa/IN: loaded serial 201011021
OK
[root@mydns ~]#

Run service named

[root@mydns ~]# service named start
Starting named:                                            [  OK  ]
[root@mydns ~]#

/etc/resolv.conf (in a RAC Node)

[root@rac1 ~]# cat /etc/resolv.conf
nameserver 192.168.80.140
search mydom.com
[root@rac1 ~]#

Test

[root@rac1 ~]# nslookup rac1
Server:         192.168.80.140
Address:        192.168.80.140#53

Name:   rac1.mydom.com
Address: 192.168.80.151

[root@rac1 ~]# nslookup rac1-vip
Server:         192.168.80.140
Address:        192.168.80.140#53

Name:   rac1-vip.mydom.com
Address: 192.168.80.171

[root@rac1 ~]# nslookup rac-scan
Server:         192.168.80.140
Address:        192.168.80.140#53

Name:   rac-scan.mydom.com
Address: 192.168.80.201
Name:   rac-scan.mydom.com
Address: 192.168.80.202
Name:   rac-scan.mydom.com
Address: 192.168.80.203

[root@rac1 ~]# nslookup rac-scan
Server:         192.168.80.140
Address:        192.168.80.140#53

Name:   rac-scan.mydom.com
Address: 192.168.80.203
Name:   rac-scan.mydom.com
Address: 192.168.80.201
Name:   rac-scan.mydom.com
Address: 192.168.80.202

Note: nslookup must return 3 IPs in a different order each time
 

[root@rac1 ~]# dig rac-scan.mydom.com

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-16.P1.el5 <<>> rac-scan.mydom.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10628
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 1, ADDITIONAL: 1

;; QUESTION SECTION:
;rac-scan.mydom.com.            IN      A

;; ANSWER SECTION:
rac-scan.mydom.com.     86400   IN      A       192.168.80.202
rac-scan.mydom.com.     86400   IN      A       192.168.80.203
rac-scan.mydom.com.     86400   IN      A       192.168.80.201

;; AUTHORITY SECTION:
mydom.com.              86400   IN      NS      mydns.mydom.com.

;; ADDITIONAL SECTION:
mydns.mydom.com.        86400   IN      A       192.168.80.140

;; Query time: 0 msec
;; SERVER: 192.168.80.140#53(192.168.80.140)
;; WHEN: Tue Apr 24 10:56:49 2012
;; MSG SIZE  rcvd: 120

[root@rac1 ~]# dig -x 192.168.80.203

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-16.P1.el5 <<>> -x 192.168.80.203
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56995
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;203.80.168.192.in-addr.arpa.   IN      PTR

;; ANSWER SECTION:
203.80.168.192.in-addr.arpa. 86400 IN   PTR     rac-scan.mydom.com.

;; AUTHORITY SECTION:
80.168.192.in-addr.arpa. 86400  IN      NS      mydns.80.168.192.in-addr.arpa.

;; Query time: 0 msec
;; SERVER: 192.168.80.140#53(192.168.80.140)
;; WHEN: Tue Apr 24 10:57:14 2012
;; MSG SIZE  rcvd: 97

[root@rac1 ~]#

 
Link:
http://www.oracle.com/technetwork/products/clustering/overview/scan-129069.pdf

- ACFS driver is not automatically loaded in Oracle Restart(Grid Infrastructur for standalone server)

see http://docs.oracle.com/cd/E14072_01/server.112/e10500/asmfs_extra.htm#CACBDGCC

- to correctly install of ACFS see: work arround

Before:

[root@oel57 ~]# lsmod | grep oracle
oracleasm               46356    1
[root@oel57 ~]#

You don’t see anything with ACFS!
 

TODO:

Creata a file acfsload

[root@oel57 ~]# vi /etc/init.d/acfsload

#!/bin/sh

# chkconfig: 2345 30 21
# description: Load Oracle ACFS drivers at system boot
/u01/app/grid/product/11.2.0/grid/bin/acfsload start -s

Note:
chkconfig: 2345 30 21: mean run in levels 2,3,4,5 start priority 30, stop priority 21
see: http://tldp.org/HOWTO/HighQuality-Apps-HOWTO/boot.html
 

[root@oel57 ~]# chmod u+x /etc/init.d/acfsload
[root@oel57 ~]# chkconfig --add acfsload
[root@oel57 ~]#
[root@oel57 ~]#  reboot

 
After:

[root@oel57 ~]# lsmod | grep oracle
 oracleacfs            787460  0
 oracleadvm            177792  0 
 oracleoks             226656  2 oracleacfs,oracleadvm
oracleasm              46356  1
[root@oel57 ~]#

 
Test:

[root@oel57 ~]# su - grid

[grid@oel57 ~]$ acfsdriverstate version
ACFS-9205: OS/ADVM,ACFS installed version = 2.6.18-8.el5(i386)/090715.1

[grid@oel57 ~]$ acfsdriverstate supported
ACFS-9200: Supported

[grid@oel57 ~]$ acfsdriverstate loaded
ACFS-9203: true

[grid@oel57 ~]$ acfsdriverstate installed
ACFS-9203: true

when install Oracle Grid Infrastructure, exactly when run script:

/u01/app/grid/product/11.2.0/grid/root.sh

you can may be got:

PRCR-1079 : Failed to start resource ora.orcl.db
ORA-01031: insufficient privileges
ORA-01031: insufficient privileges
CRS-2674: Start of 'ora.orcl.db' on 'pc5' failed

TODO:

Put user grid in secondary group dba, run:

usermod -G asmadmin,asmdba,asmoper,dba grid

 
Before:

[root@pc5 ~]# id grid
uid=501(grid) gid=1000(oinstall) groups=1000(oinstall),1300(asmadmin),1400(asmdba),1500(asmoper)

 
Run:

[root@pc5 ~]# usermod -G asmadmin,asmdba,asmoper,dba grid

 
After:

[root@pc5 ~]# id grid
uid=501(grid) gid=1000(oinstall) groups=1000(oinstall),1100(dba),1300(asmadmin),1400(asmdba),1500(asmoper)

and runInstaller again!

When you install Oracle infrastructure on Oracle Linux Version 7.5,
exactly when ran script: /u01/app/grid/product/11.2.0/grid/root.sh
you got the message:

...
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
ADVM/ACFS is not supported on oraclelinux-release-5-7.0.2
....
'UpdateNodeList' was successful.

Solution:

Create a file /tmp/.linux_release before run install grid infrastructure:

[root@oel57 ~]# cd /tmp
[root@oel57 tmp]# echo "enterprise-release-5-7.0.2" > .linux_release
[root@oel57 tmp]# cat .linux_release
enterprise-release-5-7.0.2

Why?
See: How Oracle compare the release to decide which OS to support:
/u01/app/grid/product/11.2.0/grid/lib/osds_acfslib.pm

Run the grid install again!

Check after install grid

[root@oel57 ~]# cd /u01/app/grid/product/11.2.0/grid/bin
[root@oel57 bin]# ./acfsdriverstate -orahome /u01/app/grid/product/11.2.0/grid version
ACFS-9205: OS/ADVM,ACFS installed version = 2.6.18-8.el5(i386)/090715.1

[root@oel57 ~]# lsmod | grep oracle
oracleacfs            787460  0
oracleadvm            177792  0
oracleoks             226656  2 oracleacfs,oracleadvm
oracleasm              46356  1

ADVM/ACFS seen correct installed!