Configuring multipath devices for Oracle Clusterware 11g on RHEL5/OEL5

Udev and Device Name Persistency

Unlike devlabel in the 2.4 kernel, udev (the 2.6 kernel device file naming scheme) dynamically creates device file names at boot time. However, this can give rise to the possibility that device file names may change – a device that may have once been named /dev/sdd say, may be renamed as /dev/sdf after reboot. Without specific configuration, if udev is left to dynamically name devices, the potential exists for devices referred to, or inadvertently accessed by, their arbitrary kernel-assigned name (e.g. Oracle Clusterware files; Cluster Registry, Voting disks, etc.) to fail to start or become corrupt.

Multipath and Udev

The creation of multipath devices is performed through the use of device-mapper-multipath, which although drawing upon device-mapper, is not completely integrated into the udev naming scheme. Significant modification of default udev rules is required to manipulate multipath device naming, therefore introduces potential supportability issues. Therefore, other means are required to configure device naming persistency and user-defined naming of multipath devices.

Before being able to configure udev to explicitly name devices, scsi_id(8) should first be configured to return their device identifiers. Modify the /etc/scsi_id.config file – add. or replace the existing ‘option=-b’ parameter/value pair (if exists), with ‘option=-g’, for example:

# grep -v ^# /etc/scsi_id.config
vendor="ATA",options=-p 0x80
options=-g


check the partitions names, these names are frequently gets change after each reboot.

[root@rac-node1 ~]# cat /proc/partitions
major minor #blocks name
104 0 143338560 cciss/c0d0
104 1 102400 cciss/c0d0p1
104 2 143236126 cciss/c0d0p2
8 0 20971520 sda
8 16 104857600 sdb
8 17 104856223 sdb1
8 32 104857600 sdc
8 33 104856223 sdc1
8 48 1048576 sdd
8 49 1044193 sdd1
8 64 1048576 sde
8 65 1044193 sde1
8 80 262144000 sdf
8 81 262140606 sdf1
8 96 20971520 sdg
8 112 104857600 sdh
8 113 104856223 sdh1
8 128 104857600 sdi
8 129 104856223 sdi1
8 144 1048576 sdj
8 145 1044193 sdj1
8 160 1048576 sdk
8 161 1044193 sdk1
8 176 262144000 sdl
8 177 262140606 sdl1
8 192 20971520 sdm
8 208 104857600 sdn
8 209 104856223 sdn1
8 224 104857600 sdo
8 225 104856223 sdo1
8 240 1048576 sdp
8 241 1044193 sdp1
65 0 1048576 sdq
65 1 1044193 sdq1
65 16 262144000 sdr
65 17 262140606 sdr1
65 32 20971520 sds
65 48 104857600 sdt
65 49 104856223 sdt1
65 64 104857600 sdu
65 65 104856223 sdu1
65 80 1048576 sdv
65 81 1044193 sdv1
65 96 1048576 sdw
65 97 1044193 sdw1
65 112 262144000 sdx
65 113 262140606 sdx1
253 0 20480000 dm-0
253 1 10223616 dm-1
253 2 30703616 dm-2
253 3 8192000 dm-3
253 4 10223616 dm-4
253 5 491520 dm-5
253 6 40960000 dm-6
253 7 8192000 dm-7
253 8 20971520 dm-8
253 9 104857600 dm-9
253 10 104857600 dm-10
253 11 1048576 dm-11
253 12 1048576 dm-12
253 13 262144000 dm-13
253 14 1044193 dm-14
253 15 104856223 dm-15
253 16 262140606 dm-16
253 17 1044193 dm-17
253 18 104856223 dm-18
[root@rac-node1 ~]#

using below script to know the disks with LUN number which is not changing on reboot

[root@rac-node1 ~]# for i in `cat /proc/partitions | awk '{print $4}' |grep sd`; do echo "### $i: `scsi_id -g -u -s /block/$i`"; done
### sda: 3600508b4000986c40000500000280000
### sdb: 3600508b4000986c400005000002c0000
### sdb1:
### sdc: 3600508b4000986c400005000002f0000
### sdc1:
### sdd: 3600508b4000986c40000500000350000
### sdd1:
### sde: 3600508b4000986c40000500000320000
### sde1:
### sdf: 3600508b4000986c40000500000500000
### sdf1:
### sdg: 3600508b4000986c40000500000280000
### sdh: 3600508b4000986c400005000002c0000
### sdh1:
### sdi: 3600508b4000986c400005000002f0000
### sdi1:
### sdj: 3600508b4000986c40000500000350000
### sdj1:
### sdk: 3600508b4000986c40000500000320000
### sdk1:
### sdl: 3600508b4000986c40000500000500000
### sdl1:
### sdm: 3600508b4000986c40000500000280000
### sdn: 3600508b4000986c400005000002c0000
### sdn1:
### sdo: 3600508b4000986c400005000002f0000
### sdo1:
### sdp: 3600508b4000986c40000500000350000
### sdp1:
### sdq: 3600508b4000986c40000500000320000
### sdq1:
### sdr: 3600508b4000986c40000500000500000
### sdr1:
### sds: 3600508b4000986c40000500000280000
### sdt: 3600508b4000986c400005000002c0000
### sdt1:
### sdu: 3600508b4000986c400005000002f0000
### sdu1:
### sdv: 3600508b4000986c40000500000350000
### sdv1:
### sdw: 3600508b4000986c40000500000320000
### sdw1:
### sdx: 3600508b4000986c40000500000500000
### sdx1:
[root@rac-node1 ~]#

Configure the multipath.conf file for raw and ASM disks

cat /etc/multipath.conf
multipaths {
 multipath {
 wwid 3600508b4000986c40000500000350000
 alias voting
}
 multipath {
 wwid 3600508b4000986c40000500000320000
 alias ocr
 }
 multipath {
 wwid 3600508b4000986c400005000002f0000
 alias data
 }
 multipath {
 wwid 3600508b4000986c400005000002c0000
 alias index
 }
 multipath {
 wwid 3600508b4000986c40000500000500000
 alias rman
 }
}

confirm changes in the multipath.conf file

[root@rac-node1 ~]# multipath -ll
index (3600508b4000986c400005000002c0000) dm-9 HP,HSV300
[size=100G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 0:0:1:2 sdh 8:112 [active][ready]
 \_ 1:0:0:2 sdn 8:208 [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 0:0:0:2 sdb 8:16 [active][ready]
 \_ 1:0:1:2 sdt 65:48 [active][ready]
mpath0 (3600508b4000986c40000500000280000) dm-8 HP,HSV300
[size=20G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 0:0:1:1 sdg 8:96 [active][ready]
 \_ 1:0:0:1 sdm 8:192 [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 0:0:0:1 sda 8:0 [active][ready]
 \_ 1:0:1:1 sds 65:32 [active][ready]
voting (3600508b4000986c40000500000350000) dm-11 HP,HSV300
[size=1.0G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 0:0:0:4 sdd 8:48 [active][ready]
 \_ 1:0:1:4 sdv 65:80 [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 0:0:1:4 sdj 8:144 [active][ready]
 \_ 1:0:0:4 sdp 8:240 [active][ready]
rman (3600508b4000986c40000500000500000) dm-13 HP,HSV300
[size=250G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 0:0:0:6 sdf 8:80 [active][ready]
 \_ 1:0:1:6 sdx 65:112 [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 0:0:1:6 sdl 8:176 [active][ready]
 \_ 1:0:0:6 sdr 65:16 [active][ready]
data (3600508b4000986c400005000002f0000) dm-10 HP,HSV300
[size=100G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 0:0:0:3 sdc 8:32 [active][ready]
 \_ 1:0:1:3 sdu 65:64 [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 0:0:1:3 sdi 8:128 [active][ready]
 \_ 1:0:0:3 sdo 8:224 [active][ready]
ocr (3600508b4000986c40000500000320000) dm-12 HP,HSV300
[size=1.0G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 0:0:1:5 sdk 8:160 [active][ready]
 \_ 1:0:0:5 sdq 65:0 [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 0:0:0:5 sde 8:64 [active][ready]
 \_ 1:0:1:5 sdw 65:96 [active][ready]
[root@rac-node1 ~]#

and

[root@rac-node1 ~]# dmsetup ls | sort
data (253, 10)
datap1 (253, 18)
index (253, 9)
indexp1 (253, 15)
mpath0 (253, 8)
ocr (253, 12)
ocrp1 (253, 14)
rman (253, 13)
rmanp1 (253, 16)
VolGroup00-boot (253, 5)
VolGroup00-home (253, 6)
VolGroup00-LogVol00 (253, 0)
VolGroup00-LogVol01 (253, 7)
VolGroup00-opt (253, 4)
VolGroup00-tmp (253, 3)
VolGroup00-user (253, 2)
VolGroup00-var (253, 1)
voting (253, 11)
votingp1 (253, 17)
[root@rac-node1 ~]#

Check permission and  disk by id

[root@rac-node1 ~]# ls -l /dev/disk/by-id/
total 0
lrwxrwxrwx 1 root root 9 Jun 19 15:21 scsi-3600508b4000986c40000500000280000 -> ../../sdm
lrwxrwxrwx 1 root root 9 Jun 19 15:21 scsi-3600508b4000986c400005000002c0000 -> ../../sdn
lrwxrwxrwx 1 root root 10 Jun 19 15:22 scsi-3600508b4000986c400005000002c0000-part1 -> ../../sdt1
lrwxrwxrwx 1 root root 9 Jun 19 15:21 scsi-3600508b4000986c400005000002f0000 -> ../../sdo
lrwxrwxrwx 1 root root 10 Jun 19 15:22 scsi-3600508b4000986c400005000002f0000-part1 -> ../../sdo1
lrwxrwxrwx 1 root root 9 Jun 19 15:21 scsi-3600508b4000986c40000500000320000 -> ../../sde
lrwxrwxrwx 1 root root 10 Jun 19 15:22 scsi-3600508b4000986c40000500000320000-part1 -> ../../sdw1
lrwxrwxrwx 1 root root 9 Jun 19 15:21 scsi-3600508b4000986c40000500000350000 -> ../../sdj
lrwxrwxrwx 1 root root 10 Jun 19 15:22 scsi-3600508b4000986c40000500000350000-part1 -> ../../sdp1
lrwxrwxrwx 1 root root 9 Jun 19 15:21 scsi-3600508b4000986c40000500000500000 -> ../../sdx
lrwxrwxrwx 1 root root 10 Jun 19 15:22 scsi-3600508b4000986c40000500000500000-part1 -> ../../sdx1
lrwxrwxrwx 1 root root 9 Jun 19 15:21 usb-TEAC_DVD-ROM_DW-224EV_A60020000001 -> ../../sr0
[root@rac-node1 ~]#

check exact numbers of disks (LUN)

[root@rac-node1 ~]# ls -l /dev/dm-*
brw-rw---- 1 root root 253, 10 Jun 19 15:21 /dev/dm-10
brw-rw---- 1 root root 253, 11 Jun 19 15:21 /dev/dm-11
brw-rw---- 1 root root 253, 12 Jun 19 15:21 /dev/dm-12
brw-rw---- 1 root root 253, 13 Jun 19 15:21 /dev/dm-13
brw-rw---- 1 root root 253, 14 Jun 19 15:21 /dev/dm-14
brw-rw---- 1 root root 253, 15 Jun 19 15:21 /dev/dm-15
brw-rw---- 1 root root 253, 16 Jun 19 15:21 /dev/dm-16
brw-rw---- 1 root root 253, 17 Jun 19 15:21 /dev/dm-17
brw-rw---- 1 root root 253, 18 Jun 19 15:21 /dev/dm-18
brw-rw---- 1 root root 253, 8 Jun 19 15:21 /dev/dm-8
brw-rw---- 1 root root 253, 9 Jun 19 15:21 /dev/dm-9
[root@rac-node1 ~]#

check exact numbers of disks (LUN)

[root@rac-node1 ~]# ls -l /dev/mpath/*
lrwxrwxrwx 1 root root 7 Jun 19 15:21 /dev/mpath/3600508b4000986c40000500000280000 -> ../dm-8
lrwxrwxrwx 1 root root 8 Jun 19 15:21 /dev/mpath/data -> ../dm-10
lrwxrwxrwx 1 root root 8 Jun 19 15:21 /dev/mpath/datap1 -> ../dm-18
lrwxrwxrwx 1 root root 7 Jun 19 15:21 /dev/mpath/index -> ../dm-9
lrwxrwxrwx 1 root root 8 Jun 19 15:21 /dev/mpath/indexp1 -> ../dm-15
lrwxrwxrwx 1 root root 8 Jun 19 15:21 /dev/mpath/ocr -> ../dm-12
lrwxrwxrwx 1 root root 8 Jun 19 15:21 /dev/mpath/ocrp1 -> ../dm-14
lrwxrwxrwx 1 root root 8 Jun 19 15:21 /dev/mpath/rman -> ../dm-13
lrwxrwxrwx 1 root root 8 Jun 19 15:21 /dev/mpath/rmanp1 -> ../dm-16
lrwxrwxrwx 1 root root 8 Jun 19 15:21 /dev/mpath/voting -> ../dm-11
lrwxrwxrwx 1 root root 8 Jun 19 15:21 /dev/mpath/votingp1 -> ../dm-17

check permission on each LUN

[root@rac-node1 ~]# ls -l /dev/mapper/
total 0
crw------- 1 root root 10, 62 Jun 19 15:21 control
brwxrwxr-x 1 oracle oinstall 253, 10 Jun 19 15:21 data
brwxrwxr-x 1 oracle oinstall 253, 18 Oct 18 17:37 datap1
brwxrwxr-x 1 oracle oinstall 253, 9 Jun 19 15:21 index
brwxrwxr-x 1 oracle oinstall 253, 15 Oct 18 17:37 indexp1
brw-rw---- 1 root disk 253, 8 Jun 19 15:21 mpath0
brwxrwxr-x 1 oracle oinstall 253, 12 Jun 19 15:21 ocr
brw-rw---- 1 oracle oinstall 253, 14 Jun 19 15:21 ocrp1
brwxrwxr-x 1 oracle oinstall 253, 13 Jun 19 15:21 rman
brwxrwxr-x 1 oracle oinstall 253, 16 Jun 19 15:21 rmanp1
brw-rw---- 1 root disk 253, 5 Jun 19 15:21 VolGroup00-boot
brw-rw---- 1 root disk 253, 6 Jun 19 15:21 VolGroup00-home
brw-rw---- 1 root disk 253, 0 Jun 19 15:21 VolGroup00-LogVol00
brw-rw---- 1 root disk 253, 7 Jun 19 15:21 VolGroup00-LogVol01
brw-rw---- 1 root disk 253, 4 Jun 19 15:21 VolGroup00-opt
brw-rw---- 1 root disk 253, 3 Jun 19 15:21 VolGroup00-tmp
brw-rw---- 1 root disk 253, 2 Jun 19 15:21 VolGroup00-user
brw-rw---- 1 root disk 253, 1 Jun 19 15:21 VolGroup00-var
brw-rw---- 1 oracle oinstall 253, 11 Jun 19 15:21 voting
brw-rw---- 1 oracle oinstall 253, 17 Jun 19 15:21 votingp1

Add permission to rc.local file which will auto apply on reboot of the server

[root@rac-node1 ~]# cat /etc/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
touch /var/lock/subsys/local
modprobe hangcheck-timer hangcheck_tick=30 hangcheck_margin=180
###
#OCR
chown oracle:oinstall /dev/mapper/ocr
chmod 775 /dev/mapper/ocr
chown oracle:oinstall /dev/mapper/ocrp1
chmod 775 /dev/mappper/ocrp1
raw /dev/raw/raw1 /dev/mapper/ocrp1
sleep 2
chown oracle:oinstall /dev/raw/raw1
chmod 775 /dev/raw/raw1
###
#voting
chown oracle:oinstall /dev/mapper/voting**
chmod 660 /dev/mapper/voting*
raw /dev/raw/raw2 /dev/mapper/votingp1
sleep 2
chown oracle:oinstall /dev/raw/raw2
chmod 660 /dev/raw/raw2
###data
chown oracle:oinstall /dev/mapper/data*
chmod 775 /dev/mapper/data*
chown oracle:oinstall /dev/mapper/index*
chmod 775 /dev/mapper/index*
chown oracle:oinstall /dev/mapper/rman*
chmod 775 /dev/mapper/rman*
[root@rac-node1 ~]#
Advertisements

About Sher khan

Senior Oracle DBA, Oracle 10g , Oracle 11g OCE ,Oracle 12C and Oracle 11g OPN certified Specialist, OCP Certified in Oracle 9i,10g , 11g and 12C. I am working in Oracle technology since 2004. Currently working in U.A.E Email: sher487@hotmail.com
This entry was posted in Oracle RAC. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s