Oracle Database Appliance(X5-2) Disk Online replacement

Oracle Database Appliance(X5-2) Disk Online replacement

     

• ODA 장비의 Disk 교체작업을 Online으로 하기 위하여 수행했던 내역을 정리하여 공유합니다.

 

0.기존의 Disk Group에서 삭제 및 분리

alter diskgroup DATA drop disk 'HDD_E0_S04_1463938000P1' force ; alter diskgroup RECO drop disk 'HDD_E0_S04_1463938000P2' force ;

   

• TEST를 위하여 정상인 Disk 1개를 분리시켜 Disk 교체 작업을 진행하였다.

   

1.ODA Disk 상태 확인

① OAKCLI 명령어

[root@base0 mapper]# oakcli show disk NAME PATH TYPE STATE STATE_DETAILS   e0_pd_00 /dev/sda HDD ONLINE Good e0_pd_01 /dev/sdb HDD ONLINE Good e0_pd_02 /dev/sdaa HDD ONLINE Good e0_pd_03 /dev/sdab HDD ONLINE Good e0_pd_04 /dev/sdac HDD UNKNOWN NewDiskInserted e0_pd_05 /dev/sdad HDD ONLINE Good e0_pd_06 /dev/sdae HDD ONLINE Good e0_pd_07 /dev/sdaf HDD ONLINE Good e0_pd_08 /dev/sdag HDD ONLINE Good e0_pd_09 /dev/sdah HDD ONLINE Good e0_pd_10 /dev/sdai HDD ONLINE Good e0_pd_11 /dev/sdaj HDD ONLINE Good e0_pd_12 /dev/sdak HDD ONLINE Good e0_pd_13 /dev/sdal HDD ONLINE Good e0_pd_14 /dev/sdam HDD ONLINE Good e0_pd_15 /dev/sdan HDD ONLINE Good e0_pd_16 /dev/sdao HDD ONLINE Good e0_pd_17 /dev/sdap HDD ONLINE Good e0_pd_18 /dev/sdaq HDD ONLINE Good e0_pd_19 /dev/sdar HDD ONLINE Good e0_pd_20 /dev/sdas SSD ONLINE Good e0_pd_21 /dev/sdat SSD ONLINE Good e0_pd_22 /dev/sdau SSD ONLINE Good e0_pd_23 /dev/sdav SSD ONLINE Good [root@base0 mapper]#

   

② ASM DISK 상태 확인

col GN format 99 col DN format 99 col NAME format a23   SELECT group_number GN,disk_number DN,name,state,mode_status,mount_status FROM v$asm_disk ORDER BY group_number, disk_number;     GN DN NAME STATE MODE_ST MOUNT_S --- --- ----------------------- -------- ------- ------- 0 0 NORMAL ONLINE CLOSED <- 4번 disk 제거 0 1 NORMAL ONLINE CLOSED <- 4번 disk 제거 1 0 HDD_E0_S17_1463722132P1 NORMAL ONLINE CACHED 1 1 HDD_E0_S05_1463725856P1 NORMAL ONLINE CACHED 1 2 HDD_E0_S12_1463726544P1 NORMAL ONLINE CACHED 1 3 HDD_E0_S10_1463726716P1 NORMAL ONLINE CACHED 1 4 HDD_E0_S16_1463726756P1 NORMAL ONLINE CACHED 1 5 HDD_E0_S08_1463754440P1 NORMAL ONLINE CACHED 1 6 HDD_E0_S15_1463852252P1 NORMAL ONLINE CACHED 1 7 HDD_E0_S19_1463885988P1 NORMAL ONLINE CACHED 1 8 HDD_E0_S01_1463911124P1 NORMAL ONLINE CACHED     GN DN NAME STATE MODE_ST MOUNT_S --- --- ----------------------- -------- ------- ------- 1 9 HDD_E0_S07_1463917036P1 NORMAL ONLINE CACHED 1 10 HDD_E0_S02_1463917808P1 NORMAL ONLINE CACHED 1 11 HDD_E0_S18_1463929152P1 NORMAL ONLINE CACHED 1 12 HDD_E0_S13_1463929180P1 NORMAL ONLINE CACHED 1 13 HDD_E0_S14_1463929248P1 NORMAL ONLINE CACHED 1 14 HDD_E0_S11_1463935128P1 NORMAL ONLINE CACHED 1 16 HDD_E0_S00_1463938068P1 NORMAL ONLINE CACHED 1 17 HDD_E0_S03_1463941008P1 NORMAL ONLINE CACHED 1 18 HDD_E0_S09_1463967484P1 NORMAL ONLINE CACHED 1 19 HDD_E0_S06_1463969744P1 NORMAL ONLINE CACHED 2 0 HDD_E0_S17_1463722132P2 NORMAL ONLINE CACHED     GN DN NAME STATE MODE_ST MOUNT_S --- --- ----------------------- -------- ------- ------- 2 1 HDD_E0_S05_1463725856P2 NORMAL ONLINE CACHED 2 2 HDD_E0_S12_1463726544P2 NORMAL ONLINE CACHED 2 3 HDD_E0_S10_1463726716P2 NORMAL ONLINE CACHED 2 4 HDD_E0_S16_1463726756P2 NORMAL ONLINE CACHED 2 5 HDD_E0_S08_1463754440P2 NORMAL ONLINE CACHED 2 6 HDD_E0_S15_1463852252P2 NORMAL ONLINE CACHED 2 7 HDD_E0_S19_1463885988P2 NORMAL ONLINE CACHED 2 8 HDD_E0_S01_1463911124P2 NORMAL ONLINE CACHED 2 9 HDD_E0_S07_1463917036P2 NORMAL ONLINE CACHED 2 10 HDD_E0_S02_1463917808P2 NORMAL ONLINE CACHED 2 11 HDD_E0_S18_1463929152P2 NORMAL ONLINE CACHED     GN DN NAME STATE MODE_ST MOUNT_S --- --- ----------------------- -------- ------- ------- 2 12 HDD_E0_S13_1463929180P2 NORMAL ONLINE CACHED 2 13 HDD_E0_S14_1463929248P2 NORMAL ONLINE CACHED 2 14 HDD_E0_S11_1463935128P2 NORMAL ONLINE CACHED 2 16 HDD_E0_S00_1463938068P2 NORMAL ONLINE CACHED 2 17 HDD_E0_S03_1463941008P2 NORMAL ONLINE CACHED 2 18 HDD_E0_S09_1463967484P2 NORMAL ONLINE CACHED 2 19 HDD_E0_S06_1463969744P2 NORMAL ONLINE CACHED 3 0 SSD_E0_S20_805971985P1 NORMAL ONLINE CACHED 3 1 SSD_E0_S21_805972008P1 NORMAL ONLINE CACHED 3 2 SSD_E0_S22_805972069P1 NORMAL ONLINE CACHED 3 3 SSD_E0_S23_805972099P1 NORMAL ONLINE CACHED     44 rows selected.

   

③ ISOF 상태 확인

[root@base0 mapper]# lsof | grep HDD_E0_S04 <- 비정상     [root@base0 mapper]# lsof | grep HDD_E0_S03 <- 정상 asm_dbw0_ 21756 grid 257u BLK 252,36 0t0 20941 /dev/mapper/HDD_E0_S03_1463941008p1 asm_rbal_ 21768 grid 286u BLK 252,37 0t0 20942 /dev/mapper/HDD_E0_S03_1463941008p2 asm_rbal_ 21768 grid 287u BLK 252,36 0t0 20941 /dev/mapper/HDD_E0_S03_1463941008p1 asm_gmon_ 21770 grid 272u BLK 252,36 0t0 20941 /dev/mapper/HDD_E0_S03_1463941008p1 asm_gmon_ 21770 grid 291u BLK 252,37 0t0 20942 /dev/mapper/HDD_E0_S03_1463941008p2 apx_rbal_ 22990 grid 286u BLK 252,37 0t0 20942 /dev/mapper/HDD_E0_S03_1463941008p2 oracle 26343 oracle 273u BLK 252,36 0t0 20941 /dev/mapper/HDD_E0_S03_1463941008p1 oracle 26345 oracle 273u BLK 252,36 0t0 20941 /dev/mapper/HDD_E0_S03_1463941008p1 oracle 26347 oracle 264u BLK 252,36 0t0 20941 /dev/mapper/HDD_E0_S03_1463941008p1 oracle 26349 oracle 269u BLK 252,36 0t0 20941 /dev/mapper/HDD_E0_S03_1463941008p1 oracle 26349 oracle 282u BLK 252,37 0t0 20942 /dev/mapper/HDD_E0_S03_1463941008p2 oracle 26351 oracle 272u BLK 252,36 0t0 20941 /dev/mapper/HDD_E0_S03_1463941008p1 oracle 26351 oracle 291u BLK 252,37 0t0 20942 /dev/mapper/HDD_E0_S03_1463941008p2 oracle 26357 oracle 272u BLK 252,36 0t0 20941 /dev/mapper/HDD_E0_S03_1463941008p1 oracle 26357 oracle 291u BLK 252,37 0t0 20942 /dev/mapper/HDD_E0_S03_1463941008p2 oracle 48681 oracle 263u BLK 252,36 0t0 20941 /dev/mapper/HDD_E0_S03_1463941008p1

   

2.기존 Disk 제거 및 신규 Disk 삽입

① ODA 장비에서 해당 Disk 를 확인하기 위하여 Disk Light ON

oakcli locate disk e0_pd_04 on/off

• 해당 명령어를 수행하면 ODA장비의 Disk 영역에 불빛이 들어온다.

     

     ② 신규 Disk 삽입 및 신규 정보 등록

(동일한 Disk를 다시 삽입(정상인 Disk)하면 Data Rebalancing이 자동 수행되어 Disk가 삽입될 수 있다.)

   

- Data Rebalancing 확인 명령어     select * from gv$asm_operation; -- Data rebalancing 확인

- 1번 node(Master Node)에서만 작업이 가능하다     [root@base0 mapper]# oakcli add disk e0_pd_04 Choose any of following option 0 : Exit add disk operation 1 : Try to add disk only to OAKD not to ASM 2 : Try to add disk to OAKD and ASM Enter number between 0-2 : 2 INFO: 2015-12-13 01:16:08: adddisk operation takes about 7-8 minutes, so be patience and don't abort early INFO: 2015-12-13 01:16:08: Setting up root ssh across the private network     Please enter the 'root' password : --(중요) 특수 문자 안됨, SSH 22번 포트만 가능 Please re-enter the 'root' password: --(중요) 특수 문자 안됨, SSH 22번 포트만 가능 ...........done INFO: 2015-12-13 01:17:08: Executing function dump_var_values. Dumping variable values passed to storage adapters code INFO: 2015-12-13 01:17:08: resourceName=e0_pd_04, expander=e0, slotnum=04 INFO: 2015-12-13 01:17:08: slotnumMinusLeadingZero=4, disk=HDD, _OAK_AsmDiskList=e0_data_04 e0_reco_04 INFO: 2015-12-13 01:17:08: output=/dev/sdac /dev/sde INFO: 2015-12-13 01:17:08: _OAK_MultiPathList=/dev/sdac /dev/sde, _OAK_DiskId=35000cca05741ebd0, OAK_UsrDevName=HDD_E0_S04_1463938000 INFO: 2015-12-13 01:17:08: DevName=HDD_E0_S04_1463938000, last8byet=5741ebd0, last8byetInDecimal=1463938000 INFO: 2015-12-13 01:17:08: dataDgRatio=86, recoDgRatio=14, OAK_PartitionMap=86,14 INFO: 2015-12-13 01:17:08: _OAK_OSUserType=userType:Multiuser, _OAK_Size=900185481216, _OAK_TotalSectors=1758174768 INFO: 2015-12-13 01:17:08: Executing function check_resource_status_in_asm Warning: Permanently added 'base1,192.168.0.212' (RSA) to the list of known hosts. INFO: 2015-12-13 01:17:09: Executing function: Setting up variable for node1 INFO: 2015-12-13 01:17:09: Executing function: Calling CleanPDisk on node1 INFO: Found the HDD_E0_S04 already present in the config, will remove it for 35000cca05741ebd0 INFO: Removing entry for HDD_E0_S04 and old diskName is HDD_E0_S04_1463938000 INFO: Running /sbin/kpartx -d /dev/mapper/HDD_E0_S04_1463938000 Warning: Disk has a valid GPT signature but invalid PMBR. Assuming this disk is *not* a GPT disk anymore. Use gpt kernel option to override. Use GNU Parted to correct disk. INFO: Updated /etc/multipath.conf file INFO: Calling function removePartitionMapping INFO: Running /sbin/kpartx -d /dev/mapper/HDD_E0_S04_1463938000 Warning: Disk has a valid GPT signature but invalid PMBR. Assuming this disk is *not* a GPT disk anymore. Use gpt kernel option to override. Use GNU Parted to correct disk. INFO: Calling function flushMultipathDiskPath INFO: Running /sbin/multipath -f HDD_E0_S04_1463938000 INFO: Calling function removeDevMapPath INFO: Running /sbin/dmsetup remove -f HDD_E0_S04_1463938000 device-mapper: table ioctl failed: No such device or address device-mapper: reload ioctl failed: No such device or address device-mapper: remove ioctl failed: No such device or address Command failed ERROR: Failure in running /sbin/dmsetup remove -f HDD_E0_S04_1463938000, system command return status 256 INFO: 2015-12-13 01:17:09: Executing function: Calling AddPDisk on node1 INFO: addPDisk function with ResetDisk set to true INFO: Calling function checkDiskReadyNess INFO: Running /usr/bin/sg_turs /dev/sdac INFO: Running /bin/dd if=/dev/sdac iflag=direct of=/dev/null count=100 bs=1024k 2>&1 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.570512 seconds, 184 MB/s INFO: Running /bin/dd if=/dev/zero of=/dev/sdac oflag=direct count=100 bs=1024k 2>&1 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 1.16047 seconds, 90.4 MB/s INFO: adding the 35000cca05741ebd0 to the config at the end INFO: Updated /etc/multipath.conf file INFO: Running /sbin/multipath -v2 in reloadMultipathDevMap create: HDD_E0_S04_1463938000 (35000cca05741ebd0) undef HITACHI,H109090SESUN900G size=838G features='0' hwhandler='0' wp=undef `-+- policy='round-robin 0' prio=1 status=undef |- 3:0:26:0 sde 8:64 undef ready running `- 2:0:26:0 sdac 65:192 undef ready running INFO: Running /sbin/kpartx -a /dev/mapper/HDD_E0_S04_1463938000 in reloadMultipathDevMap Warning: Disk has a valid GPT signature but invalid PMBR. Assuming this disk is *not* a GPT disk anymore. Use gpt kernel option to override. Use GNU Parted to correct disk. INFO: running parted -s /dev/mapper/HDD_E0_S04_1463938000 unit % print ….     문제 없이 Disk 삽입에 따른 환경 설정이 완료된다.

   

3.Disk 삽입 완료 후 상태확인

① OAKCLI 명령어

[root@base0 mapper]# oakcli show disk => e0_pd_04 /dev/sdac HDD UNKNOWN NewDiskInserted 상태가 good로 변경     NAME PATH TYPE STATE STATE_DETAILS   e0_pd_00 /dev/sda HDD ONLINE Good e0_pd_01 /dev/sdb HDD ONLINE Good e0_pd_02 /dev/sdaa HDD ONLINE Good e0_pd_03 /dev/sdab HDD ONLINE Good e0_pd_04 /dev/sdac HDD ONLINE Good e0_pd_05 /dev/sdad HDD ONLINE Good     -> Disk 인식 완료

   

② ASM Disk 상태 확인

SQL> col GN format 99 SQL> col DN format 99 SQL> col NAME format a23 SQL> SQL> SELECT 2 group_number GN,disk_number DN,name,state,mode_status,mount_status 3 FROM v$asm_disk 4 ORDER BY group_number, disk_number;     GN DN NAME STATE MODE_ST MOUNT_S --- --- ----------------------- -------- ------- ------- 1 0 HDD_E0_S17_1463722132P1 NORMAL ONLINE CACHED 1 1 HDD_E0_S05_1463725856P1 NORMAL ONLINE CACHED 1 2 HDD_E0_S12_1463726544P1 NORMAL ONLINE CACHED 1 3 HDD_E0_S10_1463726716P1 NORMAL ONLINE CACHED 1 4 HDD_E0_S16_1463726756P1 NORMAL ONLINE CACHED 1 5 HDD_E0_S08_1463754440P1 NORMAL ONLINE CACHED 1 6 HDD_E0_S15_1463852252P1 NORMAL ONLINE CACHED 1 7 HDD_E0_S19_1463885988P1 NORMAL ONLINE CACHED 1 8 HDD_E0_S01_1463911124P1 NORMAL ONLINE CACHED 1 9 HDD_E0_S07_1463917036P1 NORMAL ONLINE CACHED 1 10 HDD_E0_S02_1463917808P1 NORMAL ONLINE CACHED     GN DN NAME STATE MODE_ST MOUNT_S --- --- ----------------------- -------- ------- ------- 1 11 HDD_E0_S18_1463929152P1 NORMAL ONLINE CACHED 1 12 HDD_E0_S13_1463929180P1 NORMAL ONLINE CACHED 1 13 HDD_E0_S14_1463929248P1 NORMAL ONLINE CACHED 1 14 HDD_E0_S11_1463935128P1 NORMAL ONLINE CACHED 1 15 HDD_E0_S04_1463938000P1 NORMAL ONLINE CACHED 1 16 HDD_E0_S00_1463938068P1 NORMAL ONLINE CACHED 1 17 HDD_E0_S03_1463941008P1 NORMAL ONLINE CACHED 1 18 HDD_E0_S09_1463967484P1 NORMAL ONLINE CACHED 1 19 HDD_E0_S06_1463969744P1 NORMAL ONLINE CACHED 2 0 HDD_E0_S17_1463722132P2 NORMAL ONLINE CACHED 2 1 HDD_E0_S05_1463725856P2 NORMAL ONLINE CACHED     GN DN NAME STATE MODE_ST MOUNT_S --- --- ----------------------- -------- ------- ------- 2 2 HDD_E0_S12_1463726544P2 NORMAL ONLINE CACHED 2 3 HDD_E0_S10_1463726716P2 NORMAL ONLINE CACHED 2 4 HDD_E0_S16_1463726756P2 NORMAL ONLINE CACHED 2 5 HDD_E0_S08_1463754440P2 NORMAL ONLINE CACHED 2 6 HDD_E0_S15_1463852252P2 NORMAL ONLINE CACHED 2 7 HDD_E0_S19_1463885988P2 NORMAL ONLINE CACHED 2 8 HDD_E0_S01_1463911124P2 NORMAL ONLINE CACHED 2 9 HDD_E0_S07_1463917036P2 NORMAL ONLINE CACHED 2 10 HDD_E0_S02_1463917808P2 NORMAL ONLINE CACHED 2 11 HDD_E0_S18_1463929152P2 NORMAL ONLINE CACHED 2 12 HDD_E0_S13_1463929180P2 NORMAL ONLINE CACHED     GN DN NAME STATE MODE_ST MOUNT_S --- --- ----------------------- -------- ------- ------- 2 13 HDD_E0_S14_1463929248P2 NORMAL ONLINE CACHED 2 14 HDD_E0_S11_1463935128P2 NORMAL ONLINE CACHED 2 15 HDD_E0_S04_1463938000P2 NORMAL ONLINE CACHED 2 16 HDD_E0_S00_1463938068P2 NORMAL ONLINE CACHED 2 17 HDD_E0_S03_1463941008P2 NORMAL ONLINE CACHED 2 18 HDD_E0_S09_1463967484P2 NORMAL ONLINE CACHED 2 19 HDD_E0_S06_1463969744P2 NORMAL ONLINE CACHED 3 0 SSD_E0_S20_805971985P1 NORMAL ONLINE CACHED 3 1 SSD_E0_S21_805972008P1 NORMAL ONLINE CACHED 3 2 SSD_E0_S22_805972069P1 NORMAL ONLINE CACHED 3 3 SSD_E0_S23_805972099P1 NORMAL ONLINE CACHED

   

③ ISOF 상태 확인

• 정상적으로 해당 프로세스들이 올라오는 것을 확인할 수 있다.

[root@base0 mapper]# lsof | grep HDD_E0_S04 asm_dbw0_ 38776 grid 280u BLK 252,38 0t0 5509236 /dev/mapper/HDD_E0_S04_1463938000p1 asm_rbal_ 38788 grid 256u BLK 252,39 0t0 5505972 /dev/mapper/HDD_E0_S04_1463938000p2 asm_rbal_ 38788 grid 257u BLK 252,38 0t0 5509236 /dev/mapper/HDD_E0_S04_1463938000p1 apx_rbal_ 40507 grid 256u BLK 252,39 0t0 5505972 /dev/mapper/HDD_E0_S04_1463938000p2 mdb_dbw0_ 41381 grid 275u BLK 252,38 0t0 5509236 /dev/mapper/HDD_E0_S04_1463938000p1 mdb_ckpt_ 41385 grid 259u BLK 252,38 0t0 5509236 /dev/mapper/HDD_E0_S04_1463938000p1 mdb_smon_ 41389 grid 258u BLK 252,38 0t0 5509236 /dev/mapper/HDD_E0_S04_1463938000p1 mdb_rbal_ 41397 grid 257u BLK 252,38 0t0 5509236 /dev/mapper/HDD_E0_S04_1463938000p1 oracle 41476 oracle 274u BLK 252,38 0t0 5509236 /dev/mapper/HDD_E0_S04_1463938000p1 oracle 41478 oracle 265u BLK 252,38 0t0 5509236 /dev/mapper/HDD_E0_S04_1463938000p1 oracle 41480 oracle 266u BLK 252,38 0t0 5509236 /dev/mapper/HDD_E0_S04_1463938000p1 oracle 41486 oracle 272u BLK 252,38 0t0 5509236 /dev/mapper/HDD_E0_S04_1463938000p1 oracle 41490 oracle 294u BLK 252,38 0t0 5509236 /dev/mapper/HDD_E0_S04_1463938000p1 oracle 41490 oracle 295u BLK 252,39 0t0 5505972 /dev/mapper/HDD_E0_S04_1463938000p2 oracle_41 41564 grid 257u BLK 252,39 0t0 5505972 /dev/mapper/HDD_E0_S04_1463938000p2 asm_x000_ 87135 grid 271u BLK 252,38 0t0 5509236 /dev/mapper/HDD_E0_S04_1463938000p1

   

4.Parameter File

/etc/multipath.conf -> oakcli 확인 명령어 /opt/oracle/extapi/asmappl.config -> SQL 확인 명령어 multipath -r; multipath -F; multipath -v2

• 2번의 명령어를 통한 작업이 정상적으로 수행되지 않을 경우에 해당 파라미터 파일을 조작하여 작업을 진행할 수도 있음.