'DISK 교체'에 해당되는 글 1건


Oracle Database Appliance(X5-2) Disk Online replacement

     

  • ODA 장비의 Disk 교체작업을 Online으로 하기 위하여 수행했던 내역을 정리하여 공유합니다.

 

0.기존의 Disk Group에서 삭제 및 분리

alter diskgroup DATA drop disk 'HDD_E0_S04_1463938000P1' force ;

alter diskgroup RECO drop disk 'HDD_E0_S04_1463938000P2' force ;

   

  • TEST를 위하여 정상인 Disk 1개를 분리시켜 Disk 교체 작업을 진행하였다.

   

1.ODA Disk 상태 확인

① OAKCLI 명령어

[root@base0 mapper]# oakcli show disk

NAME PATH TYPE STATE STATE_DETAILS

 

e0_pd_00 /dev/sda HDD ONLINE Good

e0_pd_01 /dev/sdb HDD ONLINE Good

e0_pd_02 /dev/sdaa HDD ONLINE Good

e0_pd_03 /dev/sdab HDD ONLINE Good

e0_pd_04 /dev/sdac HDD UNKNOWN NewDiskInserted

e0_pd_05 /dev/sdad HDD ONLINE Good

e0_pd_06 /dev/sdae HDD ONLINE Good

e0_pd_07 /dev/sdaf HDD ONLINE Good

e0_pd_08 /dev/sdag HDD ONLINE Good

e0_pd_09 /dev/sdah HDD ONLINE Good

e0_pd_10 /dev/sdai HDD ONLINE Good

e0_pd_11 /dev/sdaj HDD ONLINE Good

e0_pd_12 /dev/sdak HDD ONLINE Good

e0_pd_13 /dev/sdal HDD ONLINE Good

e0_pd_14 /dev/sdam HDD ONLINE Good

e0_pd_15 /dev/sdan HDD ONLINE Good

e0_pd_16 /dev/sdao HDD ONLINE Good

e0_pd_17 /dev/sdap HDD ONLINE Good

e0_pd_18 /dev/sdaq HDD ONLINE Good

e0_pd_19 /dev/sdar HDD ONLINE Good

e0_pd_20 /dev/sdas SSD ONLINE Good

e0_pd_21 /dev/sdat SSD ONLINE Good

e0_pd_22 /dev/sdau SSD ONLINE Good

e0_pd_23 /dev/sdav SSD ONLINE Good

[root@base0 mapper]#

   

ASM DISK 상태 확인

col GN format 99

col DN format 99

col NAME format a23

 

SELECT

group_number GN,disk_number DN,name,state,mode_status,mount_status

FROM v$asm_disk

ORDER BY group_number, disk_number;

   

GN DN NAME STATE MODE_ST MOUNT_S

--- --- ----------------------- -------- ------- -------

0 0 NORMAL ONLINE CLOSED <- 4번 disk 제거

0 1 NORMAL ONLINE CLOSED <- 4번 disk 제거

1 0 HDD_E0_S17_1463722132P1 NORMAL ONLINE CACHED

1 1 HDD_E0_S05_1463725856P1 NORMAL ONLINE CACHED

1 2 HDD_E0_S12_1463726544P1 NORMAL ONLINE CACHED

1 3 HDD_E0_S10_1463726716P1 NORMAL ONLINE CACHED

1 4 HDD_E0_S16_1463726756P1 NORMAL ONLINE CACHED

1 5 HDD_E0_S08_1463754440P1 NORMAL ONLINE CACHED

1 6 HDD_E0_S15_1463852252P1 NORMAL ONLINE CACHED

1 7 HDD_E0_S19_1463885988P1 NORMAL ONLINE CACHED

1 8 HDD_E0_S01_1463911124P1 NORMAL ONLINE CACHED

   

GN DN NAME STATE MODE_ST MOUNT_S

--- --- ----------------------- -------- ------- -------

1 9 HDD_E0_S07_1463917036P1 NORMAL ONLINE CACHED

1 10 HDD_E0_S02_1463917808P1 NORMAL ONLINE CACHED

1 11 HDD_E0_S18_1463929152P1 NORMAL ONLINE CACHED

1 12 HDD_E0_S13_1463929180P1 NORMAL ONLINE CACHED

1 13 HDD_E0_S14_1463929248P1 NORMAL ONLINE CACHED

1 14 HDD_E0_S11_1463935128P1 NORMAL ONLINE CACHED

1 16 HDD_E0_S00_1463938068P1 NORMAL ONLINE CACHED

1 17 HDD_E0_S03_1463941008P1 NORMAL ONLINE CACHED

1 18 HDD_E0_S09_1463967484P1 NORMAL ONLINE CACHED

1 19 HDD_E0_S06_1463969744P1 NORMAL ONLINE CACHED

2 0 HDD_E0_S17_1463722132P2 NORMAL ONLINE CACHED

   

GN DN NAME STATE MODE_ST MOUNT_S

--- --- ----------------------- -------- ------- -------

2 1 HDD_E0_S05_1463725856P2 NORMAL ONLINE CACHED

2 2 HDD_E0_S12_1463726544P2 NORMAL ONLINE CACHED

2 3 HDD_E0_S10_1463726716P2 NORMAL ONLINE CACHED

2 4 HDD_E0_S16_1463726756P2 NORMAL ONLINE CACHED

2 5 HDD_E0_S08_1463754440P2 NORMAL ONLINE CACHED

2 6 HDD_E0_S15_1463852252P2 NORMAL ONLINE CACHED

2 7 HDD_E0_S19_1463885988P2 NORMAL ONLINE CACHED

2 8 HDD_E0_S01_1463911124P2 NORMAL ONLINE CACHED

2 9 HDD_E0_S07_1463917036P2 NORMAL ONLINE CACHED

2 10 HDD_E0_S02_1463917808P2 NORMAL ONLINE CACHED

2 11 HDD_E0_S18_1463929152P2 NORMAL ONLINE CACHED

   

GN DN NAME STATE MODE_ST MOUNT_S

--- --- ----------------------- -------- ------- -------

2 12 HDD_E0_S13_1463929180P2 NORMAL ONLINE CACHED

2 13 HDD_E0_S14_1463929248P2 NORMAL ONLINE CACHED

2 14 HDD_E0_S11_1463935128P2 NORMAL ONLINE CACHED

2 16 HDD_E0_S00_1463938068P2 NORMAL ONLINE CACHED

2 17 HDD_E0_S03_1463941008P2 NORMAL ONLINE CACHED

2 18 HDD_E0_S09_1463967484P2 NORMAL ONLINE CACHED

2 19 HDD_E0_S06_1463969744P2 NORMAL ONLINE CACHED

3 0 SSD_E0_S20_805971985P1 NORMAL ONLINE CACHED

3 1 SSD_E0_S21_805972008P1 NORMAL ONLINE CACHED

3 2 SSD_E0_S22_805972069P1 NORMAL ONLINE CACHED

3 3 SSD_E0_S23_805972099P1 NORMAL ONLINE CACHED

   

44 rows selected.

   

ISOF 상태 확인

[root@base0 mapper]# lsof | grep HDD_E0_S04 <- 비정상

   

[root@base0 mapper]# lsof | grep HDD_E0_S03 <- 정상

asm_dbw0_ 21756 grid 257u BLK 252,36 0t0 20941 /dev/mapper/HDD_E0_S03_1463941008p1

asm_rbal_ 21768 grid 286u BLK 252,37 0t0 20942 /dev/mapper/HDD_E0_S03_1463941008p2

asm_rbal_ 21768 grid 287u BLK 252,36 0t0 20941 /dev/mapper/HDD_E0_S03_1463941008p1

asm_gmon_ 21770 grid 272u BLK 252,36 0t0 20941 /dev/mapper/HDD_E0_S03_1463941008p1

asm_gmon_ 21770 grid 291u BLK 252,37 0t0 20942 /dev/mapper/HDD_E0_S03_1463941008p2

apx_rbal_ 22990 grid 286u BLK 252,37 0t0 20942 /dev/mapper/HDD_E0_S03_1463941008p2

oracle 26343 oracle 273u BLK 252,36 0t0 20941 /dev/mapper/HDD_E0_S03_1463941008p1

oracle 26345 oracle 273u BLK 252,36 0t0 20941 /dev/mapper/HDD_E0_S03_1463941008p1

oracle 26347 oracle 264u BLK 252,36 0t0 20941 /dev/mapper/HDD_E0_S03_1463941008p1

oracle 26349 oracle 269u BLK 252,36 0t0 20941 /dev/mapper/HDD_E0_S03_1463941008p1

oracle 26349 oracle 282u BLK 252,37 0t0 20942 /dev/mapper/HDD_E0_S03_1463941008p2

oracle 26351 oracle 272u BLK 252,36 0t0 20941 /dev/mapper/HDD_E0_S03_1463941008p1

oracle 26351 oracle 291u BLK 252,37 0t0 20942 /dev/mapper/HDD_E0_S03_1463941008p2

oracle 26357 oracle 272u BLK 252,36 0t0 20941 /dev/mapper/HDD_E0_S03_1463941008p1

oracle 26357 oracle 291u BLK 252,37 0t0 20942 /dev/mapper/HDD_E0_S03_1463941008p2

oracle 48681 oracle 263u BLK 252,36 0t0 20941 /dev/mapper/HDD_E0_S03_1463941008p1

   

2.기존 Disk 제거 및 신규 Disk 삽입

① ODA 장비에서 해당 Disk 를 확인하기 위하여 Disk Light ON

oakcli locate disk e0_pd_04 on/off

  • 해당 명령어를 수행하면 ODA장비의 Disk 영역에 불빛이 들어온다.

       

     ② 신규 Disk 삽입 및 신규 정보 등록

(동일한 Disk를 다시 삽입(정상인 Disk)하면 Data Rebalancing이 자동 수행되어 Disk가 삽입될 수 있다.)

   

- Data Rebalancing 확인 명령어

   

select * from gv$asm_operation; -- Data rebalancing 확인

- 1번 node(Master Node)에서만 작업이 가능하다

   

[root@base0 mapper]# oakcli add disk e0_pd_04

Choose any of following option

0 : Exit add disk operation

1 : Try to add disk only to OAKD not to ASM

2 : Try to add disk to OAKD and ASM

Enter number between 0-2 : 2

INFO: 2015-12-13 01:16:08: adddisk operation takes about 7-8 minutes, so be patience and don't abort early

INFO: 2015-12-13 01:16:08: Setting up root ssh across the private network

   

Please enter the 'root' password : --(중요) 특수 문자 안됨, SSH 22번 포트만 가능

Please re-enter the 'root' password: --(중요) 특수 문자 안됨, SSH 22번 포트만 가능

...........done

INFO: 2015-12-13 01:17:08: Executing function dump_var_values. Dumping variable values passed to storage adapters code

INFO: 2015-12-13 01:17:08: resourceName=e0_pd_04, expander=e0, slotnum=04

INFO: 2015-12-13 01:17:08: slotnumMinusLeadingZero=4, disk=HDD, _OAK_AsmDiskList=e0_data_04 e0_reco_04

INFO: 2015-12-13 01:17:08: output=/dev/sdac /dev/sde

INFO: 2015-12-13 01:17:08: _OAK_MultiPathList=/dev/sdac /dev/sde, _OAK_DiskId=35000cca05741ebd0, OAK_UsrDevName=HDD_E0_S04_1463938000

INFO: 2015-12-13 01:17:08: DevName=HDD_E0_S04_1463938000, last8byet=5741ebd0, last8byetInDecimal=1463938000

INFO: 2015-12-13 01:17:08: dataDgRatio=86, recoDgRatio=14, OAK_PartitionMap=86,14

INFO: 2015-12-13 01:17:08: _OAK_OSUserType=userType:Multiuser, _OAK_Size=900185481216, _OAK_TotalSectors=1758174768

INFO: 2015-12-13 01:17:08: Executing function check_resource_status_in_asm

Warning: Permanently added 'base1,192.168.0.212' (RSA) to the list of known hosts.

INFO: 2015-12-13 01:17:09: Executing function: Setting up variable for node1

INFO: 2015-12-13 01:17:09: Executing function: Calling CleanPDisk on node1

INFO: Found the HDD_E0_S04 already present in the config, will remove it for 35000cca05741ebd0

INFO: Removing entry for HDD_E0_S04 and old diskName is HDD_E0_S04_1463938000

INFO: Running /sbin/kpartx -d /dev/mapper/HDD_E0_S04_1463938000

Warning: Disk has a valid GPT signature but invalid PMBR.

Assuming this disk is *not* a GPT disk anymore.

Use gpt kernel option to override. Use GNU Parted to correct disk.

INFO: Updated /etc/multipath.conf file

INFO: Calling function removePartitionMapping

INFO: Running /sbin/kpartx -d /dev/mapper/HDD_E0_S04_1463938000

Warning: Disk has a valid GPT signature but invalid PMBR.

Assuming this disk is *not* a GPT disk anymore.

Use gpt kernel option to override. Use GNU Parted to correct disk.

INFO: Calling function flushMultipathDiskPath

INFO: Running /sbin/multipath -f HDD_E0_S04_1463938000

INFO: Calling function removeDevMapPath

INFO: Running /sbin/dmsetup remove -f HDD_E0_S04_1463938000

device-mapper: table ioctl failed: No such device or address

device-mapper: reload ioctl failed: No such device or address

device-mapper: remove ioctl failed: No such device or address

Command failed

ERROR: Failure in running /sbin/dmsetup remove -f HDD_E0_S04_1463938000, system command return status 256

INFO: 2015-12-13 01:17:09: Executing function: Calling AddPDisk on node1

INFO: addPDisk function with ResetDisk set to true

INFO: Calling function checkDiskReadyNess

INFO: Running /usr/bin/sg_turs /dev/sdac

INFO: Running /bin/dd if=/dev/sdac iflag=direct of=/dev/null count=100 bs=1024k 2>&1

100+0 records in

100+0 records out

104857600 bytes (105 MB) copied, 0.570512 seconds, 184 MB/s

INFO: Running /bin/dd if=/dev/zero of=/dev/sdac oflag=direct count=100 bs=1024k 2>&1

100+0 records in

100+0 records out

104857600 bytes (105 MB) copied, 1.16047 seconds, 90.4 MB/s

INFO: adding the 35000cca05741ebd0 to the config at the end

INFO: Updated /etc/multipath.conf file

INFO: Running /sbin/multipath -v2 in reloadMultipathDevMap

create: HDD_E0_S04_1463938000 (35000cca05741ebd0) undef HITACHI,H109090SESUN900G

size=838G features='0' hwhandler='0' wp=undef

`-+- policy='round-robin 0' prio=1 status=undef

|- 3:0:26:0 sde 8:64 undef ready running

`- 2:0:26:0 sdac 65:192 undef ready running

INFO: Running /sbin/kpartx -a /dev/mapper/HDD_E0_S04_1463938000 in reloadMultipathDevMap

Warning: Disk has a valid GPT signature but invalid PMBR.

Assuming this disk is *not* a GPT disk anymore.

Use gpt kernel option to override. Use GNU Parted to correct disk.

INFO: running parted -s /dev/mapper/HDD_E0_S04_1463938000 unit % print

….

   

문제 없이 Disk 삽입에 따른 환경 설정이 완료된다.

   

3.Disk 삽입 완료 후 상태확인

① OAKCLI 명령어

[root@base0 mapper]# oakcli show disk

=> e0_pd_04 /dev/sdac HDD UNKNOWN NewDiskInserted 상태가 good로 변경

   

NAME PATH TYPE STATE STATE_DETAILS

 

e0_pd_00 /dev/sda HDD ONLINE Good

e0_pd_01 /dev/sdb HDD ONLINE Good

e0_pd_02 /dev/sdaa HDD ONLINE Good

e0_pd_03 /dev/sdab HDD ONLINE Good

e0_pd_04 /dev/sdac HDD ONLINE Good

e0_pd_05 /dev/sdad HDD ONLINE Good

   

-> Disk 인식 완료

   

ASM Disk 상태 확인

SQL> col GN format 99

SQL> col DN format 99

SQL> col NAME format a23

SQL>

SQL> SELECT

2 group_number GN,disk_number DN,name,state,mode_status,mount_status

3 FROM v$asm_disk

4 ORDER BY group_number, disk_number;

   

GN DN NAME STATE MODE_ST MOUNT_S

--- --- ----------------------- -------- ------- -------

1 0 HDD_E0_S17_1463722132P1 NORMAL ONLINE CACHED

1 1 HDD_E0_S05_1463725856P1 NORMAL ONLINE CACHED

1 2 HDD_E0_S12_1463726544P1 NORMAL ONLINE CACHED

1 3 HDD_E0_S10_1463726716P1 NORMAL ONLINE CACHED

1 4 HDD_E0_S16_1463726756P1 NORMAL ONLINE CACHED

1 5 HDD_E0_S08_1463754440P1 NORMAL ONLINE CACHED

1 6 HDD_E0_S15_1463852252P1 NORMAL ONLINE CACHED

1 7 HDD_E0_S19_1463885988P1 NORMAL ONLINE CACHED

1 8 HDD_E0_S01_1463911124P1 NORMAL ONLINE CACHED

1 9 HDD_E0_S07_1463917036P1 NORMAL ONLINE CACHED

1 10 HDD_E0_S02_1463917808P1 NORMAL ONLINE CACHED

   

GN DN NAME STATE MODE_ST MOUNT_S

--- --- ----------------------- -------- ------- -------

1 11 HDD_E0_S18_1463929152P1 NORMAL ONLINE CACHED

1 12 HDD_E0_S13_1463929180P1 NORMAL ONLINE CACHED

1 13 HDD_E0_S14_1463929248P1 NORMAL ONLINE CACHED

1 14 HDD_E0_S11_1463935128P1 NORMAL ONLINE CACHED

1 15 HDD_E0_S04_1463938000P1 NORMAL ONLINE CACHED

1 16 HDD_E0_S00_1463938068P1 NORMAL ONLINE CACHED

1 17 HDD_E0_S03_1463941008P1 NORMAL ONLINE CACHED

1 18 HDD_E0_S09_1463967484P1 NORMAL ONLINE CACHED

1 19 HDD_E0_S06_1463969744P1 NORMAL ONLINE CACHED

2 0 HDD_E0_S17_1463722132P2 NORMAL ONLINE CACHED

2 1 HDD_E0_S05_1463725856P2 NORMAL ONLINE CACHED

   

GN DN NAME STATE MODE_ST MOUNT_S

--- --- ----------------------- -------- ------- -------

2 2 HDD_E0_S12_1463726544P2 NORMAL ONLINE CACHED

2 3 HDD_E0_S10_1463726716P2 NORMAL ONLINE CACHED

2 4 HDD_E0_S16_1463726756P2 NORMAL ONLINE CACHED

2 5 HDD_E0_S08_1463754440P2 NORMAL ONLINE CACHED

2 6 HDD_E0_S15_1463852252P2 NORMAL ONLINE CACHED

2 7 HDD_E0_S19_1463885988P2 NORMAL ONLINE CACHED

2 8 HDD_E0_S01_1463911124P2 NORMAL ONLINE CACHED

2 9 HDD_E0_S07_1463917036P2 NORMAL ONLINE CACHED

2 10 HDD_E0_S02_1463917808P2 NORMAL ONLINE CACHED

2 11 HDD_E0_S18_1463929152P2 NORMAL ONLINE CACHED

2 12 HDD_E0_S13_1463929180P2 NORMAL ONLINE CACHED

   

GN DN NAME STATE MODE_ST MOUNT_S

--- --- ----------------------- -------- ------- -------

2 13 HDD_E0_S14_1463929248P2 NORMAL ONLINE CACHED

2 14 HDD_E0_S11_1463935128P2 NORMAL ONLINE CACHED

2 15 HDD_E0_S04_1463938000P2 NORMAL ONLINE CACHED

2 16 HDD_E0_S00_1463938068P2 NORMAL ONLINE CACHED

2 17 HDD_E0_S03_1463941008P2 NORMAL ONLINE CACHED

2 18 HDD_E0_S09_1463967484P2 NORMAL ONLINE CACHED

2 19 HDD_E0_S06_1463969744P2 NORMAL ONLINE CACHED

3 0 SSD_E0_S20_805971985P1 NORMAL ONLINE CACHED

3 1 SSD_E0_S21_805972008P1 NORMAL ONLINE CACHED

3 2 SSD_E0_S22_805972069P1 NORMAL ONLINE CACHED

3 3 SSD_E0_S23_805972099P1 NORMAL ONLINE CACHED

  

   

ISOF 상태 확인

  • 정상적으로 해당 프로세스들이 올라오는 것을 확인할 수 있다.

[root@base0 mapper]# lsof | grep HDD_E0_S04

asm_dbw0_ 38776 grid 280u BLK 252,38 0t0 5509236 /dev/mapper/HDD_E0_S04_1463938000p1

asm_rbal_ 38788 grid 256u BLK 252,39 0t0 5505972 /dev/mapper/HDD_E0_S04_1463938000p2

asm_rbal_ 38788 grid 257u BLK 252,38 0t0 5509236 /dev/mapper/HDD_E0_S04_1463938000p1

apx_rbal_ 40507 grid 256u BLK 252,39 0t0 5505972 /dev/mapper/HDD_E0_S04_1463938000p2

mdb_dbw0_ 41381 grid 275u BLK 252,38 0t0 5509236 /dev/mapper/HDD_E0_S04_1463938000p1

mdb_ckpt_ 41385 grid 259u BLK 252,38 0t0 5509236 /dev/mapper/HDD_E0_S04_1463938000p1

mdb_smon_ 41389 grid 258u BLK 252,38 0t0 5509236 /dev/mapper/HDD_E0_S04_1463938000p1

mdb_rbal_ 41397 grid 257u BLK 252,38 0t0 5509236 /dev/mapper/HDD_E0_S04_1463938000p1

oracle 41476 oracle 274u BLK 252,38 0t0 5509236 /dev/mapper/HDD_E0_S04_1463938000p1

oracle 41478 oracle 265u BLK 252,38 0t0 5509236 /dev/mapper/HDD_E0_S04_1463938000p1

oracle 41480 oracle 266u BLK 252,38 0t0 5509236 /dev/mapper/HDD_E0_S04_1463938000p1

oracle 41486 oracle 272u BLK 252,38 0t0 5509236 /dev/mapper/HDD_E0_S04_1463938000p1

oracle 41490 oracle 294u BLK 252,38 0t0 5509236 /dev/mapper/HDD_E0_S04_1463938000p1

oracle 41490 oracle 295u BLK 252,39 0t0 5505972 /dev/mapper/HDD_E0_S04_1463938000p2

oracle_41 41564 grid 257u BLK 252,39 0t0 5505972 /dev/mapper/HDD_E0_S04_1463938000p2

asm_x000_ 87135 grid 271u BLK 252,38 0t0 5509236 /dev/mapper/HDD_E0_S04_1463938000p1

   

4.Parameter File

/etc/multipath.conf -> oakcli 확인 명령어

/opt/oracle/extapi/asmappl.config -> SQL 확인 명령어

multipath -r; multipath -F; multipath -v2 

  • 2번의 명령어를 통한 작업이 정상적으로 수행되지 않을 경우에 해당 파라미터 파일을 조작하여 작업을 진행할 수도 있음.


블로그 이미지

운명을바꾸는자

IT와 함께 살아가는 삶

,