Corrupt Data Found During RMAN Backup Troubleshoot

   

1. 이슈사항

  • RMAN Full BACKUP을 수행 할 때, 특정 TBS에서 Block Corrupt 현상이 발견되는 상태
  • 운영 시에는 Block Corrupt현상이 없으며 RMAN FULL BACKUP이 수행되는 새벽시간에만 해당 에러가 발생되고 있는 상태
  • 현재 ASM Disk의 경우에 3중(HIGH)으로 Mirroring 되어 있는 상태

ORA-27061: waiting for async I/Os failed

Linux-x86_64 Error: 5: Input/output error

Additional information: -1

Additional information: 4194304

WARNING: Read Failed. group:1 disk:4 AU:626 offset:0 size:4194304

WARNING: failed to read mirror side 1 of virtual extent 1977 logical extent 0 of file 261 in group [1.103449180] from disk DATA_0004 allocation unit 626 reason error; if possible, will try another mirror side

NOTE: successfully read mirror side 2 of virtual extent 1977 logical extent 1 of file 261 in group [1.103449180] from disk HDD_E0_S01_975274559P1 allocation unit 4776

Tue Apr 28 03:10:41 2015

Hex dump of (file 4, block 1028608) in trace file /u01/app/oracle/diag/rdbms/SID/SID1/trace/SID1_ora_21436.trc

Corrupt block relative dba: 0x010fb200 (file 4, block 1028608)

Completely zero block found during backing up datafile

Trying mirror side HDD_E1_S06_1135637411P1.

Reread of blocknum=1028608, file=+DATA/datafile/undotbs2.261.794895089. found same corrupt data

Reread of blocknum=1028608, file=+DATA/datafile/undotbs2.261.794895089. found valid data

Hex dump of (file 4, block 1028609) in trace file /u01/app/oracle/diag/rdbms/SID/SID1/trace/SID_ora_21436.trc

Corrupt block relative dba: 0x010fb201 (file 4, block 1028609)

Completely zero block found during backing up datafile

….

..

Corrupt block relative dba: 0x00800201 (file 2, block 513)

Bad header found during backing up datafile

Data in bad block:

 type: 1 format: 2 rdba: 0x00003102

 last change scn: 0xb408.00000003 seq: 0x8 flg: 0x01

 spare1: 0x8 spare2: 0x1 spare3: 0x0

 consistency value in tail: 0x00000000

 check value in block header: 0x9

 block checksum disabled

ksfdrfms:Mirror Read file=+DATA/wind/datafile/sysaux.257.797856311 fob=0x3cb15a548 bufp=0x7f1f74501000 blkno=513 nbytes=8192

ksfdrfms: Read success from mirror side=1 logical extent number=0 disk=DATA_0002 path=/dev/asmdisks/oracle_ocr02p1

Mirror I/O done from ASM disk /dev/asmdisks/oracle_ocr02p1 

Trying mirror side DATA_0002.

Reread of blocknum=513, file=+DATA/wind/datafile/sysaux.257.797856311. found same corrupt data

ksfdrnms:Mirror Read file=+DATA/wind/datafile/sysaux.257.797856311 fob=0x3cb15a548 bufp=0x7f1f74501000 nbytes=8192

ksfdrnms: Read success from mirror side=2 logical extent number=1 disk=DATA_0003 path=/dev/asmdisks/oracle_recovery01p1

Mirror I/O done from ASM disk /dev/asmdisks/oracle_recovery01p1 

Reread of blocknum=513, file=+DATA/wind/datafile/sysaux.257.797856311. found valid data

   

2. 원인 분석

  • Corrupt Data Found During RMAN Backup (문서 ID 1610350.1) 오라클 공식 문서 참조

     


3. 분석 내용

  • RMAN Full Backup이 수행 될 때에 Corrupt Data를 확인하여 Alert log에 보여주는 것으로 판단되며 현재 3중으로

    Mirroring되어 있어 운영에는 큰 영향이 없는 상태(정상적인 Block을 확인하여 백업 진행)

  • 해당 되는 TBS를 확인하여 복구작업을 진행해야 함.

       

4. 해결

  • 공식 문서에 수행해야 하는 절차가 상세하게 나와있다.
  • Full Backup 후에 DB를 내리고 Mount 상태에서 복구를 진행하면 된다.

1) Take a full backup of the database and archivelogs

2) Shutdown the database and startup mount the database

3) Restore and recovery as follows:

   

RMAN > run

{

allocate channel c1 type disk;

allocate channel c2 type disk;

allocate channel c3 type disk;

allocate channel c4 type disk;

restore datafile < list all the affected files separated by commas >;

recover database;

sql 'alter database open';

}

   

Item 1 this will backup the database if the block is corrupt on the primary side RMAN will get the good block from the mirror. So once you have a good backup restoring the datafile will restore with good blocks. 

 


'1. IT Story > DB' 카테고리의 다른 글

Oracle Backup & Recovery의 종류  (0) 2015.09.15
DBMS_CRYPTO Package를 이용한 단방향 암호화  (0) 2015.09.08
Fine Grained Auditing의 활용  (0) 2015.09.08
WRAP Utility 활용  (0) 2015.09.08
DATABASE 암호화 기술들  (0) 2015.09.04
Oracle oradebug활용 Troubleshoot  (0) 2015.09.04
Oracle RAC 11gR2 주요 Command(명령어)  (0) 2015.09.04
Oracle Flashback  (0) 2015.09.04
블로그 이미지

운명을바꾸는자

IT와 함께 살아가는 삶

,