Corrupt Data Found During RMAN Backup Troubleshoot

Corrupt Data Found During RMAN Backup Troubleshoot

   

1. 이슈사항

• RMAN Full BACKUP을 수행 할 때, 특정 TBS에서 Block Corrupt 현상이 발견되는 상태
• 운영 시에는 Block Corrupt현상이 없으며 RMAN FULL BACKUP이 수행되는 새벽시간에만 해당 에러가 발생되고 있는 상태
• 현재 ASM Disk의 경우에 3중(HIGH)으로 Mirroring 되어 있는 상태

ORA-27061: waiting for async I/Os failed Linux-x86_64 Error: 5: Input/output error Additional information: -1 Additional information: 4194304 WARNING: Read Failed. group:1 disk:4 AU:626 offset:0 size:4194304 WARNING: failed to read mirror side 1 of virtual extent 1977 logical extent 0 of file 261 in group [1.103449180] from disk DATA_0004 allocation unit 626 reason error; if possible, will try another mirror side NOTE: successfully read mirror side 2 of virtual extent 1977 logical extent 1 of file 261 in group [1.103449180] from disk HDD_E0_S01_975274559P1 allocation unit 4776 Tue Apr 28 03:10:41 2015 Hex dump of (file 4, block 1028608) in trace file /u01/app/oracle/diag/rdbms/SID/SID1/trace/SID1_ora_21436.trc Corrupt block relative dba: 0x010fb200 (file 4, block 1028608) Completely zero block found during backing up datafile Trying mirror side HDD_E1_S06_1135637411P1. Reread of blocknum=1028608, file=+DATA/datafile/undotbs2.261.794895089. found same corrupt data Reread of blocknum=1028608, file=+DATA/datafile/undotbs2.261.794895089. found valid data Hex dump of (file 4, block 1028609) in trace file /u01/app/oracle/diag/rdbms/SID/SID1/trace/SID_ora_21436.trc Corrupt block relative dba: 0x010fb201 (file 4, block 1028609) Completely zero block found during backing up datafile …. .. Corrupt block relative dba: 0x00800201 (file 2, block 513) Bad header found during backing up datafile Data in bad block:  type: 1 format: 2 rdba: 0x00003102  last change scn: 0xb408.00000003 seq: 0x8 flg: 0x01  spare1: 0x8 spare2: 0x1 spare3: 0x0  consistency value in tail: 0x00000000  check value in block header: 0x9  block checksum disabled ksfdrfms:Mirror Read file=+DATA/wind/datafile/sysaux.257.797856311 fob=0x3cb15a548 bufp=0x7f1f74501000 blkno=513 nbytes=8192 ksfdrfms: Read success from mirror side=1 logical extent number=0 disk=DATA_0002 path=/dev/asmdisks/oracle_ocr02p1 Mirror I/O done from ASM disk /dev/asmdisks/oracle_ocr02p1  Trying mirror side DATA_0002. Reread of blocknum=513, file=+DATA/wind/datafile/sysaux.257.797856311. found same corrupt data ksfdrnms:Mirror Read file=+DATA/wind/datafile/sysaux.257.797856311 fob=0x3cb15a548 bufp=0x7f1f74501000 nbytes=8192 ksfdrnms: Read success from mirror side=2 logical extent number=1 disk=DATA_0003 path=/dev/asmdisks/oracle_recovery01p1 Mirror I/O done from ASM disk /dev/asmdisks/oracle_recovery01p1  Reread of blocknum=513, file=+DATA/wind/datafile/sysaux.257.797856311. found valid data

   

2. 원인 분석

• Corrupt Data Found During RMAN Backup (문서 ID 1610350.1) 오라클 공식 문서 참조

   

3. 분석 내용

• RMAN Full Backup이 수행 될 때에 Corrupt Data를 확인하여 Alert log에 보여주는 것으로 판단되며 현재 3중으로

  Mirroring되어 있어 운영에는 큰 영향이 없는 상태(정상적인 Block을 확인하여 백업 진행)

• 해당 되는 TBS를 확인하여 복구작업을 진행해야 함.

     

4. 해결

• 공식 문서에 수행해야 하는 절차가 상세하게 나와있다.
• Full Backup 후에 DB를 내리고 Mount 상태에서 복구를 진행하면 된다.

1) Take a full backup of the database and archivelogs 2) Shutdown the database and startup mount the database 3) Restore and recovery as follows:     RMAN > run { allocate channel c1 type disk; allocate channel c2 type disk; allocate channel c3 type disk; allocate channel c4 type disk; restore datafile < list all the affected files separated by commas >; recover database; sql 'alter database open'; }     Item 1 this will backup the database if the block is corrupt on the primary side RMAN will get the good block from the mirror. So once you have a good backup restoring the datafile will restore with good blocks.