Oracle Exadata TEST Single Database Installation : Install Grid Infrastructure

4. Install Grid Infrastructure

   

1) Clone the VM from Storage Cell

① Stocell1 복제 진행

   

② 이름 지정 & 모든 네트워크 카드의 Mac 초기화

   

③ 완전한 복제 진행

 

   

④ 네트워크 및 Disk 구성 수정

-stocell1에서 내부 네트워크 추가진행

   

-stocell1에서 Disk 제거(25GB 기본 공간을 제외한 500MB, 400MB Disk 연결 삭제 후 OS Disk 제거)

   

[OS start]

   

⑤ Cell Disk 제거

[root@stocell1 oracle]# rpm -evv cell-11.2.3.2.1_LINUX.X64_130109-1

[root@stocell1 ~]# rm -rf /opt/oracle/cell*

   

⑥ Network 제거 및 재구성

-eth0.bak, eth1.bak 제거 진행

   

   

virbr0 Link encap:Ethernet HWaddr 00:00:00:00:00:00 inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:19 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:3583 (3.4 KiB)     OVM으로 OS를 설치하면 기본적으로 할당되는 IP 대역을 사용하여 내부 네트워크 구성

   

[root@exadb1 ~]# cat /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost ::1 localhost6.localdomain6 localhost6     #InfiniBand IPs 192.168.56.102 stocell1 stocell1.exatest #192.168.10.21 stocell2 stocell2.exatest -- Cell Disk를 2개를 사용할 경우 192.168.122.10 exadb1-ib exadb1-ib.exatest #Public IPs     192.168.56.101 exadb1 exadb1.exatest 192.168.56.130 exadb1-vip exadb1-vip.exatest 192.168.56.100 exadb-cluster-scan exadb-cluster-scan.exatest

   

[root@stocell1 ~]# vi /etc/sysconfig/network NETWORKING=yes NETWORKING_IPV6=yes HOSTNAME=exadb1

   

2) Install Grid Infrastructure

   

① Grid 설치 파일 준비

-p13390677_112040_Linux-x86-64_3of7.zip

[/home/grid/setup] [root@exadb1 ~]# unzip p13390677_112040_Linux-x86-64_3of7.zip

   

② Cell disk user 제거

[root@exadb1 ~]# userdel celladmin [root@exadb1 ~]# userdel cellmonitor

   

③ OS 환경 변수 추가 수정

[root@exadb1 ~]# cat /etc/sysctl.conf     fs.aio-max-nr = 1048576 fs.file-max = 6815744 kernel.shmall = 2097152 kernel.shmmax = 1375207424 kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 net.ipv4.ip_local_port_range = 9000 65500         net.core.rmem_default=262144 net.core.rmem_max=4194304 net.core.wmem_default=262144 net.core.wmem_max=2097152     --해당 설정 값 중요.     [root@exadb1 ~]# sysctl -p

[root@exadb1 ~]# cat /etc/security/limits.conf grid soft nofile 131072 grid hard nofile 131072 grid soft nproc 131072 grid hard nproc 131072 grid soft core unlimited grid hard core unlimited grid soft memlock 50000000 grid hard memlock 50000000         oracle soft nproc 2047 oracle hard nproc 16384 oracle soft nofile 1024 oracle hard nofile 65536

   

④ Group 및 User 생성

[root@exadb1 ~]# groupadd oper [root@exadb1 ~]# groupadd dba [root@exadb1 ~]# groupadd oinstall [root@exadb1 ~]# groupadd asmadmin [root@exadb1 ~]# useradd -g oinstall -G dba,oper,asmadmin grid

   

⑤ Cell Disk와 DB Instance 간에 네트워크 설정

[root@exadb1 ~]# mkdir -p /etc/oracle/cell/network-config     [root@exadb1 ~]# vi /etc/oracle/cell/network-config/cellinit.ora ipaddress1 = 192.168.56.101/24 #InfiniBand IP/network of exadb1     [root@exadb1 ~]# vi /etc/oracle/cell/network-config/cellip.ora cell = "192.168.56.102" #InfiniBand IP of stocell1     #cell = "192.168.56.103"    #InfiniBand IP of stocell2 -- DISK 확장 가능     [권한 할당] [root@exadb1 ~]# chown -R grid:oinstall /etc/oracle/cell/network-config [root@exadb1 ~]# chmod -R 775 /etc/oracle/cell/network-config

   

⑥ NTP 설정

[root@exadb1 ~]#vi /etc/sysconfig/ntpd # Drop root to id 'ntp:ntp' by default. OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid" (-x 추가)     #Set to 'yes' to sync hw clock after successful ntpdate     SYNC_HWCLOCK=no NTPDATE_OPTIONS="-t 10″

chkconfig ntpd on service ntpd start

   

⑦ Grid Infrastructure 설치

-설치 진행 전 Disk 연결 상태 확인(설치파일 unzip 이후 사용가능)

[root@exadb1 bin]# export LD_LIBRARY_PATH=/home/grid/setup/grid/stage/ext/lib [root@exadb1 bin]# ./kfod disks=all op=disks (/tmp/OraInstall2015-08-17_04-34-05PM/ext/bin/kfod) ------------------------------------------------------------------------- ------- Disk Size Path User Group ===================================================================== 1: 448 Mb o/192.168.56.102/DATA_CD_DISK01_stocell1 <unknown> <unknown> 2: 448 Mb o/192.168.56.102/DATA_CD_DISK02_stocell1 <unknown> <unknown> 3: 448 Mb o/192.168.56.102/DATA_CD_DISK03_stocell1 <unknown> <unknown> 4: 448 Mb o/192.168.56.102/DATA_CD_DISK04_stocell1 <unknown> <unknown> 5: 448 Mb o/192.168.56.102/DATA_CD_DISK05_stocell1 <unknown> <unknown> 6: 448 Mb o/192.168.56.102/DATA_CD_DISK06_stocell1 <unknown> <unknown> 7: 448 Mb o/192.168.56.102/DATA_CD_DISK07_stocell1 <unknown> <unknown> 8: 448 Mb o/192.168.56.102/DATA_CD_DISK08_stocell1 <unknown> <unknown> 9: 448 Mb o/192.168.56.102/DATA_CD_DISK09_stocell1 <unknown> <unknown> 10: 448 Mb o/192.168.56.102/DATA_CD_DISK10_stocell1 <unknown> <unknown> 11: 448 Mb o/192.168.56.102/DATA_CD_DISK11_stocell1 <unknown> <unknown> 12: 448 Mb o/192.168.56.102/DATA_CD_DISK12_stocell1 <unknown> <unknown> [root@exadb1 bin]#

   

-Local에서 진행할 경우, xhost +를 활용하여 설치 진행

[root@stocell1 media]# xhost + [root@stocell1 media]# su – grid [grid@stocell1 grid]$ ./runInstaller

   

-xmanager를 통하여 진행할 경우, passive mode를 수행한 뒤에

[grid@exadb1 ~]$ export DISPLAY=192.168.56.1:0.0 [grid@stocell1 grid]$ ./runInstaller

   

   

   

   

-Install and Configure Oracle Grid Infrastucture for a Cluster 선택

   

   

-Advanced Installation 선택

   

   

- /etc/hosts 설정에서 scan ip 설정 중요

   

   

-ssh connectivity를 선택하여 Setup 진행(Grid, Oracle)

   

   

-Public과 Private의 IP 대역대가 달라야 한다.

   

   

-Oracle ASM 선택

   

   

-disk 그룹이 안보일 경우에 change discovery Path에서 o/*/* 로 검색 (AU Size 4MB, External로 선택)

1. Normal - 2-way mirroring, 2배의 디스크가 필요 (실제 데이터가 저장되는 공간이 100G면, 100G 디스크 2개가 필요)     2. High - 3-way mirroring, 3배의 디스크 필요     3. External - ASM mirroring 기능 사용하지 않음, Hardware RAID 기능으로 디스크를 보호하고 있을 경우에만 사용하길 권장

   

   

   

   

-선택 없이 Next 진행

   

   

   

   

-해당 Check 내역에 나오는 RPM 및 설정을 모두 설정해준다.

-단 Task resolv.conf lntegrity는 무시하고 진행해도 된다.

   

-root.sh까지 수행하면 Grid 구성은 완료된다.

[root@exadb1 ~]# /home/grid/app/oraInventory/orainstRoot.sh Changing permissions of /home/grid/app/oraInventory. Adding read,write permissions for group. Removing read,write,execute permissions for world.     Changing groupname of /home/grid/app/oraInventory to oinstall. The execution of the script is complete.     [root@exadb1 ~]# /home/grid/11.2.0/grid/root.sh Performing root user operation for Oracle 11g     The following environment variables are set as: ORACLE_OWNER= grid ORACLE_HOME= /home/grid/11.2.0/grid     Enter the full pathname of the local bin directory: [/usr/local/bin]: Copying dbhome to /usr/local/bin ... Copying oraenv to /usr/local/bin ... Copying coraenv to /usr/local/bin ...         Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Using configuration parameter file: /home/grid/11.2.0/grid/crs/install/crsconfig_params Creating trace directory User ignored Prerequisites during installation Installing Trace File Analyzer OLR initialization - successful root wallet root wallet cert root cert export peer wallet profile reader wallet pa wallet peer wallet keys pa wallet keys peer cert request pa cert request peer cert pa cert peer root cert TP profile reader root cert TP pa root cert TP peer pa cert TP pa peer cert TP profile reader pa cert TP profile reader peer cert TP peer user cert pa user cert Adding Clusterware entries to inittab CRS-2672: Attempting to start 'ora.mdnsd' on 'exadb1' CRS-2676: Start of 'ora.mdnsd' on 'exadb1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'exadb1' CRS-2676: Start of 'ora.gpnpd' on 'exadb1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'exadb1' CRS-2672: Attempting to start 'ora.gipcd' on 'exadb1' CRS-2676: Start of 'ora.gipcd' on 'exadb1' succeeded CRS-2676: Start of 'ora.cssdmonitor' on 'exadb1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'exadb1' CRS-2672: Attempting to start 'ora.diskmon' on 'exadb1' CRS-2676: Start of 'ora.diskmon' on 'exadb1' succeeded CRS-2676: Start of 'ora.cssd' on 'exadb1' succeeded     ASM created and started successfully.     Disk Group MGMT created successfully.     clscfg: -install mode specified Successfully accumulated necessary OCR keys. Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. CRS-4256: Updating the profile Successful addition of voting disk 2d929fc5b7f24fd6bfc418ae56169886. Successfully replaced voting disk group with +MGMT. CRS-4256: Updating the profile CRS-4266: Voting file(s) successfully replaced ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 2d929fc5b7f24fd6bfc418ae56169886 (o/192.168.56.102/DATA_CD_DISK01_stocell1) [MGMT] Located 1 voting disk(s). CRS-2672: Attempting to start 'ora.asm' on 'exadb1' CRS-2676: Start of 'ora.asm' on 'exadb1' succeeded CRS-2672: Attempting to start 'ora.MGMT.dg' on 'exadb1' CRS-2676: Start of 'ora.MGMT.dg' on 'exadb1' succeeded Configure Oracle Grid Infrastructure for a Cluster ... succeeded [root@exadb1 ~]#

   

   

-위 에러는 무시하고 진행을 완료 한다. (해당 에러에 대한 상세 로그를 확인해보면 좋다)

   

⑧ Grid 구성 확인

[Cell Disk alert log 확인] Tue Aug 18 20:31:42 2015 Heartbeat with diskmon started on exadb1 Tue Aug 18 20:31:49 2015 Resilvering incompatible ASM instance (exadb1, pid: 15062) connected. Any griddisks with resilvering tables will be dropped and recreated. Tue Aug 18 20:32:39 2015 Heartbeat with diskmon stopped on exadb1

   

[Grid 확인] [grid@exadb1 bin]$ ./crsctl stat res -t -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.LISTENER.lsnr ONLINE ONLINE exadb1 ora.MGMT.dg ONLINE ONLINE exadb1 ora.asm ONLINE ONLINE exadb1 Started ora.gsd OFFLINE OFFLINE exadb1 ora.net1.network ONLINE ONLINE exadb1 ora.ons ONLINE ONLINE exadb1 ora.registry.acfs ONLINE ONLINE exadb1 -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE exadb1 ora.cvu 1 ONLINE ONLINE exadb1 ora.exadb1.vip 1 ONLINE ONLINE exadb1 ora.oc4j 1 ONLINE ONLINE exadb1 ora.scan1.vip 1 ONLINE ONLINE exadb1 [grid@exadb1 bin]$

   

⑨ 구성 시 에러 발생 내역

2015-08-18 20:28:56.305: [/home/grid/11.2.0/grid/bin/oraagent.bin(13738)]CRS-5019:All OCR locations are on ASM disk groups [MGMT], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/home/grid/11.2.0/grid/log/exadb1/agent/ohasd/oraagent_grid/oraagent_grid.log". 2015-08-18 20:28:57.349: [/home/grid/11.2.0/grid/bin/oraagent.bin(13738)]CRS-5019:All OCR locations are on ASM disk groups [MGMT], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/home/grid/11.2.0/grid/log/exadb1/agent/ohasd/oraagent_grid/oraagent_grid.log".     --Root.sh를 수행할 때 CRS Alert log에서 위 에러가 계속해서 발생되면서 다음 Step으로 진행되지 않을 경우에 /etc/sysctl.con의 설정 값이 정확하게 되었는지 확인

   

[참고 문서 및 사이트]

• http://www.oracle.com/index.html
• http://oracletechdba.blogspot.in/2014/08/exadata-simulator-vm.html
• https://dbaesp.wordpress.com/2013/12/11/exadata-virtual-test-environment-for-oce-prep-storage-cell-node/
• http://uhesse.com/2011/01/19/exadata-part-i-smart-scan/
• http://www.oracle.com/technetwork/oem/exa-mgmt/em12c-exadata-discovery-cookbook-1662643.pdf
• http://digital.kyobobook.co.kr/digital/ebook/ebookDetail.ink?barcode=480140001714P
• http://pat98.tistory.com/category
• Exadata의 Base Document (MOS ID 888828.1]