Oracle Exadata TEST Single Database Installation : Storage Cell disk

3. Storage Cell disk

• Stocell1의 환경 구성을 기반으로 계속해서 진행한다. (CPU 2개, Memory 2Gb, Network 이더넷1,2 구성)

     

1) Install Oracle Linux 5

• Oracle Linux 5 설치 파일 (V40139-01.iso) 준비해야 하며 다른 버전으로 할 경우에 이슈가 있을 수 있어 되도록이면 Linux 5버전을 추천한다.
• 게스트 확장 설치를 할 경우에 진행에 있어 설치 파일 이관 및 공유폴더 설정 등 편리한 점이 많다.(하지만 RHEL5의 경우에게스트 확장설치를 할 경우에 OS가 깨지는 현상이 있어 게스트 확장 설치를 하지 않고 진행해도 이상 없다.)

  

     

[root@stocell1 media]# cd VBOXADDITIONS_4.3.30_101610/ /media/VBOXADDITIONS_4.3.30_101610 [root@stocell1 VBOXADDITIONS_4.3.30_101610]# ls 32Bit autorun.sh runasroot.sh VBoxWindowsAdditions-amd64.exe 64Bit cert VBoxLinuxAdditions.run VBoxWindowsAdditions.exe AUTORUN.INF OS2 VBoxSolarisAdditions.pkg VBoxWindowsAdditions-x86.exe [root@stocell1 VBOXADDITIONS_4.3.30_101610]# [root@stocell1 VBOXADDITIONS_4.3.30_101610]# ./VBoxLinuxAdditions.run

   

① OVM 새로 만들기

-OVM 이름 & Oracle Linux 64bit로 설치 진행

   

-파일 설치 위치 및 파일 크기 지정(25GB를 권장, 그 이상으로 할당하면 된다.)

-OVM 만들기 선택

   

② ISO 설치 파일 추가

   

③ Oracle Linux 5 설치

-따로 설정할 것은 없고 아래 세부 설정에서만 선택하면 된다.

-Next를 계속 진행해서 설치를 진행하면 된다.

   

-Software Development

-Virtualization

-Clustering

-Storage Clustering

   

   

-설치가 완료 되면 추가 설정을 진행하게 되는데 여기에서 바오하벽과 SELinux를 모두 비활성화로 진행한다.

   

-설치 완료

   

④ Network 설정 확인

-DHCP로 자동 할당 받아서 사용

-무선 와이파이로 인터넷을 사용.(192.168.0.* 대역)

   

   

-eth1의 경우에는 VirtualBox Host-Only Ethernet Adapter를 통해서 IP를 할당 받음

-192.168.56.* 대역을 사용

   

   

[root@stocell1 ~]# cat /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 stocell1 localhost.localdomain localhost ::1 localhost6.localdomain6 localhost6

[root@stocell1 VBOXADDITIONS_4.3.30_101610]# ifconfig -a eth0 Link encap:Ethernet HWaddr 08:00:27:4A:70:23 inet addr:192.168.0.14 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fe4a:7023/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:20133 errors:0 dropped:0 overruns:0 frame:0 TX packets:37 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1263813 (1.2 MiB) TX bytes:10165 (9.9 KiB)     eth1 Link encap:Ethernet HWaddr 08:00:27:C2:28:D1 inet addr:192.168.56.102 Bcast:192.168.56.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fec2:28d1/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:338 errors:0 dropped:0 overruns:0 frame:0 TX packets:275 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:35142 (34.3 KiB) TX bytes:37744 (36.8 KiB)     lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:9865 errors:0 dropped:0 overruns:0 frame:0 TX packets:9865 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:8315266 (7.9 MiB) TX bytes:8315266 (7.9 MiB)

   

   

⑤ 불필요한 기능을 Stop

-해당 내역 때문에 OVM이 느려지는 현상이 발생될 수 있으므로 정지 시켜둔다.

[root@stocell1 ~]# service yum-updatesd stop [root@stocell1 ~]# chkconfig yum-updatesd off

   

2) Install Storage cell S/W

-설치 파일을 전송할 때, 공유폴더에 넣어서도 가능하고 FTP로 전송해서 진행해도 된다.

   

① 설치 파일 전송

-Oracle Database Machine Exadata Storage Cell 설치 파일 (V36290-01.zip) 설치 파일을 OVM에 넣어준다.

   

② 알집 풀기

[root@stocell1 ~]# unzip V36290-01.zip

[root@stocell1 ~]# cd dl180/boot/cellbits/

-rwxrwx— 1 root root 245231205 Jan  9  2013 cell.bin 해당 파일을 Copy 해서 다른 공간에 unzip을 해도 된다.     [root@stocell1 ~]# unzip cell.bin Archive:  cell.bin warning [cell.bin]:  6408 extra bytes at beginning or within zipfile (attempting to process anyway) inflating: cell-11.2.3.2.1_LINUX.X64_130109-1.x86_64.rpm inflating: jdk-1_5_0_15-linux-amd64.rpm Install jdk [root@stocell1 ~]# rpm -ivh jdk-1_5_0_15-linux-amd64.rpm Preparing…                ########################################### [100%] 1:jdk                    ########################################### [100%] Preparing to install cell rpm (thanks to Lee ..)     [권한 설정] [root@stocell1 ~]# mkdir /var/log/oracle [root@stocell1 ~]# chmod 775 /var/log/oracle (It will be used also by celladmin user …) Install cell sw [root@stocell1 ~]# rpm -ivh cell-11.2.3.2.1_LINUX.X64_130109-1.x86_64.rpm Preparing…                ########################################### [100%] Pre Installation steps in progress … 1:cell                   ########################################### [100%] Post Installation steps in progress … Set cellusers group for /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/cellsrv/deploy/log directory Set 775 permissions for /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/cellsrv/deploy/log directory / /     Installation SUCCESSFUL.     -RPM 설치 시에 사전에 필요한 RPM이 있을 수 있어 필요한 RPM은 yum으로 설치하고 진행 하면 된다. -RPM 2개를 설치하면 끝난다.

   

   

3) Add Virtual Disks

① Disk 추가(새 디스크 만들기)

   

   

   

   

-500MB 12개 추가(stocell_disk1~12), 400MB 6개 추가(fd_stocell_disk1~6) 진행

   

4) Start Storage cell S/W

-Disk 추가 이후에 Stocell1 OVM Start 진행

-로그인시에 아래와 같은 에러가 발생하면 export DISPLAY=:0 를 추가해 재 로그인 하면 된다.

[root@stocell1]# vi /etc/bashrc ….. export DISPLAY=:0     -해당 내역을 추가하고 다시 로그인을 진행하면 된다.

   

① OS 환경 변수 추가

/etc/sysctl.ctl     fs.file-max = 65536

/etc/security/limit.conf      * soft nofile 65536 * hard nofile 65536

   

[root@stocell1 ~]# modprobe rds [root@stocell1 ~]# modprobe rds_tcp [root@stocell1 ~]# modprobe rds_rdma

[root@stocell1 ~]# vi /etc/modprobe.d/rds.conf install rds /sbin/modprobe --ignore-install rds && /sbin/modprobe rds_tcp && /sbin/modprobe rds_rdma [해당 내역을 삽입]

[설정 확인] [root@stocell1 ~]# lsmod | grep rds rds_rdma 106561 0 rds_tcp 48097 0 rds 155561 2 rds_rdma,rds_tcp rdma_cm 73429 2 rds_rdma,ib_iser ib_core 108097 7 rds_rdma,ib_iser,rdma_cm,ib_cm,iw_cm,ib_sa,ib_mad

   

② Cell disk 추가 및 설정

-500MB 12개, 400MB 6개 Disk 내역 확인 [root@stocell1 ~]# fdisk -l 2>/dev/null |grep "B," Disk /dev/sda: 26.8 GB, 26843545600 bytes Disk /dev/sdb: 524 MB, 524288000 bytes Disk /dev/sdc: 524 MB, 524288000 bytes Disk /dev/sdd: 524 MB, 524288000 bytes Disk /dev/sde: 524 MB, 524288000 bytes Disk /dev/sdf: 524 MB, 524288000 bytes Disk /dev/sdg: 524 MB, 524288000 bytes Disk /dev/sdh: 524 MB, 524288000 bytes Disk /dev/sdi: 524 MB, 524288000 bytes Disk /dev/sdj: 524 MB, 524288000 bytes Disk /dev/sdk: 524 MB, 524288000 bytes Disk /dev/sdl: 524 MB, 524288000 bytes Disk /dev/sdm: 524 MB, 524288000 bytes Disk /dev/sdn: 419 MB, 419430400 bytes Disk /dev/sdo: 419 MB, 419430400 bytes Disk /dev/sdp: 419 MB, 419430400 bytes Disk /dev/sdq: 419 MB, 419430400 bytes Disk /dev/sdr: 419 MB, 419430400 bytes Disk /dev/sds: 419 MB, 419430400 bytes Disk /dev/dm-0: 22.5 GB, 22515023872 bytes Disk /dev/dm-1: 4194 MB, 4194304000 bytes (500MB data, 400MB flash)

[root@stocell1 unix]# echo $T_WORK /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks

cd /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks --디렉토리가 없을 경우 생성할 것(mkdir disks/raw)

[할당된 disk 설정] cd /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw     ln -s /dev/sdb stocell1_DISK01 ln -s /dev/sdc stocell1_DISK02 ln -s /dev/sdd stocell1_DISK03 ln -s /dev/sde stocell1_DISK04 ln -s /dev/sdf stocell1_DISK05 ln -s /dev/sdg stocell1_DISK06 ln -s /dev/sdh stocell1_DISK07 ln -s /dev/sdi stocell1_DISK08 ln -s /dev/sdj stocell1_DISK09 ln -s /dev/sdk stocell1_DISK10 ln -s /dev/sdl stocell1_DISK11 ln -s /dev/sdm stocell1_DISK12 ln -s /dev/sdn stocell1_FLASH01 ln -s /dev/sdo stocell1_FLASH02 ln -s /dev/sdp stocell1_FLASH03 ln -s /dev/sdq stocell1_FLASH04 ln -s /dev/sdr stocell1_FLASH05 ln -s /dev/sds stocell1_FLASH06

   

③ Cell Disk Service 설정

[root@stocell1 ~]# su – celladmin

[celladmin@stocell1 ~]$ cellcli -e alter cell restart services all     Stopping the RS, CELLSRV, and MS services… CELL-01509: Restart Server (RS) not responding. Starting the RS, CELLSRV, and MS services… Getting the state of RS services… running Starting CELLSRV services… The STARTUP of CELLSRV services was not successful. CELL-01547: CELLSRV startup failed due to unknown reasons. Starting MS services… The STARTUP of MS services was successful. The error in not unknown (as stated) but well known and expected

[InfiniBand Storage Network 설정] [celladmin@stocell1 ~]$ cellcli -e create cell stocell1 interconnect1=eth1 Cell stocell1 successfully created Starting CELLSRV services… The STARTUP of CELLSRV services was successful. Flash cell disks, FlashCache, and FlashLog will be created… CellDisk FD_00_stocell1 successfully created CellDisk FD_01_stocell1 successfully created CellDisk FD_02_stocell1 successfully created CellDisk FD_03_stocell1 successfully created CellDisk FD_04_stocell1 successfully created CellDisk FD_05_stocell1 successfully created Flash log stocell1_FLASHLOG successfully created Flash cache stocell1_FLASHCACHE successfully created

[Cell Disks 설정] [celladmin@stocell1 ~]$ cellcli -e create celldisk all CellDisk CD_DISK01_stocell1 successfully created CellDisk CD_DISK02_stocell1 successfully created CellDisk CD_DISK03_stocell1 successfully created CellDisk CD_DISK04_stocell1 successfully created CellDisk CD_DISK05_stocell1 successfully created CellDisk CD_DISK06_stocell1 successfully created CellDisk CD_DISK07_stocell1 successfully created CellDisk CD_DISK08_stocell1 successfully created CellDisk CD_DISK09_stocell1 successfully created CellDisk CD_DISK10_stocell1 successfully created CellDisk CD_DISK11_stocell1 successfully created CellDisk CD_DISK12_stocell1 successfully created

[Grid disks 설정] [celladmin@stocell1 ~]$ cellcli -e create griddisk all harddisk prefix=DATA GridDisk DATA_CD_DISK01_stocell1 successfully created GridDisk DATA_CD_DISK02_stocell1 successfully created GridDisk DATA_CD_DISK03_stocell1 successfully created GridDisk DATA_CD_DISK04_stocell1 successfully created GridDisk DATA_CD_DISK05_stocell1 successfully created GridDisk DATA_CD_DISK06_stocell1 successfully created GridDisk DATA_CD_DISK07_stocell1 successfully created GridDisk DATA_CD_DISK08_stocell1 successfully created GridDisk DATA_CD_DISK09_stocell1 successfully created GridDisk DATA_CD_DISK10_stocell1 successfully created GridDisk DATA_CD_DISK11_stocell1 successfully created GridDisk DATA_CD_DISK12_stocell1 successfully created

[celladmin@stocell1 ~]$ cellcli -e alter cell restart services all     Stopping the RS, CELLSRV, and MS services... The SHUTDOWN of services was successful. Starting the RS, CELLSRV, and MS services... Getting the state of RS services... running Starting CELLSRV services... The STARTUP of CELLSRV services was successful. Starting MS services... The STARTUP of MS services was successful.

   

[cell disk alert log] [root@stocell1 trace]# cd /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/log/diag/asm/cell/stocell1/trace [root@stocell1 trace]# ls -al alert* -rw-rw---- 1 root celladmin 156850 Aug 21 10:00 alert.log .. .. [RS] Started Service MS with pid 5390 Fri Aug 21 10:00:51 2015 Smart Flash Logging enabled on FlashLog stocell1_FLASHLOG (1842424628), size=80MB, cdisk=FD_05_stocell1 Fri Aug 21 10:00:51 2015 Smart Flash Logging enabled on FlashLog stocell1_FLASHLOG (3996181628), size=80MB, cdisk=FD_04_stocell1 Fri Aug 21 10:00:51 2015 Smart Flash Logging enabled on FlashLog stocell1_FLASHLOG (3466652180), size=80MB, cdisk=FD_03_stocell1 Fri Aug 21 10:00:51 2015 Smart Flash Logging enabled on FlashLog stocell1_FLASHLOG (1448144612), size=80MB, cdisk=FD_02_stocell1 Fri Aug 21 10:00:51 2015 Smart Flash Logging enabled on FlashLog stocell1_FLASHLOG (2992946252), size=80MB, cdisk=FD_00_stocell1 Smart Flash Logging enabled on FlashLog stocell1_FLASHLOG (1465279500), size=80MB, cdisk=FD_01_stocell1 FlashLog: initialization complete Fri Aug 21 10:00:51 2015 CELLSRV Server startup complete [RS] Started Service CELLSRV with pid 5397 Fri Aug 21 10:00:57 2015 Info: Assigning flash CD FD_00_stocell1 to group#1 Info: Assigning flash CD FD_01_stocell1 to group#2 Fri Aug 21 10:00:57 2015 Info: Assigning flash CD FD_02_stocell1 to group#3 Info: Assigning flash CD FD_03_stocell1 to group#0 Info: Assigning flash CD FD_04_stocell1 to group#1 Info: Assigning flash CD FD_05_stocell1 to group#2

   

-stocell1 OVM이 완료된 상태로 해당 내역을 복제하여 exadata 구성을 진행한다.

-해당 stocell1을 바탕으로 DB 1개+Cell Disk 2개 구성, DB 2개+Cell Disk1개 구성(RAC)등 다양하게 활용할 수 있으므로

OVM export(내보내기)로 백업 받아 두고 사용하면 좋다.

   

④ Cellcli 명렁어

cellcli 명령어

1. check cell status cellcli -e list cell cellcli -e list cell detail     2. Physical Disk Information cellcli -e list physicaldisk cellcli -e list physicaldisk E15SBS detail     3. LUN Disks Detail cellcli -e list lun cellcli -e list lun 0_0 detail  cellcli -e list lun 5_3 detail     4. Cell Disks Report cellcli -e list celldisk cellcli -e list celldisk CD_08_exadatalcel10 detail cellcli -e list celldisk attributes name, devicePartition where size>200g; cellcli -e list celldisk attributes name,status,size     5. Grid Disk Knowledge cellcli -e list griddisk cellcli -e list celldisk CD_02_exadatalcel10 detail cellcli -e list griddisk DATA_DMORL_CD_02_exadatalcel10 detail cellcli -e list griddisk DBFS_DG_CD_02_exadatalcel10 detail cellcli -e list griddisk RECO_DMORL_CD_02_exadatalcel10 detail     6. Display Exadate Alerts cellcli -e list alerthistory cellcli -e list alerthistory 8_1 detail cellcli -e  list alerthistory where severity like '[warning|critical]' cellcli -e  list alertdefinition detail     7. Restart Cell Services cellcli -e list cell detail cellcli -e alter cell restart services rs cellcli -e alter cell restart services ms cellcli -e alter cell restart services cellsrv cellcli -e alter cell restart services all cellcli -e alter cell shutdown services rs cellcli -e alter cell shutdown services ms cellcli -e alter cell shutdown services cellsrv

   

⑤ 구성 시에 에러(CELL-01518)

/opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/deploy/config/cellinit.ora     [celladmin@cell ~]$ cellcli -e create cell cell1 interconnect1=eth1 CELL-01518: Stop CELLSRV. Create Cell cannot continue with CELLSRV running.     [celladmin@cell ~]$ cellcli CellCLI: Release 11.2.3.2.0 - Production on Tue Jan 20 22:14:40 CST 2015 Copyright (c) 2007, 2012, Oracle.  All rights reserved. Cell Efficiency Ratio: 1     CellCLI> alter cell shutdown services cellsrv     Stopping CELLSRV services...  The SHUTDOWN of CELLSRV services was successful.     CellCLI> exit quitting     [celladmin@cell ~]$ cellcli -e create cell cell1 interconnect1=eth1 Cell cell1 successfully created

   

   

[참고 문서 및 사이트]

• http://www.oracle.com/index.html
• http://oracletechdba.blogspot.in/2014/08/exadata-simulator-vm.html
• https://dbaesp.wordpress.com/2013/12/11/exadata-virtual-test-environment-for-oce-prep-storage-cell-node/
• http://uhesse.com/2011/01/19/exadata-part-i-smart-scan/
• http://www.oracle.com/technetwork/oem/exa-mgmt/em12c-exadata-discovery-cookbook-1662643.pdf
• http://digital.kyobobook.co.kr/digital/ebook/ebookDetail.ink?barcode=480140001714P
• http://pat98.tistory.com/category
• http://pat98.tistory.com/m/post/732
• Exadata의 Base Document (MOS ID 888828.1]