'엑사테이터 설치'에 해당되는 글 1건

2015.08.21 Oracle Exadata TEST Single Database Installation : Storage Cell disk 3

Oracle Exadata TEST Single Database Installation : Storage Cell disk

1. IT Story/DB 2015. 8. 21. 11:37

3. Storage Cell disk

Stocell1의 환경 구성을 기반으로 계속해서 진행한다. (CPU 2개, Memory 2Gb, Network 이더넷1,2 구성)

1) Install Oracle Linux 5

Oracle Linux 5 설치 파일 (V40139-01.iso) 준비해야 하며 다른 버전으로 할 경우에 이슈가 있을 수 있어 되도록이면 Linux 5버전을 추천한다.
게스트 확장 설치를 할 경우에 진행에 있어 설치 파일 이관 및 공유폴더 설정 등 편리한 점이 많다.(하지만 RHEL5의 경우에게스트 확장설치를 할 경우에 OS가 깨지는 현상이 있어 게스트 확장 설치를 하지 않고 진행해도 이상 없다.)

[root@stocell1 media]# cd VBOXADDITIONS_4.3.30_101610/

/media/VBOXADDITIONS_4.3.30_101610

[root@stocell1 VBOXADDITIONS_4.3.30_101610]# ls

32Bit autorun.sh runasroot.sh VBoxWindowsAdditions-amd64.exe

64Bit cert VBoxLinuxAdditions.run VBoxWindowsAdditions.exe

AUTORUN.INF OS2 VBoxSolarisAdditions.pkg VBoxWindowsAdditions-x86.exe

[root@stocell1 VBOXADDITIONS_4.3.30_101610]#

[root@stocell1 VBOXADDITIONS_4.3.30_101610]# ./VBoxLinuxAdditions.run

① OVM 새로 만들기

-OVM 이름 & Oracle Linux 64bit로 설치 진행

-파일 설치 위치 및 파일 크기 지정(25GB를 권장, 그 이상으로 할당하면 된다.)

-OVM 만들기 선택

② ISO 설치 파일 추가

③ Oracle Linux 5 설치

-따로 설정할 것은 없고 아래 세부 설정에서만 선택하면 된다.

-Next를 계속 진행해서 설치를 진행하면 된다.

-Software Development

-Virtualization

-Clustering

-Storage Clustering

-설치가 완료 되면 추가 설정을 진행하게 되는데 여기에서 바오하벽과 SELinux를 모두 비활성화로 진행한다.

-설치 완료

④ Network 설정 확인

-DHCP로 자동 할당 받아서 사용

-무선 와이파이로 인터넷을 사용.(192.168.0.* 대역)

-eth1의 경우에는 VirtualBox Host-Only Ethernet Adapter를 통해서 IP를 할당 받음

-192.168.56.* 대역을 사용

[root@stocell1 ~]# cat /etc/hosts

# Do not remove the following line, or various programs

# that require network functionality will fail.

127.0.0.1 stocell1 localhost.localdomain localhost

::1 localhost6.localdomain6 localhost6

[root@stocell1 VBOXADDITIONS_4.3.30_101610]# ifconfig -a

eth0 Link encap:Ethernet HWaddr 08:00:27:4A:70:23

inet addr:192.168.0.14 Bcast:192.168.0.255 Mask:255.255.255.0

inet6 addr: fe80::a00:27ff:fe4a:7023/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:20133 errors:0 dropped:0 overruns:0 frame:0

TX packets:37 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:1263813 (1.2 MiB) TX bytes:10165 (9.9 KiB)

eth1 Link encap:Ethernet HWaddr 08:00:27:C2:28:D1

inet addr:192.168.56.102 Bcast:192.168.56.255 Mask:255.255.255.0

inet6 addr: fe80::a00:27ff:fec2:28d1/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:338 errors:0 dropped:0 overruns:0 frame:0

TX packets:275 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:35142 (34.3 KiB) TX bytes:37744 (36.8 KiB)

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

inet6 addr: ::1/128 Scope:Host

UP LOOPBACK RUNNING MTU:16436 Metric:1

RX packets:9865 errors:0 dropped:0 overruns:0 frame:0

TX packets:9865 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:8315266 (7.9 MiB) TX bytes:8315266 (7.9 MiB)

⑤ 불필요한 기능을 Stop

-해당 내역 때문에 OVM이 느려지는 현상이 발생될 수 있으므로 정지 시켜둔다.

[root@stocell1 ~]# service yum-updatesd stop

[root@stocell1 ~]# chkconfig yum-updatesd off

2) Install Storage cell S/W

-설치 파일을 전송할 때, 공유폴더에 넣어서도 가능하고 FTP로 전송해서 진행해도 된다.

① 설치 파일 전송

-Oracle Database Machine Exadata Storage Cell 설치 파일 (V36290-01.zip) 설치 파일을 OVM에 넣어준다.

② 알집 풀기

[root@stocell1 ~]# unzip V36290-01.zip

[root@stocell1 ~]# cd dl180/boot/cellbits/

-rwxrwx— 1 root root 245231205 Jan 9 2013 cell.bin

해당 파일을 Copy 해서 다른 공간에 unzip을 해도 된다.

[root@stocell1 ~]# unzip cell.bin

Archive: cell.bin

warning [cell.bin]: 6408 extra bytes at beginning or within zipfile

(attempting to process anyway)

inflating: cell-11.2.3.2.1_LINUX.X64_130109-1.x86_64.rpm

inflating: jdk-1_5_0_15-linux-amd64.rpm

Install jdk

[root@stocell1 ~]# rpm -ivh jdk-1_5_0_15-linux-amd64.rpm

Preparing… ########################################### [100%]

1:jdk ########################################### [100%]

Preparing to install cell rpm (thanks to Lee ..)

[권한 설정]

[root@stocell1 ~]# mkdir /var/log/oracle

[root@stocell1 ~]# chmod 775 /var/log/oracle

(It will be used also by celladmin user …)

Install cell sw

[root@stocell1 ~]# rpm -ivh cell-11.2.3.2.1_LINUX.X64_130109-1.x86_64.rpm

Preparing… ########################################### [100%]

Pre Installation steps in progress …

1:cell ########################################### [100%]

Post Installation steps in progress …

Set cellusers group for /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/cellsrv/deploy/log directory

Set 775 permissions for /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/cellsrv/deploy/log directory

Installation SUCCESSFUL.

-RPM 설치 시에 사전에 필요한 RPM이 있을 수 있어 필요한 RPM은 yum으로 설치하고 진행 하면 된다.

-RPM 2개를 설치하면 끝난다.

3) Add Virtual Disks

① Disk 추가(새 디스크 만들기)

-500MB 12개 추가(stocell_disk1~12), 400MB 6개 추가(fd_stocell_disk1~6) 진행

4) Start Storage cell S/W

-Disk 추가 이후에 Stocell1 OVM Start 진행

-로그인시에 아래와 같은 에러가 발생하면 export DISPLAY=:0 를 추가해 재 로그인 하면 된다.

[root@stocell1]# vi /etc/bashrc

…..

export DISPLAY=:0

-해당 내역을 추가하고 다시 로그인을 진행하면 된다.

① OS 환경 변수 추가

/etc/sysctl.ctl

fs.file-max = 65536

/etc/security/limit.conf

* soft nofile 65536

* hard nofile 65536

[root@stocell1 ~]# modprobe rds

[root@stocell1 ~]# modprobe rds_tcp

[root@stocell1 ~]# modprobe rds_rdma

[root@stocell1 ~]# vi /etc/modprobe.d/rds.conf

install rds /sbin/modprobe --ignore-install rds && /sbin/modprobe rds_tcp && /sbin/modprobe rds_rdma

[해당 내역을 삽입]

[설정 확인]

[root@stocell1 ~]# lsmod | grep rds

rds_rdma 106561 0

rds_tcp 48097 0

rds 155561 2 rds_rdma,rds_tcp

rdma_cm 73429 2 rds_rdma,ib_iser

ib_core 108097 7 rds_rdma,ib_iser,rdma_cm,ib_cm,iw_cm,ib_sa,ib_mad

② Cell disk 추가 및 설정

-500MB 12개, 400MB 6개 Disk 내역 확인

[root@stocell1 ~]# fdisk -l 2>/dev/null |grep "B,"

Disk /dev/sda: 26.8 GB, 26843545600 bytes

Disk /dev/sdb: 524 MB, 524288000 bytes

Disk /dev/sdc: 524 MB, 524288000 bytes

Disk /dev/sdd: 524 MB, 524288000 bytes

Disk /dev/sde: 524 MB, 524288000 bytes

Disk /dev/sdf: 524 MB, 524288000 bytes

Disk /dev/sdg: 524 MB, 524288000 bytes

Disk /dev/sdh: 524 MB, 524288000 bytes

Disk /dev/sdi: 524 MB, 524288000 bytes

Disk /dev/sdj: 524 MB, 524288000 bytes

Disk /dev/sdk: 524 MB, 524288000 bytes

Disk /dev/sdl: 524 MB, 524288000 bytes

Disk /dev/sdm: 524 MB, 524288000 bytes

Disk /dev/sdn: 419 MB, 419430400 bytes

Disk /dev/sdo: 419 MB, 419430400 bytes

Disk /dev/sdp: 419 MB, 419430400 bytes

Disk /dev/sdq: 419 MB, 419430400 bytes

Disk /dev/sdr: 419 MB, 419430400 bytes

Disk /dev/sds: 419 MB, 419430400 bytes

Disk /dev/dm-0: 22.5 GB, 22515023872 bytes

Disk /dev/dm-1: 4194 MB, 4194304000 bytes

(500MB data, 400MB flash)

[root@stocell1 unix]# echo $T_WORK

/opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks

cd /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks

--디렉토리가 없을 경우 생성할 것(mkdir disks/raw)

[할당된 disk 설정]

cd /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw

ln -s /dev/sdb stocell1_DISK01

ln -s /dev/sdc stocell1_DISK02

ln -s /dev/sdd stocell1_DISK03

ln -s /dev/sde stocell1_DISK04

ln -s /dev/sdf stocell1_DISK05

ln -s /dev/sdg stocell1_DISK06

ln -s /dev/sdh stocell1_DISK07

ln -s /dev/sdi stocell1_DISK08

ln -s /dev/sdj stocell1_DISK09

ln -s /dev/sdk stocell1_DISK10

ln -s /dev/sdl stocell1_DISK11

ln -s /dev/sdm stocell1_DISK12

ln -s /dev/sdn stocell1_FLASH01

ln -s /dev/sdo stocell1_FLASH02

ln -s /dev/sdp stocell1_FLASH03

ln -s /dev/sdq stocell1_FLASH04

ln -s /dev/sdr stocell1_FLASH05

ln -s /dev/sds stocell1_FLASH06

③ Cell Disk Service 설정

[root@stocell1 ~]# su – celladmin

[celladmin@stocell1 ~]$ cellcli -e alter cell restart services all

Stopping the RS, CELLSRV, and MS services…

CELL-01509: Restart Server (RS) not responding.

Starting the RS, CELLSRV, and MS services…

Getting the state of RS services… running

Starting CELLSRV services…

The STARTUP of CELLSRV services was not successful.

CELL-01547: CELLSRV startup failed due to unknown reasons.

Starting MS services…

The STARTUP of MS services was successful.

The error in not unknown (as stated) but well known and expected

[InfiniBand Storage Network 설정]

[celladmin@stocell1 ~]$ cellcli -e create cell stocell1 interconnect1=eth1

Cell stocell1 successfully created

Starting CELLSRV services…

The STARTUP of CELLSRV services was successful.

Flash cell disks, FlashCache, and FlashLog will be created…

CellDisk FD_00_stocell1 successfully created

CellDisk FD_01_stocell1 successfully created

CellDisk FD_02_stocell1 successfully created

CellDisk FD_03_stocell1 successfully created

CellDisk FD_04_stocell1 successfully created

CellDisk FD_05_stocell1 successfully created

Flash log stocell1_FLASHLOG successfully created

Flash cache stocell1_FLASHCACHE successfully created

[Cell Disks 설정]

[celladmin@stocell1 ~]$ cellcli -e create celldisk all

CellDisk CD_DISK01_stocell1 successfully created

CellDisk CD_DISK02_stocell1 successfully created

CellDisk CD_DISK03_stocell1 successfully created

CellDisk CD_DISK04_stocell1 successfully created

CellDisk CD_DISK05_stocell1 successfully created

CellDisk CD_DISK06_stocell1 successfully created

CellDisk CD_DISK07_stocell1 successfully created

CellDisk CD_DISK08_stocell1 successfully created

CellDisk CD_DISK09_stocell1 successfully created

CellDisk CD_DISK10_stocell1 successfully created

CellDisk CD_DISK11_stocell1 successfully created

CellDisk CD_DISK12_stocell1 successfully created

[Grid disks 설정]

[celladmin@stocell1 ~]$ cellcli -e create griddisk all harddisk prefix=DATA

GridDisk DATA_CD_DISK01_stocell1 successfully created

GridDisk DATA_CD_DISK02_stocell1 successfully created

GridDisk DATA_CD_DISK03_stocell1 successfully created

GridDisk DATA_CD_DISK04_stocell1 successfully created

GridDisk DATA_CD_DISK05_stocell1 successfully created

GridDisk DATA_CD_DISK06_stocell1 successfully created

GridDisk DATA_CD_DISK07_stocell1 successfully created

GridDisk DATA_CD_DISK08_stocell1 successfully created

GridDisk DATA_CD_DISK09_stocell1 successfully created

GridDisk DATA_CD_DISK10_stocell1 successfully created

GridDisk DATA_CD_DISK11_stocell1 successfully created

GridDisk DATA_CD_DISK12_stocell1 successfully created

[celladmin@stocell1 ~]$ cellcli -e alter cell restart services all

Stopping the RS, CELLSRV, and MS services...

The SHUTDOWN of services was successful.

Starting the RS, CELLSRV, and MS services...

Getting the state of RS services... running

Starting CELLSRV services...

The STARTUP of CELLSRV services was successful.

Starting MS services...

The STARTUP of MS services was successful.

[cell disk alert log]

[root@stocell1 trace]# cd /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/log/diag/asm/cell/stocell1/trace

[root@stocell1 trace]# ls -al alert*

-rw-rw---- 1 root celladmin 156850 Aug 21 10:00 alert.log

[RS] Started Service MS with pid 5390

Fri Aug 21 10:00:51 2015

Smart Flash Logging enabled on FlashLog stocell1_FLASHLOG (1842424628), size=80MB, cdisk=FD_05_stocell1

Fri Aug 21 10:00:51 2015

Smart Flash Logging enabled on FlashLog stocell1_FLASHLOG (3996181628), size=80MB, cdisk=FD_04_stocell1

Fri Aug 21 10:00:51 2015

Smart Flash Logging enabled on FlashLog stocell1_FLASHLOG (3466652180), size=80MB, cdisk=FD_03_stocell1

Fri Aug 21 10:00:51 2015

Smart Flash Logging enabled on FlashLog stocell1_FLASHLOG (1448144612), size=80MB, cdisk=FD_02_stocell1

Fri Aug 21 10:00:51 2015

Smart Flash Logging enabled on FlashLog stocell1_FLASHLOG (2992946252), size=80MB, cdisk=FD_00_stocell1

Smart Flash Logging enabled on FlashLog stocell1_FLASHLOG (1465279500), size=80MB, cdisk=FD_01_stocell1

FlashLog: initialization complete

Fri Aug 21 10:00:51 2015

CELLSRV Server startup complete

[RS] Started Service CELLSRV with pid 5397

Fri Aug 21 10:00:57 2015

Info: Assigning flash CD FD_00_stocell1 to group#1

Info: Assigning flash CD FD_01_stocell1 to group#2

Fri Aug 21 10:00:57 2015

Info: Assigning flash CD FD_02_stocell1 to group#3

Info: Assigning flash CD FD_03_stocell1 to group#0

Info: Assigning flash CD FD_04_stocell1 to group#1

Info: Assigning flash CD FD_05_stocell1 to group#2

-stocell1 OVM이 완료된 상태로 해당 내역을 복제하여 exadata 구성을 진행한다.

-해당 stocell1을 바탕으로 DB 1개+Cell Disk 2개 구성, DB 2개+Cell Disk1개 구성(RAC)등 다양하게 활용할 수 있으므로

OVM export(내보내기)로 백업 받아 두고 사용하면 좋다.

④ Cellcli 명렁어

cellcli 명령어

1. check cell status

cellcli -e list cell

cellcli -e list cell detail

2. Physical Disk Information

cellcli -e list physicaldisk

cellcli -e list physicaldisk E15SBS detail

3. LUN Disks Detail

cellcli -e list lun

cellcli -e list lun 0_0 detail

cellcli -e list lun 5_3 detail

4. Cell Disks Report

cellcli -e list celldisk

cellcli -e list celldisk CD_08_exadatalcel10 detail

cellcli -e list celldisk attributes name, devicePartition where size>200g;

cellcli -e list celldisk attributes name,status,size

5. Grid Disk Knowledge

cellcli -e list griddisk

cellcli -e list celldisk CD_02_exadatalcel10 detail

cellcli -e list griddisk DATA_DMORL_CD_02_exadatalcel10 detail

cellcli -e list griddisk DBFS_DG_CD_02_exadatalcel10 detail

cellcli -e list griddisk RECO_DMORL_CD_02_exadatalcel10 detail

6. Display Exadate Alerts

cellcli -e list alerthistory

cellcli -e list alerthistory 8_1 detail

cellcli -e list alerthistory where severity like '[warning|critical]'

cellcli -e list alertdefinition detail

7. Restart Cell Services

cellcli -e list cell detail

cellcli -e alter cell restart services rs

cellcli -e alter cell restart services ms

cellcli -e alter cell restart services cellsrv

cellcli -e alter cell restart services all

cellcli -e alter cell shutdown services rs

cellcli -e alter cell shutdown services ms

cellcli -e alter cell shutdown services cellsrv

⑤ 구성 시에 에러(CELL-01518)

/opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/deploy/config/cellinit.ora

[celladmin@cell ~]$ cellcli -e create cell cell1 interconnect1=eth1

CELL-01518: Stop CELLSRV. Create Cell cannot continue with CELLSRV running.

[celladmin@cell ~]$ cellcli

CellCLI: Release 11.2.3.2.0 - Production on Tue Jan 20 22:14:40 CST 2015

Cell Efficiency Ratio: 1

CellCLI> alter cell shutdown services cellsrv

Stopping CELLSRV services...

The SHUTDOWN of CELLSRV services was successful.

CellCLI> exit

quitting

[celladmin@cell ~]$ cellcli -e create cell cell1 interconnect1=eth1

Cell cell1 successfully created

[참고 문서 및 사이트]

'1. IT Story > DB' 카테고리의 다른 글

OCR, VOTE ASM Disk에서 RawDevice로 Disk 변경 TEST (0)	2015.08.26
ODA 기본 OS 상태(OVM환경)에서 Grid & DB Installation (0)	2015.08.26
Oracle Exadata TEST Single Database Installation : Install Database S/W & TEST Database (1)	2015.08.21
Oracle Exadata TEST Single Database Installation : Install Grid Infrastructure (1)	2015.08.21
Oracle Exadata TEST Single Database Installation : OVM TEST Environment (2)	2015.08.21
Oracle Exadata TEST Single Database Installation : Overview (0)	2015.08.20
Enterprise Manager을 이용하여 Rman backup설정 TEST (0)	2015.08.07
Oracle RAC 11.2.0.3 with ASM / Vmware Grid 설치 (0)	2015.08.07