安装环境说明与虚拟机规化:

安装环境
主机操作系统:windows 10
虚拟机VMware15:两台CentOS-7.6
Oracle Database software: Oracle11gR2 (11.2.0.1.0)
Cluster software: Oracle grid infrastructure 11gR2(11.2.0.1.0)
共享存储:ASM
oracle传送门 https://www.oracle.com/technetwork/database/enterprise-edition/downloads/112010-linx8664soft-100572.html

虚拟机规划

  1. 安装虚拟机时注意分配两个网卡,一个网卡为Host Only方式,用于两台虚拟机节点的通讯,另一个网卡为NAT方式,用于连接外网,这里由虚拟机自动分配IP(为了以后省事还是配静态好)。每台主机的内存和swap规划为至少2.5G。硬盘规划为:boot 500M,其他空间分配为LVM方式管理,LVM划分2.5G为swap,其他为/。 主机名设为rac1、rac2 (注意这里两个虚拟机最好在不同的硬盘中,否则I/O会很吃力,否则万分痛苦!)
  2. 由于采用的是共享存储ASM,而且搭建集群需要共享空间作注册盘(OCR)和投票盘(votingdisk)。VMware创建共享存储方式:
    进入VMware安装目录,cmd命令下:
    E:\Program Files (x86)\VMware\VMware Workstation
vmware-vdiskmanager.exe -c -s 1000Mb -a lsilogic -t 2 F:\asm\ocr.vmdk
vmware-vdiskmanager.exe -c -s 1000Mb -a lsilogic -t 2 F:\asm\ocr2.vmdk
vmware-vdiskmanager.exe -c -s 1000Mb -a lsilogic -t 2 F:\asm\votingdisk.vmdk
vmware-vdiskmanager.exe -c -s 20000Mb -a lsilogic -t 2 F:\asm\data.vmdk
vmware-vdiskmanager.exe -c -s 10000Mb -a lsilogic -t 2 F:\asm\backup.vmdk

创建了两个1G的ocr盘,一个1G的投票盘,一个20G的数据盘,一个10G的备份盘。

3、虚拟机添加磁盘,选定虚拟机然后编辑配置,添加磁盘选择刚刚添加的虚拟磁盘,大概如下(2台都执行)

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_运维


虚拟机操作系统配置(没有特别注明都是2台均执行)

  1. 关闭防火墙
    查看防火墙状态
    #firewall-cmd --state停止firewall
    #systemctl stop firewalld.service禁止firewall开机启动
    #systemctl disable firewalld.service
  2. 创建必要的用户、组和目录,并授权
#groupadd -g 1000 oinstall
#groupadd -g 1020 asmadmin
#groupadd -g 1021 asmdba
#groupadd -g 1022 asmoper
#groupadd -g 1031 dba
#groupadd -g 1032 oper
#useradd -u 1100 -g oinstall -G asmadmin,asmdba,asmoper,oper,dba grid
#useradd -u 1101 -g oinstall -G dba,asmdba,oper oracle
#mkdir -p /u01/app/11.2.0/grid
#mkdir -p /u01/app/grid
#mkdir /u01/app/oracle
#chown -R grid:oinstall /u01
#chown oracle:oinstall /u01/app/oracle
#chmod -R 775 /u01/
  1. 系统文件设置a.修改内核配置,在文件/etc/sysctl.conf 的末尾处添加如下信息
1. #vi /etc/sysctl.conf 
kernel.msgmnb = 65536 
kernel.msgmax = 65536 
kernel.shmmax = 68719476736 
kernel.shmall = 4294967296 
fs.aio-max-nr = 1048576 
fs.file-max = 6815744 
kernel.shmall = 2097152 
kernel.shmmax = 1306910720 
kernel.shmmni = 4096 
kernel.sem = 250 32000 100 128 
net.ipv4.ip_local_port_range = 9000 65500 
net.core.rmem_default = 262144 
net.core.rmem_max = 4194304 
net.core.wmem_default = 262144 
net.core.wmem_max = 1048586 
net.ipv4.tcp_wmem = 262144 262144 262144 
net.ipv4.tcp_rmem = 4194304 4194304 4194304

使配置生效

#sysctl -pb.配置oracle、grid用户的shell限制,在文件/etc/security/limits.conf 的末尾加入如下内容

#vi /etc/security/limits.conf 
grid soft nproc 2047 
grid hard nproc 16384 
grid soft nofile 1024 
grid hard nofile 65536 
oracle soft nproc 2047 
oracle hard nproc 16384 
oracle soft nofile 1024 
oracle hard nofile 65536
c.配置login,在文件/etc/pam.d/login的末尾加入如下内容
#vi /etc/pam.d/login 
session required pam_limits.so

d.修改操作系统发行版本名,将文件/etc/centos-release的原有内容注释,添加redheat-7(因为oracle不支持centos7安装)

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_bc_02


e.使用安装介质作为yum源安装依赖的包

配置本地源的方式,自己先进行配置:

#mount /dev/sr0 /mnt/cdrom/ 
#vi /etc/yum.repos.d/dvd.repo 
[dvd] 
name=dvd 
baseurl=file:///mnt/cdrom 
gpgcheck=0 
enabled=1 
#yum clean all 
#yum makecache 
#yum install gcc gcc-c++ glibc glibc-devel ksh libgcc libstdc++ libstdc++-devel* make sysstat

4.配置IP和hosts、hostname(这里的hostname在系统安装时已经设定为rac1和rac2,ip由虚拟机自动分配,因此只配hosts),在文件/etc/hosts 的末尾添加如下内容

#vi /etc/hosts 
192.168.149.129 rac1
192.168.149.201 rac1-vip
192.168.96.128 rac1-priv
192.168.149.130 rac2
192.168.149.202 rac2-vip
192.168.96.129 rac2-priv192.168.149.100 scan-ip

5.配置grid和oracle用户环境变量

Oracle_sid需要根据节点不同进行修改 
#su - grid 
$ vi .bash_profile
export TMP=/tmp
export TMPDIR=$TMP
export ORACLE_SID=+ASM1 # RAC1使用这个配置
export ORACLE_SID=+ASM2 # RAC2使用这个配置
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=/u01/app/11.2.0/grid
export PATH=/usr/sbin:$PATH
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
umask 022

需要注意的是ORACLE_UNQNAME是数据库名(这里用orcl),创建数据库时指定多个节点是会创建多个实例,ORACLE_SID指的是数据库实例名

#su - oracle 
$vi .bash_profile
export TMP=/tmp
export TMPDIR=$TMP
export ORACLE_SID=orcl1 # RAC1使用这个配置
export ORACLE_SID=orcl2 # RAC2使用这个配置
export ORACLE_UNQNAME=orcl
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1
export TNS_ADMIN=$ORACLE_HOME/network/admin
export PATH=/usr/sbin:$PATH
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib


$ source .bash_profile使配置文件生效(修改环境变量都是在最末行追加,保存后都要source使配置生效)

7.配置oracle、grid用户ssh互信

#su – oracle
$mkdir ~/.ssh
$chmod 755 .ssh 
$/usr/bin/ssh-keygen -t rsa
$/usr/bin/ssh-keygen -t dsa
下面步骤只在节点1执行,将所有的key文件汇总到一个总的认证文件中
$ssh rac1 cat ~/.ssh/id_rsa.pub >> authorized_keys
$ssh rac2 cat ~/.ssh/id_rsa.pub >> authorized_keys
$ssh rac1 cat ~/.ssh/id_dsa.pub >> authorized_keys
$ssh rac2 cat ~/.ssh/id_dsa.pub >> authorized_keys
$ cd ~/.ssh/
$ scp authorized_keys rac2:~/.ssh/  拷贝完整的key到节点2
[oracle@rac2 .ssh]chmod 600 authorized_keys  此处注意是登陆到节点2修改权限
下面步骤2个节点都要执行,否则后续安装会报错,相当重要,得到的结果应该是一个时间
$ssh rac1 date
$ssh rac2 date
$ssh rac1-priv date
$ssh rac2-priv date

grid的互信参照oracle用户处理即可

8.配置裸盘
(1)fdisk -l 通过该命令查看磁盘,可看到我们后面添加上去的磁盘信息,磁盘从sdb到sdf
(2)分区、格式化磁盘。由于是共享磁盘,所以只需要在一个节点上执行即可
在节点1上格式化,以/dev/sdb为例:

#fdisk /dev/sdb
The number of cylinders for this disk is set to 3824.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-3824, default 1): 
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-3824, default 3824): 
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.


格式化完毕后得到如下分区
#ls /dev/sd*
/dev/sda /dev/sda2 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sda1 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf
(3)在节点上添加裸设备,注意2个节点都要执行

#vi /etc/udev/rules.d/99-oracle-asmdevices.rules
ACTION=="add", KERNEL=="sdb1", RUN+="/bin/raw /dev/raw/raw1 %N"
ACTION=="add", KERNEL=="sdc1", RUN+="/bin/raw /dev/raw/raw2 %N"
ACTION=="add", KERNEL=="sdd1", RUN+="/bin/raw /dev/raw/raw3 %N"
ACTION=="add", KERNEL=="sde1", RUN+="/bin/raw /dev/raw/raw4 %N"
ACTION=="add", KERNEL=="sdf1", RUN+="/bin/raw /dev/raw/raw5 %N"
KERNEL=="raw[1]", MODE="0660", OWNER="grid", GROUP="asmadmin"
KERNEL=="raw[2]", MODE="0660", OWNER="grid", GROUP="asmadmin"
KERNEL=="raw[3]", MODE="0660", OWNER="grid", GROUP="asmadmin"
KERNEL=="raw[4]", MODE="0660", OWNER="grid", GROUP="asmadmin"
KERNEL=="raw[5]", MODE="0660", OWNER="grid", GROUP="asmadmin"

(4)启动裸设备,2个节点都执行
#start_udev
(5)查看裸设备,2个节点都要查看

#raw -qa
/dev/raw/raw1: bound to major 8, minor 17
/dev/raw/raw2: bound to major 8, minor 33
/dev/raw/raw3: bound to major 8, minor 49
/dev/raw/raw4: bound to major 8, minor 65
/dev/raw/raw5: bound to major 8, minor 81


9.挂载安装软件文件夹
这里是主机windows系统开启文件夹共享,将压缩包解压后供虚拟机挂载即可 ,这里的用户名密码目录仅供参考
mkdir -p /home/grid/db
mount -t cifs -o username=share,password=123456 //192.168.248.1/DB /home/grid/db
mkdir -p /home/oracle/db
mount -t cifs -o username=share,password=123456 //192.168.248.1/DB /home/oracle/db

10.安装用于环境检查的cvuqdisk(双机)

#cd /home/grid/db/rpm
#rpm -ivh cvuqdisk-1.0.7-1.rpm

11.手动运行cvu使用验证程序验证Oracle集群要求

#su - grid
$cd /home/grid/db/
$./runcluvfy.sh stage -pre crsinst -n rac1,rac2 -fixup -verbose

注:这里检查见过出现很多32位(i386或i686)的软件包校验不通过,我直接忽略不处理也没有影响系统使用,因此可以忽略,另外NTP、pdksh的报错也可以忽略,校验时2机报 glibc不存在,因此只能重新打包,从网上下载包

glibc-2.17-260.el7_6.4.i686.rpm ,直接安装存在依赖
error: Failed dependencies:
libfreebl3.so is needed by glibc-2.17-260.el7_6.4.i686
libfreebl3.so(N×××AWHASH_3.12.3) is needed by glibc-2.17-260.el7_6.4.i686
百度查询需要打包 nss-softokn-freebl-3.36.0-5.el7_5.i686.rpm ,直接打包报错:
error: Failed dependencies:
libc.so.6 is needed by nss-softokn-freebl-3.36.0-5.el7_5.i686
libc.so.6(GLIBC_2.0) is needed by nss-softokn-freebl-3.36.0-5.el7_5.i686
libc.so.6(GLIBC_2.1) is needed by nss-softokn-freebl-3.36.0-5.el7_5.i686
libc.so.6(GLIBC_2.1.3) is needed by nss-softokn-freebl-3.36.0-5.el7_5.i686
libc.so.6(GLIBC_2.3) is needed by nss-softokn-freebl-3.36.0-5.el7_5.i686
libc.so.6(GLIBC_2.3.4) is needed by nss-softokn-freebl-3.36.0-5.el7_5.i686
libc.so.6(GLIBC_2.4) is needed by nss-softokn-freebl-3.36.0-5.el7_5.i686
libdl.so.2 is needed by nss-softokn-freebl-3.36.0-5.el7_5.i686
libdl.so.2(GLIBC_2.0) is needed by nss-softokn-freebl-3.36.0-5.el7_5.i686
libdl.so.2(GLIBC_2.1) is needed by nss-softokn-freebl-3.36.0-5.el7_5.i686
通过2个包一起安装解决:
#rpm -ivh glibc-2.17-260.el7_6.4.i686.rpm nss-softokn-freebl-3.36.0-5.el7_5.i686.rpm 
Preparing... ################################# [100%]
Updating / installing...
1:nss-softokn-freebl-3.36.0-5.el7_5################################# [ 50%]
2:glibc-2.17-260.el7_6.4 ################################# [100%]

安装Grid,只需要在1个节点执行即可,另外一个节点自动同步
1.安装流程
(1)开启安装(这里为了更好的可用性用Xmanager来远程安装,配置参考)

#xhost +
#su - grid
$export DISPLAY=Windows主机IP:0.0
$cd cd /home/grid/db/
$./runInstaller

图形化流程

跳过更新-安装集群

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_虚拟机搭建Kubernetes的优点_03


自定义安装

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_bc_04


安装语音默认英文

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_bc_05


定义集群名字,SCAN Name 为hosts中定义的scan-ip,GNS去勾选

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_数据库_06


界面只有第一个节点rac1,点击“Add”把第二个节点rac2加上,信息可以参照rac1填写

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_bc_07


虚拟机搭建Kubernetes的优点 虚拟机搭建rac_bc_08


选择网卡,默认即可

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_bc_09


配置ASM,这里选择前面配置的裸盘raw1,raw2,raw3,冗余方式为External即不冗余。因为是不用于,所以也可以只选一个设备。这里的设备是用来做OCR注册盘和votingdisk投票盘的。

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_bc_10


虚拟机搭建Kubernetes的优点 虚拟机搭建rac_数据库_11


配置ASM实例需要为具有sysasm权限的sys用户,具有sysdba权限的asmsnmp用户设置密码,这里设置统一密码,点击OK即可

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_虚拟机搭建Kubernetes的优点_12


不选择智能管理

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_运维_13


检查ASM实例权限分组情况

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_虚拟机搭建Kubernetes的优点_14


选择grid软件安装路径和base目录 ,这里默认即可

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_oracle_15


选择grid安装清单目录

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_运维_16


环境检测出现ntp等刚刚跑脚本检测出的错误,选择忽略全部

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_虚拟机搭建Kubernetes的优点_17


安装概要,点击安装后开始安装并且自动复制到rac2

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_oracle_18


安装grid完成,提示需要root用户依次执行脚本orainstRoot.sh ,root.sh (一定要先在rac1执行完脚本后,才能在其他节点执行)

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_bc_19


rac1:

#/u01/app/oraInventory/orainstRoot.sh

Changing permissions of /u01/app/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.

The execution of the script is complete.

#/u01/app/11.2.0/grid/root.sh

Performing root user operation for Oracle 11g

The following environment variables are set as:

ORACLE_OWNER= grid

ORACLE_HOME= /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]: (回车)

Copying dbhome to /usr/local/bin ...

Copying oraenv to /usr/local/bin ...

Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params

Creating trace directory

User ignored Prerequisites during installation

OLR initialization - successful

root wallet

root wallet cert

root cert export

peer wallet

profile reader wallet

pa wallet

peer wallet keys

pa wallet keys

peer cert request

pa cert request

peer cert

pa cert

peer root cert TP

profile reader root cert TP

pa root cert TP

peer pa cert TP

pa peer cert TP

profile reader pa cert TP

profile reader peer cert TP

peer user cert

pa user cert

Adding Clusterware entries to upstart

CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'

CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'

CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'

CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'

CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded

CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'rac1'

CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'

CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded

CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded

ASM created and started successfully.

Disk Group OCR created successfully.

clscfg: -install mode specified

Successfully accumulated necessary OCR keys.

Creating OCR keys for user 'root', privgrp 'root'..

Operation successful.

CRS-4256: Updating the profile

Successful addition of voting disk 496abcfc4e214fc9bf85cf755e0cc8e2.

Successfully replaced voting disk group with +OCR.

CRS-4256: Updating the profile

CRS-4266: Voting file(s) successfully replaced

STATE File Universal Id File Name Disk group

  1. ONLINE 496abcfc4e214fc9bf85cf755e0cc8e2 (/dev/raw/raw1) [OCR]
    Located 1 voting disk(s).
    CRS-2672: Attempting to start 'ora.asm' on 'rac1'
    CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.OCR.dg' on 'rac1'
    CRS-2676: Start of 'ora.OCR.dg' on 'rac1' succeeded
    Configure Oracle Grid Infrastructure for a Cluster ... succeeded

在rac2用同样的方法跑2个脚本,完成后点击上图中的ok按钮再执行下一步,忽略报错完成安装
2.安装grid后的资源检查

#su - grid
$ crsctl check crs

CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online12345
检查Clusterware资源
$crs_stat -t -v

Name Type R/RA F/FT Target State Host
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE rac1
ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE rac1
ora.OCR.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE rac1
ora.cvu ora.cvu.type 0/5 0/0 ONLINE ONLINE rac1
ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE rac1
ora.oc4j ora.oc4j.type 0/1 0/2 ONLINE ONLINE rac1
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE rac1
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE rac1
ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.gsd application 0/5 0/0 OFFLINE OFFLINE
ora.rac1.ons application 0/3 0/0 ONLINE ONLINE rac1
ora.rac1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac1
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE rac2
ora....C2.lsnr application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.gsd application 0/5 0/0 OFFLINE OFFLINE
ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2
ora.rac2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac2
ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE rac1
检查集群节点

$ olsnodes -n
rac1    1
rac2    2

检查两个节点上的Oracle TNS监听器进程

$ ps -ef|grep lsnr|grep -v 'grep'|grep -v 'ocfs'|awk '{print$9}'
LISTENER_SCAN1
LISTENER

确认针对Oracle Clusterware文件的Oracle ASM功能:
如果在 Oracle ASM 上安装过了OCR和表决磁盘文件,则以Grid Infrastructure 安装所有者的身份,使用给下面的命令语法来确认当前正在运行已安装的Oracle ASM:

$ srvctl status asm -a
ASM is running on rac2,rac1
ASM is enabled.

3.为数据和快速恢复去创建ASM磁盘组(只在rac1执行即可)

#su - grid 
$ asmca

这里看到安装grid时配置的OCR盘已存在

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_oracle_20


点击create,添加DATA盘,使用裸盘raw4

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_oracle_21


同样创建FRA盘,使用裸盘raw5

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_数据库_22


ASM磁盘组的情况

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_运维_23


ASM的实例

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_oracle_24


安装Oracle database软件(RAC)

1.安装流程(只在rac1执行即可,图形化设置参照asm的)

#su - oracle

$ cd db/database

$ ./runInstaller

进入图形化界面,跳过更新选项接着选择只安装数据库软件

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_oracle_25


虚拟机搭建Kubernetes的优点 虚拟机搭建rac_虚拟机搭建Kubernetes的优点_26


选择Oracel Real Application Clusters database installation按钮(默认),点击select all 按钮 确保勾选所有的节点

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_oracle_27


默认英文安装

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_运维_28


选择安装企业版软件

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_数据库_29


选择安装Oracle软件路径,其中ORACLE_BASE,ORACLE_HOME均选择之前配置好的,默认即可

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_bc_30


oracle权限授予用户组,默认即可

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_虚拟机搭建Kubernetes的优点_31


安装前的预检查,忽略所有检查异常项

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_bc_32


显示安装RAC的概要信息

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_oracle_33


开始安装,会自动复制到其他节点

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_数据库_34


安装过程的异常解决方案:

/u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib//libnmectl.a(nmectlt.o): In function nmectlt_genSudoProps':<br/>nmectlt.c:(.text+0x76): undefined reference toB_DestroyKeyObject'

nmectlt.c:(.text+0x7f): undefined reference to B_DestroyKeyObject'<br/>nmectlt.c:(.text+0x88): undefined reference toB_DestroyKeyObject'

nmectlt.c:(.text+0x91): undefined reference to `B_DestroyKeyObject'

INFO: collect2: error: ld returned 1 exit status

INFO: /u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib/ins_emagent.mk:176: recipe for target '/u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib/emdctl' failed

make[1]: Leaving directory '/u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib'

INFO: /u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib/ins_emagent.mk:52: recipe for target 'emdctl' failed

INFO: make[1]: [/u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib/emdctl] Error 1
INFO: make:
[emdctl] Error 2

INFO: End output from spawned process.

INFO: ----------------------------------

INFO: Exception thrown from action: make

Exception Name: MakefileException

Exception String: Error in invoking target 'agent nmhs' of makefile '/u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib/ins_emagent.mk'. See '/u01/app/oraInventory/logs/installActions2017-05-02_12-37-15PM.log' for details.

Exception Severity: 1

方案:

修改“emdctl”的编译参数,编辑“/u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib/ins_emagent.mk”文件,将

171 #===========================

172 # emdctl

173 #===========================

174

175 $(SYSMANBIN)emdctl:

176 $(MK_EMAGENT_NMECTL)

改为

171 #===========================

172 # emdctl

173 #===========================

174

175 $(SYSMANBIN)emdctl:

176 $(MK_EMAGENT_NMECTL) -lnnz11

然后点击retry

NFO: Exception thrown from action: make

Exception Name: MakefileException

Exception String: Error in invoking target 'irman ioracle' of makefile '/u01/app/oracle/product/11.2.0/db_1/rdbms/lib/ins_rdbms.mk'. See '/u01/app/oraInventory/logs/installActions2019-04-30_03-12-13PM.log' for details.

解决方法如下:

cd $ORACLE_HOME/rdbms/admin

/usr/bin/make -f $ORACLE_HOME/rdbms/lib/ins_rdbms.mk irman

然后点击retry安装完,在每个节点用root用户执行脚本

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_oracle_35


#/u01/app/oracle/product/11.2.0/db_1/root.sh

Performing root user operation for Oracle 11g

The following environment variables are set as:

ORACLE_OWNER= oracle

ORACLE_HOME= /u01/app/oracle/product/11.2.0/db_1

Enter the full pathname of the local bin directory: [/usr/local/bin]: (回车)

The contents of "dbhome" have not changed. No need to overwrite.

The contents of "oraenv" have not changed. No need to overwrite.

The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Finished product-specific root actions.

执行过程出错

CRS-4124: Oracle High Availability Services startup failed.

CRS-4000: Command Start failed, or completed with errors.

ohasd failed to start: Inappropriate ioctl for device

ohasd failed to start at /u01/app/11.2.0/grid/crs/install/rootcrs.pl line 443.

这是11.0.2.1的经典问题,一个bug,解决办法也很简单,开2个窗口一个跑root.sh在执行root.sh命令出现Adding daemon to inittab的时候在另外一个窗口反复执行/bin/dd if=/var/tmp/.oracle/npohasd of=/dev/null bs=1024 count=1

直到没有出现/bin/dd: opening`/var/tmp/.oracle/npohasd': No such file or directory

重新执行root.sh之前别忘了删除配置:/u01/app/11.2.0/grid/crs/install/roothas.pl -deconfig -force-verbose

参考:

到此完成数据库软件安装

2.创建集群数据库

在节点rac1上用oracle用户执行dbca创建RAC数据库(图形设置参照之前)

#su - oracle

$dbca

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_虚拟机搭建Kubernetes的优点_36


这里如果看到的部署RAC安装可能是RAC没有启动,grid用户检查RAC资源

$ crsctl check crs

CRS-4639: Could not contact Oracle High Availability Services

$ crs_stat -t -v

CRS-0184: Cannot communicate with the CRS daemon.

看到没有起点,可以在root用户下手动拉起服务

#/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

选择创建数据库

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_虚拟机搭建Kubernetes的优点_37


选择自定义数据库(也可以是通用)

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_oracle_38


配置类型选择Admin-Managed,输入全局数据库名orcl,每个节点实例SID前缀为orcl,选择双节点

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_oracle_39


选择默认,配置OEM,启用数据库自动维护任务

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_数据库_40


统一设置sys,system,dbsnmp,sysman用户的密码为oracle

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_运维_41


使用ASM存储,使用OMF(oracle的自动管理文件),数据区选择之前创建的DATA磁盘组 (

a.这里有可能出现没有磁盘的问题,参考解决方案

本次是修改了grid下oracle的权限6751

b.还有可能出现磁盘没有挂载的情况,解决方案:

$ asmcmd

asmcmd>lsdg --查看当前挂载情况

asmcmd>mount DATA 挂载名为data的磁盘)

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_虚拟机搭建Kubernetes的优点_42


设置ASM密码为oracle_4U

(这里会报"ORA-01031: insufficient privileges"

先删除文件/u01/app/11.2.0/grid/dbs/orapw+ASM,然后新生成文件,此时的密码为oracle_4U

[grid@node1 bin]$ orapwd file=/u01/app/11.2.0/grid/dbs/orapw+ASM password=oracle_4U

[grid@node1 dbs]$ scp orapw+ASM Node2:/u01/app/11.2.0/grid/dbs/ --拷贝到另外一个节点

[grid@node1 bin]$ sqlplus / as sysasm

SQL>create user asmsnmp identified by oracle_4U;

SQL> grant sysdba to asmsnmp;

然后点击retry即可)

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_虚拟机搭建Kubernetes的优点_43


指定数据闪回区,选择之前创建好的FRA磁盘组,不开归档

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_oracle_44


组件选择,默认

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_oracle_45


选择字符集AL32UTF8

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_bc_46


选择默认的数据存储信息

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_bc_47


开始创建数据库,勾选生成数据库的脚本

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_数据库_48


数据库的概要信息

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_虚拟机搭建Kubernetes的优点_49


开始安装

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_运维_50


虚拟机搭建Kubernetes的优点 虚拟机搭建rac_运维_51


安装过程可能出现的问题:ORA-12547: TNS:lost contact

检查$ORACLE_HOME/bin/oracle 和 $ORACLE_HOME/rdbms/lib/config.o 的文件大小是否为0,

如果大小为0,需要重新编译oracle软件:

[oracle@normal backup]$ relink all

writing relink log to: /u01/app/oracle/product/11.2.0/db_1/install/relink.log

完成数据库实例安装

虚拟机搭建Kubernetes的优点 虚拟机搭建rac_数据库_52


RAC维护
1.查看服务状态(gsd 是为了兼容9i,10G以后不再启动,忽略offline)
#su - grid
$crs_stat -t
Name Type Target State Host
ora.DATA.dg ora....up.type ONLINE ONLINE rac1
ora.FRA.dg ora....up.type ONLINE ONLINE rac1
ora....ER.lsnr ora....er.type ONLINE ONLINE rac1
ora....N1.lsnr ora....er.type ONLINE ONLINE rac1
ora.OCR.dg ora....up.type ONLINE ONLINE rac1
ora.asm ora.asm.type ONLINE ONLINE rac1
ora.cvu ora.cvu.type ONLINE ONLINE rac1
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE rac1
ora.oc4j ora.oc4j.type ONLINE ONLINE rac1
ora.ons ora.ons.type ONLINE ONLINE rac1
ora.orcl.db ora....se.type ONLINE ONLINE rac1
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application OFFLINE OFFLINE
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip ora....t1.type ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application OFFLINE OFFLINE
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip ora....t1.type ONLINE ONLINE rac2
ora.scan1.vip ora....ip.type ONLINE ONLINE rac1

2.检查集群状态

$srvctl status database -d orcl 
Instance orcl1 is running on node rac1 
Instance orcl2 is running on node rac2

3.检查CRS状态
本地

$crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

集群

$ crsctl check cluster
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

4.查看集群中节点配置信息

$ olsnodes
rac1
rac2
$ olsnodes -n
rac1    1
rac2    2
$ olsnodes -n -i -s -t
rac1    1       rac1-vip        Active  Unpinned
rac2    2       rac2-vip        Active  Unpinned

5.查看集群件的表决磁盘信息

.$ crsctl query css votedisk
STATE    File Universal Id                File Name Disk group
 1. ONLINE   496abcfc4e214fc9bf85cf755e0cc8e2 (/dev/raw/raw1) [OCR]
Located 1 voting disk(s).

6.查看集群SCAN VIP信息(参考示例)

$ srvctl config scan
SCAN name: scan-ip, Network: 1/192.168.248.0/255.255.255.0/eth0
SCAN VIP name: scan1, IP: /scan-ip/192.168.248.110
$ srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521

7.启、停集群数据库
整个集群的数据库启停
进入grid用户

$ srvctl stop database -d orcl
$ srvctl status database -d orcl
$ srvctl start database -d orcl
关闭所有节点 
进入root用户 
关闭所有节点 
# /u01/app/11.2.0/grid/bin/crsctl stop crs

实际只关闭了当前结点
启动过程出现的异常(由于内存不够后面降低了内存重新启动虚拟机):
报错:ORA-00845: MEMORY_TARGET not supported on this system
$ srvctl start database -d orcl
PRCR-1079 : Failed to start resource ora.orcl.db
ORA-00845: MEMORY_TARGET not supported on this system
CRS-2674: Start of 'ora.orcl.db' on 'rac1' failed
CRS-2632: There are no more servers to try to place resource 'ora.orcl.db' on that would satisfy its placement policy
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
ORA-00845: MEMORY_TARGET not supported on this system
CRS-2674: Start of 'ora.orcl.db' on 'rac2' failed
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
解决办法:
加大/dev/shm,使他大于系统参数,由于停机后无法获取系统参数,在此可以设置一个总内存大小:
#cat /etc/fstab | grep tmpfs
tmpfs /dev/shm tmpfs defaults,size=4G 0 0
#mount -o remount,size=4G /dev/shm
#df -h | grep shm
tmpfs 4.0G 0 4.0G 0% /dev/shm
接着重新启动即可,然后进入数据库修改内存
$ sqlplus / as sysdba
SQL> show parameter target; 其中/dev/shm必须大于MEMORY_TARGET,需要做修改下面2个值,改完需要重新启动数据库
sql> alter system set memory_max_target=12288M scope=spfile;
sql> alter system set memory_target=12288M scope=spfile;


EM管理(oracle用户,其中一个节点执行即可,访问地址https://节点IP:1158/em 用户名密码用上文设置的system之类的)

$ emctl status dbconsole
$ emctl start dbconsole
$ emctl stop dbconsole
本地sqlplus连接
安装oracle客户端版修改tsnames.ora 
D:\develop\app\orcl\product\11.2.0\client_1\network\admin\tsnames.ora 
添加
ORCL =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.248.110)(PORT = 1521))
)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = orcl)
)
)12345678910这里的HOST写的是scan-ip
sqlplus sys/oracle@RAC_ORCL as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on 星期四 4月 14 14:37:30 2016
Copyright (c) 1982, 2010, Oracle. All rights reserved.
连接到:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> select instance_name, status from v$instance;INSTANCE_NAME STATUS

orcl1 OPEN

当开启第二个命令行窗口连接时,发现实例名为orcl2,可以看出,scan-ip的加入可以具有负载均衡的作用。

如果连接的时候有报错:ORA-12545
解决方案:
a.可以在本机hosts文件添加双机的vip
b.或者修改数据库的监听配置
SQL> show parameter local

NAME TYPE VALUE

local_listener string (ADDRESS=(PROTOCOL=TCP)(HOST=scan-vip)(PORT=1521))
parallel_force_local boolean FALSE修改local_listener主机名为VIP-IP地址
SQL> alter system set local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST= 192.168.149.202)(PORT=1521))' sid='orcl2';System altered.
SQL> alter system set local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST= 192.168.149.201)(PORT=1521))' sid='orcl1';
System altered.

容灾演练(模拟单机运行)
参考文档