如何在云环境下部署oracle18c RAC(京东云公有云)

理论篇

安装oracle RAC　主要需要几个条件

• • 
    
• 私有网络和公共网络隔离（不在同一网段）
• 共享存储
• 网络需要支持mulitcast
  • 
    

我们可以利用京东云提供的基础设施解决以上问题

• • 
    
• 通过创建"创建无预设CIDR私有网络"的vpc进行网络隔离，实现public和private网段隔离
• 通过创建"多点挂载存储"实现ASM所需的共享存储
• 通过n2n点对点vpn实现multicast支持
  • 
    

实施篇

网络配置

• • 
    
• 创建“创建无预设CIDR私有网络”vpc
  控制台=>网络=>私有网络(vpc)=>创建=>选择"创建无预设CIDR私有网络"
• 创建子网
  分别创建两个不同网段的子网
  10.0.1.0/24
  192.168.1.0/24
  • 
    

主机创建

创建共享磁盘

控制台=>弹性计算=>云硬盘=>创建=>选择"多点挂载"（目前仅华东有此选项）

创建弹性网卡

192.168.1.10 rac1private

192.168.1.20 rac2private

修改hosts文件

vi /etc/hosts

#oracle rac public ip
10.0.0.10 rac1
10.0.0.20 rac2

#oracle rac vip
10.0.0.11 rac1vip
10.0.0.21 rac2vip

#oracle rac scanip
10.0.0.30 scan-rac

#oracle rac private ip
192.168.２.10 rac1private
192.168.２.20 rac2private

这里可以利用弹性网卡来绑定多个ip地址来实现public ip和vip

创建用户

​​https://docs.oracle.com/en/database/oracle/oracle-database/18/cwlin/identifying-an-oracle-software-owner-user-account.html#GUID-0A95F4B1-1045-455D-9897-A23012E4E27F​​

用户和组：
/usr/sbin/groupadd -g 54321 oinstall
/usr/sbin/groupadd -g 54322 dba
/usr/sbin/groupadd -g 54323 oper
/usr/sbin/groupadd -g 54324 backupdba
/usr/sbin/groupadd -g 54325 dgdba
/usr/sbin/groupadd -g 54326 kmdba
/usr/sbin/groupadd -g 54327 asmdba
/usr/sbin/groupadd -g 54328 asmoper
/usr/sbin/groupadd -g 54329 asmadmin
/usr/sbin/groupadd -g 54330 racdba

/usr/sbin/useradd -u 54321 -g oinstall -G dba,asmdba,backupdba,dgdba,kmdba,racdba,oper oracle
/usr/sbin/useradd -u 54331 -g oinstall -G dba,asmdba,asmoper,asmadmin grid

echo oracle | passwd --stdin oracle
echo oracle | passwd --stdin grid

11、目录：
mkdir -p /u01/app/18.3.0/grid
mkdir -p /u01/app/grid
mkdir -p /u01/app/oracle
chown -R grid:oinstall /u01
chown oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/

12、环境变量

节点1：
--grid
export ORACLE_SID=+ASM1
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=/u01/app/18.3.0/grid
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
#export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

--oracle用户
export ORACLE_SID=db1
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=/u01/app/oracle/product/18.3.0/db_1
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
#export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib


节点2：
--gird
export ORACLE_SID=+ASM2
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=/u01/app/18.3.0/grid
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
#export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

--oracle用户
export ORACLE_SID=db2
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=/u01/app/oracle/product/18.3.0/db_1
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
#export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

• 验证用户组

$ grep "oinstall" /etc/group
oinstall:x:54321:grid,oracle
You can then use the ID command to verify that the Oracle installation owners you intend to use have the Oracle Inventory group as their primary group. For example:

$ id oracle
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba), 
54323(oper),54324(backupdba),54325(dgdba),54326(kmdba),54327(asmdba),54330(racdba)

$ id grid
uid=54331(grid) gid=54321(oinstall) groups=54321(oinstall),54322(dba),
54327(asmdba),54328(asmoper),54329(asmadmin),54330(racdba)

操作系统参数

• vi /etc/sysctl.conf
  sysctl -p /etc/sysctl.d/*.conf

fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 67108864
kernel.shmmax = 274877906944
kernel.shmmni = 4096
#kernel.sem = 250 32000 100 128
kernel.sem = 5010 641280 5010 128
net.ipv4.ip_local_port_range = 9000 65500
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 16384 4194304
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576

• 限制文件：vi /etc/security/limits.conf

grid soft nproc 2047 
grid hard nproc 16384 
grid soft nofile 1024 
grid hard nofile 65536 
grid soft stack 10240 
oracle soft nproc 2047 
oracle hard nproc 16384 
oracle soft nofile 1024 
oracle hard nofile 65536 
oracle soft stack 10240
* soft memlock 524288000
* hard memlock 524288000

• • 
    
• 
  

vi /etc/pam.d/login 文件
session required pam_limits.so

vi /etc/fstab
tmpfs /dev/shm tmpfs size=5000m 0 0

/etc/sysconfig/network
NOZEROCONF=yes

vi /etc/profile

• 
  

if [ $USER = "oracle" ] || [ $USER = "grid" ]; then
    if [ $SHELL = "/bin/ksh" ]; then
        ulimit -p 16384
        ulimit -n 65536
    else
        ulimit -u 16384 -n 65536
    fi
    umask 022
fi

安装必要包

yum install -y gcc-c++
yum install -y bc
yum install -y binutils
yum install -y compat-libcap1
yum install -y compat-libstdc++-33.i686    
yum install -y compat-libstdc++-33.x86_64
yum install -y glibc-2.17-36.el7.i686
yum install -y glibc-2.17-36.el7.x86_64
yum install -y glibc-devel.i686
yum install -y glibc-devel.x86_64
yum install -y ksh
yum install -y libaio.i686
yum install -y libaio.x86_64
yum install -y libaio-devel.i686
yum install -y libaio-devel.x86_64
yum install -y libX11.i686
yum install -y libX11.x86_64
yum install -y libXau.i686
yum install -y libXau.x86_64
yum install -y libXi.i686
yum install -y libXi.x86_64
yum install -y libXtst.i686
yum install -y libXtst.el7.x86_64
yum install -y libgcc.i686
yum install -y libgcc.x86_64
yum install -y libstdc++.i686
yum install -y libstdc++
yum install -y libstdc++-devel.i686
yum install -y libstdc++-devel.x86_64
yum install -y libxcb.i686
yum install -y libxcb.x86_64
yum install -y make
yum install -y nfs-utils 
yum install -y net-tools
yum install -y python
yum install -y python-configshell
yum install -y python-rtslib
yum install -y python-six
yum install -y smartmontools
yum install -y sysstat
yum install -y targetcli

• 为保证xwindow正常显示的必要安装

yum groupinstall "X Window System" "Fonts" -y 

准备asm磁盘组（所有节点操作）

• asmlib方式

  参考文档：​​​https://docs.microsoft.com/zh-tw/azure/virtual-machines/workloads/oracle/configure-oracle-asm​​​

• • • 
      
  • 从 ​​​https://www.oracle.com/technetwork/server-storage/linux/asmlib/rhel7-2773795.html​​​下载rpm
  • 安装
    rpm -Uvh oracleasm-support-2.1.11-2.el7.x86_64.rpm
    yum -y install kmod-oracleasm.x86_64
    rpm -ivh oracleasmlib-2.0.12-1.el7.x86_64.rpm
  • 配置asm
    • 
      
• ​​/usr/sbin/oracleasm configure -i ​​​​Configuring the Oracle ASM library driver. This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets ('[]'). Hitting <ENTER> without typing an answer will keep that current value. Ctrl-C will abort. Default user to own the driver interface []: grid Default group to own the driver interface []: asmadmin Start Oracle ASM library driver on boot (y/n) [n]: y Scan for Oracle ASM disks on boot (y/n) [y]: y Writing Oracle ASM library driver configuration: done ​​此处需要reboot
• • 创建asm磁盘组
• ​​oracleasm createdisk ocr /dev/vdb1 oracleasm createdisk data /dev/vdc1 oracleasm scandisks oracleasm listdisks ​​
• • 清除磁盘数据

• dd if=/dev/zero of=/dev/vdb1 bs=1M count=1000
  
  

n2n安装及配置

• • 
    
• 安装
  ​​yum install -y gcc yum install -y openssl-devel ​​​​git clone https://github.com/ntop/n2n.git cd n2n git checkout 2.4-stable make make PREFIX=/opt/n2n install ​​
• 安装完毕后启动supernode服务
  ​​nohup /opt/n2n/sbin/supernode -l 65530 & ​​
• 在RAC节点安装ntop软件（计划加入vpn网络的节点都需要安装ntop）​​scp -r /opt/n2n 10.0.0.10:/opt scp -r /opt/n2n 10.0.0.20:/opt ​​
• • 节点1
• ​​/opt/n2n/sbin/edge -d edge1 -a 192.168.2.10 -s 255.255.255.0 -c dtstack -k dtstack -l 192.168.1.40:65530 -E -r ​​
• • 节点2

• /opt/n2n/sbin/edge -d edge1 -a 192.168.2.20 -s 255.255.255.0 -c dtstack -k  dtstack -l 192.168.1.40:65530 -E -r
  
  

• 测试multicast(非必要，主要是验证multicast无异常)
  ​​./mcasttest.pl -n rac1,rac2 -i edge1 ​​
  • 
    

ssh互信

• • 
    
• 分别在grid和oracle用户，每个节点上执行。
• • 创建ssh目录
• ​​mkdir .ssh chmod 700 ~/.ssh cd ~/.ssh ​​
• • 生成秘钥

• ssh-keygen -t rsa
  ssh-keygen -t dsa
  
  

• 在rac1上将密钥添加到授权文件中,grid和oracle用户分别执行
• • 本机的拷贝
• ​​cat id_dsa.pub >> authorized_keys cat id_rsa.pub >> authorized_keys ​​
• • 拷贝另一台主机的秘钥

• ssh rac2 cat ~/.ssh/id_dsa.pub >> authorized_keys
  ssh rac2 cat ~/.ssh/id_rsa.pub >> authorized_keys
  
  

• • 将本机的authorized_keys的文件拷贝到另一台主机上

• scp authorized_keys rac2:/home/grid/.ssh/
  scp authorized_keys rac2:/home/oracle/.ssh/
  
  

• 修改两个主机的授权密钥的权限
  ​​chmod 600 ~/.ssh/authorized_keys ​​
• 测试
• • 获取服务器公钥
• ​​ssh-keyscan rac1 >> ~/.ssh/known_hosts ssh-keyscan rac2 >> ~/.ssh/known_hosts ssh-keyscan rac1private >> ~/.ssh/known_hosts ssh-keyscan rac2private >> ~/.ssh/known_hosts ​​
• • 测试

• ssh rac1 date
  ssh rac2 date
  ssh rac1private date
  ssh rac2private date
  
  

  • 
    

安装前检查

$ORACLE_HOME/runcluvfy.sh stage -pre crsinst -n rac1,rac2 -verbose

• 错误处理
• • PRVE-0421
    修改/etc/fstab
• ​​tmpfs /dev/shm tmpfs size=5000m 0 0 ​​

安装GI--grid用户下

• • 
    
• 解压安装压缩包

cd /u01/app/12.2.0/grid/grid
unzip /oraclesoftware/linuxx64_12201_grid_home.zip

安装cvudisk RPM包
root 用户安装
cvuqdisk存于oracle GI安装介质的cv/rpm目录下

cd /u01/app/18.3.0/grid
export CVUQDISK_GRP=asmadmin
rpm -ivh cv/rpm/cvuqdisk-1.0.10-1.rpm

scp cv/rpm/cvuqdisk-1.0.10-1.rpm 10.0.0.20:/root
export CVUQDISK_GRP=asmadmin
rpm -ivh /root/cvuqdisk-1.0.10-1.rpm

grid用户执行$HOME_ORACLE/gridSetup.sh

• 
  

安装DB--oracle用户下

• • 
    
• 解压安装压缩包 ​​cd /u01/app/oracle/product/18.3.0/db_1 unzip /oraclesoftware/LINUX.X64_180000_db_home.zip ​​
• oracle用户执行$HOME_ORACLE/runInstaller.sh
  • 
    

查看crs服务状态

crsctl status res -t