Oracle11gR2集群安装

集群基础环境

Redhat7，Oracle11gR2

集群IP规划

IP	RAC1	RAC2
public IP	192.168.100.11	192.168.100.12
private IP	10.10.10.11	10.10.10.12
VIP	192.168.100.13	192.168.100.14
SCAN IP	192.168.100.15	192.168.100.15

集群共享存储规划

DATA	3 x 3G
FRA	1 x 2G
OCR	1 x 2G

1.本地yum配置

df -h
mount /dev/cdrom /mnt
cd /etc/yum.repos.d/
mkdir bak
mv CentOS-* bak/

vi local.repo
[centos]
name=centos
baseurl=file:///mnt	
gpgcheck=0
enable=1

yum clean all 

2.安装依赖包

yum -y install ntsysv
yum -y install system-config-securitylevel-tui
yum -y install system-config-network-tui
yum -y install authconfig-gtk
yum -y install system-config-keyboard
yum install compat-libcap1 smartmontools -y
yum install -y openssh-clients
yum install binutils* -y
yum install compat* -y
yum install elfutils* -y
yum install gcc* -y
yum install glibc* -y
yum install kernel* -y
yum install ksh* -y
yum install libaio* -y
yum install libgcc* -y
yum install libgomp* -y
yum install libstdc* -y
yum install make* -y
yum install sysstat* -y

Oracle安装包这里上传到/tmp目录下
yum install -y oracle-validated-1.0.0-28.el5.i386.rpm	该包另外上传
cvuqdisk包：cvuqdisk-1.0.9-1.rpm是在/tmp/grid/rpm/下
rpm -vih cvuqdisk-1.0.9-1.rpm

3.关掉防火墙

systemctl stop firewalld
systemctl disable firewalld
systemctl status firewalld

关闭selinux的方式
vi /etc/selinux/config
SELINUX=disabled

或

sed -ri 's/^(SELINUX=).*/\1disabled/g'  /etc/selinux/config 

4.添加主机信息

vi  /etc/hosts 

192.168.100.11 rac1 rac1-public
10.10.10.11 rac1-priv
192.168.100.13 rac1-vip

192.168.100.12 rac2 rac2-public
10.10.10.12 rac2-priv
192.168.100.14 rac2-vip

192.168.100.15 rac-scan

5.添加组和用户

groupadd -g 1000 oinstall 
groupadd -g 1200 asmadmin 
groupadd -g 1201 asmdba 
groupadd -g 1202 asmoper 
groupadd -g 1300 dba 
groupadd -g 1301 oper 
useradd -m -u 1100 -g oinstall -G asmadmin,asmdba,asmoper -d /home/grid -s /bin/bash grid 
useradd -m -u 1101 -g oinstall -G dba,oper,asmdba -d /home/oracle -s /bin/bash oracle 


确认用户信息：
[root@rac1 ~]# id oracle 
uid=502(oracle) gid=507(oinstall) groups=507(oinstall),502(dba),503(oper),506(asmdba) 
[root@rac1 ~]# id grid 
uid=1100(grid) gid=507(oinstall) groups=507(oinstall),504(asmadmin),506(asmdba),505(asmoper)
修改密码：
passwd oracle
passwd grid 

6.创建目录结构

mkdir -p /u01/app/11.2.0/grid
mkdir -p /u01/app/grid
chown -R grid:oinstall /u01 
mkdir -p /u01/app/oracle/11.2.0/db_1 
chown -R oracle:oinstall /u01/app/oracle 
chmod -R 775 /u01 

7.添加grid环境变量

注：二节点的ORACLE_SID

vi /home/grid/.bash_profile

export ORACLE_SID=+ASM1 
#export ORACLE_SID=+ASM2 
export ORACLE_HOME=/u01/app/11.2.0/grid
export ORACLE_BASE=/u01/app/grid
export PATH=$ORACLE_HOME/bin:$PATH:/usr/local/bin/:. 
export TEMP=/tmp export TMP=/tmp 
export TMPDIR=/tmp 
umask 022

8.添加oracle环境变量

注：二节点的ORACLE_SID

vi /home/oracle/.bash_profile

ORACLE_SID=dave1; 
export ORACLE_SID 
#ORACLE_SID=dave2; 
export ORACLE_SID ORACLE_UNQNAME=dave; 
export ORACLE_UNQNAME JAVA_HOME=/usr/local/java; 
export JAVA_HOME ORACLE_BASE=/u01/app/oracle; 
export ORACLE_BASE ORACLE_HOME=$ORACLE_BASE/11.2.0/db_1; 
export ORACLE_HOME ORACLE_TERM=xterm; 
export ORACLE_TERM NLS_DATE_FORMAT="YYYY:MM:DD HH24:MI:SS"; 
export NLS_DATE_FORMAT NLS_LANG=american_america.ZHS16GBK; 
export NLS_LANG TNS_ADMIN=$ORACLE_HOME/network/admin; 
export TNS_ADMIN ORA_NLS11=$ORACLE_HOME/nls/data; 
export ORA_NLS11 PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin:$ORA_CRS_HOME/bin 
PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin 
export PATH LD_LIBRARY_PATH=$ORACLE_HOME/lib 
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib 
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib 
export LD_LIBRARY_PATH CLASSPATH=$ORACLE_HOME/JRE CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib 
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib 
export CLASSPATH
THREADS_FLAG=native; 
export THREADS_FLAG 
export TEMP=/tmp 
export TMPDIR=/tmp
umask 022

9.资源限制

vi /etc/security/limits.conf

grid soft nproc 2047 
grid hard nproc 16384 
grid soft nofile 1024 
grid hard nofile 65536 
oracle soft nproc 2047 
oracle hard nproc 16384 
oracle soft nofile 1024 
oracle hard nofile 65536

10.修改login

vi /etc/pam.d/login

session required pam_limits.so

11.内核限制

vi /etc/sysctl.conf

fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576

kernel.shmall = 2097152
kernel.shmmax = 4294967295
kernel.shmmni = 4096
修改后让内核参数生效
sysctl -p

12.shell 的限制

vi /etc/profile

if [ /$USER = "oracle" ] || [ /$USER = "grid" ]; then 
    if [ /$SHELL = "/bin/ksh" ]; then 
        ulimit -p 16384 
        ulimit -n 65536 
    else 
        ulimit -u 16384 -n 65536 
    fi 
    umask 022 
fi

13.UDEV 进行持久化

linux 6：
for i in b c d e f;
do
echo "KERNEL==\"sd*\", BUS==\"scsi\", PROGRAM==\"/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/\$name\",RESULT==\"`/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/sd$i`\", NAME=\"asm-disk$i\", OWNER=\"grid\",GROUP=\"asmadmin\", MODE=\"0660\""     >> /etc/udev/rules.d/99-oracle-asmdevices.rules
done
 
生效
/sbin/start_udev

---------------------------------------------------------
linux 7：

for i in b c d;
do
echo "KERNEL==\"sd*\",  ENV{DEVTYPE}==\"disk\",SUBSYSTEM==\"block\", PROGRAM==\"/lib/udev/scsi_id -g -u -d \$devnode\",RESULT==\"`/usr/lib/udev/scsi_id -g -u  /dev/sd$i`\",RUN+=\"/bin/sh -c 'mknod /dev/asm-disk$i b  \$major \$minor; chown grid:asmadmin /dev/asm-disk$i; chmod 0660 /dev/asm-disk$i'\"">>/etc/udev/rules.d/99-oracle-asmdevices.rules
done


查看 rules 文件： 
cat /etc/udev/rules.d/99-oracle-asmdevices.rules

KERNEL=="sd?",SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name",RESULT=="1ATA_VBOX_HARDDISK_VBccd57a70-a527fd7f", SYMLINK+="asm-diskb",OWNER="grid", GROUP="asmadmin",MODE="0660"
KERNEL=="sd?",SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name",RESULT=="1ATA_VBOX_HARDDISK_VB3465839e-60fc8676", SYMLINK+="asm-diskc",OWNER="grid", GROUP="asmadmin",MODE="0660"
KERNEL=="sd?",SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name",RESULT=="1ATA_VBOX_HARDDISK_VBd87076a8-c1d17bbb", SYMLINK+="asm-diskd",OWNER="grid", GROUP="asmadmin",MODE="0660"

# 加载rules文件,重新加载udev rule
/sbin/udevadm control --reload
# 检查新的设备名称
/sbin/udevadm trigger --type=devices --action=change

配置互信

使用root用户
cd /tmp/datebase/sshsetup
./sshUserSetup.sh -user grid -hosts "rac1 rac2" -advanced -noPromptPassphrase 

报错1：
Failed to start the Clusterware. Last 20 lines of the alert log follow: 
2020-12-29 16:22:03.055: 
[client(17339)]CRS-2101:The OLR was formatted using version 3.

原因是：redaht6和7的启动方式不同

解决步骤:

touch /usr/lib/systemd/system/ohas.service;
chmod 777 /usr/lib/systemd/system/ohas.service;
vi /usr/lib/systemd/system/ohas.service
添加：
[Unit]
Description=Oracle High Availability Services
After=syslog.target

[Service]
ExecStart=/etc/init.d/init.ohasd run >/dev/null 2>&1 Type=simple
Restart=always

[Install]
WantedBy=multi-user.target


systemctl daemon-reload;
systemctl enable ohas.service;
systemctl start ohas.service;
systemctl status ohas.service;

报错2：
【Oracle】Linux7安装11g
Error in invoking target 'agent nmhs' of makefile
 '/u01/app/oracle/11.2.0/db_1/sysman/lib/ins_emagent.mk'. 
 See '/u01/app/oraInventory/logs/installActions2021-04-23_11-49-31AM.log' 
 for details.

解决方案
在makefile中添加链接libnnz11库的参数
修改$ORACLE_HOME/sysman/lib/ins_emagent.mk，将
$(MK_EMAGENT_NMECTL)修改为：$(MK_EMAGENT_NMECTL) -lnnz11
建议修改前备份原始文件
cd $ORACLE_HOME/sysman/lib
cp ins_emagent.mk ins_emagent.mk.bak
vi ins_emagent.mk
进入vi编辑器后  命令模式输入/NMECTL 进行查找，快速定位要修改的行
在后面追加参数-lnnz11        第一个是字母l   后面两个是数字1

#===========================

$(SYSMANBIN)emdctl:
        $(MK_EMAGENT_NMECTL) -lnnz11 

#===========================


在点Retry

13.安装集群软件

14.添加共享磁盘

15.建库

等待完成即可