我们的文章会在微信公众号“Oracle恢复实录”和博客网站“www.htz.pw” 同步更新 ,欢迎关注收藏,也欢迎大家转载,但是请在文章开始地方标注文章出处,谢谢!

由于博客中有大量代码,通过页面浏览效果更佳。

本篇文章为ASMFD系列第三篇:ASMFD的安装。前面两篇文档链接如下:

ASMFD系列一:ASMFD的介绍与常见问题

ASMFD系列二:AFD在CentOS环境中部署

在第一篇中我们说过ASMFD简化Oracle的配置、提供统一的磁盘名字和权限的管理、减少Oracle数据库对主机配置的依赖和提升数据库部署的易用性,ASMFD的安装同样是非常简单,一条命令就可以完成ASMFD的安装。ASMFD安装根据当前数据库环境是否运行分为初始化环境安装、已运行的环境安装,两种环境执行的安装命令不同,需要注意的事项也各不相同,下面将分别进行详细的说明。

1 已运行的环境

针对已经在运行集群的环境中,需要首先获取当前环境的固定磁盘名和修改磁盘权限的方案,不同的方案对应安装ASMFD时考虑的操作步骤各不一样。如asmlib环境中,因为ASMFD已经具备asmlib的功能,并且固定磁盘名字跟就是使用的asmlib的技术,即ASMFD与asmlib不兼容,在安装ASMFD之前需要卸载asmlib;在udev环境中,ASMFD与udev完全兼用,不需要卸载udev。其实ASMFD技术在修改权限时,其实是通过udev来实现的,并且Oracle自己的技术实现的。本节内容会比较多,会在后续专门出二篇博客来分别介绍asmlib与udev环境中安装ASMFD与迁移到ASMFD环境。

2 初始化安装

在上一节中,介绍了正在运行的环境中如何安装ASMFD,本节中将介绍在数据库软件安装时一起安装ASMFD。

2.1 确认内核版本与ASMFD的兼容性已经具备

ASMFD与操作系统类型和内核是完全绑定的,所以在确认需要部署的数据库版本后,需要确认此版本的ASMFD与操作系统内核是否完全兼容,是否得到Oracle认证,兼容列表参考MOS:MOS:ACFS Support On OS Platforms (Certification Matrix). (Doc ID 1369107.1),在本次测试中,数据库版本选择目前最新版本19.17,操作系统选择Oracle Linux 7.9.

2.2 集群解压和补丁升级

下面命令直接在root用户执行就可以

chown grid:oinstall /oracle
su - grid -c "mkdir -p \$ORACLE_HOME"
su - grid -c "unzip -qo /oracle/soft/oracleinstall/19c/LINUX.X64_193000_grid_home.zip -d \$ORACLE_HOME"
su - grid -c "unzip -qo /oracle/soft/oracleinstall/19c/patch/19.17_ojvm/p34449117_190000_Linux-x86-64.zip -d /tmp"
su - grid -c "unzip -qo /oracle/soft/oracleinstall/opatch/p6880880_190000_Linux-x86-64.zip -d \$ORACLE_HOME"jubei
su - grid -c "\$ORACLE_HOME/OPatch/opatch version"
su - grid -c "\$ORACLE_HOME/gridSetup.sh -applyRU /tmp/34449117/34416665"

2.3 验证ASMFD与系统内核的兼容性

通过afddriverstate就可以验证是否兼容,如果Linux系统为CentOS,请参考ASMFD系列二:AFD在CentOS环境中部署

[root@19crac1 /]# /oracle/app/19.3.0/grid/bin/afddriverstate supported
AFD-9200: Supported
[root@19crac1 /]# uname -r
5.4.17-2011.6.2.el7uek.x86_64

2.4 确认集群OCR磁盘

生产环境中ASM所使用的磁盘基本都是通过多路径软件生成的MD磁盘,千万记住,生产环境中,切勿直接使用非聚合的磁盘,否则磁盘一条链路异常就可能导致磁盘不可访问,数据库实例和ASM实例异常。说句题外话,之前见过很多客户的环境,虽然购买高端存储,也配置了多路径磁盘,但是ASM使用的磁盘是在udev里面配置的单路径磁盘,并没有使用多路径磁盘。为什么会出现这种情况呢?原因很简单,在传统的运维架构中,主机、存储、数据库是分别由不同的运维人员负责,也可能是由不同的三方厂商负责,每个厂商的职责划分也是相当明确的;首先主机厂商是负责主机安装操作,安装完系统后其它的就不管了;接下来存储厂商负责划盘,配置多路径,至于上面怎么部署Oracle RAC他不负责,是否需要通过udev来改磁盘的名字和权限跟他也没有关系;接下来到了数据库厂商,因为数据库安装大家都认为是一个很简单的事情,安装好能用就表示可以了,但是数据库集群安装,对安装人员要求具备主机、存储、数据库的能力并且还需要达到资深的级别。如果现场实施人员并非资深级别的人员,很难全部具备上面提到的能力。此时工程师就按照自己平时自己环境的测试记录(自己环境基本上无多多径)或者有一些是网站上随便找一篇博客照着配置udev就可以,安装上数据库,应用测试无问题,安装工作结束。其实这样的环境部署以后出问题的概率也不会太大,除非是后期光纤链路及所经过的硬件设备异常,数据库才会报IO错误,此时又因为本身链路异常,所以不会更多深入的去分析磁盘本身的高可用,而硬件成为了背锅侠。原来跟同事交流面试时,就常常说,拿Oracle数据库集群安装这个话题来面试候选者,可以大概知道候选者在数据库上的技术能力和候选者自己对知识的求知欲。虽然这个问题比较片面,但是也具备一些参考的价值。因为数据库集群的安装,不仅仅是数据库的知识,还需要对多种操作系统、存储及系统IO操作、多路径软件、SFRAC、HACMP等等知识点非常熟悉,这样才能搭建出一套真正具备高可用的数据库集群环境。这里就拿ASMFD技术来说,使用ASMFD技术后,不再需要对udev技术非常熟悉,这样就可减少数据库工程师跨知识域的学习Linux udev的知识。为了防止这些情况的发生,不管从甲方和乙方的角度,我们要重视数据库集群这一简单的安装工作,工作本身虽然简单,但是要求的知识面广且需要深入;工作本身虽然简单,但是如果实施不好,会留下大量的隐患,可能后期90%以上的故障都是由于安装不规范导致的。可通过制定安装标准、制定严格的测试场景和标准、严格的实施人员技术要求就可以改善这种情况。Oracle也在不断的从产品、技术方案上给客户提供一条龙的解决方案,杜绝再次发生上述的情况。如Oracle自己的一体机解决方案,就根本不需要考虑上述的问题,在技术上Oracle一体机有自己全套的技术,在初始化上线的时候,全部都已经配置完成;在人员分工责任上,Oracle一体机将主机、存储、数据库工程师三和一,不再有责任不明确的现象;在实施和流程上,一体机具有自动化实施部署工具来简化部署和统一部署标准,有专有的流程来保证实施的合理性和技术的正确性。如Oracle Acs服务团队推出的数据库安装和混沌工程解决方案,有成套的标准实施方案和测试场景,帮助客户建立一个最佳的数据库平台。本次测试环境中,采用Linux自带的mpath多路径技术,多路径非本篇文章的重点,暂时先跳过,如果感兴趣可以去www.htz.pw搜。通过mulltipathd -ll后,三个OCR的磁盘信息分别如下:

/dev/mapper/ocr1
 /dev/mapper/ocr2
 /dev/mapper/ocr3

2.5 配置AFD磁盘标签

通过asmcmd afd_label命令,结合--init参数就可以完成对磁盘打AFD的标签,如果遇到已经是AFD磁盘了,可以通过添加--migrate参数。

[root@19crac1 34416665]# export ORACLE_HOME=/oracle/app/19.3.0/grid
[root@19crac1 34416665]# export ORACLE_BASE=/tmp
[root@19crac1 34416665]# export PATH=$ORACLE_HOME/bin:$PATH
[root@19crac1 34416665]# asmcmd afd_label OCR1 /dev/mapper/ocr1 --init
Creating diag directory /tmp/diag
disk /dev/mapper/ocr1 is already provisioned for ASM


[root@19crac1 34416665]# asmcmd afd_label OCR1 /dev/mapper/ocr1 --migrate --init
[root@19crac1 34416665]# asmcmd afd_label OCR2 /dev/mapper/ocr2 --migrate --init
[root@19crac1 34416665]# asmcmd afd_label OCR3 /dev/mapper/ocr3 --migrate --init
查看生产的磁盘路径信息
[root@19crac1 34416665]# ls -l /dev/oracleafd/disks/*
-rw-rw-r--. 1 grid oinstall 39 Dec 30 17:19 /dev/oracleafd/disks/OCR1
-rw-rw-r--. 1 grid oinstall 39 Dec 30 17:19 /dev/oracleafd/disks/OCR2
-rw-rw-r--. 1 grid oinstall 39 Dec 30 17:19 /dev/oracleafd/disks/OCR3

2.6 配置集群与安装ASMFD

由于资源有限,这个环境中的集群为单机集群版本,在ASMFD安装上,单机和RAC版本操作都是一样的,无任何差异。

如需要启用ASMFD功能,需要添加_oracle.install.asm.configureAFD=true_,同时在安装GRID时,会扫描磁盘,所以此时需要GRID用户对磁盘具有读的权限。本次演示仅通过操作系统chown来修改磁盘的属主,并没有使用udev来修改磁盘的属主与权限,原因很简单,一次性使用,那个简单实用那个。

[root@19crac1 network-scripts]# chown grid:oinstall /dev/mapper/ocr[1,2,3][root@19crac1 network-scripts]# ls -l /dev/mapper/ocr[1,2,3]lrwxrwxrwx. 1 root root 7 Dec 30 17:37 /dev/mapper/ocr1 -> ../dm-9lrwxrwxrwx. 1 root root 7 Dec 30 17:37 /dev/mapper/ocr2 -> ../dm-8lrwxrwxrwx. 1 root root 7 Dec 30 17:37 /dev/mapper/ocr3 -> ../dm-5[root@19crac1 network-scripts]# ls -l /dev/dm-9brw-rw----. 1 grid oinstall 252, 9 Dec 30 17:37 /dev/dm-9
$ORACLE_HOME/gridSetup.sh -silent -skipPrereqs -waitForCompletion  -printtime  \INVENTORY_LOCATION=/oracle/app/oraInventory \SELECTED_LANGUAGES=en \oracle.install.option=CRS_CONFIG \ORACLE_BASE=/oracle/app/grid \ORACLE_HOME=/oracle/app/19.3.0/grid \oracle.install.asm.OSDBA=dba \oracle.install.asm.OSOPER=oinstall \oracle.install.asm.OSASM=oinstall \oracle.install.crs.config.scanType=LOCAL_SCAN \oracle.install.crs.config.gpnp.scanName=rac19c-scan \oracle.install.crs.config.gpnp.scanPort=1521 \oracle.install.crs.config.gpnp.configureGNS=false \oracle.install.crs.config.ClusterConfiguration=STANDALONE \oracle.install.crs.config.clusterName=rac19ccluster \oracle.install.crs.config.gpnp.configureGNS=false \oracle.install.crs.config.clusterNodes=19crac1:19crac1-vip \oracle.install.crs.config.networkInterfaceList=ens192:10.10.10.0:1,ens224:10.10.11.0:5 \oracle.install.crs.configureGIMR=false \oracle.install.asm.configureGIMRDataDG=false \oracle.install.asm.configureAFD=true \oracle.install.crs.configureRHPS=false \oracle.install.crs.config.storageOption=FLEX_ASM_STORAGE \oracle.install.crs.config.useIPMI=false \oracle.install.asm.SYSASMPassword=Htzoracle123 \oracle.install.asm.diskGroup.diskDiscoveryString=/dev/mapper/* \oracle.install.asm.diskGroup.name=rac19c_ocr \oracle.install.asm.diskGroup.disksWithFailureGroupNames=/dev/mapper/ocr1,,/dev/mapper/ocr2,,/dev/mapper/ocr3, \oracle.install.asm.diskGroup.disks=/dev/mapper/ocr1,/dev/mapper/ocr2,/dev/mapper/ocr3 \oracle.install.asm.diskGroup.redundancy=NORMAL \oracle.install.asm.diskGroup.AUSize=4 \oracle.install.asm.monitorPassword=Htzoracle123 \oracle.install.asm.configureGIMRDataDG=false

后续操作为执行root.sh和executeConfigTools,跟普通环境一样,此处省略。

2.7 查看ASMFD标签

查询ASM的ASMFD磁盘标签

[root@19crac1 ~]# ls -l /dev/dm*
brw-rw---- 1 root disk 252,  0 Dec 31 02:46 /dev/dm-0
brw-rw---- 1 root disk 252,  1 Dec 31 02:46 /dev/dm-1
brw-rw---- 1 root disk 252, 10 Dec 31 02:46 /dev/dm-10
brw-rw---- 1 root disk 252,  2 Dec 31 02:46 /dev/dm-2
brw-rw---- 1 root disk 252,  3 Dec 31 02:46 /dev/dm-3
brw-rw---- 1 root disk 252,  4 Dec 31 02:46 /dev/dm-4
brw-rw---- 1 root disk 252,  5 Dec 31 02:46 /dev/dm-5
brw-rw---- 1 root disk 252,  6 Dec 31 02:46 /dev/dm-6
brw-rw---- 1 root disk 252,  7 Dec 31 02:46 /dev/dm-7
brw-rw---- 1 root disk 252,  8 Dec 31 02:46 /dev/dm-8
brw-rw---- 1 root disk 252,  9 Dec 31 02:46 /dev/dm-9

[grid@19crac1 ~]$ asmcmd afd_lsdsk
--------------------------------------------------------------------------------
Label                     Filtering   Path
================================================================================
OCR1                        ENABLED   /dev/mapper/ocr1
OCR2                        ENABLED   /dev/mapper/ocr2
OCR3                        ENABLED   /dev/mapper/ocr3

[grid@19crac1 ~]$ asmcmd lsdsk --discovery
Path
AFD:OCR1
AFD:OCR2
AFD:OCR3

2.8 ASMFD环境磁盘搜索路径

在ASMFD环境中,ASM在mount磁盘时,首先根据asm_diskstring搜索磁盘的路径,如果asm_diskstring中有AFD的关键字,磁盘ASM调用ASMFD接口,扫描ASMFD的磁盘。ASMFD再根据afd_diskstring参数配置的值,去扫描物理磁盘信息。如当前环境的配置信息如下:asm_diskstring参数配置和查询当前配置的值

[grid@19crac1 ~]$ asmcmd dsset '/dev/mapper/*','AFD:*','/dev/asm_*'
[grid@19crac1 ~]$ asmcmd dsget
parameter:/dev/mapper/*, AFD:*, /dev/asm_*
profile:/dev/mapper/*,AFD:*,/dev/asm_*

afd_diskstring参数当前配置的值

[grid@19crac1 ~]$ asmcmd afd_dsget
AFD discovery string: /dev/mapper/*

2.9 ASMFD安装日志

下面是从安装日志中选取部分ASMFD安装是比较重要的日志信息,感兴趣的可以读下一下。大概的过程是执行afdroot install -v执行ASMFD安装,安装过程中复制afd.ko、libafd19.so,加载模块文件,创建ASMFD的udev规则文件,复制afd启动文件,配置init.d服务

2022-12-30 17:58:37: The 'ROOTCRS_AFDINST' is either in START/FAILED state
2022-12-30 17:58:37: Installing AFD
2022-12-30 17:58:37: Executing cmd: /oracle/app/19.3.0/grid/bin/afdroot install -v
2022-12-30 17:58:55: Command output:
>  AFD-627: AFD distribution files found.
>  AFD-9155: Checking for existing 'oracleafd.ko' driver installation.
>  AFD-634: Removing previous AFD installation.
>  AFD-635: Previous AFD components successfully removed.
>  AFD-9294: updating file /etc/sysconfig/oracledrivers.conf
>  AFD-636: Installing requested AFD software.
>  AFD-626: AFD driver media location is '/oracle/app/19.3.0/grid/usm/install/Oracle/EL7UEK/x86_64/5.4.17-2011.0.7/5.4.17-2011.0.7-x86_64/bin'
>  AFD-9504: Copying file '/oracle/app/19.3.0/grid/usm/install/Oracle/EL7UEK/x86_64/5.4.17-2011.0.7/5.4.17-2011.0.7-x86_64/bin/oracleafd.ko' to the path '/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/extra/oracle/oracleafd.ko'
>  AFD-9504: Copying file '/oracle/app/19.3.0/grid/bin/afdboot' to the path '/sbin/afdboot'
>  AFD-637: Loading installed AFD drivers.
>  AFD-9321: Creating udev for AFD.
>  AFD-9323: Creating module dependencies - this may take some time.
>  AFD-9154: Loading 'oracleafd.ko' driver.
>  AFD-9504: Copying file '/oracle/app/19.3.0/grid/lib/libafd19.so' to the path '/opt/oracle/extapi/64/asm/orcl/1/'
>  AFD-649: Verifying AFD devices.
>  AFD-9156: Detecting control device '/dev/oracleafd/admin'.
>  AFD-638: AFD installation correctness verified.
>  AFD-9294: updating file /etc/sysconfig/oracledrivers.conf
>  End Command output
2022-12-30 17:58:55: Successfully executed 'afdroot install'
2022-12-30 17:58:55: AFD Diskstring: /dev/mapper/*
2022-12-30 17:58:55: Executing cmd: /oracle/app/19.3.0/grid/bin/asmcmd afd_scan
2022-12-30 17:58:56: init file = /oracle/app/19.3.0/grid/crs/init/afd.sles
2022-12-30 17:58:56: Copying file /oracle/app/19.3.0/grid/crs/init/afd.sles to /etc/init.d directory
2022-12-30 17:58:56: Setting afd permission in /etc/init.d directory
2022-12-30 17:58:56: Removing "/etc/rc.d/rc3.d/S96afd"
2022-12-30 17:58:56: Creating a link "/etc/rc.d/rc3.d/S96afd" pointing to /etc/init.d/afd
2022-12-30 17:58:56: Removing "/etc/rc.d/rc5.d/S96afd"
2022-12-30 17:58:56: Creating a link "/etc/rc.d/rc5.d/S96afd" pointing to /etc/init.d/afd
2022-12-30 17:58:56: Removing "/etc/rc.d/rc0.d/K15afd"
2022-12-30 17:58:56: Creating a link "/etc/rc.d/rc0.d/K15afd" pointing to /etc/init.d/afd
2022-12-30 17:58:56: Removing "/etc/rc.d/rc1.d/K15afd"
2022-12-30 17:58:56: Creating a link "/etc/rc.d/rc1.d/K15afd" pointing to /etc/init.d/afd
2022-12-30 17:58:56: Removing "/etc/rc.d/rc2.d/K15afd"
2022-12-30 17:58:56: Creating a link "/etc/rc.d/rc2.d/K15afd" pointing to /etc/init.d/afd
2022-12-30 17:58:56: Removing "/etc/rc.d/rc4.d/K15afd"
2022-12-30 17:58:56: Creating a link "/etc/rc.d/rc4.d/K15afd" pointing to /etc/init.d/afd
2022-12-30 17:58:56: Removing "/etc/rc.d/rc6.d/K15afd"
2022-12-30 17:58:56: Creating a link "/etc/rc.d/rc6.d/K15afd" pointing to /etc/init.d/afd
2022-12-30 17:58:56: The file afd has been successfully linked to the RC directories
2022-12-30 17:58:56: Invoking "/oracle/app/19.3.0/grid/bin/cluutil -ckpt -oraclebase /oracle/app/grid -writeckpt -name ROOTCRS_AFDINST -state SUCCESS"
2022-12-30 17:58:56: trace file=/oracle/app/grid/crsdata/19crac1/crsconfig/cluutil3.log
2022-12-30 17:58:56: Running as user grid: /oracle/app/19.3.0/grid/bin/cluutil -ckpt -oraclebase /oracle/app/grid -writeckpt -name ROOTCRS_AFDINST -state SUCCESS
2022-12-30 17:58:56: Removing file /tmp/7kiFGF1d_H
2022-12-30 17:58:56: Successfully removed file: /tmp/7kiFGF1d_H
2022-12-30 17:58:56: pipe exit code: 0
2022-12-30 17:58:56: /bin/su successfully executed

创建ASM的日志如下:
[main] [ 2022-12-30 18:00:30.268 CST ] [UsmcaLogger.logInfo:156]  /dev/mapper/ocr1 is AFDCapable
[main] [ 2022-12-30 18:00:30.268 CST ] [UsmcaLogger.logInfo:156]  Label predefined for /dev/mapper/ocr1 Label OCR1
[main] [ 2022-12-30 18:00:30.268 CST ] [UsmcaLogger.logInfo:156]  Final label OCR1 for device /dev/mapper/ocr1
[main] [ 2022-12-30 18:00:30.269 CST ] [UsmcaLogger.logInfo:156]  /dev/mapper/ocr2 is AFDCapable
[main] [ 2022-12-30 18:00:30.269 CST ] [UsmcaLogger.logInfo:156]  Label predefined for /dev/mapper/ocr2 Label OCR2
[main] [ 2022-12-30 18:00:30.269 CST ] [UsmcaLogger.logInfo:156]  Final label OCR2 for device /dev/mapper/ocr2
[main] [ 2022-12-30 18:00:30.269 CST ] [UsmcaLogger.logInfo:156]  /dev/mapper/ocr3 is AFDCapable
[main] [ 2022-12-30 18:00:30.269 CST ] [UsmcaLogger.logInfo:156]  Label predefined for /dev/mapper/ocr3 Label OCR3
[main] [ 2022-12-30 18:00:30.269 CST ] [UsmcaLogger.logInfo:156]  Final label OCR3 for device /dev/mapper/ocr3
****************
[main] [ 2022-12-30 18:00:30.269 CST ] [AFDToolUtil.getAFDToolBin:76]  Obtained path for AFDTool: /oracle/app/19.3.0/grid/bin/afdtool
[main] [ 2022-12-30 18:00:30.269 CST ] [AFDToolUtil.executeCommand:120]  AFDTool command to be executed: [/oracle/app/19.3.0/grid/bin/afdtool, -getdevlist, -nohdr]
[main] [ 2022-12-30 18:00:30.312 CST ] [AFDToolUtil.executeCommand:130]  Result obtained from AFDTool command execution:
OCR1                      /dev/mapper/ocr1
OCR2                      /dev/mapper/ocr2
OCR3                      /dev/mapper/ocr3
*****************8
[main] [ 2022-12-30 18:00:30.313 CST ] [UsmcaLogger.logInfo:156]  Is label OCR1 found true
[main] [ 2022-12-30 18:00:30.313 CST ] [UsmcaLogger.logExit:139]  Exiting oracle.sysman.assistants.usmca.backend.USMDiskGroupManager Method : isLabelFound
[main] [ 2022-12-30 18:00:30.313 CST ] [UsmcaLogger.logInfo:156]  Skipping label creation since it exists already OCR1
[main] [ 2022-12-30 18:00:30.313 CST ] [UsmcaLogger.logInfo:156]  Is label OCR2 found true
[main] [ 2022-12-30 18:00:30.313 CST ] [UsmcaLogger.logExit:139]  Exiting oracle.sysman.assistants.usmca.backend.USMDiskGroupManager Method : isLabelFound
[main] [ 2022-12-30 18:00:30.313 CST ] [UsmcaLogger.logInfo:156]  Skipping label creation since it exists already OCR2
[main] [ 2022-12-30 18:00:30.313 CST ] [UsmcaLogger.logInfo:156]  Is label OCR3 found true
[main] [ 2022-12-30 18:00:30.313 CST ] [UsmcaLogger.logExit:139]  Exiting oracle.sysman.assistants.usmca.backend.USMDiskGroupManager Method : isLabelFound
[main] [ 2022-12-30 18:00:30.313 CST ] [UsmcaLogger.logInfo:156]  Skipping label creation since it exists already OCR3
*******************
[main] [ 2022-12-30 18:00:30.346 CST ] [UsmcaLogger.logInfo:156]  Diskstring in createDG to be updated: '/dev/mapper/*','AFD:*'
[main] [ 2022-12-30 18:00:30.346 CST ] [UsmcaLogger.logInfo:156]   update param sql ALTER SYSTEM SET asm_diskstring='/dev/mapper/*','AFD:*' SID='*'
*******************

[main] [ 2022-12-30 18:00:30.433 CST ] [UsmcaLogger.logInfo:156]  CREATE DISKGROUP SQL: CREATE DISKGROUP rac19c_ocr NORMAL REDUNDANCY  DISK 'AFD:OCR1' SIZE 2048M
 DISK 'AFD:OCR2' SIZE 2048M
 DISK 'AFD:OCR3' SIZE 2048M
 ATTRIBUTE 'compatible.asm'='19.0.0.0','compatible.advm'='19.0.0.0','au_size'='4M'


网名:认真就输
博客地址 :http://www.htz.pw 51CTO地址 :https://blog.51cto.com/luoping


![图片](https://mmbiz.qpic.cn/mmbiz_png/4WNGGBpyqywCxsAe4hkmJ5sducCnxjWP6OL0lxBHnnAnAkp7aujZSFdCwUo7aEoCOo1D3L4UYaw4cN2cXwvJjQ/640?wx_fmt=png)