非Kubernetes工作负载（RHEL7.5/7.8-CentOS7.9服务器）加入网格（续2）

原创

山西空管技术支持 2024-01-09 09:12:58 博主文章分类：服务网格 ©著作权

文章标签 bc centos 服务器 文章分类 运维

©著作权归作者所有：来自51CTO博客作者山西空管技术支持的原创作品，请联系作者获取转载授权，否则将追究法律责任

摘要：

make；gcc；glibc；nmap；istio-sidecar；nslookup

***:CentOS 7.9的配置步骤与RHEL7.8完全一样！

1、RHEL7.5服务器运行istio报错___未解决！

RHEL7.5服务器指AirNet系统的FDP1服务器，运行在vsphere虚拟机上；centos7默认的glibc函数库的版本为2.17（centos7.9也是ldd (GNU libc) 2.17），无法运行一些对glibc版本有要求的中间件。

root@debian:/home# /lib/x86_64-linux-gnu/libc.so.6     <---debian的glibc版本
GNU C Library (Debian GLIBC 2.36-9+deb12u3) stable release version 2.36
Compiled by GNU CC version 12.2.0.
[root@FDP1 build]# cat /proc/version
Linux version 3.10.0-862.el7.x86_64 (mockbuild@x86-034.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC) ) #1 SMP Wed Mar 21 18:14:51 EDT 2018
[root@FDP1 build]# gcc -v
gcc 版本 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC)
[root@FDP1 build]# make -v
GNU Make 3.82
[root@FDP1 bin]#  tail /var/log/istio/istio.err.log /var/log/istio/istio.log -Fq -n 100
/usr/local/bin/envoy: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /usr/local/bin/envoy)
/usr/local/bin/envoy: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /usr/local/bin/envoy)
/usr/local/bin/envoy: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /usr/local/bin/envoy)
/usr/local/bin/envoy: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by /usr/local/bin/envoy)
2024-01-08T23:51:06.855978Z     error   Envoy exited with error: exit status 1

（某博客提示）对于centos系统，目前官方声称只运行centos8(截止istio1.11.2），但经过测试发现 centos7.6 .1801 也可以支持，只是需要安装一个glibc 扩展包即可。
Glibc 不能在源码目录中编译，它必须在一个额外分开的目录中编译。这样在编译发生错误的时候，就可以删除整个编译目录重新开始--->"mkdir build && cd build"
安装GLIBC_2.18后，报错少一条“version `GLIBC_2.18' not found (required by /usr/local/bin/envoy)”

# tar -xvzf glibc-2.18.tar.gz
# mkdir build && cd build
# ../configure --prefix=/usr --disable-profile --enable-add-ons --with-headers=/usr/include --with-binutils=/usr/bin
# make & make install
[root@FDP1 bin]#  tail /var/log/istio/istio.err.log /var/log/istio/istio.log -Fq -n 100
/usr/local/bin/envoy: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /usr/local/bin/envoy)
/usr/local/bin/envoy: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /usr/local/bin/envoy)
/usr/local/bin/envoy: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by /usr/local/bin/envoy)
2024-01-08T23:58:19.345534Z     error   Envoy exited with error: exit status 1

（某博客提示）可直接升级到glibc2.31(升级过程中会自动安装缺失的版本)。升级glibc2.31前需要确保基础环境满足以下条件：gcc版本要在9以上(默认4.8.5)，make版本要在4.0以上(默认3.82)。升级gcc和make。
当glibc版本为2.17时千万不要直接升级到2.25！！！2.17与2.25直接差4个版本(2.18、2.22、2.23、2.24)，经反复测试确认发现直接升级到2.25时不会自动安装缺失的版本，而2.25又对之前的版本有依赖(个人猜测)，强行安装2.25不但安装失败，且会造成系统崩溃、异常(比如无法使用ls、cp等命令，无法进行远程连接)。进行make升级

# yum -y install gcc gcc+  //安装依赖
# wget https://mirrors.aliyun.com/gnu/make/make-4.3.tar.gz
# tar -xf make-4.3.tar.gz
# cd make-4.3
# mkdir build
# ./configure --prefix=/opt/make  //指定安装到具体的目录下，此示例表示将make安装到/opt下
# make && make install
[root@FDP1 make-4.3]# ln -sf /opt/make/bin/make /usr/bin/make
[root@FDP1 make-4.3]# make -v
GNU Make 4.3
[root@FDP1 istio-vm]# ls
ls: relocation error: /lib64/libpthread.so.0: symbol h_errno, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

出现问题，大部分命令使用报错，按（记一次centos curl升级引发的灾难 - 知乎 (zhihu.com)）步骤执行后，部分命令OK，例如ls，但是系统问题还是很多。

2、RHEL7.8 按博客（centos7升级glibc2.25避坑指南_glibc升级-CSDN博客）安装make,gcc,glibc，OK!!!

[root@Redhat7 usr]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.8 (Maipo)
[root@Redhat7 usr]# cat /proc/version
Linux version 3.10.0-1127.el7.x86_64 (mockbuild@x86-034.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) ) #1 SMP Tue Feb 18 16:39:12 EST 2020
--->make升级
# wget https://ftp.gnu.org/gnu/make/make-4.3.tar.gz --no-check-certificate
# tar xf make-4.3.tar.gz
# yum -y install gcc gcc+
# mkdir build
# cd build
#  ../configure --prefix=/opt/make
# make && make install
#  ln -sf /opt/make/bin/make /usr/bin/make
[root@Redhat7 build]# make -v
GNU Make 4.3
--->gcc升级
# yum install -y gcc-c++ glibc-devel mpfr-devel libmpc-devel gmp-devel glibc-devel.i686
tar -xf gcc-9.3.0.tar.gz
# cd gcc-9.3.0
# ./contrib/download_prerequisites
# mkdir build
# cd build
#  ../configure --enable-checking=release --enable-language=c,c++ --disable-multilib --prefix=/usr
# make -j8
# make install
//若安装目录不是/usr，则需要在编译完成后配置环境变量、建立软连接,配置环境变量
[root@Redhat7 build]# gcc -v
配置为：../configure --enable-checking=release --enable-language=c,c++ --disable-multilib --prefix=/usr
gcc 版本 9.3.0 (GCC) 
--->glibc升级
[root@Redhat7 usr]# strings /lib64/libc.so.6 | grep -E "^GLIBC" | sort -V -r | uniq
GLIBC_PRIVATE
GLIBC_2.17
GLIBC_2.16
GLIBC_2.15 
......
[root@Redhat7 istio]# wget https://mirrors.aliyun.com/gnu/glibc/glibc-2.31.tar.gz
[root@Redhat7 istio]# tar -xf glibc-2.31.tar.gz
[root@Redhat7 istio]# cd glibc-2.31
[root@Redhat7 glibc-2.31]# cat INSTALL | grep -E "newer|later" | grep "*"
   * GNU 'make' 4.0 or newer
   * GCC 6.2 or newer
   * GNU 'binutils' 2.25 or later
   * GNU 'texinfo' 4.7 or later
   * GNU 'bison' 2.7 or later
   * GNU 'sed' 3.02 or newer
   * Python 3.4 or later
   * GDB 7.8 or later with support for Python 2.7/3.4 or later
   * GNU 'gettext' 0.10.36 or later
//以上查看安装glibc的前提依赖，对于不满足的依赖需要进行升级，使用yum -y install xxx 升级或安装即可
[root@Redhat7 glibc-2.31]# yum install binutils texinfo bison sed gettext
[root@Redhat7 build]# mkdir build & cd build
[root@Redhat7 build]# ../configure  --prefix=/usr --disable-profile --enable-add-ons --with-headers=/usr/include --with-binutils=/usr/bin --disable-sanity-checks --disable-werror
configure: error: 
*** These critical programs are missing or too old: python
[root@Redhat7 build]# yum install python3
# make -j8
# make install
/usr/bin/ld: cannot find -lnss_test2
collect2: error: ld returned 1 exit status
Execution of gcc -B/usr/bin/ failed!
The script has found some problems with your installation!
Please read the FAQ and the README file and check the following:
- Did you change the gcc specs file (necessary after upgrading from
  Linux libc5)?
- Are there any symbolic links of the form libXXX.so to old libraries?
  Links like libm.so -> libm.so.5 (where libm.so.5 is an old library) are wrong,
  libm.so should point to the newly installed glibc file - and there should be
  only one such link (check e.g. /lib and /usr/lib)
You should restart this script from your build directory after you've
fixed all problems!
Btw. the script doesn't work if you're installing GNU libc not as your
primary library!
make[1]: *** [Makefile:120: install] Error 1
make[1]: Leaving directory '/home/istio/glibc-2.31'
make: *** [Makefile:12：install] 错误 2
//博客中提示此错误可忽略，以上未处理；以下解决新启动远程终端时报一个WARNING
# make localedata/install-locales
root@Redhat7 ~]# strings /lib64/libc.so.6 | grep -E "^GLIBC" | sort -V -r | uniq
GLIBC_PRIVATE
GLIBC_2.30
GLIBC_2.29
......
GLIBC_2.2.5

make install时出现报错，博客中提示此错误可忽略，未做处理。
当glibc版本为2.17时千万不要直接升级到2.25（将会出现操作系统崩溃的情况，如：大部分命令不可用、无法远程登录、yum报错等）！！！2.17与2.25直接差4个版本(2.18、2.22、2.23、2.24)，经反复测试确认发现直接升级到2.25时不会自动安装缺失的版本，而2.25又对之前的版本有依赖(个人猜测)，强行安装2.25不但安装失败，且会造成系统崩溃、异常(比如无法使用ls、cp等命令，无法进行远程连接)。经反复测试确认，可直接升级到glibc2.31(升级过程中会自动安装缺失的版本)。

3、RHEL7.8运行istio

# rpm -ivh istio-sidecar.rpm     //重装前需删除 yum erase  istio-sidecar
# export CLUSTER="kubernetes"  CLUSTER_NETWORK=""  VM_NETWORK="" SERVICE_ACCOUNT="airnet-fdp1" WORK_DIR="/home/fdp1-airnet" VM_APP="airnet-fdp1" VM_NAMESPACE="vm-airnet"
# kubectl create serviceaccount "${SERVICE_ACCOUNT}" -n "${VM_NAMESPACE}"
# istioctl x workload entry configure -f workloadgroup.yaml -o "${WORK_DIR}" --clusterID "${CLUSTER}" --autoregister --namespace vm-airnet --internalIP 192.168.31.159 --ingressService istio-eastwestgateway.istio-system.svc --ingressIP 192.168.31.212 --capture-dns
# scp ./* 192.168.31.159:/home/fdp1-airnet/
# systemctl daemon-reload & systemctl restart istio
[root@Redhat7 fdp1-airnet]# systemctl status istio
   Active: active (running) since 二 2024-01-09 13:02:23 CST; 10ms ago
[root@Redhat7 fdp1-airnet]#  tail /var/log/istio/istio.err.log /var/log/istio/istio.log -Fq -n 100
2024-01-09T05:06:17.890387Z     error   failed to start xds proxy: failed to listen on unix socket "etc/istio/proxy/XDS": listen unix etc/istio/proxy/XDS: bind: no such file or directory
[root@Redhat7 istio]# tailf /var/log/istio/istio.log
2024-01-09T07:13:50.424620Z     info    xdsproxy        connected to upstream XDS server: istiod.istio-system.svc:27537
2024-01-09T07:13:50.433760Z     warn    xdsproxy        upstream [22] terminated with unexpected error rpc error: code = FailedPrecondition desc = auto-registration WorkloadEntry of Redhat7.vm-airnet failed: cannot find WorkloadGroup vm-airnet/airnet-fdp1
2024-01-09T07:14:11.326758Z     info    xdsproxy        connected to upstream XDS server: istiod.istio-system.svc:27537
2024-01-09T07:14:11.678918Z     info    ads     ADS: new connection for node:Redhat7.vm-airnet-2
2024-01-09T07:14:11.678928Z     info    ads     ADS: new connection for node:Redhat7.vm-airnet-1
2024-01-09T07:14:11.679278Z     info    cache   returned workload certificate from cache        ttl=23h56m39.320729897s
2024-01-09T07:14:11.679283Z     info    cache   returned workload trust anchor from cache       ttl=23h56m39.320722815s
2024-01-09T07:14:11.680768Z     info    ads     SDS: PUSH request for node:Redhat7.vm-airnet resources:1 size:1.1kB resource:ROOTCA
2024-01-09T07:14:11.680775Z     info    ads     SDS: PUSH request for node:Redhat7.vm-airnet resources:1 size:4.0kB resource:default
2024-01-09T07:14:13.910510Z     info    Readiness succeeded in 3m23.220497697s
# k apply -f workloadgroup.yaml
[root@k8s-master01 airnet-fdp1]#  k -n vm-airnet get workloadentry
NAME                          AGE   ADDRESS
airnet-fdp1-192.168.31.159    27s   192.168.31.159
# rpm -qf `which nslookup`
bind-utils-9.11.4-26.P2.el7_9.15.x86_64
# yum install bind-utils
[root@Redhat7 fdp1-airnet]# nslookup helloworld.vm-airnet.svc
Server:         114.114.114.114
Address:        114.114.114.114#53
Name:   helloworld.vm-airnet.svc
Address: 10.16.75.100
[root@Redhat7 istio]# yum install net-tools
# route add -net 172.29.115.128/26 gw  192.168.31.217  //所有k8s主机的路由
[root@Redhat7 fdp1-airnet]# curl helloworld.vm-airnet.svc:5000/hello
Hello version: v1, instance: helloworld-v1-7cb486975f-hqkdh

与服务网格连通测试:"upstream connect error"
Linux Redhat7.8安装Source RPM nmap-7.94.tar.bz2 (includes Nmap, Zenmap, Ncat, and Nping)报错“nse_openssl.h:8:10: 致命错误：openssl/bn.h：没有那个文件或目录”

[root@Redhat7 ~]# yum install nmap
[root@Redhat7 istio]# nc -lvv 8888
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Listening on :::8888
Ncat: Listening on 0.0.0.0:8888
[root@k8s-master01 airnet-fdp1]#   k -n vm-airnet get workloadentry  airnet-fdp1-192.168.31.159 -oyaml
    status: "False"
    type: Healthy
[root@Redhat7 istio]# ./forum-linux-amd64              --->status: "True"
Server is listening in port:8080
[root@k8s-master01 airnet-fdp1]# k -n vm-airnet exec -ti tool-97f9cfd66-4j9b2  -- curl fdp1.vm-airnet.svc:/api/users
upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: delayed connect error: 113
[root@k8s-master01 ~]# traceroute 192.168.31.159
traceroute to 192.168.31.159 (192.168.31.159), 30 hops max, 60 byte packets
 1  192.168.31.159 (192.168.31.159)  0.452 ms !X  0.405 ms !X  0.378 ms !X
[root@k8s-master01 ~]# curl 192.168.31.159:8080/api/users
curl: (7) Failed connect to 192.168.31.159:8080; 没有到主机的路由

traceroute：！X（通信管理禁止） !X (communication administratively prohibited)（VirualBox虚拟机服务无法访问——No route to host 问题分析 - 知乎 (zhihu.com)）
关闭防火墙测试OK:(error: 113Requirements for IP Version 4 Routers_does the router need to fragment the datagram)
默认情况下，Istio CA 会生成一个自签名的根证书和密钥，并使用它们来签署工作负载证书。Istio 向 Pod 中注入的 Envoy 代理中有两个进程——envoy 和 pilot-agent。pilot-agent 为 Envoy 生成私钥，通过 UNIX Domain Socekt（UDS）使用 Secret Discovery Service（SDS）向 CA 发起证书签名请求（CSR），如果你没有配置 CA 插件的话，Istio 默认向 istiod 发起 CSR。（Istio 中的证书管理方式介绍 · Jimmy Song）

[root@Redhat7 fdp1-airnet]#  systemctl start firewalld.service    // systemctl disable firewalld.service
[root@k8s-master02 ~]# k -n vm-airnet exec -ti tool-97f9cfd66-4j9b2  -- curl fdp1.vm-airnet.svc.cluster.local:/api/users
upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: delayed connect error: 113
[root@Redhat7 ~]# systemctl stop firewalld.service   
// 假如以下测试异常的话，systemctl daemon-reload & systemctl restart istio 再测试OK
[root@k8s-master02 ~]# k -n vm-airnet exec -ti tool-97f9cfd66-4j9b2  -- curl fdp1.vm-airnet.svc.cluster.local:/api/users
upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: TLS_error:|268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER:TLS_error_end:TLS_error_end
# systemctl daemon-reload & systemctl restart istio
[root@k8s-master02 ~]# k -n vm-airnet exec -ti tool-97f9cfd66-4j9b2  -- curl -i fdp1.vm-airnet.svc.cluster.local:/api/users
HTTP/1.1 200 OK
[root@k8s-master01 ~]# curl -i 192.168.31.159:8080/api/users
HTTP/1.1 200 OK
[root@k8s-master01 ~]#  traceroute 192.168.31.159
traceroute to 192.168.31.159 (192.168.31.159), 30 hops max, 60 byte packets
 1  192.168.31.159 (192.168.31.159)  0.451 ms  0.402 ms  0.375 ms

或者执行如下命令(其实关闭防火墙一定程度上就等于允许所有包通过)

# iptables -P INPUT ACCEPT  //-P chain target, Change policy on chain to target -P :设置默认策略的（设定默认门是关着的还是开着的）
[root@Redhat7 ~]# iptables -nL -t filter |grep 'Chain INPUT'
Chain INPUT (policy ACCEPT)
[root@Redhat7 ~]# iptables -P INPUT DROP                        //设置默认策略
[root@Redhat7 ~]# iptables -nL -t filter |grep 'Chain INPUT'    //查看默认policy状态：DROP
Chain INPUT (policy DROP)
# iptables -F       //-F [chain]   Delete all rules in  chain or all chains。该命令会将所有规则和链都清空
[root@k8s-node04 ~]#  traceroute 192.168.31.159
traceroute to 192.168.31.159 (192.168.31.159), 30 hops max, 60 byte packets
 1  192.168.31.159 (192.168.31.159)  0.591 ms  0.535 ms  0.505 ms
[root@k8s-node04 ~]# curl -i 192.168.31.159:8080/api/users
HTTP/1.1 200 OK

4、CentOS Linux release 7.9.2009配置为ACC3席位，加入网格

 cd /home/airnet-acc3/
 export CLUSTER="kubernetes"  CLUSTER_NETWORK=""  VM_NETWORK="" SERVICE_ACCOUNT="airnet-acc3" WORK_DIR="/home/airnet-acc3" VM_APP="airnet-acc3" VM_NAMESPACE="vm-airnet"
 kubectl create serviceaccount "${SERVICE_ACCOUNT}" -n "${VM_NAMESPACE}"
 istioctl x workload entry configure -f workloadgroup.yaml -o "${WORK_DIR}" --clusterID "${CLUSTER}" --autoregister --namespace vm-airnet --internalIP 192.168.31.104 --ingressService istio-eastwestgateway.istio-system.svc --ingressIP 192.168.31.212 --capture-dns
 scp ./* 192.168.31.104:/home/airnet-acc3/
    mesh.yaml                         --->   discoveryAddress: istiod.istio-system.svc:27537
    /var/lib/istio/envoy/sidecar.env  --->   CA_ADDR=istiod.istio-system.svc:27537
 mkdir -p /etc/certs
 mkdir -p /var/run/secrets/tokens
 mkdir -p /etc/istio/proxy
  cp root-cert.pem /etc/certs/root-cert.pem
  cp  istio-token /var/run/secrets/tokens/istio-token
  cp  cluster.env /var/lib/istio/envoy/cluster.env
  cp  mesh.yaml /etc/istio/config/mesh
  chown -R istio-proxy /var/lib/istio /etc/certs /etc/istio/proxy /etc/istio/config /var/run/secrets /etc/certs/root-cert.pem
 
  /home/forum-linux-amd64 > /dev/null 2>&1 &
  systemctl daemon-reload & systemctl restart istio
  k -n vm-airnet exec -ti tool-97f9cfd66-4j9b2  -- curl acc3-istio.vm-airnet.svc.cluster.local:/api/healthz
  k -n vm-airnet exec -ti tool-97f9cfd66-4j9b2  -- curl acc3-istio.vm-airnet.svc.cluster.local:/api/users