1、将AirNet系统SDFP2服务器迁移到镜像。基础镜像（https://hub.docker.com/r/xplenty/rhel7-pod-infrastructure/tags）；遗留问题：网络及IP地址的配置，运行./mid不正常。
docker pull xplenty/rhel7-pod-infrastructure:v3.4 //基础镜像Red Hat Enterprise Linux Server release 7.3 (Maipo)
进入镜像安装应用程序和库文件
docker run -itd -h "sdfp2" -v /home/AirNet:/home/mi xplenty/rhel7-pod-infrastructure:v3.4 /bin/bash
docker exec -ti -u root 813e66551ded /bin/bash //-u root 使用root用户
执行脚本：install_zsam_20220120/install
运行进程mnt,按提示拷贝lib文件到/usr/lib64/，并建立软链接。
[root@813e66551ded bin]# /home/cdatc/AirNet/bin/mnt
./mnt: error while loading shared libraries: libnetsnmp.so.31: cannot open shared object file: No such file or directory
cp libgtop-2.0.so.10 /usr/lib64/
ln -s libgtop-2.0.so.10.0.0 libgtop-2.0.so.10
（libgtop-2.0.so.10，libnetsnmp.so.31.0.2，libGL.so.1.2.0，libxcb-dri3.so.0.0.0，libxcb-xfixes.so.0.0.0，
libxcb-present.so.0.0.0，libxcb-sync.so.1.0.0，libxshmfence.so.1.0.0，libglapi.so.0.0.0，libXext.so.6.4.0，
libXdamage.so.1.1.0，libXfixes.so.3.1.0，libX11-xcb.so.1.0.0，libX11.so.6.3.0，libxcb.so.1.1.0，libxcb-glx.so.0.0.0，
libxcb-dri2.so.0.0.0，libXxf86vm.so.1.0.0，libdrm.so.2.4.0，libXau.so.6.0.0，libpcap.so.1）
再次运行./mnt提示：
[ERROR] can not find this hostname from network.xml
lock failed:datatype no ini: Permission denied
[root@SDFP2 home]# docker run -itd -h "sdfp2" -v /home/AirNet:/home/mi airnet-sdfp2:v1.0 /bin/bash //使用-h "sdfp2"参数指定容器的hostname为SDFP2
Docker容器保存为镜像文件
docker commit 813e66551ded airnet-sdfp2:v1.0 //docker images名称不支持大写字母
docker save -o /home/airnet-sdfp2:v1.0.tar airnet-sdfp2:v1.0 //2.4G 需要优化
[root@sdfp2 bin]# ./bafp //bafp进程还需要这个库文件
./bafp: error while loading shared libraries: libpcap.so.1: cannot open shared object file: No such file or directory
！！！步骤梳理：
！-->新制作安装包：加了23个lib库文件，修改network.xml,license.ini（主机名修改为小写sdfp2）
# docker run -itd -h "sdfp2" -v /home/sdfp2:/home/mi xplenty/rhel7-pod-infrastructure:v3.4 /bin/bash
09cecf3a72f13b380be2c66686075737dbd435b101d126e336eb33fba8b2747b
# docker exec -ti -u root 09cecf3a72f1 /bin/bash
# /home/mi/install/install_zsam_20220120/install
##enter cdatc install...
Fri Feb 18 00:43:13 UTC 2022
......
installation finished
# docker commit -a "myname" -m "update lib64 network license" -p 09cecf3a72f1 sdfp2:v1.0 //-p在commit时，将容器暂停
[root@k8s-node03 AirNet]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
sdfp2 v1.0 26e8465213d1 33 seconds ago 2.5GB
# docker save -o /home/airnet-sdfp2:v1.1.tar airnet-sdfp2:v1.1 (2.5G)
！-->现象：假如宿主机磁盘占用率高，出现自动删除刚创建的该镜像的现象！清理磁盘空间后ok.
# k apply -f sdfp2-netshoot.yaml
# k exec -ti sdfp2-netshoot-dp-b5d887cb9-wxrct -c airnet-sdfp2 -- bash
[root@sdfp2 bin]# /usr/local/redis/bin/redis-server /usr/local/redis/conf/redis.conf 1>/dev/null 2>&1
[root@sdfp2 bin]# /usr/local/redis/bin/redis-server /usr/local/redis/conf/redis_6378.conf 1>/dev/null 2>&1
# bash -c ./mnt >/dev/null 2>&1 & //mnt会自动将mid（物理机主机名#hoatname sdfp2）,rfp,afp拉起来
关于k8s-node03多IP问题的补充（calico没法指定使用ens4多个ip中的其中一个，因为每个节点的IP都不一样，需通过设置网关实现）
kubectl set env daemonset/calico-node -n kube-system IP_AUTODETECTION_METHOD=can-reach=192.168.31.1 //这样设置后ok
替代以下的IP检测方法，因为使用interface的方法指定不了IP.
kubectl set env daemonset/calico-node -n kube-system IP_AUTODETECTION_METHOD=interface='enp.*|eth.*|en.*'
！-->docker build制作镜像，默认Dockerfile
FROM xplenty/rhel7-pod-infrastructure:v3.4
MAINTAINER "mi"
#LABEL version="1.0" description="first build" by="mi_zy"
WORKDIR "/home/"
ADD install_zsam_20220217.tar.gz .
COPY ./_install /home/install_zsam_20220120/mi
USER root
RUN chmod -R +x /home/install_zsam_20220120/mi
RUN ["install_zsam_20220120/mi"]
CMD ls /home/install_zsam_20220120/
#docker build -t mi:v1 . //生成镜像mi:v1，大小5.23GB
部署pod，运行OK!
apiVersion: v1
kind: Pod
metadata:
name: sdfp2-pod
spec:
hostNetwork: true
containers:
- image: mi:v1
name: sdfp2
imagePullPolicy: IfNotPresent
securityContext:
runAsUser: 0
command: ["/bin/sh","-c","--"]
args: ["/usr/local/redis/bin/redis-server /usr/local/redis/conf/redis.conf 1>/dev/null 2>&1 ; /usr/local/redis/bin/redis-server /usr/local/redis/conf/redis_6378.conf 1>/dev/null 2>&1 ;/home/cdatc/AirNet/bin/mnt >/dev/null 2>&1 ;"]
nodeSelector:
kubernetes.io/hostname: k8s-node03
修改k8s-node03节点的名称hostname MSDP2，和IP:192.168.5.53/24，部署# k apply -f sdfp2-build.yaml后，镜像作为MSDP2服务器运行ok! 以下示例是改成FDP2，实测也OK!
# vi /etc/sysconfig/network-scripts/ifcfg-ens4 //IPADDR=192.168.5.59
# nmcli c reload
# nmcli c up ens4
# hostname FDP2 //网络修改IP生效后(nmcli c up ens4),设置主机名
# k apply -f sdfp2-build.yaml
*:mnt运行后，自动启动mid进程，只有在2个redis-server进程都起来后，mnt才自动启动rfp,afp进程。
！-->制作YAML,kind: Pod hostNetwork: true ，
apiVersion: v1
kind: Pod
metadata:
name: sdfp2-pod
spec:
hostNetwork: true
containers:
- image: sdfp2:v1.0
name: sdfp2
imagePullPolicy: IfNotPresent
securityContext:
runAsUser: 0
command: ["/bin/sh","-c","--"]
args: ["/usr/local/redis/bin/redis-server /usr/local/redis/conf/redis.conf 1>/dev/null 2>&1 ; /usr/local/redis/bin/redis-server /usr/local/redis/conf/redis_6378.conf 1>/dev/null 2>&1 ;/home/cdatc/AirNet/bin/mnt >/dev/null 2>&1 ;"]
nodeSelector:
kubernetes.io/hostname: k8s-node03
！-->修改k8s-node03物理机主机名#hoatname sdfp2后，运行#k apply -f sdfp2-host.yaml正常。
——Pod（sdfp2-host.yaml）运行后，由于k8s-node03宿主机有多个IP，组播默认使用网卡ens4配置的第一个IP，所以需要修改ifcfg-ens4配置文件，将sdfp2指定的IP(192.168.5.51)配置为第一个IP。192.168.5.51.47974 > 225.1.0.2.40202，否则不可用。
[root@k8s-node03 sdfp2]# tcpdump -n -i ens4 port 40202
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens4, link-type EN10MB (Ethernet), capture size 262144 bytes
23:07:13.494442 IP 192.168.5.51.47974 > 225.1.0.2.40202: UDP, length 36
23:07:14.194037 IP 192.168.5.51.43067 > 225.1.0.2.40202: UDP, length 1452
——后测试，Pod组播使用网卡属于别的网段第一个IP，SMC监控和SDD雷达信号也OK!{因为是“组播业务”}，但是该网卡的多个IP中必须存在network.xml指定的IP(192.168.5.51->到组播255.1.0.2；
192.168.6.51->到组播255.2.0.2)，mid进程由此决定使用不同的组播地址输出，只是组播出来时数据包的源IP会转而使用网卡ens4配置的第一个IP（192.168.31.10）
# tcpdump -n -i eth0 port 40202 |grep 192.168.31.10
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:36:48.605724 IP 192.168.31.10.60480 > 225.1.0.2.40202: UDP, length 874
15:36:48.605882 IP 192.168.31.10.60480 > 225.1.0.2.40202: UDP, length 904
15:36:48.705841 IP 192.168.31.10.41965 > 225.1.0.2.40202: UDP, length 36
Pod退出时IP:192.168.31.10会退出leave之前加入的组播组。
15:20:58.465893 IP 192.168.31.10.37685 > 225.2.0.2.42200: UDP, length 299
15:20:58.475544 IP 192.168.31.10 > 224.0.0.2: igmp leave 225.2.225.225
15:20:58.476234 IP 192.168.31.10 > 224.0.0.2: igmp leave 225.2.0.2
15:20:58.476712 IP 192.168.31.10 > 224.0.0.2: igmp leave 225.2.0.1
15:20:58.496764 IP 192.168.31.10 > 224.0.0.2: igmp leave 233.1.21.1
*：出现服务器例如MSDP2进程运行正常，SMC监控该主机为灰色，但是可以主备切换的现象，原因是时间不一致，不能大于3秒。对时后OK。
#vim SMC1_smc_20220311.log
...Error SDFP time is not sync
# vim /etc/chrony.conf
server 192.168.5.50 iburst minpoll 4 maxpoll 5
# chronyc sources
# systemctl restart chronyd
2、然后将上面的镜像移到k8s；yaml中指定容器中运行进程的用户为root
containers:
securityContext: //类似docker exec -u root，指定容器中运行进程的用户（root用户ID：0 ）
runAsUser: 0
3、在 Pod /Deployment中指定pod的 hostname 和 subdomain；或者向 Pod 的 /etc/hosts 文件中添加条目，可以在 Pod 级别覆盖对主机名的解析。在 1.7 版本，用户可以通过 PodSpec 的 hostAliases 字段来添加这些自定义的条目。主机名必须是小写lowercase。
——kubelet 管理 Pod 中每个容器的 hosts 文件，避免 Docker 在容器已经启动之后去修改该文件。因为该文件是托管性质的文件，无论容器重启或 Pod 重新调度，用户修改该 hosts 文件的任何内容，都会在 Kubelet 重新安装后被覆盖。因此，不建议修改该文件的内容。
修改network.xml对应主机名为小写“sdfp2”（原为node hostname="SDFP2"），运行./mnt正常。
spec:
hostAliases:
- ip: "127.0.0.1"
hostnames:
- "sdfp2"
- ip: "172.16.135.135"
hostnames:
- "sdfp2"
hostname: sdfp2
[root@k8s-node03 AirNet]# k apply -f sdfp2-netshoot-priv.yaml
service/svc-sdfp2netshoot unchanged
The Deployment "sdfp2-netshoot-dp" is invalid: spec.template.spec.hostAliases.hostnames: Invalid value: "SDFP1": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
4、问题？：ip==>（SDFP2）192.168.5.51怎么实现，或者验证一下是不是使用hosts的主机名访问：
[root@ACSDD5 etc]# more hosts //只有c网网段，所以airnet服务器之间不会使用主机名SDFP2访问服务。
192.168.7.50 SDFP1
192.168.7.51 SDFP2
5、假如IP正确之后，还需要在yaml中配置端口。
bash-5.1# netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 1 sdfp2:59068 localhost:46010 SYN_SENT
tcp 0 0 sdfp2:46972 192.168.6.50:46010 ESTABLISHED
tcp 0 0 sdfp2:50388 localhost:46010 ESTABLISHED
6、SDFP服务器，启动redis方式：
# cat /etc/rc.local
#auto start redis
/usr/local/redis/bin/redis-server /usr/local/redis/conf/redis.conf 1>/dev/null 2>&1
#auto start redis for maps
/usr/local/redis/bin/redis-server /usr/local/redis/conf/redis_6378.conf 1>/dev/null 2>&1
手动启动mnt方式：执行命令# /usr/bin/sudo /bin/sh /home/cdatc/.local/bin/start_mnt，不能直接执行脚本/home/cdatc/.local/bin/start_mnt
root@SDFP1 cdatc]# more .bash_profile (这里指定路径$HOME/.local/bin，会自动执行其下的脚本start_mnt？)
# User specific environment and startup programs
PATH=$PATH:$HOME/.local/bin:$HOME/bin
[root@sdfp2 autostart]# cat /home/cdatc/.config/autostart/mnt.desktop
[Desktop Entry]
Type=Application
Name=CDATC MNT
Comment=start mnt you log in
Exec=/usr/bin/sudo /bin/sh /home/cdatc/.local/bin/start_mnt
[root@SDFP1 cdatc]# cat /home/cdatc/.local/bin/start_mnt 中以下是后台启动命令 ($HOME/.local/bin/start_mnt)
cd /home/cdatc/AirNet/bin
sudo $myenv setsid bash -c ./mnt >/dev/null 2>&1 &
手动启动mnt后，mnt会自动启动mid进程：bash -c setsid /home/cdatc/AirNet/bin/mid 1>/dev/null 2>&1（该命令格式是mnt程序里写的）。但是在镜像里没有自动启动mid。
7、某节点calico-node显示READY 0/1，原因是master节点网卡比较多，calico选择了错误的网卡，修改calico.yaml，指定正确的网卡名称即可。
describe pod显示报错：
Readiness probe failed: 2021-10-02 08:02:33.072 [INFO][205] confd/health.go 180: Number of node(s) with BGP peering established = 0
calico/node is not ready: BIRD is not ready: BGP not established with 192.168.232.133,192.168.232.134
--------增加在env中增加：---------------------------
– name: IP_AUTODETECTION_METHOD
value: “interface=ens*”
systemctl disable --now NetworkManager //不关闭，引起用于容器间互通的calico不稳定
另2022-3-1的k8s-node03，READY：0/1；STATUS：Pending“kubelet cgroup driver: \"cgroupfs\" is different from docker cgroup driver: \"systemd\"”
# /usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
E0301 08:27:01.632045 233214 server.go:302] "Failed to run kubelet" err="failed to run Kubelet: misconfiguration: kubelet cgroup driver: \"cgroupfs\" is different from docker cgroup driver: \"systemd\""
[root@k8s-node03 kubelet.service.d]# docker info
Cgroup Driver: systemd
kubectl set env daemonset/calico-node -n kube-system IP_AUTODETECTION_METHOD=can-reach=192.168.31.1 //这样设置后ok
