跨主机容器间通信网络
- flannel是CoreOS提供用于解决Dokcer跨主机通讯的覆盖网络(overlay network)工具。
- 数据包在主机间转发是由 backend （后端，网络类型）实现的,flannel提供了多种 backend，最常用的有 vxlan 和 host-gw，我们将在本节讨论这两种 backend。
安装配置etcd，在192.168.70.106上执行
- etcd是构建一个高可用的分布式键值(key-value)数据库。是go语言编写的。
- etcd，充当服务注册与发现中心。
- etcd在生产环境中一般推荐集群（至少三台etcd节点组成，一个leader,两个follwers）方式部署。
- etcd目前默认使用2379端口提供HTTP API服务，用于客户端连接，也就是说某个应用要把数据存储到etcd中则通过此端口。2380端口用于集群间同步数据，也就是说这个端口用与多个etcd间通信。
也可以wget 下载包
[root@docker01 ~]# ls
anaconda-ks.cfg etcd-v3.3.11-linux-amd64.tar.gz hostfile
[root@docker01 ~]# tar zxf etcd-v3.3.11-linux-amd64.tar.gz
[root@docker01 ~]# ls
anaconda-ks.cfg etcd-v3.3.11-linux-amd64 etcd-v3.3.11-linux-amd64.tar.gz hostfile
[root@docker01 ~]# mv etcd-v3.3.11-linux-amd64 /opt/etcd
[root@docker01 ~]# cd /opt/etcd/
[root@docker01 etcd]# ls
Documentation etcd etcdctl README-etcdctl.md README.md READMEv2-etcdctl.md
[root@docker01 etcd]# /opt/etcd/etcd
//需要将etcd监听ip地址修改主机的ip地址，即修改etcd监听的ip，192.168.70.106。
[root@docker01 etcd]# /opt/etcd/etcd --data-dir /opt/etcd/default.etcd --listen-client-urls http://192.168.70.106:2379,http://127.0.0.1:2379 --advertise-client-urls http://192.168.70.106:2379,http://127.0.0.1:237
//后台运行
[root@docker01 etcd]# nohup /opt/etcd/etcd --data-dir /opt/etcd/default.etcd --listen-client-urls http://192.168.70.106:2379,http://127.0.0.1:2379 --advertise-client-urls http://192.168.70.106:2379,http://127.0.0.1:2379 &
[root@docker01 etcd]# ls //信息输出在nohup.out文件
default.etcd Documentation etcd etcdctl nohup.out README-etcdctl.md README.md READMEv2-etcdctl.md
[root@docker01 etcd]# netstat -anplt | grep 2379
tcp 0 0 192.168.70.106:2379 0.0.0.0:* LISTEN 1716/etcd
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 1716/etcd
tcp 0 0 127.0.0.1:2379 127.0.0.1:39854 ESTABLISHED 1716/etcd
tcp 0 0 127.0.0.1:39854 127.0.0.1:2379 ESTABLISHED 1716/etcd
tcp 0 0 192.168.70.106:2379 192.168.70.106:58388 ESTABLISHED 1716/etcd
tcp 0 0 192.168.70.106:58388 192.168.70.106:2379 ESTABLISHED 1716/etcd
[root@docker01 etcd]# netstat -anplt | grep etcd
tcp 0 0 192.168.70.106:2379 0.0.0.0:* LISTEN 1716/etcd
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 1716/etcd
tcp 0 0 127.0.0.1:2380 0.0.0.0:* LISTEN 1716/etcd
tcp 0 0 127.0.0.1:2379 127.0.0.1:39854 ESTABLISHED 1716/etcd
tcp 0 0 127.0.0.1:39854 127.0.0.1:2379 ESTABLISHED 1716/etcd
tcp 0 0 192.168.70.106:2379 192.168.70.106:58388 ESTABLISHED 1716/etcd
tcp 0 0 192.168.70.106:58388 192.168.70.106:2379 ESTABLISHED 1716/etcd
//测试etcd是否可用，（set键值对 get查询 foo值 ）
[root@docker01 etcd]# /opt/etcd/etcdctl --endpoints=http://192.168.70.106:2379 set foo "bar"
bar
[root@docker01 etcd]# /opt/etcd/etcdctl --endpoints=http://192.168.70.106:2379 set name "lisi"
lisi
[root@docker01 etcd]# /opt/etcd/etcdctl --endpoints=http://192.168.70.106:2379 get name
lisi
[root@docker01 etcd]# /opt/etcd/etcdctl --endpoints=http://192.168.70.106:2379 get foo
bar
//执行PUT操作 192.168.70.100
[root@docker02 ~]# curl -v -X PUT http://192.168.70.106:2379/v2/keys/zhuzhi?value="beijing"
* About to connect() to 192.168.70.106 port 2379 (#0)
* Trying 192.168.70.106...
* Connected to 192.168.70.106 (192.168.70.106) port 2379 (#0)
> PUT /v2/keys/zhuzhi?value=beijing HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 192.168.70.106:2379
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< X-Etcd-Cluster-Id: cdf818194e3a8c32
< X-Etcd-Index: 11
< X-Raft-Index: 14
< X-Raft-Term: 4
< Date: Tue, 30 May 2023 14:39:11 GMT
< Content-Length: 174
<
{"action":"set","node":{"key":"/zhuzhi","value":"beijing","modifiedIndex":11,"createdIndex":11},"prevNode":{"key":"/zhuzhi","value":"","modifiedIndex":10,"createdIndex":10}}
* Connection #0 to host 192.168.70.106 left intact
安装flannel，在docker01和docker02上执行
//192.168.70.106主机
[root@docker01 ~]# ls
anaconda-ks.cfg etcd-v3.3.11-linux-amd64.tar.gz flannel-v0.10.0-linux-amd64.tar.gz hostfile
[root@docker01 ~]# scp flannel-v0.10.0-linux-amd64.tar.gz 192.168.70.100:~
The authenticity of host '192.168.70.100 (192.168.70.100)' can't be established.
ECDSA key fingerprint is SHA256:/SEgdMNlDnxDO43UvYjXg5hX6dld66fOyGLmkAhfgnY.
ECDSA key fingerprint is MD5:7a:56:38:66:a5:be:d4:9b:f4:c0:6c:ce:da:96:56:20.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.70.100' (ECDSA) to the list of known hosts.
root@192.168.70.100's password:
flannel-v0.10.0-linux-amd64.tar.gz 100% 9479KB 55.0MB/s 00:00
[root@docker01 ~]# tar zxf flannel-v0.10.0-linux-amd64.tar.gz
[root@docker01 ~]# ls
anaconda-ks.cfg flanneld hostfile README.md
etcd-v3.3.11-linux-amd64.tar.gz flannel-v0.10.0-linux-amd64.tar.gz mk-docker-opts.sh
[root@docker01 ~]# cp flanneld mk-docker-opts.sh /usr/bin/
//192.168.70.100主机
[root@docker02 ~]# ls
anaconda-ks.cfg flannel-v0.10.0-linux-amd64.tar.gz
[root@docker02 ~]# tar zxf flannel-v0.10.0-linux-amd64.tar.gz
[root@docker02 ~]# ls
anaconda-ks.cfg flanneld flannel-v0.10.0-linux-amd64.tar.gz mk-docker-opts.sh README.md
[root@docker02 ~]# cp flanneld mk-docker-opts.sh /usr/bin/
将 flannel 网络的配置信息保存到 etcd,在etcd上执行下面的操作
[root@docker01 ~]# mkdir /opt/flannel
[root@docker01 ~]# vi /opt/flannel/flannel-config.json
[root@docker01 ~]# /opt/etcd/etcdctl --endpoints http://127.0.0.1:2379 set /docker/network/config < /opt/flannel/flannel-config.json
{
"Network": "10.2.0.0/16",
"SubnetLen": 24,
"SubnetMin": "10.2.1.0",
"SubnetMax": "10.2.254.0",
"Backend":{
"Type": "vxlan"
}
}
[root@docker01 ~]# /opt/etcd/etcdctl --endpoints http://127.0.0.1:2379 get /docker/network/config
{
"Network": "10.2.0.0/16",
"SubnetLen": 24,
"SubnetMin": "10.2.1.0",
"SubnetMax": "10.2.254.0",
"Backend":{
"Type": "vxlan"
}
}
启动Flannel
//在docker01和docker02上执行,192.168.70.106
[root@docker01 ~]# /usr/bin/flanneld -etcd-endpoints=http://192.168.70.106:2379 -iface=ens33 -etcd-prefix=/docker/network
//192.168.70.100
[root@docker02 ~]# /usr/bin/flanneld -etcd-endpoints=http://192.168.70.106:2379 -iface=ens33 -etcd-prefix=/docker/network
//查看
[root@docker01 ~]# /opt/etcd/etcdctl ls /docker/network/subnets
/docker/network/subnets/10.2.98.0-24
/docker/network/subnets/10.2.52.0-24
[root@docker01 ~]# ip a
[root@docker01 ~]# ip r
default via 192.168.70.254 dev ens33 proto static metric 100
10.2.52.0/24 via 10.2.52.0 dev flannel.1 onlink
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.70.0/24 dev ens33 proto kernel scope link src 192.168.70.106 metric 100
//启动脚本192.168.70.106
[root@docker01 ~]# vi /etc/systemd/system/flanneld.service
# systemctl daemon-reload
# systemctl start flanneld
# systemctl status flanneld
# systemctl enable flanneld
//启动脚本192.168.70.100
[root@docker02 ~]# vi /etc/systemd/system/flanneld.service
# systemctl daemon-reload
# systemctl start flanneld
# systemctl status flanneld
# systemctl enable flanneld
配置 Docker 连接 flannel,在docker01和docker02上执行
//192.168.70.106
[root@docker01 ~]# cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.2.0.0/16
FLANNEL_SUBNET=10.2.98.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
[root@docker01 ~]# vi /usr/lib/systemd/system/docker.service
[root@docker01 ~]# systemctl daemon-reload
[root@docker01 ~]# systemctl restart docker
[root@docker01 ~]# ip addr show docker0
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:80:32:c3:86 brd ff:ff:ff:ff:ff:ff
inet 10.2.98.1/24 brd 10.2.98.255 scope global docker0
valid_lft forever preferred_lft forever
//192.168.70.100
[root@docker02 ~]# cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.2.0.0/16
FLANNEL_SUBNET=10.2.52.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
[root@docker02 ~]# ip r
default via 192.168.70.254 dev ens33 proto static metric 100
10.2.52.0/24 dev docker0 proto kernel scope link src 10.2.52.1
10.2.98.0/24 via 10.2.98.0 dev flannel.1 onlink
192.168.70.0/24 dev ens33 proto kernel scope link src 192.168.70.100 metric 100
[root@docker02 ~]# ip addr show docker0
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:d4:7f:00:b6 brd ff:ff:ff:ff:ff:ff
inet 10.2.52.1/24 brd 10.2.52.255 scope global docker0
valid_lft forever preferred_lft forever
将容器连接到 flannel 网络
//192.168.70.106
[root@docker01 ~]# docker run -it --name bbox1 busybox
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
5: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue
link/ether 02:42:0a:02:62:02 brd ff:ff:ff:ff:ff:ff
inet 10.2.98.2/24 brd 10.2.98.255 scope global eth0
valid_lft forever preferred_lft forever
/ # ping 10.2.52.2
PING 10.2.52.2 (10.2.52.2): 56 data bytes
64 bytes from 10.2.52.2: seq=0 ttl=62 time=0.832 ms
64 bytes from 10.2.52.2: seq=1 ttl=62 time=1.373 ms
//192.168.70.100
[root@docker02 ~]# docker run -it --name bbox1 busybox
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
325d69979d33: Pull complete
Digest: sha256:560af6915bfc8d7630e50e212e08242d37b63bd5c1ccf9bd4acccf116e262d5b
Status: Downloaded newer image for busybox:latest
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
5: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue
link/ether 02:42:0a:02:34:02 brd ff:ff:ff:ff:ff:ff
inet 10.2.52.2/24 brd 10.2.52.255 scope global eth0
valid_lft forever preferred_lft forever
- vxlan 二次封装，覆盖网络。把原来的报文UDP封装一层，从etcd中查询出10.2.3.0/24的子网的宿主机host的ip 192.168.1.10,通过配置的iface发送出去。
- flannel 是没有 DNS 服务的,容器无法通过 hostname 通信。
- vxlan和 host-gw的区别;
1.host-gw把每个主机都配置成网关，主机知道其他主机的subnet和转发地址。
vxlan则是在主机之间创建隧道，不同的主机的容器都在一个大的网段内，比如：10.2.0.0/16
2.虽然vxlan与host-gw使用不同的机制，但对于容器之间还是可以相互通信的
3.由于vxlan需要对数据进行打包和拆包，性能稍逊于host-gw
host-gw
//192.168.70.106
[root@docker01 ~]# vi /opt/flannel/flannel-config.json
[root@docker01 ~]# /opt/etcd/etcdctl --endpoints http://127.0.0.1:2379 set /docker/network/config < /opt/flannel/flannel-config.json
{
"Network": "10.2.0.0/16",
"SubnetLen": 24,
"SubnetMin": "10.2.1.0",
"SubnetMax": "10.2.254.0",
"Backend":{
"Type": "host-gw"
}
}
[root@docker01 ~]# /opt/etcd/etcdctl --endpoints http://127.0.0.1:2379 get /docker/network/config
{
"Network": "10.2.0.0/16",
"SubnetLen": 24,
"SubnetMin": "10.2.1.0",
"SubnetMax": "10.2.254.0",
"Backend":{
"Type": "host-gw"
}
}
[root@docker01 ~]# systemctl restart flanneld
//192.168.70.100
[root@docker02 ~]# systemctl restart flanneld
//106
[root@docker01 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.70.254 0.0.0.0 UG 100 0 0 ens33
10.2.52.0 192.168.70.100 255.255.255.0 UG 0 0 0 ens33
10.2.98.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
192.168.70.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
//100
[root@docker02 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.70.254 0.0.0.0 UG 100 0 0 ens33
10.2.52.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
10.2.98.0 192.168.70.106 255.255.255.0 UG 0 0 0 ens33
192.168.70.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
//106,由vxlan改成host-gw 通信
[root@docker01 ~]# cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.2.0.0/16
FLANNEL_SUBNET=10.2.98.1/24
FLANNEL_MTU=1500
FLANNEL_IPMASQ=true
[root@docker01 ~]# vi /usr/lib/systemd/system/docker.service
[root@docker01 ~]# systemctl daemon-reload
[root@docker01 ~]# systemctl restart docker
//100
[root@docker02 ~]# vi /usr/lib/systemd/system/docker.service
[root@docker02 ~]# systemctl daemon-reload
[root@docker02 ~]# systemctl restart docker
- host-gw 不会封装数据包，而是在主机的路由表中创建到其他主机 subnet 的路由条目，从而实现容器跨主机通信
