浅谈Openstack网络
本版本为Stein版本，采用OVS网络
测试环境：1控制节点，2个计算节点
目录
- 浅谈Openstack网络
- 一、网络结构图
- 二、OVS分析
- 三、DHCP服务
- 四、Route服务
- 五、浮动IP
- 六、br-ex
- 七、安全组
一、网络结构图
借用网上2张图
图1👇：
图2👇：
二、OVS分析
如图2
vm1 表示一台实例
qbr-xxx 表示Linux虚拟网桥
可通过命令 #brctl show //compute node查看
[root@compute02 ~]# brctl show|grep bfda
qbrbfda8db6-7b 8000.5e414374148d no qvbbfda8db6-7b tapbfda8db6-7b可看到虚拟网桥（qbrbfda8db6-7b）上会有2个虚拟网口，如图2对应的A和C
br-int: 可理解为虚拟机二层交换机 实现同子网内不同实例间的访问，同时也具备VLAN tag的功能，可区分不同网段 可通过：#ovs-vsctl show 查看 br-int上有对应的端口，对应图2的D
[root@compute02 ~]# ovs-vsctl show|grep bfda
Port "qvobfda8db6-7b"
Interface "qvobfda8db6-7b" 除此之外图2的E口也可看到，也是br-int上的虚拟端口，用来连接br-tun的：
[root@compute02 ~]# ovs-vsctl show|grep patch-tun
options: {peer=patch-tun}
Port patch-tun
Interface patch-tun br-tun：实现多个computer node间的网络互通
当同子网内的实例不在同一个computer node的时候，就需要通过br-tun转发数据。过程：
- 实例发出的packet，经过Linux bridge到达br-int，被打上 VLAN ID Tag
- 到达br-tun，将VLAN ID转化为Tunnel ID，从GRE Tunnel 发出，到达另一个compute node
- 在另一个compute节点上相反的过程
在br-tun节点上也会有虚拟机接口与br-int相连接，对应图2的F口
[root@compute02 ~]# ovs-vsctl show|grep patch-in
Port patch-int
Interface patch-int
options: {peer=patch-int}不同compute node间的br-tun连接，如图2的G，H口
[root@compute02 ~]# ovs-vsctl show
6febea71-608a-4161-af6b-7bfb24fea765
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge br-tun
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
datapath_type: system
Port "vxlan-0aa80711"
Interface "vxlan-0aa80711"
type: vxlan
options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="10.168.7.132", out_key=flow, remote_ip="10.168.7.17"}
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port br-tun
Interface br-tun
type: internal
Port "vxlan-0aa80710"
Interface "vxlan-0aa80710"
type: vxlan
options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="10.168.7.132", out_key=flow, remote_ip="10.168.7.16"}如上，br-tun虚拟网桥上有2个vxlan-xxx端口（图2只有1个compute node,我的测试环境有2个compute node），分别连接两个compute node 的br-tun，这样network node和2台compute node就通过br-tun的vxlan-xxx端口打通了。
综上，同一conpute node上的网络和不同compute node间的网络就都打通了
三、DHCP服务
每个子网都有对应的DHCP服务，由network node提供，每个dhcp都有对应的namespace以此隔离不通的dhcp服务
[root@controller ~]# ip netns list
qdhcp-96c5a292-af46-4ae8-ae42-cf47ab21dd7a (id: 8)
qrouter-db3c68f9-e7ba-4a75-abff-ad78bccb34de (id: 15)
qrouter-c8edeee3-d95d-45f2-9290-4c21e7928404
qdhcp-fb12f0d5-40c5-41a5-8ea8-156ea71db5fa (id: 14)
qdhcp-0dd98fc2-d30a-433d-bd9a-466970c8b72b (id: 13)
qdhcp-8e73b70d-0f88-4803-94ad-870e99e36428 (id: 12)
qdhcp-07b7235a-b6a3-46c3-af32-d3bd341038e9 (id: 11)
qdhcp-4fba4040-4b9c-4631-8563-9647d1eed5b8 (id: 10)
qdhcp-6a840bae-e75d-4e14-9a63-10424c5d5f4d (id: 9)
qdhcp-0c17d541-2035-472f-8441-0d3750e137a2 (id: 7)
qdhcp-45b132ab-0fba-4ffb-8b91-ad12cf083d15 (id: 4)
qdhcp-74024eeb-1506-45a2-a776-fa30ff114071 (id: 6)
qdhcp-69663c23-ad87-4ffd-931d-5e3e85eb0bb2 (id: 5)
qdhcp-4633daf8-4495-4c4a-ab66-ab16875aa036 (id: 2)
qdhcp-f678d5c8-b5e0-4783-97aa-28f0db680172 (id: 3)
qdhcp-28c483f7-e520-4fc8-b363-ac8f4a1cb74f (id: 0)
qrouter-be384c66-1834-4265-8f8a-9c85c016ee79 (id: 1)[root@controller ~]# ip netns exec qdhcp-96c5a292-af46-4ae8-ae42-cf47ab21dd7a ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
tap46db5863-b8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 172.12.0.2 netmask 255.255.255.0 broadcast 172.12.0.255
inet6 fe80::f816:3eff:fe6b:8949 prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:6b:89:49 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 5 bytes 446 (446.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0可见，在对应的ns里面有对应虚拟网口tap46db5863-b8，对应图2的P口。另一端在br-int上，对应图2的O口[root@controller ~]# ovs-vsctl show|grep -C10 46db
type: patch
options: {peer=int-br-ex}
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "tap645ced8c-52"
tag: 14
Interface "tap645ced8c-52"
type: internal
Port "tap46db5863-b8"
tag: 19
Interface "tap46db5863-b8"
type: internal
Port "tapefd340f0-9d"
tag: 32
Interface "tapefd340f0-9d"
type: internal
Port "qr-9959547d-8f"
tag: 4095
Interface "qr-9959547d-8f"
type: internal
Port "tapc39b592a-14"四、Route服务
Router可提供上网功能，每个Router也是对应一个namespace
[root@controller ~]# ip netns list
qdhcp-b613251e-b959-4e8a-b959-515823a51d30 (id: 17)
qdhcp-74bd9951-523f-4732-96e9-5b7eccf01407 (id: 16)
qdhcp-96c5a292-af46-4ae8-ae42-cf47ab21dd7a (id: 8)
qrouter-db3c68f9-e7ba-4a75-abff-ad78bccb34de (id: 15)
qrouter-c8edeee3-d95d-45f2-9290-4c21e7928404
qrouter-be384c66-1834-4265-8f8a-9c85c016ee79 (id: 1)[root@controller ~]# ip netns exec qrouter-be384c66-1834-4265-8f8a-9c85c016ee79 ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 52 bytes 5219 (5.0 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 52 bytes 5219 (5.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qg-77688753-4e: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.100.18 netmask 255.255.255.0 broadcast 192.168.100.255
inet6 fe80::f816:3eff:fe14:8bf2 prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:14:8b:f2 txqueuelen 1000 (Ethernet)
RX packets 94919400 bytes 93216606173 (86.8 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 36054483 bytes 4787900088 (4.4 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qr-104bbda7-ca: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 192.168.198.1 netmask 255.255.255.0 broadcast 192.168.198.255
inet6 fe80::f816:3eff:feb9:3ec prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:b9:03:ec txqueuelen 1000 (Ethernet)
RX packets 15972 bytes 466577 (455.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 19 bytes 1250 (1.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qr-3b8aae49-1a: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 192.168.199.1 netmask 255.255.255.0 broadcast 192.168.199.255
inet6 fe80::f816:3eff:feea:57f9 prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:ea:57:f9 txqueuelen 1000 (Ethernet)
RX packets 16105 bytes 475741 (464.5 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 85 bytes 8071 (7.8 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qr-42ff6e11-9f: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 10.11.11.1 netmask 255.255.255.0 broadcast 10.11.11.255
inet6 fe80::f816:3eff:feb0:d3a3 prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:b0:d3:a3 txqueuelen 1000 (Ethernet)
RX packets 36987969 bytes 4483620005 (4.1 GiB)
RX errors 0 dropped 123 overruns 0 frame 0
TX packets 50672255 bytes 91520023500 (85.2 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0其中，qr-xxxxx是子网的网关，和br-int上的qr-xxxx组成一对，对应图2的N和M口
qg-xxxx是出网的网卡，相当于Router的WAN口，对应图2的K口
NOTE：旧版本的 qg-XXX 是挂载在 br-ex 上的，新版本则挂载在 br-int。虽然位置改变了，但功能不变，这个地方我还没有搞清楚，在我这个版本的br-ex上看不到qg-xxx，在br-int可以看到qg-xxx,希望懂的大佬可以分享下
五、浮动IP
在Router的namespace里面配置nat表实现的
[root@controller ~]# ip netns exec qrouter-be384c66-1834-4265-8f8a-9c85c016ee79 iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 53108 packets, 6285K bytes)
pkts bytes target prot opt in out source destination
3148K 438M neutron-l3-agent-PREROUTING all -- * * 0.0.0.0/0 0.0.0.0/0
Chain INPUT (policy ACCEPT 1728 packets, 231K bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 52 packets, 3878 bytes)
pkts bytes target prot opt in out source destination
156 10876 neutron-l3-agent-OUTPUT all -- * * 0.0.0.0/0 0.0.0.0/0
Chain POSTROUTING (policy ACCEPT 657 packets, 34260 bytes)
pkts bytes target prot opt in out source destination
606K 37M neutron-l3-agent-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0
589K 36M neutron-postrouting-bottom all -- * * 0.0.0.0/0 0.0.0.0/0
Chain neutron-l3-agent-OUTPUT (1 references)
pkts bytes target prot opt in out source destination
0 0 DNAT all -- * * 0.0.0.0/0 192.168.100.41 to:10.11.11.81
0 0 DNAT all -- * * 0.0.0.0/0 192.168.100.63 to:10.11.11.63
0 0 DNAT all -- * * 0.0.0.0/0 192.168.100.16 to:10.11.11.25
0 0 DNAT all -- * * 0.0.0.0/0 192.168.100.237 to:10.11.11.66
0 0 DNAT all -- * * 0.0.0.0/0 192.168.100.131 to:10.11.11.210
0 0 DNAT all -- * * 0.0.0.0/0 192.168.100.239 to:10.11.11.30
0 0 DNAT all -- * * 0.0.0.0/0 192.168.100.33 to:10.11.11.118
0 0 DNAT all -- * * 0.0.0.0/0 192.168.100.43 to:10.11.11.148
Chain neutron-l3-agent-POSTROUTING (1 references)
pkts bytes target prot opt in out source destination
17074 1023K ACCEPT all -- * !qg-77688753-4e 0.0.0.0/0 0.0.0.0/0 ! ctstate DNAT
Chain neutron-l3-agent-PREROUTING (1 references)
pkts bytes target prot opt in out source destination
241 14460 REDIRECT tcp -- qr-+ * 0.0.0.0/0 169.254.169.254 tcp dpt:80 redir ports 9697
5 348 DNAT all -- * * 0.0.0.0/0 192.168.100.41 to:10.11.11.81
259 13468 DNAT all -- * * 0.0.0.0/0 192.168.100.63 to:10.11.11.63
11 758 DNAT all -- * * 0.0.0.0/0 192.168.100.16 to:10.11.11.25
4 1416 DNAT all -- * * 0.0.0.0/0 192.168.100.237 to:10.11.11.66
581 39061 DNAT all -- * * 0.0.0.0/0 192.168.100.131 to:10.11.11.210
2799 151K DNAT all -- * * 0.0.0.0/0 192.168.100.239 to:10.11.11.30
4 260 DNAT all -- * * 0.0.0.0/0 192.168.100.33 to:10.11.11.118
46 8716 DNAT all -- * * 0.0.0.0/0 192.168.100.43 to:10.11.11.148
Chain neutron-l3-agent-float-snat (1 references)
pkts bytes target prot opt in out source destination
631 37931 SNAT all -- * * 10.11.11.81 0.0.0.0/0 to:192.168.100.41
1270 80695 SNAT all -- * * 10.11.11.63 0.0.0.0/0 to:192.168.100.63
2054 136K SNAT all -- * * 10.11.11.25 0.0.0.0/0 to:192.168.100.16
19283 1135K SNAT all -- * * 10.11.11.66 0.0.0.0/0 to:192.168.100.237
154K 9113K SNAT all -- * * 10.11.11.210 0.0.0.0/0 to:192.168.100.131
11594 779K SNAT all -- * * 10.11.11.30 0.0.0.0/0 to:192.168.100.239
54356 3424K SNAT all -- * * 10.11.11.118 0.0.0.0/0 to:192.168.100.33
102K 5455K SNAT all -- * * 10.11.11.148 0.0.0.0/0 to:192.168.100.43
Chain neutron-l3-agent-snat (1 references)
pkts bytes target prot opt in out source destination
589K 36M neutron-l3-agent-float-snat all -- * * 0.0.0.0/0 0.0.0.0/0
163K 11M SNAT all -- * qg-77688753-4e 0.0.0.0/0 0.0.0.0/0 to:192.168.100.18
0 0 SNAT all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0x2/0xffff ctstate DNAT to:192.168.100.18
Chain neutron-postrouting-bottom (1 references)
pkts bytes target prot opt in out source destination
589K 36M neutron-l3-agent-snat all -- * * 0.0.0.0/0 0.0.0.0/0 /* Perform source NAT on outgoing traffic. */六、br-ex
br-ex 绑定一个物理端口，用来出网
[root@controller ~]# ovs-vsctl show
c46120c7-9020-477f-93c6-33dda84ba833
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Manager "tcp:127.0.0.1:6640"
is_connected: true
Bridge br-ex
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port br-ex
Interface br-ex
type: internal
Port "em4"
Interface "em4"
Port phy-br-ex
Interface phy-br-ex
type: patch
options: {peer=int-br-ex}其中”em4”为出网网卡，并将其器绑定在br-ex，将上网网络接到em4，并在openstack的Route上配置对应上网地址即可七、安全组
实例的安全组是在对应的compute node上通过iptables实现的，安全组里面的规则会体现在iptables里面
iptables通过控制实例的接口，类似tap-xxxx，
如某一个实例的接口为：(0db0e25b-000f)
在compute node上可以看到对应的iptables规则
[root@compute01 ~]# iptables -nvL neutron-openvswi-o0db0e25b-0
Chain neutron-openvswi-o0db0e25b-0 (2 references)
pkts bytes target prot opt in out source destination
0 0 RETURN udp -- * * 0.0.0.0 255.255.255.255 udp spt:68 dpt:67 /* Allow DHCP client traffic. */
26207 1994K neutron-openvswi-s0db0e25b-0 all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 RETURN udp -- * * 0.0.0.0/0 0.0.0.0/0 udp spt:68 dpt:67 /* Allow DHCP client traffic. */
0 0 DROP udp -- * * 0.0.0.0/0 0.0.0.0/0 udp spt:67 dpt:68 /* Prevent DHCP Spoofing by VM. */
288 29909 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */
1 40 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */
19835 1470K neutron-openvswi-sg-fallback all -- * * 0.0.0.0/0 0.0.0.0/0 /* Send unmatched traffic to the fallback chain. */并且，forward链默认是拒绝的，所以实例不关闭端口安全和安全组是不能转发流量的，这里我没有贴出所有的表，因为openstack自建的链太多了，需要自己去找到对应的链，我只贴出了最终的一个策略。如下最后一条
[root@compute01 ~]# iptables -nvL neutron-openvswi-i0db0e25b-0
Chain neutron-openvswi-i0db0e25b-0 (1 references)
pkts bytes target prot opt in out source destination
1019 73957 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */
0 0 RETURN udp -- * * 0.0.0.0/0 172.16.0.249 udp spt:67 dpt:68
2 781 RETURN udp -- * * 0.0.0.0/0 255.255.255.255 udp spt:67 dpt:68
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */而且不关闭端口安全，浮动IP不能生效
有不对的地方，欢迎指正
本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。
