目录
1、实验目的
2、实验环境
实验拓扑
数据准备
服务器配置数据
vSwitch配置数据
Leaf1/Leaf2/Leaf3/Spine1/Spine2配置数据
EVPN相关配置数据
3、配置文件
4、配置步骤
4.1 基础配置
4.2 配置业务接入点
4.3 配置BGP EVPN Peer
4.4 配置VPN实例和EVPN实例
4.5 使能头端复制功能
4.6 配置VXLAN三层网关
4.7 配置BGP对邻居发布IRB路由
4.8 配置BGP对邻居发布IP前缀路由
4.9 其他配置
5、结果验证
5.1 检查EVPN Peer是否正常建立
5.2 查看EVPN路由
5.2.1 Leaf1初始bgp evpn路由信息
5.2.2 Type 3路由
5.2.3 Type 5路由
5.2.4 Type 2路由
5.3、同网段，不同Leaf服务器Ping测试
5.4、不同网段，同一Leaf下服务器Ping测试
5.5、不同网段，不同Leaf下服务器Ping
5.6、模拟外部和服务器之间Ping
5.7、arp广播抑制和代答
1、实验目的
了解分布式网关以及BGP EVPN工作机制。
2、实验环境
实验拓扑
ENSP软件版本：V100R003C00SPC100，CE12800软件版本： Version 8.180 (CE12800 V200R005C10SPC607B607),该版本可在模拟器上完美支持VXLAN。
- 按照实验环境进行组网。
- 基础配置：配置OSPF，相互能学习到Loopback地址。
- 完成BGP EVPN配置和分布式网关相关配置。
- 服务器vm1能Ping通vm3。
- vm2能Ping通vm3
- vm1、vm2、vm3能Ping通R1地址177.1.1.1。
- 检查VXLAN/EVPN/路由状态。
数据准备
服务器配置数据
名称 | IP地址 | 网关 | VLAN编号 |
VM-1 | 192.168.1.1/24 | 192.168.1.254 | 10 |
VM-2 | 192.168.2.1/24 | 192.168.2.254 | 20 |
VM-3 | 192.168.1.2/24 | 192.168.1.254 | 30 |
vSwitch配置数据
vSwitch-1 | 划分vlan10/20，和Leaf1 Trunk连接 |
vSwitch-2 | 划分vlan30，和Leaf2 Acess连接 |
Leaf1/Leaf2/Leaf3/Spine1/Spine2配置数据
设备 | LoopBack0 (Router-ID) | LoopBack10 (VTEP IP) | VID | BD | L2VNI |
Leaf-1 | 10.1.1.1 | 20.1.1.1 | 10 | 10 | 10 |
20 | 20 | 20 | |||
Leaf-2 | 10.1.1.2 | 20.1.1.2 | 30 | 10 | 10 |
Leaf-3 | 10.1.1.3 | 20.1.1.3 | |||
Spine-1 | 10.1.1.4 | ||||
Spine-2 | 10.1.1.5 |
本实验场景，Spine不做vxlan封装，只是三层转发，不需要VTEP地址。Leaf3没有业务接入点，不需配置二层vxlan/bridge domain。
互联端口地址：10.1.xy.x or y/24。
EVPN相关配置数据
BD | L2VNI | RD | RT | L3VNI | RD | RT | |
Leaf-1 | 10 | 10 | 10:1 | 10:1 1000:1(ert) | 100 | 100:1 | 100:1 1000:1(evpn) |
20 | 20 | 20:1 | 20:1 1000:1(ert) | ||||
Leaf-2 | 10 | 10 | 10:1 | 10:1 1000:1(ert) | 100 | 100:1 | 100:1 1000:1(evpn) |
Leaf-3 | 100 | 100:1 | 100:1 1000:1(evpn) |
Leaf-3没有L2VPN配置，只是通过L3VNI和Leaf1/Leaf2进行通信。
3、配置文件
详见ENSP工程文件
4、配置步骤
4.1 基础配置
- 配置vm1/vm2/vm3IP地址/网关；
- 配置vSwitch-1，端口划分vlan，配置和Leaf1的Trunk连接，vSwitch-2端口划分vlan30，和Leaf2 Acess连接；
a. vSwitch-1配置如下：
#
interface Ethernet0/0/1
port link-type trunk
port trunk allow-pass vlan 10 20
#
interface Ethernet0/0/2
port link-type access
port default vlan 10
#
interface Ethernet0/0/3
port link-type access
port default vlan 20
#
b. vSwitch-2配置如下：
interface Ethernet0/0/1
port link-type access
port default vlan 30
#
interface Ethernet0/0/2
port link-type access
port default vlan 30- 配置Spine1/Leaf1/Leaf2/Leaf3的loopback地址，互联地址，配置OSPF，使得Loopback地址可达。
a. Spine-1配置如下：
sysname Spine-1
#
interface GE1/0/0
undo portswitch
undo shutdown
ip address 10.1.45.4 255.255.255.0
ospf network-type p2p
#
interface GE1/0/1
undo portswitch
undo shutdown
ip address 10.1.14.4 255.255.255.0
ospf network-type p2p
#
interface GE1/0/2
undo portswitch
undo shutdown
ip address 10.1.24.4 255.255.255.0
ospf network-type p2p
#
interface GE1/0/3
undo portswitch
undo shutdown
ip address 10.1.34.4 255.255.255.0
ospf network-type p2p
#
interface LoopBack0
ip address 10.1.1.4 255.255.255.255
#
ospf 1 router-id 10.1.1.4
area 0.0.0.0
network 10.1.1.4 0.0.0.0
network 10.1.14.4 0.0.0.0
network 10.1.24.4 0.0.0.0
network 10.1.34.4 0.0.0.0
network 10.1.45.4 0.0.0.0
#
b. Spine-2配置如下：
sysname Spine-2
#
interface GE1/0/0
undo portswitch
undo shutdown
ip address 10.1.45.5 255.255.255.0
ospf network-type p2p
#
interface GE1/0/1
undo portswitch
undo shutdown
ip address 10.1.15.5 255.255.255.0
ospf network-type p2p
#
interface GE1/0/2
undo portswitch
undo shutdown
ip address 10.1.25.5 255.255.255.0
ospf network-type p2p
#
interface GE1/0/3
undo portswitch
undo shutdown
ip address 10.1.35.5 255.255.255.0
ospf network-type p2p
#
interface LoopBack0
ip address 10.1.1.5 255.255.255.255
#
ospf 1 router-id 10.1.1.5
area 0.0.0.0
network 10.1.1.5 0.0.0.0
network 10.1.15.5 0.0.0.0
network 10.1.25.5 0.0.0.0
network 10.1.35.5 0.0.0.0
network 10.1.45.5 0.0.0.0
#
c. Leaf-1配置如下：
sysname Leaf-1
#
interface GE1/0/0
undo portswitch
undo shutdown
ip address 10.1.14.1 255.255.255.0
ospf network-type p2p
#
interface GE1/0/2
undo portswitch
undo shutdown
ip address 10.1.15.1 255.255.255.0
ospf network-type p2p
#
interface LoopBack0
ip address 10.1.1.1 255.255.255.255
#
interface LoopBack1
ip address 20.1.1.1 255.255.255.255
#
ospf 1 router-id 10.1.1.1
area 0.0.0.0
network 10.1.1.1 0.0.0.0
network 10.1.14.1 0.0.0.0
network 10.1.15.1 0.0.0.0
network 20.1.1.1 0.0.0.0
#
d. Leaf-2配置如下：
sysname Leaf-2
#
interface GE1/0/0
undo portswitch
undo shutdown
ip address 10.1.24.2 255.255.255.0
ospf network-type p2p
#
interface GE1/0/2
undo portswitch
undo shutdown
ip address 10.1.25.2 255.255.255.0
ospf network-type p2p
#
interface LoopBack0
ip address 10.1.1.2 255.255.255.255
#
interface LoopBack1
ip address 20.1.1.2 255.255.255.255
#
ospf 1 router-id 10.1.1.2
area 0.0.0.0
network 10.1.1.2 0.0.0.0
network 10.1.24.2 0.0.0.0
network 10.1.25.2 0.0.0.0
network 20.1.1.2 0.0.0.0
#
e. Leaf-3配置如下：
sysname Leaf-3
#
interface GE1/0/0
undo portswitch
undo shutdown
ip address 10.1.34.3 255.255.255.0
ospf network-type p2p
#
interface GE1/0/2
undo portswitch
undo shutdown
ip address 10.1.35.3 255.255.255.0
ospf network-type p2p
#
interface LoopBack0
ip address 10.1.1.3 255.255.255.255
#
interface LoopBack1
ip address 20.1.1.3 255.255.255.255
#
ospf 1 router-id 10.1.1.3
area 0.0.0.0
network 10.1.1.3 0.0.0.0
network 10.1.34.3 0.0.0.0
network 10.1.35.3 0.0.0.0
network 20.1.1.3 0.0.0.04.2 配置业务接入点
分别在Leaf1/Leaf2配置业务接入点。
1) Leaf1配置：
#
bridge-domain 10
#
bridge-domain 20
#
interface GE1/0/1
undo shutdown
#
interface GE1/0/1.10 mode l2
encapsulation dot1q vid 10
bridge-domain 10
#
interface GE1/0/1.20 mode l2
encapsulation dot1q vid 20
bridge-domain 20
#
2) Leaf2配置：
bridge-domain 20
#
interface GE1/0/1
undo shutdown
#
interface GE1/0/1.20 mode l2
encapsulation untag
bridge-domain 204.3 配置BGP EVPN Peer
1) Spine1(RR)配置：
#
evpn-overlay enable
#
bgp 100
router-id 10.1.1.4
peer 10.1.1.1 as-number 100
peer 10.1.1.1 connect-interface LoopBack0
peer 10.1.1.2 as-number 100
peer 10.1.1.2 connect-interface LoopBack0
peer 10.1.1.3 as-number 100
peer 10.1.1.3 connect-interface LoopBack0
peer 10.1.1.5 as-number 100
peer 10.1.1.5 connect-interface LoopBack0
#
ipv4-family unicast
undo peer 10.1.1.1 enable
undo peer 10.1.1.2 enable
undo peer 10.1.1.3 enable
undo peer 10.1.1.5 enable
#
l2vpn-family evpn
undo policy vpn-target
peer 10.1.1.1 enable
peer 10.1.1.1 reflect-client
peer 10.1.1.2 enable
peer 10.1.1.2 reflect-client
peer 10.1.1.3 enable
peer 10.1.1.3 reflect-client
peer 10.1.1.5 enable
peer 10.1.1.5 reflect-client
#
2) Spine2(RR)配置：
#
evpn-overlay enable
#
bgp 100
router-id 10.1.1.5
peer 10.1.1.1 as-number 100
peer 10.1.1.1 connect-interface LoopBack0
peer 10.1.1.2 as-number 100
peer 10.1.1.2 connect-interface LoopBack0
peer 10.1.1.3 as-number 100
peer 10.1.1.3 connect-interface LoopBack0
peer 10.1.1.4 as-number 100
peer 10.1.1.4 connect-interface LoopBack0
#
ipv4-family unicast
undo peer 10.1.1.1 enable
undo peer 10.1.1.2 enable
undo peer 10.1.1.3 enable
undo peer 10.1.1.4 enable
#
l2vpn-family evpn
undo policy vpn-target
peer 10.1.1.1 enable
peer 10.1.1.1 reflect-client
peer 10.1.1.2 enable
peer 10.1.1.2 reflect-client
peer 10.1.1.3 enable
peer 10.1.1.3 reflect-client
peer 10.1.1.4 enable
peer 10.1.1.4 reflect-client
#
3) Leaf1 BGP EVPN配置：
#
evpn-overlay enable
#
bgp 100
router-id 10.1.1.1
peer 10.1.1.4 as-number 100
peer 10.1.1.4 connect-interface LoopBack0
peer 10.1.1.5 as-number 100
peer 10.1.1.5 connect-interface LoopBack0
#
ipv4-family unicast
undo peer 10.1.1.4 enable
undo peer 10.1.1.5 enable
#
l2vpn-family evpn
policy vpn-target
peer 10.1.1.4 enable
peer 10.1.1.5 enable
#
4) Leaf2 BGP EVPN配置：
#
evpn-overlay enable
#
bgp 100
router-id 10.1.1.2
peer 10.1.1.4 as-number 100
peer 10.1.1.4 connect-interface LoopBack0
peer 10.1.1.5 as-number 100
peer 10.1.1.5 connect-interface LoopBack0
#
ipv4-family unicast
undo peer 10.1.1.4 enable
undo peer 10.1.1.5 enable
#
l2vpn-family evpn
policy vpn-target
peer 10.1.1.4 enable
peer 10.1.1.5 enable
#
5) Leaf3 BGP EVPN配置：
#
evpn-overlay enable
#
bgp 100
router-id 10.1.1.3
peer 10.1.1.4 as-number 100
peer 10.1.1.4 connect-interface LoopBack0
peer 10.1.1.5 as-number 100
peer 10.1.1.5 connect-interface LoopBack0
#
ipv4-family unicast
undo peer 10.1.1.4 enable
undo peer 10.1.1.5 enable
#
l2vpn-family evpn
policy vpn-target
peer 10.1.1.4 enable
peer 10.1.1.5 enable
#Undo policy vpn-target
缺省情况下，PE对收到的VPNv4路由进行VPN-target过滤。通过过滤的路由会被加入到路由表中，没有通过过滤的路由将被丢弃。因此，如果PE没有配置VPN实例，或者VPN实例没有配置VPN-Target，则PE丢弃所有收到的VPNv4路由。
Spine1/2(RR)不配置VPN实例，但是RR需要保存所有EVPN路由信息，以通告给对端Leaf。这种情况下，RR应接收所有的EVPN路由信息，不对它们进行VPN-Target过滤。
4.4 配置VPN实例和EVPN实例
Leaf1配置，Leaf2/Leaf3类似，其中Leaf3没有配置bridge-domain，所以没有evpn实例：
1) Leaf1配置VPN实例和EVPN实例，如下：
ip vpn-instance vpn1
ipv4-family
route-distinguisher 100:1
vpn-target 100:1 export-extcommunity
vpn-target 1000:1 export-extcommunity evpn
vpn-target 100:1 import-extcommunity
vpn-target 1000:1 import-extcommunity evpn
vxlan vni 100
#
bridge-domain 10
vxlan vni 10
evpn
route-distinguisher 10:1
vpn-target 10:1 export-extcommunity
vpn-target 1000:1 export-extcommunity
vpn-target 10:1 import-extcommunity
#
bridge-domain 20
vxlan vni 20
evpn
route-distinguisher 20:1
vpn-target 20:1 export-extcommunity
vpn-target 1000:1 export-extcommunity
vpn-target 20:1 import-extcommunity
#
2) Leaf2配置VPN实例和EVPN实例，如下：
ip vpn-instance vpn1
ipv4-family
route-distinguisher 100:1
vpn-target 100:1 export-extcommunity
vpn-target 1000:1 export-extcommunity evpn
vpn-target 100:1 import-extcommunity
vpn-target 1000:1 import-extcommunity evpn
vxlan vni 100
#
bridge-domain 10
vxlan vni 10
evpn
route-distinguisher 10:1
vpn-target 10:1 export-extcommunity
vpn-target 1000:1 export-extcommunity
vpn-target 10:1 import-extcommunity
#
bridge-domain 20
vxlan vni 20
evpn
route-distinguisher 20:1
vpn-target 20:1 export-extcommunity
vpn-target 1000:1 export-extcommunity
vpn-target 20:1 import-extcommunity
#
3) Leaf3配置VPN实例，如下：
ip vpn-instance vpn1
ipv4-family
route-distinguisher 100:1
vpn-target 100:1 export-extcommunity
vpn-target 1000:1 export-extcommunity evpn
vpn-target 100:1 import-extcommunity
vpn-target 1000:1 import-extcommunity evpn
vxlan vni 100L3VPN（ip vpn-instance)下配置vpn-target 1000:1 export-extcommunity evpn，主要对该L3VPN产生的ip prefix，在生成BGP Update——Type5类EVPN路由时，通过MPBGP EVPN传输时，携带RT：1000:1，用于远端L3VPN过滤接收该前缀路由。
L2VPN（evpn)下配置vpn-target 1000:1 export-extcommunity，主要对evpn产生的Type2类EVPN路由，即mac/ip信息，提取其中的ip信息（32位主机路由），通过MPBGP EVPN传输时，携带RT：1000:1，用于远端L3VPN过滤接收该主机路由。
4.5 使能头端复制功能
1) Leaf1配置，如下：
interface Nve1
source 20.1.1.1
vni 10 head-end peer-list protocol bgp
vni 20 head-end peer-list protocol bgp
#
2) Leaf2配置，如下：
interface Nve1
source 20.1.1.2
vni 10 head-end peer-list protocol bgp
vni 20 head-end peer-list protocol bgp
#
3) Leaf3配置，如下：
interface Nve1
source 20.1.1.3
只需要配置NVE端口，指定源地址即可，无需配置头端复制（没有BUM traffic）。配置vni 10 head-end peer-list protocol bgp后，会生成BGP Update——Type3类EVPN路由（inclusive multicast route)，携带vni10和VTEP地址为20.1.1.1，告诉其他VTEP，自己是对应VNI（即L2VNI）的成员，远端收到后，判断路由可达则建立VXLAN隧道，同时会把该VTEP接入到自己对应VNI的头端复制列表中（用于BUM流量的发送）。
通过dis vxlan peer查看vni对应的头端复制列表。
4.6 配置VXLAN三层网关
1) Leaf1配置如下：
interface Vbdif10
ip binding vpn-instance vpn1
ip address 192.168.1.254 255.255.255.0
mac-address 0001-0001-0001
vxlan anycast-gateway enable
arp collect host enable
#
interface Vbdif20
ip binding vpn-instance vpn1
ip address 192.168.2.254 255.255.255.0
mac-address 0002-0002-0002
vxlan anycast-gateway enable
arp collect host enable
#
2) Leaf2配置如下：
interface Vbdif10
ip binding vpn-instance vpn1
ip address 192.168.1.254 255.255.255.0
mac-address 0001-0001-0001
vxlan anycast-gateway enable
arp collect host enable
#
interface Vbdif20
ip binding vpn-instance vpn1
ip address 192.168.2.254 255.255.255.0
mac-address 0002-0002-0002
vxlan anycast-gateway enable
arp collect host enable配置说明：
vxlan anycast-gateway enable当用户希望网关作为分布式网关，并且需要网关只学习用户侧主机发送的ARP、ND或DHCP报文时，可以执行该命令。使能分布式网关功能之后：
网关只处理收到的用户侧主机发送的ARP、ND或DHCP报文，并生成主机路由。
网关删除已经学到的网络侧的ARP、ND或DHCP报文，同时删除相应的主机路由。
arp collect host enable使三层网关能够获取主机信息表。
在配置分布式网关部署方式的VXLAN（BGP EVPN方式）场景中，当VXLAN网关之间发布的路由类型为IRB时，需配置arp collect host enable命令，用来发布主机路由。
4.7 配置BGP对邻居发布IRB路由
Leaf1/Leaf2类似，配置向邻居发布：
1) Leaf1配置如下：
bgp 100
l2vpn-family evpn
peer 10.1.1.4 advertise irb
peer 10.1.1.5 advertise irb
#
2) Leaf2配置如下：
bgp 100
l2vpn-family evpn
peer 10.1.1.4 advertise irb
peer 10.1.1.5 advertise irb
#
3) Spnie1配置如下：bgp 100
l2vpn-family evpn
undo policy vpn-target
peer 10.1.1.1 advertise irb
peer 10.1.1.2 advertise irb
peer 10.1.1.3 advertise irb
peer 10.1.1.5 advertise irb
4) Spnie2配置如下：
bgp 100
l2vpn-family evpn
undo policy vpn-target
peer 10.1.1.1 advertise irb
peer 10.1.1.2 advertise irb
peer 10.1.1.3 advertise irb
peer 10.1.1.4 advertise irbSpine设备作为RR，也需要使能对等体发布irb路由功能，否则irb路由经过反射器后不会再给客户。
4.8 配置BGP对邻居发布IP前缀路由
只有在如下两种场景中，Leaf节点可发布网段路由：
Leaf节点连接的网段在整个VXLAN网络中是唯一的，而且有效的主机明细路由数量较大，此时可发布主机IP所在的网段路由，从而减轻Leaf节点上路由存储的压力。
VXLAN网络中的主机需要访问外部网络，此时Leaf节点可在VXLAN网络中发布其连接的外部网段路由，从而使其他Leaf节点学习到去往外部网络的路由。
Leaf3配置IP前缀路由发布，如下：
bgp 100
ipv4-family vpn-instance vpn1
import-route ospf 2
advertise l2vpn evpn
#
advertise l2vpn evpn
//让vpn-instance的三层路由通过bgp evpn传递Leaf1和Leaf2上可使用import-route命令发布前缀路由，将192.168.1.0/24和192.168.2.0/24的ip vpn实例路由转换成evpn的tpye-5类型发布出去。但是若使用了该命令之后，由于leaf1和Leaf2上面都承载了192.168.1.0/24和192.168.2.0/24的网段，Leaf3在根据BGP选路原则选路的时候默认均只会选择Leaf1发送的前缀路由。而且这样在实际传输过程中可能会出现未知问题，故Leaf1和Leaf2发布IRB路由即可，同时终端在入网之后应用协议进程自身会交互报文，顾不用担心Leaf1和Leaf2作为网关设备无法收集到VM1、VM2和VM3的IRB路由信息。
若Leaf1和Leaf2发布了关于192.168.1.0/24的前缀信息，那么当Leaf3上访问192.168.1.3（该地址实际没有被VM使用），则Leaf3还是会通过L3 VXLAN隧道传递给Leaf1或者Leaf2，造成网络没必要的流量负担，而使用IRB路由可以避免该问题的出现。
另外若Leaf3下R1上的有主动访问192.168.1.1的需求，则需要结合实际情况判断是否在Leaf1上发布IP前缀路由，因为IBR是结合ARP表项信息收集的32位主机路由，倘若192.168.1.1的ARP表项超时，则Leaf3关于192.168.1.1的32位IRB路由也会消失，这种场景下会出现无法访问192.168.1.1。
4.9 其他配置
配置R1和Leaf3互联和路由：
1) R1配置：
sysname R1
#
interface GigabitEthernet0/0/1
ip address 10.1.36.6 255.255.255.0
#
interface LoopBack0
ip address 177.1.1.1 255.255.255.255
#
ospf 1 router-id 10.1.36.6
area 0.0.0.0
network 177.1.1.1 0.0.0.0
network 10.1.36.6 0.0.0.0
#
2) Leaf3配置：
interface GE1/0/1
undo portswitch
undo shutdown
ip binding vpn-instance vpn1
ip address 10.1.36.3 255.255.255.0
#
ospf 2 router-id 10.1.36.3 vpn-instance vpn1
import-route bgp
area 0.0.0.0
network 10.1.36.3 0.0.0.0
#5、结果验证
5.1 检查EVPN Peer是否正常建立
在Spine上检查BGP EVPN的邻居状态，如下：
[~Spine-1]display bgp evpn peer
BGP local router ID : 10.1.1.4
Local AS number : 100
Total number of peers : 4
Peers in established state : 3
Peer V AS MsgRcvd MsgSent OutQ Up/Down State PrefRcv
10.1.1.1 4 100 11 15 0 00:00:59 Established 10
10.1.1.2 4 100 10 15 0 00:01:01 Established 8
10.1.1.3 4 100 5 12 0 00:00:58 Established 2
10.1.1.5 4 100 0 0 0 01:31:30 Active 05.2 查看EVPN路由
5.2.1 Leaf1初始bgp evpn路由信息
1) 在Leaf1上查看BGP EVPN邻居状态，如下：
[~Leaf-1]display bgp evpn peer
BGP local router ID : 10.1.1.1
Local AS number : 100
Total number of peers : 2
Peers in established state : 1
Peer V AS MsgRcvd MsgSent OutQ Up/Down State PrefRcv
10.1.1.4 4 100 20 15 0 00:05:34 Established 5
10.1.1.5 4 100 0 0 0 01:33:55 Active 0
2) 此时查看Leaf1的全局路由信息，如下：
[~Leaf-1]display ip routing-table vpn-instance vpn1
Proto: Protocol Pre: Preference
Route Flags: R - relay, D - download to fib, T - to vpn-instance, B - black hole route
------------------------------------------------------------------------------
Routing Table : vpn1
Destinations : 10 Routes : 10
Destination/Mask Proto Pre Cost Flags NextHop Interface
10.1.36.0/24 IBGP 255 2 RD 20.1.1.3 VXLAN
177.1.1.1/32 IBGP 255 2 RD 20.1.1.3 VXLAN //type5的IP前缀路由
192.168.1.0/24 Direct 0 0 D 192.168.1.254 Vbdif10
192.168.1.2/32 IBGP 255 0 RD 20.1.1.2 VXLAN //type2的IRB路由
192.168.1.254/32 Direct 0 0 D 127.0.0.1 Vbdif10
192.168.1.255/32 Direct 0 0 D 127.0.0.1 Vbdif10
192.168.2.0/24 Direct 0 0 D 192.168.2.254 Vbdif20
192.168.2.254/32 Direct 0 0 D 127.0.0.1 Vbdif20
192.168.2.255/32 Direct 0 0 D 127.0.0.1 Vbdif20
255.255.255.255/32 Direct 0 0 D 127.0.0.1 InLoopBack0
3) 查看Leaf1的EVPN路由信息，如下：
[~Leaf-1]display bgp evpn all routing-table
Local AS number : 100
BGP Local router ID is 10.1.1.1
Status codes: * - valid, > - best, d - damped, x - best external, a - add path,
h - history, i - internal, s - suppressed, S - Stale
Origin : i - IGP, e - EGP, ? - incomplete
EVPN address family:
Number of Mac Routes: 4
Route Distinguisher: 10:1
Network(EthTagId/MacAddrLen/MacAddr/IpAddrLen/IpAddr) NextHop
*> 0:48:0001-0001-0001:0:0.0.0.0 0.0.0.0
*> 0:48:5489-9805-6139:32:192.168.1.1 0.0.0.0
Route Distinguisher: 20:1
Network(EthTagId/MacAddrLen/MacAddr/IpAddrLen/IpAddr) NextHop
*> 0:48:0002-0002-0002:0:0.0.0.0 0.0.0.0
*> 0:48:5489-9897-18ac:32:192.168.2.1 0.0.0.0
EVPN-Instance 10:
Number of Mac Routes: 2
Network(EthTagId/MacAddrLen/MacAddr/IpAddrLen/IpAddr) NextHop
*> 0:48:0001-0001-0001:0:0.0.0.0 0.0.0.0
*> 0:48:5489-9805-6139:32:192.168.1.1 0.0.0.0
EVPN-Instance 20:
Number of Mac Routes: 2
Network(EthTagId/MacAddrLen/MacAddr/IpAddrLen/IpAddr) NextHop
*> 0:48:0002-0002-0002:0:0.0.0.0 0.0.0.0
*> 0:48:5489-9897-18ac:32:192.168.2.1 0.0.0.0
EVPN address family:
Number of Inclusive Multicast Routes: 4
Route Distinguisher: 10:1
Network(EthTagId/IpAddrLen/OriginalIp) NextHop
*> 0:32:20.1.1.1 0.0.0.0
*>i 0:32:20.1.1.2 20.1.1.2
Route Distinguisher: 20:1
Network(EthTagId/IpAddrLen/OriginalIp) NextHop
*> 0:32:20.1.1.1 0.0.0.0
*>i 0:32:20.1.1.2 20.1.1.2
EVPN-Instance 10:
Number of Inclusive Multicast Routes: 2
Network(EthTagId/IpAddrLen/OriginalIp) NextHop
*> 0:32:20.1.1.1 0.0.0.0
*>i 0:32:20.1.1.2 20.1.1.2
EVPN-Instance 20:
Number of Inclusive Multicast Routes: 2
Network(EthTagId/IpAddrLen/OriginalIp) NextHop
*> 0:32:20.1.1.1 0.0.0.0
*>i 0:32:20.1.1.2 20.1.1.2
EVPN address family:
Number of Ip Prefix Routes: 2
Route Distinguisher: 100:1
Network(EthTagId/IpPrefix/IpPrefixLen) NextHop
*>i 0:10.1.36.0:24 20.1.1.3
*>i 0:177.1.1.1:32 20.1.1.3
EVPN-Instance __vni100__:
Number of Ip Prefix Routes: 2
Network(EthTagId/IpPrefix/IpPrefixLen) NextHop
*>i 0:10.1.36.0:24 20.1.1.3
*>i 0:177.1.1.1:32 20.1.1.3
4条type3路由，分别为Leaf1和Leaf2关于VNI10和20产生的。
4条type2路由，2条mac路由，2条由arp路由；
2条type5路由，为Leaf3引入的ospf路由；
4) 2条type5路由，由L3VPN注入进来的:
[~Leaf-1]display bgp vpnv4 all routing-table
BGP Local router ID is 10.1.1.1
Status codes: * - valid, > - best, d - damped, h - history,
i - internal, s - suppressed, S - Stale
Origin : i - IGP, e - EGP, ? - incomplete
Total number of routes from all PE: 4
Route Distinguisher: 100:1
Network NextHop MED LocPrf PrefVal Path/Ogn
*> 192.168.1.0 0.0.0.0 0 0 ?
*> 192.168.1.254/32 0.0.0.0 0 0 ?
*> 192.168.2.0 0.0.0.0 0 0 ?
*> 192.168.2.254/32 0.0.0.0 0 0 ?
VPN-Instance vpn1, Router ID 10.1.1.1:
Total Number of Routes: 4
Network NextHop MED LocPrf PrefVal Path/Ogn
*> 192.168.1.0 0.0.0.0 0 0 ?
*> 192.168.1.254/32 0.0.0.0 0 0 ?
*> 192.168.2.0 0.0.0.0 0 0 ?
*> 192.168.2.254/32 0.0.0.0 0 0 ?
具体查看一条前缀路由（比如：0:192.168.1.0:24）的注入方式，可以看到此路由是“Imported route.":
[~Leaf-1]display bgp evpn all routing-table prefix-route 0:177.1.1.1:32
BGP local router ID : 10.1.1.1
Local AS number : 100
Total routes of Route Distinguisher(100:1): 1
BGP routing table entry information of 0:177.1.1.1:32:
Label information (Received/Applied): 100/NULL
From: 10.1.1.4 (10.1.1.4)
Route Duration: 0d00h18m18s
Relay IP Nexthop: 10.1.14.4
Relay Tunnel Out-Interface:
Original nexthop: 20.1.1.3
Qos information : 0x0
Ext-Community:RT <100 : 1>, RT <1000 : 1>, OSPF DOMAIN ID <0.0.0.0 : 0>, OSPF ROUTER ID <10.1.36.3 : 0>, OSPF RT <0.0.0.0 : 0 : 1>, Tunnel Type <VxLan(8)>, Router's MAC <707b-e8e6-5d85>
AS-path Nil, origin incomplete, MED 2, localpref 100, pref-val 0, valid, internal, best, select, pre 255, IGP cost 2
Originator: 10.1.1.3
Cluster list: 10.1.1.4
Route Type: 5 (Ip Prefix Route)
Ethernet Tag ID: 0, IP Prefix/Len: 177.1.1.1/32, ESI: 0000.0000.0000.0000.0000, GW IP Address: 0.0.0.0
Not advertised to any peer yet
EVPN-Instance __vni100__:
Number of Ip Prefix Routes: 1
BGP routing table entry information of 0:177.1.1.1:32:
Route Distinguisher: 100:1
Remote-Cross route
Label information (Received/Applied): 100/NULL
From: 10.1.1.4 (10.1.1.4)
Route Duration: 0d00h18m18s
Relay Tunnel Out-Interface: VXLAN
Original nexthop: 20.1.1.3
Qos information : 0x0
Ext-Community:RT <100 : 1>, RT <1000 : 1>, OSPF DOMAIN ID <0.0.0.0 : 0>, OSPF ROUTER ID <10.1.36.3 : 0>, OSPF RT <0.0.0.0 : 0 : 1>, Tunnel Type <VxLan(8)>, Router's MAC <707b-e8e6-5d85>
AS-path Nil, origin incomplete, MED 2, localpref 100, pref-val 0, valid, internal, best, select, pre 255
Originator: 10.1.1.3
Cluster list: 10.1.1.4
Route Type: 5 (Ip Prefix Route)
Ethernet Tag ID: 0, IP Prefix/Len: 177.1.1.1/32, ESI: 0000.0000.0000.0000.0000, GW IP Address: 0.0.0.0
Not advertised to any peer yet
5) 在Leaf1上，Leaf1连接Spine1的端口抓包，查看BGP update信息。5.2.2 Type 3路由
首先在Leaf1和Leaf2之间建立BGP EVPN对等体。然后，在Leaf1和Leaf2上分别创建二层广播域，并在二层广播域下配置关联的二层VNI。接下来在二层广播域下创建EVPN实例，配置本端EVPN实例的RD、出方向VPN-Target（ERT）、入方向VPN-Target（IRT）。在配置完本端VTEP IP地址后，Leaf1和Leaf2会生成BGPEVPN路由并发送给对端，该路由携带本端EVPN实例的出方向VPN-Target和BGPEVPN协议新定义的Type3路由即Inclusive Multicast路由。其中，Inclusive Multicast路由如下图所示，由前缀和PMSI属性组成，VTEP IP地址存放在前缀的Originating Router's IP Address字段中，二层VNI存放在PMSI属性的MPLSLabel字段中。VTEP IP地址同时也放在下一跳属性中。
a. Leaf1发给Spine1的bgp update type3抓包信息：
Frame 483: 510 bytes on wire (4080 bits), 510 bytes captured (4080 bits) on interface 0
Ethernet II, Src: 4e:01:00:11:01:00 (4e:01:00:11:01:00), Dst: 4e:01:00:41:01:01 (4e:01:00:41:01:01)
Internet Protocol Version 4, Src: 10.1.1.1, Dst: 10.1.1.4
Transmission Control Protocol, Src Port: 50046, Dst Port: 179, Seq: 65, Ack: 65, Len: 456
Border Gateway Protocol - UPDATE Message
Marker: ffffffffffffffffffffffffffffffff
Length: 108
Type: UPDATE Message (2)
Withdrawn Routes Length: 0
Total Path Attribute Length: 85
Path attributes
Path Attribute - ORIGIN: INCOMPLETE
Path Attribute - AS_PATH: empty
Path Attribute - LOCAL_PREF: 100
Path Attribute - EXTENDED_COMMUNITIES
Flags: 0xc0, Optional, Transitive: Optional, Transitive, Complete
Type Code: EXTENDED_COMMUNITIES (16)
Length: 24
Carried extended communities: (3 communities)
Community Transitive Two-Octet AS Route Target: 10:1
Community Transitive Two-Octet AS Route Target: 1000:1
Community Transitive Opaque Encapsulation: VXLAN Encapsulation
Path Attribute - PMSI_TUNNEL_ATTRIBUTE
Flags: 0xc0, Optional, Transitive: Optional, Transitive, Complete
Type Code: PMSI_TUNNEL_ATTRIBUTE (22)
Length: 9
Flags: 0
Tunnel Type: Ingress Replication (6)
0000 0000 0000 0000 0000 .... = MPLS Label: 0//十六进制a转换为十进制为10，即L2VNI为10.
Tunnel ID: tunnel end point -> 20.1.1.1
Tunnel type ingress replication IP end point: 20.1.1.1
Path Attribute - MP_REACH_NLRI
Flags: 0x90, Optional, Length: Optional, Non-transitive, Complete, Extended Length
Type Code: MP_REACH_NLRI (14)
Length: 28
Address family identifier (AFI): Layer-2 VPN (25)
Subsequent address family identifier (SAFI): EVPN (70)
Next hop network address (4 bytes) //VTEP IP地址同时也放在下一跳属性中,即20.1.1.1
Number of Subnetwork points of attachment (SNPA): 0
Network layer reachability information (19 bytes)
EVPN NLRI: Inclusive Multicast Route
AFI: Inclusive Multicast Route (3)
Length: 17
Route Distinguisher: 0000000a00000001 (10:1)
Ethernet Tag ID: 0
IP Address Length: 32
IPv4 address: 20.1.1.1 //Originating Router's IP Address为20.1.1.1
Border Gateway Protocol - UPDATE Message
Border Gateway Protocol - UPDATE Message
Marker: ffffffffffffffffffffffffffffffff
Length: 108
Type: UPDATE Message (2)
Withdrawn Routes Length: 0
Total Path Attribute Length: 85
Path attributes
Path Attribute - ORIGIN: INCOMPLETE
Path Attribute - AS_PATH: empty
Path Attribute - LOCAL_PREF: 100
Path Attribute - EXTENDED_COMMUNITIES
Flags: 0xc0, Optional, Transitive: Optional, Transitive, Complete
Type Code: EXTENDED_COMMUNITIES (16)
Length: 24
Carried extended communities: (3 communities)
Community Transitive Two-Octet AS Route Target: 20:1
Community Transitive Two-Octet AS Route Target: 1000:1
Community Transitive Opaque Encapsulation: VXLAN Encapsulation
Path Attribute - PMSI_TUNNEL_ATTRIBUTE
Flags: 0xc0, Optional, Transitive: Optional, Transitive, Complete
Type Code: PMSI_TUNNEL_ATTRIBUTE (22)
Length: 9
Flags: 0
Tunnel Type: Ingress Replication (6)
0000 0000 0000 0000 0001 .... = MPLS Label: 1 //十六进制14转换为十进制20，即L2VNI为20
Tunnel ID: tunnel end point -> 20.1.1.1
Tunnel type ingress replication IP end point: 20.1.1.1
Path Attribute - MP_REACH_NLRI
Flags: 0x90, Optional, Length: Optional, Non-transitive, Complete, Extended Length
Type Code: MP_REACH_NLRI (14)
Length: 28
Address family identifier (AFI): Layer-2 VPN (25)
Subsequent address family identifier (SAFI): EVPN (70)
Next hop network address (4 bytes)
Number of Subnetwork points of attachment (SNPA): 0
Network layer reachability information (19 bytes)
EVPN NLRI: Inclusive Multicast Route
AFI: Inclusive Multicast Route (3)
Length: 17
Route Distinguisher: 0000001400000001 (20:1)
Ethernet Tag ID: 0
IP Address Length: 32
IPv4 address: 20.1.1.1 //Originating Router's IP Address为20.1.1.1
Border Gateway Protocol - UPDATE Message
b. Spine1发给Leaf1的bgp update type3抓包信息：
Frame 487: 1012 bytes on wire (8096 bits), 1012 bytes captured (8096 bits) on interface 0
Ethernet II, Src: 4e:01:00:41:01:01 (4e:01:00:41:01:01), Dst: 4e:01:00:11:01:00 (4e:01:00:11:01:00)
Internet Protocol Version 4, Src: 10.1.1.4, Dst: 10.1.1.1
Transmission Control Protocol, Src Port: 179, Dst Port: 50046, Seq: 577, Ack: 551, Len: 958
Border Gateway Protocol - UPDATE Message
Border Gateway Protocol - UPDATE Message
Marker: ffffffffffffffffffffffffffffffff
Length: 122
Type: UPDATE Message (2)
Withdrawn Routes Length: 0
Total Path Attribute Length: 99
Path attributes
Path Attribute - ORIGIN: INCOMPLETE
Path Attribute - AS_PATH: empty
Path Attribute - LOCAL_PREF: 100
Path Attribute - ORIGINATOR_ID: 10.1.1.2
Path Attribute - CLUSTER_LIST: 10.1.1.4
Path Attribute - EXTENDED_COMMUNITIES
Flags: 0xc0, Optional, Transitive: Optional, Transitive, Complete
Type Code: EXTENDED_COMMUNITIES (16)
Length: 24
Carried extended communities: (3 communities)
Community Transitive Two-Octet AS Route Target: 10:1
Community Transitive Two-Octet AS Route Target: 1000:1
Community Transitive Opaque Encapsulation: VXLAN Encapsulation
Path Attribute - PMSI_TUNNEL_ATTRIBUTE
Path Attribute - MP_REACH_NLRI
Flags: 0x90, Optional, Length: Optional, Non-transitive, Complete, Extended Length
Type Code: MP_REACH_NLRI (14)
Length: 28
Address family identifier (AFI): Layer-2 VPN (25)
Subsequent address family identifier (SAFI): EVPN (70)
Next hop network address (4 bytes)
Number of Subnetwork points of attachment (SNPA): 0
Network layer reachability information (19 bytes)
EVPN NLRI: Inclusive Multicast Route
AFI: Inclusive Multicast Route (3)
Length: 17
Route Distinguisher: 0000000a00000001 (10:1)
Ethernet Tag ID: 0
IP Address Length: 32
IPv4 address: 20.1.1.2
Border Gateway Protocol - UPDATE Message
Border Gateway Protocol - UPDATE Message
Border Gateway Protocol - UPDATE Message
Marker: ffffffffffffffffffffffffffffffff
Length: 122
Type: UPDATE Message (2)
Withdrawn Routes Length: 0
Total Path Attribute Length: 99
Path attributes
Path Attribute - ORIGIN: INCOMPLETE
Path Attribute - AS_PATH: empty
Path Attribute - LOCAL_PREF: 100
Path Attribute - ORIGINATOR_ID: 10.1.1.2
Path Attribute - CLUSTER_LIST: 10.1.1.4
Path Attribute - EXTENDED_COMMUNITIES
Path Attribute - PMSI_TUNNEL_ATTRIBUTE
Path Attribute - MP_REACH_NLRI
Border Gateway Protocol - UPDATE Message
Border Gateway Protocol - UPDATE Message
这个type3路由就是Leaf2产生的，vni 10/20触发产生。
c. 在Leaf1查看bgp evpn type3路由信息：
[~Leaf-1]display bgp evpn all routing-table inclusive-route
Local AS number : 100
BGP Local router ID is 10.1.1.1
Status codes: * - valid, > - best, d - damped, x - best external, a - add path,
h - history, i - internal, s - suppressed, S - Stale
Origin : i - IGP, e - EGP, ? - incomplete
EVPN address family:
Number of Inclusive Multicast Routes: 4
Route Distinguisher: 10:1
Network(EthTagId/IpAddrLen/OriginalIp) NextHop
*> 0:32:20.1.1.1 0.0.0.0
*>i 0:32:20.1.1.2 20.1.1.2
Route Distinguisher: 20:1
Network(EthTagId/IpAddrLen/OriginalIp) NextHop
*> 0:32:20.1.1.1 0.0.0.0
*>i 0:32:20.1.1.2 20.1.1.2
EVPN-Instance 10:
Number of Inclusive Multicast Routes: 2
Network(EthTagId/IpAddrLen/OriginalIp) NextHop
*> 0:32:20.1.1.1 0.0.0.0
*>i 0:32:20.1.1.2 20.1.1.2
EVPN-Instance 20:
Number of Inclusive Multicast Routes: 2
Network(EthTagId/IpAddrLen/OriginalIp) NextHop
*> 0:32:20.1.1.1 0.0.0.0
*>i 0:32:20.1.1.2 20.1.1.2
[~Leaf-1]
一共4条type3路由，2条为Leaf1关于EVPN-Instance 10和EVPN-Instance 20产生的，如上紫色标记部分，2条为Leaf2关于EVPN-Instance 10和EVPN-Instance 20产生的，如上红色标记部分；
查看具体的type3路由的命令：[~Leaf-1]display bgp evpn all routing-table inclusive-route 0:32:20.1.1.1Leaf1和Leaf2在收到对端发来的BGP EVPN路由后，首先检查该路由携带的EVPN实例的出方向VPN-Target，如果与本端EVPN实例的入方向VPN-Target相等，则接收该路由，否则丢弃该路由。在接收该路由后，Leaf1和Leaf2将获取其中携带的对端VTEP IP地址（从下一跳属性获取）和二层VNI，如果对端VTEP IP地址是三层路由可达的，则建立一条到对端的VXLAN隧道；同时，本端会创建一个基于VNI的头端复制表并将对端VTEP IP地址加入其中，用于后续BUM报文转发。
a. 查看Leaf1 vxlan peer建立：
[~Leaf-1]display vxlan peer
Number of peers : 2
Vni ID Source Destination Type
--------------------------------------------------------------
10 20.1.1.1 20.1.1.2 dynamic
20 20.1.1.1 20.1.1.2 dynamic
type是动态的，通过bgp evpn协议建立的。
这就是头端复制列表。
b. 查看Leaf1 vxla隧道，Leaf1和Leaf2、Leaf3都建立了vxlan tunnel:
[~Leaf-1]display vxlan tunnel
Number of vxlan tunnel : 2
Tunnel ID Source Destination State Type
--------------------------------------------------------------
4026531843 20.1.1.1 20.1.1.2 up dynamic
4026531844 20.1.1.1 20.1.1.3 up dynamic
注意由于Leaf3没有配置L2VNI，所以Leaf1和Leaf3之间没有BUM流量的头端复制列表（就是没有vxlan peer)。
可以理解tunnel主要用于单播流量迭代到tunnel上（进行vxlan封装）。
关于和Leaf3的VXLAN隧道其实是通过tpye5类路由建立的L3隧道。实际抓包并没有发现Leaf3产生的type3类路由。VPN-Target是一种BGP扩展团体属性，一个EVPN实例可以配置出方向和入方向两类VPN Target，两端EVPN实例的VPN-Target要相互匹配（即本端EVPN实例配置的出方向VPN-target值需要与对端EVPN实例配置的入方向VPN-target值相等），才能相互交换EVPN路由，否则VXLAN隧道无法建立成功。如果仅有一端匹配成功可以接收路由，则此端设备上可以建立通往另一端设备的隧道，但是无法传输数据报文，因为另一端设备在收到报文后，将会检查本端是否有通往对端的VXLAN隧道，如果没有的话，将会丢弃报文。
5.2.3 Type 5路由
Spine1发送给Leaf1的bgp update type5的抓包信息（实际实验环境中为Leaf3产生）：
Frame 31: 768 bytes on wire (6144 bits), 768 bytes captured (6144 bits) on interface 0
Ethernet II, Src: 4e:01:00:11:01:01 (4e:01:00:11:01:01), Dst: 4e:01:00:21:01:00 (4e:01:00:21:01:00)
Internet Protocol Version 4, Src: 10.1.1.4, Dst: 10.1.1.1
Transmission Control Protocol, Src Port: 179, Dst Port: 65055, Seq: 65, Ack: 65, Len: 714
Border Gateway Protocol - UPDATE Message
Border Gateway Protocol - UPDATE Message
[Malformed Packet: BGP]
[Expert Info (Error/Malformed): Malformed Packet (Exception occurred)]
Border Gateway Protocol - UPDATE Message
Border Gateway Protocol - UPDATE Message
[Malformed Packet: BGP]
Border Gateway Protocol - UPDATE Message
Marker: ffffffffffffffffffffffffffffffff
Length: 202
Type: UPDATE Message (2)
Withdrawn Routes Length: 0
Total Path Attribute Length: 179
Path attributes
Path Attribute - ORIGIN: INCOMPLETE
Path Attribute - AS_PATH: empty
Path Attribute - MULTI_EXIT_DISC: 2
Path Attribute - LOCAL_PREF: 100
Path Attribute - ORIGINATOR_ID: 10.1.1.3
Path Attribute - CLUSTER_LIST: 10.1.1.4
Path Attribute - EXTENDED_COMMUNITIES
Flags: 0xc0, Optional, Transitive, Complete
Type Code: EXTENDED_COMMUNITIES (16)
Length: 56
Carried extended communities: (7 communities)
Route Target: 100:1 [Transitive 2-Octet AS-Specific] //Leaf3上vpn-target 100:1 export-extcommunity
Route Target: 1000:1 [Transitive 2-Octet AS-Specific] //Leaf3上 vpn-target 1000:1 export-extcommunity evpn
OSPF Domain Identifier: 0:0 [Transitive 2-Octet AS-Specific]
OSPF Router ID: 10.1.36.3:0 [Transitive IPv4-Address-Specific]
OSPF Route Type: Area: 0.0.0.0, Type: Router [Transitive Opaque]
Encapsulation: VXLAN Encapsulation [Transitive Opaque]
Unknown subtype 0x03: 0x707b 0xe8e6 0x5d85 [Transitive EVPN] //Router's MAC，Leaf3上VTEP的MAC地址707b-08e6-5db5，即20.1.1.3
Path Attribute - MP_REACH_NLRI
Flags: 0x90, Optional, Extended-Length, Non-transitive, Complete
Type Code: MP_REACH_NLRI (14)
Length: 81
Address family identifier (AFI): Layer-2 VPN (25)
Subsequent address family identifier (SAFI): EVPN (70)
Next hop network address (4 bytes) //该处可能是wireshark版本问题，该字段没有直接显示，但是实际上为Leaf3 VETP的IP地址，且为十六进制格式。
//Laef1收到该tpye5类型的信息，看到Leaf3的VTEP地址，判断路由可达，故建立VXLAN隧道。
Number of Subnetwork points of attachment (SNPA): 0
Network layer reachability information (72 bytes)
EVPN NLRI: IP Prefix route
Route Type: IP Prefix route (5)
Length: 34
Route Distinguisher: 0000006400000001 (100:1)
ESI: 00:00:00:00:00:00:00:00:00:00
Ethernet Tag ID: 0
IP prefix length: 24
IPv4 address: 10.1.36.0
IPv4 Gateway address: 0.0.0.0
MPLS Label Stack: 6, (BOGUS: Bottom of Stack NOT set!)
MPLS Label: 1006,//该处可能为wireshark版本问题，该字段没有直接显示，但实际为L3 VNI的值，且为十六进制格式。
//转换为十进制后值为100，即Leaf1结合Leaf3的VTEP地址，建立L3的VXLAN隧道。
EVPN NLRI: IP Prefix route
Route Type: IP Prefix route (5)
Length: 34
Route Distinguisher: 0000006400000001 (100:1)
ESI: 00:00:00:00:00:00:00:00:00:00
Ethernet Tag ID: 0
IP prefix length: 32
IPv4 address: 177.1.1.1
IPv4 Gateway address: 0.0.0.0
MPLS Label Stack: 6, (BOGUS: Bottom of Stack NOT set!)
MPLS Label: 1006,
说明：
type5携带L3VPN的export RT值：100:1和1000:1（evpn）；
type5只携带L3VNI：100；
Type5携带Next hop network address，即为Leaf3的VTEP的IP地址；
type5携带Router’s MAC ，即为Leaf3的NVE（VTEP）端口mac地址：
<Leaf-3>display interface Nve1
Nve1 current state : UP (ifindex: 17)
Line protocol current state : UP
Description:
IP Sending Frames' Format is PKTFMT_ETHNT_2, Hardware address is 707b-e8e6-5d85
Tpye5作为前缀路由不携带携带L2VNI，通过普通IP VPN实例的RT值100:1匹配本地的IP VPN实例，然后生成路由条目。若leaf1和leaf2也发布了直连192.168.1.0/24和192.168.2.0/24的IP前缀路由，则针对这两个网段，组成分布式网关，彼此发送相同的192.168.1.0/24和192.168.2.0/24的路由。具体信息这里不做展示。
Leaf1查看bgp evpn type5的路由表：
<Leaf-1>display bgp evpn all routing-table prefix-route
Local AS number : 100
BGP Local router ID is 10.1.1.1
Status codes: * - valid, > - best, d - damped, x - best external, a - add path,
h - history, i - internal, s - suppressed, S - Stale
Origin : i - IGP, e - EGP, ? - incomplete
EVPN address family:
Number of Ip Prefix Routes: 2
Route Distinguisher: 100:1
Network(EthTagId/IpPrefix/IpPrefixLen) NextHop
*>i 0:10.1.36.0:24 20.1.1.3
*>i 0:177.1.1.1:32 20.1.1.3
EVPN-Instance __vni100__:
Number of Ip Prefix Routes: 2
Network(EthTagId/IpPrefix/IpPrefixLen) NextHop
*>i 0:10.1.36.0:24 20.1.1.3
*>i 0:177.1.1.1:32 20.1.1.3查看具体的bgp evpn type5路由信息，如0:177.1.1.1:32：
<Leaf-1>display bgp evpn all routing-table prefix-route 0:177.1.1.1:32
BGP local router ID : 10.1.1.1
Local AS number : 100
Total routes of Route Distinguisher(100:1): 1
BGP routing table entry information of 0:177.1.1.1:32:
Label information (Received/Applied): 100/NULL
From: 10.1.1.4 (10.1.1.4)
Route Duration: 0d00h24m23s
Relay IP Nexthop: 10.1.14.4
Relay Tunnel Out-Interface:
Original nexthop: 20.1.1.3
Qos information : 0x0
Ext-Community:RT <100 : 1>, RT <1000 : 1>, OSPF DOMAIN ID <0.0.0.0 : 0>, OSPF ROUTER ID <10.1.36.3 : 0>, OSPF RT <0.0.0.0 : 0 : 1>, Tunnel Type <VxLan(8)>, Router's MAC <707b-e8e6-5d85>
AS-path Nil, origin incomplete, MED 2, localpref 100, pref-val 0, valid, internal, best, select, pre 255, IGP cost 2
Originator: 10.1.1.3
Cluster list: 10.1.1.4
Route Type: 5 (Ip Prefix Route)
Ethernet Tag ID: 0, IP Prefix/Len: 177.1.1.1/32, ESI: 0000.0000.0000.0000.0000, GW IP Address: 0.0.0.0
Not advertised to any peer yet
EVPN-Instance __vni100__:
Number of Ip Prefix Routes: 1
BGP routing table entry information of 0:177.1.1.1:32:
Route Distinguisher: 100:1
Remote-Cross route
Label information (Received/Applied): 100/NULL
From: 10.1.1.4 (10.1.1.4)
Route Duration: 0d00h24m23s
Relay Tunnel Out-Interface: VXLAN
Original nexthop: 20.1.1.3
Qos information : 0x0
Ext-Community:RT <100 : 1>, RT <1000 : 1>, OSPF DOMAIN ID <0.0.0.0 : 0>, OSPF ROUTER ID <10.1.36.3 : 0>, OSPF RT <0.0.0.0 : 0 : 1>, Tunnel Type <VxLan(8)>, Router's MAC <707b-e8e6-5d85>
AS-path Nil, origin incomplete, MED 2, localpref 100, pref-val 0, valid, internal, best, select, pre 255
Originator: 10.1.1.3
Cluster list: 10.1.1.4
Route Type: 5 (Ip Prefix Route)
Ethernet Tag ID: 0, IP Prefix/Len: 177.1.1.1/32, ESI: 0000.0000.0000.0000.0000, GW IP Address: 0.0.0.0
Not advertised to any peer yet
去往Leaf3的177.1.1.1/32的路由，下一跳为vtep地址20.1.1.3,vxlan封装。查看路由关联隧道信息：
<Leaf-1>display ip routing-table vpn-instance vpn1 177.1.1.1 verbose
Route Flags: R - relay, D - download to fib, T - to vpn-instance, B - black hole route
------------------------------------------------------------------------------
Routing Table : vpn1
Summary Count : 1
Destination: 177.1.1.1/32
Protocol: IBGP Process ID: 0
Preference: 255 Cost: 2
NextHop: 20.1.1.3 Neighbour: 0.0.0.0
State: Active Adv Relied Age: 00h29m42s
Tag: 0 Priority: low
Label: NULL QoSInfo: 0x0
IndirectID: 0xAB00009C
RelayNextHop: 0.0.0.0 Interface: VXLAN
TunnelID: 0x0000000027f000000e Flags: RD
<Leaf-1>display tunnel all
Tunnel ID Type Destination Status
-----------------------------------------------------------------------------
0x0000000027f000000d vxlan_nvo3 20.1.1.2 UP
0x0000000027f000000e vxlan_nvo3 20.1.1.3 Up5.2.4 Type 2路由
type2路由分三种：mac route, arp route, irb route。如下所示：
leaf1发给spine1的mac路由抓包：
mac路由抓包：
Frame 28: 510 bytes on wire (4080 bits), 510 bytes captured (4080 bits) on interface 0
Ethernet II, Src: 4e:01:00:21:01:00 (4e:01:00:21:01:00), Dst: 4e:01:00:11:01:01 (4e:01:00:11:01:01)
Internet Protocol Version 4, Src: 10.1.1.1, Dst: 10.1.1.4
Transmission Control Protocol, Src Port: 65056, Dst Port: 179, Seq: 65, Ack: 65, Len: 456
Border Gateway Protocol - UPDATE Message
Border Gateway Protocol - UPDATE Message
Marker: ffffffffffffffffffffffffffffffff
Length: 120
Type: UPDATE Message (2)
Withdrawn Routes Length: 0
Total Path Attribute Length: 97
Path attributes
Path Attribute - ORIGIN: INCOMPLETE
Path Attribute - AS_PATH: empty
Path Attribute - LOCAL_PREF: 100
Path Attribute - EXTENDED_COMMUNITIES
Flags: 0xc0, Optional, Transitive, Complete
Type Code: EXTENDED_COMMUNITIES (16)
Length: 32
Carried extended communities: (4 communities)
Route Target: 10:1 [Transitive 2-Octet AS-Specific]
Route Target: 1000:1 [Transitive 2-Octet AS-Specific]
Encapsulation: VXLAN Encapsulation [Transitive Opaque]
MAC Mobility: Sticky MAC [Transitive EVPN]
Path Attribute - MP_REACH_NLRI
Flags: 0x90, Optional, Extended-Length, Non-transitive, Complete
Type Code: MP_REACH_NLRI (14)
Length: 44
Address family identifier (AFI): Layer-2 VPN (25)
Subsequent address family identifier (SAFI): EVPN (70)
Next hop network address (4 bytes)
Number of Subnetwork points of attachment (SNPA): 0
Network layer reachability information (35 bytes)
EVPN NLRI: MAC Advertisement Route
Route Type: MAC Advertisement Route (2)
Length: 33
Route Distinguisher: 0000000a00000001 (10:1)
ESI: 00:00:00:00:00:00:00:00:00:00
Ethernet Tag ID: 0
MAC Address Length: 48
MAC Address: EquipTra_01:00:01 (00:01:00:01:00:01)
IP Address Length: 0
IP Address: NOT INCLUDED
[Expert Info (Note/Protocol): IP Address: NOT INCLUDED]
a. 这是一条mac route，发送的mac地址为启用了分布式网关的int vbdif 10端口的mac地址：另外该处MP_REACH_NLRI并没有携带L2VNI值，Leaf2收到之后通过对比其携带的RT值（10：1和1000:1），发现其中10：1和本地BD10对应的EVPN实例iRT相同，则将该路由放置在对应的EVPN实例中，且打上L2VNI。如下所示：
[~Leaf-2]display bgp evpn all routing-table mac 0:48:0001-0001-0001:0:0.0.0.0
BGP local router ID : 10.1.1.2
Local AS number : 100
Total routes of Route Distinguisher(10:1): 2
BGP routing table entry information of 0:48:0001-0001-0001:0:0.0.0.0:
Label information (Received/Applied): 10/NULL
From: 0.0.0.0 (0.0.0.0)
Route Duration: 0d04h04m10s
Direct Out-interface:
Original nexthop: 20.1.1.2
Qos information : 0x0
Ext-Community:RT <10 : 1>, RT <1000 : 1>, Tunnel Type <VxLan(8)>, Mac Mobility <flag:1 seq:0 res:0>
AS-path Nil, origin incomplete, pref-val 0, valid, local, best, select, pre 255
Route Type: 2 (MAC Advertisement Route)
Ethernet Tag ID: 0, MAC Address/Len: 0001-0001-0001/48, IP Address/Len: 0.0.0.0/0, ESI:0000.0000.0000.0000.0000
Advertised to such 1 peers:
10.1.1.4
BGP routing table entry information of 0:48:0001-0001-0001:0:0.0.0.0:
Label information (Received/Applied): 10/NULL
From: 10.1.1.4 (10.1.1.4)
Route Duration: 0d01h04m10s
Relay IP Nexthop: 10.1.24.4
Relay Tunnel Out-Interface:
Original nexthop: 20.1.1.1
Qos information : 0x0
Ext-Community:RT <10 : 1>, RT <1000 : 1>, Tunnel Type <VxLan(8)>, Mac Mobility <flag:1 seq:0 res:0>
AS-path Nil, origin incomplete, localpref 100, pref-val 0, valid, internal, pre 255, IGP cost 2, not preferred for route type
Originator: 10.1.1.1
Cluster list: 10.1.1.4
Route Type: 2 (MAC Advertisement Route)
Ethernet Tag ID: 0, MAC Address/Len: 0001-0001-0001/48, IP Address/Len: 0.0.0.0/0, ESI:0000.0000.0000.0000.0000
Not advertised to any peer yet
EVPN-Instance 10:
Number of Mac Routes: 2
BGP routing table entry information of 0:48:0001-0001-0001:0:0.0.0.0:
Route Distinguisher: 10:1
Label information (Received/Applied): 10/NULL
From: 0.0.0.0 (0.0.0.0)
Route Duration: 0d04h04m12s
Direct Out-interface:
Original nexthop: 20.1.1.2
Qos information : 0x0
Ext-Community:Tunnel Type <VxLan(8)>, Mac Mobility <flag:1 seq:0 res:0>
AS-path Nil, origin incomplete, pref-val 0, valid, local, best, select, pre 255
Route Type: 2 (MAC Advertisement Route)
Ethernet Tag ID: 0, MAC Address/Len: 0001-0001-0001/48, IP Address/Len: 0.0.0.0/0, ESI:0000.0000.0000.0000.0000
Not advertised to any peer yet
BGP routing table entry information of 0:48:0001-0001-0001:0:0.0.0.0:
Route Distinguisher: 10:1
Remote-Cross route
Label information (Received/Applied): 10/NULL
From: 10.1.1.4 (10.1.1.4)
Route Duration: 0d01h04m12s
Relay Tunnel Out-Interface: VXLAN
Original nexthop: 20.1.1.1
Qos information : 0x0
Ext-Community:RT <10 : 1>, RT <1000 : 1>, Tunnel Type <VxLan(8)>, Mac Mobility <flag:1 seq:0 res:0>
AS-path Nil, origin incomplete, localpref 100, pref-val 0, valid, internal, pre 255, not preferred for mac mobility
Originator: 10.1.1.1
Cluster list: 10.1.1.4
Route Type: 2 (MAC Advertisement Route)
Ethernet Tag ID: 0, MAC Address/Len: 0001-0001-0001/48, IP Address/Len: 0.0.0.0/0, ESI:0000.0000.0000.0000.0000
Not advertised to any peer yet
b. Leaf1发送的mac route，经过spine反射之后交给leaf2，由于分布式网关场景下leaf1和Lea2leaf2的网关地址和mac地址都一直，因此Leaf2有0001:0001:0001的mac route，本地优先。arp route 在分布式对称转发场景中不会产生，原因在于配置过程中指定了向BGP EVPN邻居发送的路由类型为IRB路由，如peer 10.1.1.4 advertise irb。
leaf1发给spine1的irb路由抓包：
irb路由抓包：
Frame 30: 764 bytes on wire (6112 bits), 764 bytes captured (6112 bits) on interface 0
Ethernet II, Src: 4e:01:00:21:01:00 (4e:01:00:21:01:00), Dst: 4e:01:00:11:01:01 (4e:01:00:11:01:01)
Internet Protocol Version 4, Src: 10.1.1.1, Dst: 10.1.1.4
Transmission Control Protocol, Src Port: 65055, Dst Port: 179, Seq: 65, Ack: 65, Len: 710
Border Gateway Protocol - UPDATE Message
Border Gateway Protocol - UPDATE Message
[Malformed Packet: BGP]
Border Gateway Protocol - UPDATE Message
Marker: ffffffffffffffffffffffffffffffff
Length: 127
Type: UPDATE Message (2)
Withdrawn Routes Length: 0
Total Path Attribute Length: 104
Path attributes
Path Attribute - ORIGIN: INCOMPLETE
Path Attribute - AS_PATH: empty
Path Attribute - LOCAL_PREF: 100
Path Attribute - EXTENDED_COMMUNITIES
Flags: 0xc0, Optional, Transitive, Complete
Type Code: EXTENDED_COMMUNITIES (16)
Length: 32
Carried extended communities: (4 communities)
Route Target: 10:1 [Transitive 2-Octet AS-Specific]
Route Target: 1000:1 [Transitive 2-Octet AS-Specific]
Encapsulation: VXLAN Encapsulation [Transitive Opaque]
Unknown subtype 0x03: 0x707b 0xe80d 0x0337 [Transitive EVPN] //
Path Attribute - MP_REACH_NLRI
Flags: 0x90, Optional, Extended-Length, Non-transitive, Complete
Type Code: MP_REACH_NLRI (14)
Length: 51
Address family identifier (AFI): Layer-2 VPN (25)
Subsequent address family identifier (SAFI): EVPN (70)
Next hop network address (4 bytes)
Number of Subnetwork points of attachment (SNPA): 0
Network layer reachability information (42 bytes)
EVPN NLRI: MAC Advertisement Route
Route Type: MAC Advertisement Route (2)
Length: 40
Route Distinguisher: 0000000a00000001 (10:1)
ESI: 00:00:00:00:00:00:00:00:00:00
Ethernet Tag ID: 0
MAC Address Length: 48
MAC Address: HuaweiTe_05:61:39 (54:89:98:05:61:39)
IP Address Length: 32
IPv4 address: 192.168.1.1
[Malformed Packet: BGP]
Border Gateway Protocol - UPDATE Message
Border Gateway Protocol - UPDATE Message
[Malformed Packet: BGP]
Border Gateway Protocol - UPDATE Message
[Malformed Packet: BGP]
a. 这是一条ibr route，发送的mac地址为leaf1下挂的vm1的IP和MAC信息，其vm1在向外通信的时候，网关根据arp信息生成irb路由；
b. Leaf1根据网关配置的arp collect host enable命令来实现arp信息收集，然后生成ibr路由；
c. Leaf1发送的ibr route，经过spine反射之后交给leaf2，同mac route一样该处MP_REACH_NLRI并没有携带L2VNI值，Leaf2收到之后通过对比其携带的RT值（10：1和1000:1），发现其中10：1和本地BD10对应的EVPN实例iRT相同，则将该路由放置在对应的EVPN实例中，且本地打上L2VNI值，同时发现1000:1和本地IP VPN实例的iRT相同，则将该路由放置在对应的IP VPN实例中，且本地打上L3VNI值。
d. Tpye2中arp和irb区别在于，ibr携带L2VNI、L3VNI以及Router's MAC，但是抓包过程中发现并没有携带L2VNI和L3VNI信息，该处实际不影响本地路由接收，原因如步骤c中所述。具体信息如下：
[~Leaf-2]display bgp evpn all routing-table mac-route 0:48:5489-9805-6139:32:192.168.1.1
BGP local router ID : 10.1.1.2
Local AS number : 100
Total routes of Route Distinguisher(10:1): 1
BGP routing table entry information of 0:48:5489-9805-6139:32:192.168.1.1:
Label information (Received/Applied): 10 100/NULL
From: 10.1.1.4 (10.1.1.4)
Route Duration: 0d00h00m15s
Relay IP Nexthop: 10.1.24.4
Relay Tunnel Out-Interface:
Original nexthop: 20.1.1.1
Qos information : 0x0
Ext-Community:RT <10 : 1>, RT <1000 : 1>, Tunnel Type <VxLan(8)>, Router's MAC <707b-e80d-0337>
AS-path Nil, origin incomplete, localpref 100, pref-val 0, valid, internal, best, select, pre 255, IGP cost 2
Originator: 10.1.1.1
Cluster list: 10.1.1.4
Route Type: 2 (MAC Advertisement Route)
Ethernet Tag ID: 0, MAC Address/Len: 5489-9805-6139/48, IP Address/Len: 192.168.1.1/32, ESI:0000.0000.0000.0000.0000
Not advertised to any peer yet
EVPN-Instance 10:
Number of Mac Routes: 1
BGP routing table entry information of 0:48:5489-9805-6139:32:192.168.1.1:
Route Distinguisher: 10:1
Remote-Cross route
Label information (Received/Applied): 10 100/NULL //10为L2VNI ，100为L3VNI
From: 10.1.1.4 (10.1.1.4)
Route Duration: 0d00h00m15s
Relay Tunnel Out-Interface: VXLAN
Original nexthop: 20.1.1.1
Qos information : 0x0
Ext-Community:RT <10 : 1>, RT <1000 : 1>, Tunnel Type <VxLan(8)>, Router's MAC <707b-e80d-0337>
AS-path Nil, origin incomplete, localpref 100, pref-val 0, valid, internal, best, select, pre 255
Originator: 10.1.1.1
Cluster list: 10.1.1.4
Route Type: 2 (MAC Advertisement Route)
Ethernet Tag ID: 0, MAC Address/Len: 5489-9805-6139/48, IP Address/Len: 192.168.1.1/32, ESI:0000.0000.0000.0000.0000
Not advertised to any peer yet
EVPN-Instance __vni100__:
Number of Mac Routes: 1
BGP routing table entry information of 0:48:5489-9805-6139:32:192.168.1.1:
Route Distinguisher: 10:1
Remote-Cross route
Label information (Received/Applied): 10 100/NULL
From: 10.1.1.4 (10.1.1.4)
Route Duration: 0d00h00m16s
Relay Tunnel Out-Interface: VXLAN
Original nexthop: 20.1.1.1
Qos information : 0x0
Ext-Community:RT <10 : 1>, RT <1000 : 1>, Tunnel Type <VxLan(8)>, Router's MAC <707b-e80d-0337>
AS-path Nil, origin incomplete, localpref 100, pref-val 0, valid, internal, best, select, pre 255
Originator: 10.1.1.1
Cluster list: 10.1.1.4
Route Type: 2 (MAC Advertisement Route)
Ethernet Tag ID: 0, MAC Address/Len: 5489-9805-6139/48, IP Address/Len: 192.168.1.1/32, ESI:0000.0000.0000.0000.0000
Not advertised to any peer yet5.3、同网段，不同Leaf服务器Ping测试
执行操作：192.168.1.1Ping 192.168.1.2，并在Leaf1连Spine1端口抓包，其来回路径为VM1-vSwitch1-Leaf1-Spine1-Leaf2-vSwtich2-VM3；
vm1首次和vm3通信，发送arp报文学习vm3的mac地址；
Leaf1通过原mac学习到vm1的MAC地址、BDID（二层广播域标识）和报文入接口的对应关系，并在本地MAC表中生成vm1的MAC表项，其出接口为GE1/0/1.10。
[~Leaf-1]display mac-address dynamic
Flags: * - Backup
BD : bridge-domain Age : dynamic MAC learned time in seconds
-------------------------------------------------------------------------------
MAC Address VLAN/VSI/BD Learned-From Type Age
-------------------------------------------------------------------------------
5489-9805-6139 -/-/10 GE1/0/1.10 dynamic -
-------------------------------------------------------------------------------
Total items: 1
[~Leaf-1]Leaf1上VTEP根据对应的二层广播域获取对应VNI10的头端复制隧道列表，依据获取的隧道列表进行报文复制，并进行VXLAN封装。然后将封装后的报文从出接口转发出去。
a. Leaf1关于VNI10的头端复制隧道列表如下：
[~Leaf-1]display vxlan peer
Number of peers : 2
Vni ID Source Destination Type Out Vni ID
-------------------------------------------------------------------------------
10 20.1.1.1 20.1.1.2 dynamic 10
20 20.1.1.1 20.1.1.2 dynamic 20
[~Leaf-1]
b. 封装报文信息如下：
Frame 346: 110 bytes on wire (880 bits), 110 bytes captured (880 bits) on interface 0
Ethernet II, Src: 38:42:a9:01:01:00 (38:42:a9:01:01:00), Dst: 38:42:a9:04:01:01 (38:42:a9:04:01:01)
Internet Protocol Version 4, Src: 20.1.1.1, Dst: 20.1.1.2
User Datagram Protocol, Src Port: 4789, Dst Port: 4789
Virtual eXtensible Local Area Network
Flags: 0x0800, VXLAN Network ID (VNI)
Group Policy ID: 0
VXLAN Network Identifier (VNI): 10
Reserved: 0
Ethernet II, Src: HuaweiTe_05:61:39 (54:89:98:05:61:39), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
Address Resolution Protocol (request)Leaf2上VTEP收到VXLAN报文后，根据UDP目的端口号、源/目的IP地址、VNI判断VXLAN报文的合法有效性。然后依据VNI获取对应的二层广播域，进行VXLAN解封装，获取内层二层报文。同时学习到vm1的MAC地址、BDID（二层广播域标识）和报文入VTEP的对应关系，形成mac转发表，如下：
[~Leaf-2-GE1/0/1.20]display mac-address dynamic
Flags: * - Backup
BD : bridge-domain Age : dynamic MAC learned time in seconds
-------------------------------------------------------------------------------
MAC Address VLAN/VSI/BD Learned-From Type Age
-------------------------------------------------------------------------------
5489-9805-6139 -/-/10 20.1.1.1 dynamic -
-------------------------------------------------------------------------------Leaf2检查内层二层报文的目的MAC，发现是BUM MAC，在对应的二层广播域内的非VXLAN隧道侧进行广播处理，即：Leaf2分别从本地MAC表中找到非VXLAN隧道侧的所有出接口（GE1/0/1.20）和封装信息，为报文添加VLAN Tag30，转发给对应的终端vm3。
vm3收到该arp报文解封装之后，发现是请求自己的mac地址，则通过单播的方式响应arp报文，单播的目的mac地址为5489-9805-6139。
Leaf2收到该arp应答报文后，通过原mac学习到vm1的MAC地址、BDID（二层广播域标识）和报文入接口的对应关系，并在本地MAC表中生成vm1的MAC表项，其出接口为GE1/0/1.20。
<Leaf-2>display mac-address dynamic
Flags: * - Backup
BD : bridge-domain Age : dynamic MAC learned time in seconds
-------------------------------------------------------------------------------
MAC Address VLAN/VSI/BD Learned-From Type Age
-------------------------------------------------------------------------------
5489-9805-6139 -/-/10 20.1.1.1 dynamic -
5489-988b-4f15 -/-/10 GE1/0/1.20 dynamic -
-------------------------------------------------------------------------------
Total items: 2Leaf2同时查看mac表项，发现目的mac地址出接口为20.1.1.1，且VNI为10，随即查看VXLAN隧道信息进行封装。
a. Leaf2的VXLAN隧道信息如下：
<Leaf-2>display vxlan tunnel
Number of vxlan tunnel : 2
Tunnel ID Source Destination State Type Uptime
--------------------------------------------------------------------------------
---
4026531841 20.1.1.2 20.1.1.3 up dynamic 00:41:53
4026531842 20.1.1.2 20.1.1.1 up dynamic 00:41:51
<Leaf-2>
b. 封装报文信息如下：
Frame 347: 110 bytes on wire (880 bits), 110 bytes captured (880 bits) on interface 0
Ethernet II, Src: 38:42:a9:04:01:01 (38:42:a9:04:01:01), Dst: 38:42:a9:01:01:00 (38:42:a9:01:01:00)
Internet Protocol Version 4, Src: 20.1.1.2, Dst: 20.1.1.1
User Datagram Protocol, Src Port: 4789, Dst Port: 4789
Virtual eXtensible Local Area Network
Flags: 0x0800, VXLAN Network ID (VNI)
Group Policy ID: 0
VXLAN Network Identifier (VNI): 10
Reserved: 0
Ethernet II, Src: HuaweiTe_8b:4f:15 (54:89:98:8b:4f:15), Dst: HuaweiTe_05:61:39 (54:89:98:05:61:39)
Address Resolution Protocol (reply)Leaf1上VTEP收到VXLAN报文后，根据UDP目的端口号、源/目的IP地址、VNI判断VXLAN报文的合法有效性。然后依据VNI获取对应的二层广播域，进行VXLAN解封装，获取内层二层报文。同时学习到vm3的MAC地址、BDID（二层广播域标识）和报文入VTEP的对应关系，形成mac转发表，如下：
[~Leaf-1]display mac-address dynamic
Flags: * - Backup
BD : bridge-domain Age : dynamic MAC learned time in seconds
-------------------------------------------------------------------------------
MAC Address VLAN/VSI/BD Learned-From Type Age
-------------------------------------------------------------------------------
5489-988b-4f15 -/-/10 20.1.1.2 dynamic -
5489-9805-6139 -/-/10 GE1/0/1.10 dynamic -
-------------------------------------------------------------------------------
Total items: 2Leaf2检查内层二层报文的目的MAC，发现是单播MAC，查看mac表项，找到出接口（GE1/0/1.10）和封装信息，为报文添加VLAN Tag30，转发给对应的终端vm1。
vm1收到arp应答报文后学习到vm3的mac地址后完成报文封装。
Leaf1收到后查看mac表项，发现目的mac地址的出接口为20.1.1.2，且VNI为10，随即查看VXLAN隧道信息进行封装。
报文封装：
Frame 348: 124 bytes on wire (992 bits), 124 bytes captured (992 bits) on interface 0
Ethernet II, Src: 38:42:a9:01:01:00 (38:42:a9:01:01:00), Dst: 38:42:a9:04:01:01 (38:42:a9:04:01:01)
Internet Protocol Version 4, Src: 20.1.1.1, Dst: 20.1.1.2
User Datagram Protocol, Src Port: 4789, Dst Port: 4789
Virtual eXtensible Local Area Network
Flags: 0x0800, VXLAN Network ID (VNI)
Group Policy ID: 0
VXLAN Network Identifier (VNI): 10
Reserved: 0
Ethernet II, Src: HuaweiTe_05:61:39 (54:89:98:05:61:39), Dst: HuaweiTe_8b:4f:15 (54:89:98:8b:4f:15)
Internet Protocol Version 4, Src: 192.168.1.1, Dst: 192.168.1.2
Internet Control Message ProtocolLeaf2上VTEP收到VXLAN报文后，根据UDP目的端口号、源/目的IP地址、VNI判断VXLAN报文的合法有效性。然后依据VNI获取对应的二层广播域，进行VXLAN解封装，获取内层icmp request 报文，根据目的mac地址查到mac表项，即从Leaf2的VNI10 对应的GE1/0/1.20接口转发出去。
vm3收到icmp request报文后随即回复icmp reply报文，其仍然是单播转发，其过程同icmp request类似。
需要注意，在ibr对称型分布式网关场景中，二层通信没有BGP EVPN没有相关mac的学习信息。同网段通信，完全依靠数据平面arp完成，控制平面的BGP EVPN没有触发任何动作（没有update，没有mac学习）。因为同网段通信，并不解析网关地址，所以没有触发bgp evpn控制平面学习。
5.4、不同网段，同一Leaf下服务器Ping测试
执行操作：192.168.1.1 Ping 192.168.2.1，并在Leaf1连Spine1端口抓包，其来回路径为VM1-vSwitch1-Leaf1-vSwtich2-VM2。
该过程同集中转发场景中三层互访一至，也和传统的单臂路由场景类似。
5.5、不同网段，不同Leaf下服务器Ping
执行操作：192.168.2.1 Ping 192.168.1.2，并在Leaf1连Spine1端口抓包，其来回路径为VM2-vSwitch1-Leaf1-Leaf2-vSwtich2-VM3。
Leaf1收到来自vm2的报文，检测到报文的目的MAC是网关接口MAC，判断该报文需要进行三层转发。
Leaf1根据报文的入接口找到对应的二层广播域，然后找到绑定该广播域VBDIF接口的L3VPN实例。根据报文的目的IP地址，查找该L3VPN实例下的路由表，获取该路由对应的三层VNI，以及下一跳地址。再根据出接口是VXLAN隧道，判断需要进行VXLAN封装：
a. 根据tpye2-ibr路由学习到192.168.1.2/32位路由，如下：
[~Leaf-1]display ip routing-table vpn-instance vpn1
Proto: Protocol Pre: Preference
Route Flags: R - relay, D - download to fib, T - to vpn-instance, B - black hole
route
------------------------------------------------------------------------------
Routing Table : vpn1
Destinations : 10 Routes : 10
Destination/Mask Proto Pre Cost Flags NextHop Interface
10.1.36.0/24 IBGP 255 2 RD 20.1.1.3 VXLAN
177.1.1.1/32 IBGP 255 2 RD 20.1.1.3 VXLAN
192.168.1.0/24 Direct 0 0 D 192.168.1.254 Vbdif10
192.168.1.2/32 IBGP 255 0 RD 20.1.1.2 VXLAN //该路由条目有irb路由
192.168.1.254/32 Direct 0 0 D 127.0.0.1 Vbdif10
192.168.1.255/32 Direct 0 0 D 127.0.0.1 Vbdif10
192.168.2.0/24 Direct 0 0 D 192.168.2.254 Vbdif20
192.168.2.254/32 Direct 0 0 D 127.0.0.1 Vbdif20
192.168.2.255/32 Direct 0 0 D 127.0.0.1 Vbdif20
255.255.255.255/32 Direct 0 0 D 127.0.0.1 InLoopBack0
b. Leaf1关于192.168.1.2/32路由的详细信息如下：
[~Leaf-1]display ip routing-table vpn-instance vpn1 192.168.1.2 verbose
Route Flags: R - relay, D - download to fib, T - to vpn-instance, B - black hole
route
------------------------------------------------------------------------------
Routing Table : vpn1
Summary Count : 1
Destination: 192.168.1.2/32
Protocol: IBGP Process ID: 0
Preference: 255 Cost: 0
NextHop: 20.1.1.2 Neighbour: 10.1.1.4
State: Active Adv Relied Age: 00h02m17s
Tag: 0 Priority: low
Label: NULL QoSInfo: 0x0
IndirectID: 0x1000080 Instance:
RelayNextHop: 0.0.0.0 Interface: VXLAN
TunnelID: 0x0000000027f0000001 Flags: RD
c. Leaf1上关于192.168.1.2/32的EVPN路由如下：
[~Leaf-1]display bgp evpn all routing-table mac-route 0:48:5489-988b-4f15:32:192.168.1.2
EVPN-Instance __RD_1_100_1__:
Number of Mac Routes: 1
BGP routing table entry information of 0:48:5489-988b-4f15:32:192.168.1.2:
Route Distinguisher: 10:1
Remote-Cross route
Label information (Received/Applied): 10 100/NULL
From: 10.1.1.4 (10.1.1.4)
Route Duration: 0d00h17m49s
Relay Tunnel Out-Interface: VXLAN
Original nexthop: 20.1.1.2
Qos information : 0x0
Ext-Community: RT <10 : 1>, RT <1000 : 1>, Tunnel Type <VxLan>, Router's MAC <707b-e85f-46d1>
AS-path Nil, origin incomplete, localpref 100, pref-val 0, valid, internal, best, select, pre 255
Originator: 10.1.1.2
Cluster list: 10.1.1.4
Route Type: 2 (MAC Advertisement Route)
Ethernet Tag ID: 0, MAC Address/Len: 5489-988b-4f15/48, IP Address/Len: 192.168.1.2/32, ESI:0000.0000.0000.0000.0000
Not advertised to any peer yet
d. Leaf1关于VXLAN隧道信息如下：
[~Leaf-1] display tunnel all
Tunnel ID Type Destination
Status
--------------------------------------------------------------------------------
--------
0x0000000027f0000001 vxlan_nvo3 20.1.1.2
UP
0x0000000027f0000002 vxlan_nvo3 20.1.1.3
UP
根据VXLAN隧道的目的IP和源IP地址，获取对应的MAC地址，并将内层目的MAC和源MAC替换。
a. 内层源MAC地址为vm2网关地址192.168.2.254(IP VPN实例入口)接口的mac地址，即0002-0002-0002；
b. 内网目的MAC地址为Leaf2上VTEP的mac地址，该MAC地址通过irb路由中Router's MAC 携带。
将三层VNI封装到报文中。
外层封装VXLAN隧道的目的IP和源IP地址，源MAC地址为Leaf1的出接口MAC地址，目的MAC地址为网络下一跳的MAC地址。报文封装如下
报文封装
Frame 2084: 124 bytes on wire (992 bits), 124 bytes captured (992 bits) on interface 0
Ethernet II, Src: 38:42:a9:01:01:00 (38:42:a9:01:01:00), Dst: 38:42:a9:04:01:01 (38:42:a9:04:01:01)
Internet Protocol Version 4, Src: 20.1.1.1, Dst: 20.1.1.2
User Datagram Protocol, Src Port: 4789, Dst Port: 4789
Virtual eXtensible Local Area Network
Flags: 0x0800, VXLAN Network ID (VNI)
Group Policy ID: 0
VXLAN Network Identifier (VNI): 100
Reserved: 0
Ethernet II, Src: NetSys_02:00:02 (00:02:00:02:00:02), Dst: HuaweiTe_5f:46:d1 (70:7b:e8:5f:46:d1)
Destination: HuaweiTe_5f:46:d1 (70:7b:e8:5f:46:d1)
Source: NetSys_02:00:02 (00:02:00:02:00:02)
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 192.168.2.1, Dst: 192.168.1.2
Internet Control Message Protocol封装后的报文根据外层MAC和IP信息在IP网络中传输，送达Leaf2。
Leaf2收到VXLAN报文后进行解封装，检测到报文外层的目的MAC是自己的MAC地址，判断该报文需要进行三层转发，进行外层解封装，后发现内层的目的MAC是自己VTEP的MAC地址，继续解封装；
Leaf2根据报文携带的三层VNI找到对应的L3VPN实例，查找该L3VPN实例下的路由表，发现192.1681.2所在的网段路由获取报文的下一跳是网关接口地址，然后进行ARP迭代，将目的MAC地址替换为VM3的MAC地址，源MAC地址替换为Leaf2的网关地址为192.168.1.254的MAC地址，转发给vm3。
vm3给vm2回复的报文转发流程类似。
5.6、模拟外部和服务器之间Ping
Leaf3配置IP前缀路由发布，因此R1上177.1.1.1/32位路由条目通过bgp evpn type 5 prefix-route update, 携带如下信息：
a. type5携带L3VPN的export RT值：100:1和1000:1（evpn）；
b. type5只携带L3VNI：100；
c. Type5携带Next hop network address，即为Leaf3的VTEP的IP地址；
d. type5携带Router’s MAC ，即为Leaf3的NVE（VTEP）端口mac地址：
具体报文和路由信息详见章节5.2.3。Leaf1和2配置对邻居发布IRB路由，因此Leaf1和Leaf2通过ARP信息收集的主机信息（Leaf1上192.168.1.1/32和192.168.2.1/32；Leaf2上192.1681.2/32）会通过bgp evpn type 2 irb update，以Leaf1为例携带信息如下：
a.type2携带EVPN实例的export RT值：10:1和1000:1
b.type2不携带L2 VNI和L3VNI //该处和资料有出入，可能是模拟器软件版本原因，但是实际不影响对端路由接收，分析详见章节5.2.4
c.Type2携带Next hop network address，即为Leaf1的VTEP的IP地址；
d.Type2携带Router’s MAC ，即为Leaf1的NVE（VTEP）端口mac地址：Leaf2根据该路由信息用于不用网段，不同Leaf下互访（即192.168.1.2和192.168.1.1之间），Leaf3根据该路由信息用于外部和服务器之间互访。以177.1.1.1 ping 192.168.1.1为例具体流程如下：
a. R1产生icmp request报文，如下：
Frame 4: 98 bytes on wire (784 bits), 98 bytes captured (784 bits) on interface 0
Ethernet II, Src: HuaweiTe_dc:48:25 (54:89:98:dc:48:25), Dst: 38:42:a9:03:01:01 (38:42:a9:03:01:01)
Internet Protocol Version 4, Src: 177.1.1.1, Dst: 192.168.1.1
Internet Control Message Protocol
b. Leaf3从接口GE1/0/1接口收到该icmp request，匹配至IP VPN实例中，查找路由信息如下：
<Leaf-3>display ip routing-table vpn-instance vpn1 192.168.1.1
Proto: Protocol Pre: Preference
Route Flags: R - relay, D - download to fib, T - to vpn-instance, B - black hole route
------------------------------------------------------------------------------
Routing Table : vpn1
Summary Count : 1
Destination/Mask Proto Pre Cost Flags NextHop Interface
192.168.1.1/32 IBGP 255 0 RD 20.1.1.1 VXLAN
<Leaf-3>display ip routing-table vpn-instance vpn1 192.168.1.1 verbose
Route Flags: R - relay, D - download to fib, T - to vpn-instance, B - black hole route
------------------------------------------------------------------------------
Routing Table : vpn1
Summary Count : 1
Destination: 192.168.1.1/32
Protocol: IBGP Process ID: 0
Preference: 255 Cost: 0
NextHop: 20.1.1.1 Neighbour: 10.1.1.4
State: Active Adv Relied Age: 00h23m24s
Tag: 0 Priority: low
Label: NULL QoSInfo: 0x0
IndirectID: 0x1000072 Instance:
RelayNextHop: 0.0.0.0 Interface: VXLAN
TunnelID: 0x0000000027f0000004 Flags: RD
<Leaf-3>
<Leaf-3>display tunnel all
Tunnel ID Type Destination Status
----------------------------------------------------------------------------------------
0x0000000027f0000002 vxlan_nvo3 20.1.1.2 UP
0x0000000027f0000004 vxlan_nvo3 20.1.1.1 UP
<Leaf-3>display bgp evpn all routing-table mac-route 0:48:5489-9805-6139:32:192.168.1.1
EVPN-Instance __RD_1_100_1__:
Number of Mac Routes: 1
BGP routing table entry information of 0:48:5489-9805-6139:32:192.168.1.1:
Route Distinguisher: 10:1
Remote-Cross route
Label information (Received/Applied): 10 100/NULL
From: 10.1.1.4 (10.1.1.4)
Route Duration: 0d00h21m43s
Relay Tunnel Out-Interface: VXLAN
Original nexthop: 20.1.1.1
Qos information : 0x0
Ext-Community: RT <10 : 1>, RT <1000 : 1>, Tunnel Type <VxLan>, Router's MAC <707b-e80d-0337>
AS-path Nil, origin incomplete, localpref 100, pref-val 0, valid, internal, best, select, pre 255
Originator: 10.1.1.1
Cluster list: 10.1.1.4
Route Type: 2 (MAC Advertisement Route)
Ethernet Tag ID: 0, MAC Address/Len: 5489-9805-6139/48, IP Address/Len: 192.168.1.1/32, ESI:0000.0000.0000.0000.0000
Not advertised to any peer yet
c. Leaf3匹配路由，进行VXLAN封装：
根据VXLAN隧道的目的IP和源IP地址，获取对应的MAC地址，并将内层目的MAC和源MAC替换。
i.内层源MAC地址为GE1/0/1(IP VPN实例入口)接口的mac地址，即38:42:a9:03:01:01；
ii.内网目的MAC地址为Leaf1上VTEP的mac地址，该MAC地址通过irb路由中Router's MAC 携带。
将三层VNI封装到报文中。
外层封装VXLAN隧道的目的IP和源IP地址，源MAC地址为Leaf1的出接口MAC地址，目的MAC地址为网络下一跳的MAC地址。
d. 报文封装如下：
Frame 3306: 148 bytes on wire (1184 bits), 148 bytes captured (1184 bits) on interface 0
Ethernet II, Src: 38:42:a9:04:01:01 (38:42:a9:04:01:01), Dst: 38:42:a9:01:01:00 (38:42:a9:01:01:00)
Internet Protocol Version 4, Src: 20.1.1.3, Dst: 20.1.1.1
User Datagram Protocol, Src Port: 4789, Dst Port: 4789
Virtual eXtensible Local Area Network
Flags: 0x0800, VXLAN Network ID (VNI)
Group Policy ID: 0
VXLAN Network Identifier (VNI): 100
Reserved: 0
Ethernet II, Src: 38:42:a9:03:01:01 (38:42:a9:03:01:01), Dst: HuaweiTe_0d:03:37 (70:7b:e8:0d:03:37)
Internet Protocol Version 4, Src: 177.1.1.1, Dst: 192.168.1.1
Internet Control Message Protocol封装后的报文根据外层MAC和IP信息在IP网络中传输，送达Leaf1。
Leaf1收到VXLAN报文后进行解封装，检测到报文外层的目的MAC是自己的MAC地址，判断该报文需要进行三层转发，进行外层解封装，后发现内层的目的MAC是自己VTEP的MAC地址，继续解封装；
Leaf1根据报文携带的三层VNI找到对应的L3VPN实例，查找该L3VPN实例下的路由表，发现192.168.1.1所在的网段路由获取报文的下一跳是网关接口地址，然后进行ARP迭代，将目的MAC地址替换为VM1的MAC地址，源MAC地址替换为Leaf1的网关地址为192.168.1.254的MAC地址，转发给vm1。
vm1给R1回复的报文转发流程类似。
5.7、arp广播抑制和代答
arp广播抑制，作用域L2和L3网关。
在终端租户初次互通过程中，终端租户会发送ARP广播请求报文，而ARP请求报文会在二层网络内广播。为了抑制ARP广播请求报文给网络带来的广播风暴，可在VXLAN二层网关设备上使能ARP广播抑制功能。但是，ARP广播抑制功能的实现依赖于三层网关上的主机信息表（包括主机IP地址、MAC地址、VTEP地址和VNI ID，主机信息表通过arp collect host enable获取）。
a. 配置如下：
在bridge-domain下配置arp broadcast-suppress，使能ARP广播抑制功能。
Leaf1和Leaf2上类似，如下：
bridge-domain 10
arp broadcast-suppress enable
#
bridge-domain 20
arp broadcast-suppress enable
#
b. 结果验证：
vm1访问vm3同网段，不同leaf互访，leaf1收到vm1发送的arp报文后，会根据irb路由信息进行arp广播抑制，如下：
Frame 6: 114 bytes on wire (912 bits), 114 bytes captured (912 bits) on interface 0
Ethernet II, Src: 38:42:a9:03:01:00 (38:42:a9:03:01:00), Dst: 38:42:a9:02:01:01 (38:42:a9:02:01:01)
Internet Protocol Version 4, Src: 20.1.1.1, Dst: 20.1.1.2
User Datagram Protocol, Src Port: 4789, Dst Port: 4789
Virtual eXtensible Local Area Network
Flags: 0x0800, VXLAN Network ID (VNI)
Group Policy ID: 0
VXLAN Network Identifier (VNI): 10
Reserved: 0
Ethernet II, Src: HuaweiTe_05:61:39 (54:89:98:05:61:39), Dst: HuaweiTe_8b:4f:15 (54:89:98:8b:4f:15)
Destination: HuaweiTe_8b:4f:15 (54:89:98:8b:4f:15)
Source: HuaweiTe_05:61:39 (54:89:98:05:61:39)
Type: 802.1Q Virtual LAN (0x8100)
802.1Q Virtual LAN, PRI: 6, DEI: 0, ID: 10
Address Resolution Protocol (request)arp proxy/arp 代答，作用与L2网关。
arp l2-proxy enable ----BD下配置二层代答功能，可以配合arp collect host一起使用，即三层网关通过arp收集通过BGP EVPN更新至二层网关，二层网关收到arp请求报文后判断是否进行代答处理。
ARP二层代理是一种有效分散ARP报文处理压力的方法，其核心思想是隔离ARP广播域，对接收到的ARP请求报文尽量进行本地优先代答。执行arp l2-proxy enable命令使能ARP二层代理功能之后，设备在收到ARP报文时，会先查看自己能否获取到该ARP请求报文的目的用户的信息，如果能够获取就直接进行ARP代答，否则按照原先的转发流程转发该报文。
已经配置了VBDIF接口的Leaf上无法使能该功能。
本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。
