gobgp服务无损变更：graceful restart特性
场景
当我们的bgp网关在对外宣告bgp路由的时候，如果我们网关有新的特性要发布，那么此时如果把网关停止再启动新版本，此时bgp路由会有短暂撤回再播出的过程，会有网络抖动
期待的行为：无损变更
我们希望bgp网关服务在变更的时候，播出去的路由能够在bgp网关中断时，继续保持一段时间，除非过了这段时间，bgp网关仍无法正常启动，对端网络设备再进行路由撤回
graceful restart特性
- bgp服务非正常退出时，会启动优雅重启特性，此时路由不会马上撤回
- bgp服务是被SIGTERM信号终止的时候，则会马上回撤路由
解析：
通过此配置，如果与对等方协商了优雅重启功能，则对等方启动优雅重启帮助程序，当 gobgpd 非自愿死亡或 SIGINT 时，SIGKILL 信号发送到 gobgpd。请注意，当 SIGTERM 信号发送到 gobgpd 时，优雅重启协商的对等点不会启动优雅重启帮助程序，因为 gobgpd 在它死亡之前会向这些对等点发送通知消息
graceful restart演示：gobgp graceful restart example
场景：
192.168.128.132节点与192.168.128.134节点建立bgp连接，132向134宣告路由，同时132会模拟退出后，让134进行路由保持的特性，也即：graceful restart
192.168.128.132节点的bgp config文件：
[global.config]
as = 65001
router-id = "192.168.128.132"
[[neighbors]]
[neighbors.config]
neighbor-address = "192.168.128.134"
peer-as = 65001
[neighbors.graceful-restart.config]
enabled = true
restart-time = 30
[[neighbors.afi-safis]]
[neighbors.afi-safis.config]
afi-safi-name = "ipv4-unicast"
[neighbors.afi-safis.mp-graceful-restart.config]
enabled = true
[neighbors.afi-safis.long-lived-graceful-restart.config]
enabled = true
restart-time = 30192.168.128.134节点的bgp config文件：
[global.config]
as = 65001
router-id = "192.168.128.134"
[[neighbors]]
[neighbors.config]
neighbor-address = "192.168.128.132"
peer-as = 65001
[neighbors.graceful-restart.config]
enabled = true
long-lived-enabled = true
restart-time = 30
notification-enabled = true
[[neighbors.afi-safis]]
[neighbors.afi-safis.config]
afi-safi-name = "ipv4-unicast"
[neighbors.afi-safis.mp-graceful-restart.config]
enabled = true
[neighbors.afi-safis.long-lived-graceful-restart.config]
enabled = true
restart-time = 30启动两个bgp server：
sudo ./gobgpd -f bgp-graceful.conf -l debug -p -r在132上宣告一条路由：./gobgp global rib -a ipv4 add 192.168.3.0/24 origin igp
Key=192.168.128.134 Topic=config
INFO[0000] Add a peer configuration Key=192.168.128.134 Topic=Peer
DEBU[0000] IdleHoldTimer expired Duration=0 Key=192.168.128.134 Topic=Peer
DEBU[0000] state changed Key=192.168.128.134 Topic=Peer new=BGP_FSM_ACTIVE old=BGP_FSM_IDLE reason=idle-hold-timer-expired
DEBU[0005] try to connect Key=192.168.128.134 Topic=Peer
DEBU[0005] state changed Key=192.168.128.134 Topic=Peer new=BGP_FSM_OPENSENT old=BGP_FSM_ACTIVE reason=new-connection
DEBU[0005] state changed Key=192.168.128.134 Topic=Peer new=BGP_FSM_OPENCONFIRM old=BGP_FSM_OPENSENT reason=open-msg-received
INFO[0005] Peer Up Key=192.168.128.134 State=BGP_FSM_OPENCONFIRM Topic=Peer
DEBU[0005] state changed Key=192.168.128.134 Topic=Peer new=BGP_FSM_ESTABLISHED old=BGP_FSM_OPENCONFIRM reason=open-msg-negotiated
DEBU[0005] Now syncing, suppress sending updates. start deferral timer Duration=360 Key=192.168.128.134 Topic=Server
DEBU[0005] received update Key=192.168.128.134 Topic=Peer attributes="[]" nlri="[]" withdrawals="[]"
DEBU[0005] EOR received AddressFamily=ipv4-unicast Key=192.168.128.134 Topic=Peer
INFO[0005] sync finished Topic=Server
DEBU[0005] sent update Key=192.168.128.134 State=BGP_FSM_ESTABLISHED Topic=Peer attributes="[]" nlri="[]" withdrawals="[]"
DEBU[0012] create Destination Nlri=192.168.3.0/24 Topic=Table
DEBU[0012] sent update Key=192.168.128.134 State=BGP_FSM_ESTABLISHED Topic=Peer attributes="[{Origin: i} {Nexthop: 192.168.128.132} {LocalPref: 100}]" nlri="[192.168.3.0/24]" withdrawals="[]"在134上查看从132邻居学到的路由：
luzejia@luzejia-virtual-machine:~/Desktop$ ./gobgp neighbor 192.168.128.132 adj-in
ID Network Next Hop AS_PATH Age Attrs
0 192.168.3.0/24 192.168.128.132 00:00:02 [{Origin: i} {LocalPref: 100}]使用ctrl + c将bgp server停掉，可以看到做了一些清理现场的行为，让134对端知道你是正常退出，不需要启动优雅重启，直接回撤路由即可
sudo ./gobgpd -f bgp-graceful.conf -l debug -p -r
INFO[0000] gobgpd started
INFO[0000] Finished reading the config file Topic=Config
INFO[0000] Add Peer Key=192.168.128.134 Topic=config
INFO[0000] Add a peer configuration Key=192.168.128.134 Topic=Peer
DEBU[0000] IdleHoldTimer expired Duration=0 Key=192.168.128.134 Topic=Peer
DEBU[0000] state changed Key=192.168.128.134 Topic=Peer new=BGP_FSM_ACTIVE old=BGP_FSM_IDLE reason=idle-hold-timer-expired
DEBU[0001] Accepted a new passive connection Key=192.168.128.134 Topic=Peer
DEBU[0001] stop connect loop Key=192.168.128.134 Topic=Peer
DEBU[0001] state changed Key=192.168.128.134 Topic=Peer new=BGP_FSM_OPENSENT old=BGP_FSM_ACTIVE reason=new-connection
DEBU[0001] peer has restarted, skipping wait for EOR Key=192.168.128.134 State=BGP_FSM_OPENSENT Topic=Peer
DEBU[0001] state changed Key=192.168.128.134 Topic=Peer new=BGP_FSM_OPENCONFIRM old=BGP_FSM_OPENSENT reason=open-msg-received
INFO[0001] Peer Up Key=192.168.128.134 State=BGP_FSM_OPENCONFIRM Topic=Peer
DEBU[0001] state changed Key=192.168.128.134 Topic=Peer new=BGP_FSM_ESTABLISHED old=BGP_FSM_OPENCONFIRM reason=open-msg-negotiated
INFO[0001] sync finished Key=192.168.128.134 Topic=Server
DEBU[0001] received update Key=192.168.128.134 Topic=Peer attributes="[]" nlri="[]" withdrawals="[]"
DEBU[0001] EOR received AddressFamily=ipv4-unicast Key=192.168.128.134 Topic=Peer
DEBU[0001] sent update Key=192.168.128.134 State=BGP_FSM_ESTABLISHED Topic=Peer attributes="[]" nlri="[]" withdrawals="[]"
DEBU[0008] create Destination Nlri=192.168.3.0/24 Topic=Table
DEBU[0008] sent update Key=192.168.128.134 State=BGP_FSM_ESTABLISHED Topic=Peer attributes="[{Origin: i} {Nexthop: 192.168.128.132} {LocalPref: 100}]" nlri="[192.168.3.0/24]" withdrawals="[]"
^CINFO[0021] stopping gobgpd server
INFO[0021] Delete a peer configuration Key=192.168.128.134 Topic=Peer
INFO[0021] Peer Down Key=192.168.128.134 Reason=dying State=BGP_FSM_ESTABLISHED Topic=Peer
DEBU[0021] freed fsm.h Key=192.168.128.134 State=BGP_FSM_ESTABLISHED Topic=Peer134上观察路由，发现被回撤：
luzejia@luzejia-virtual-machine:~/Desktop$ ./gobgp neighbor 192.168.128.132 adj-in
neighbor 192.168.128.132's BGP session is not established查看134的日志，原因是识别到了132是peer down，然后回撤路由
INFO[0028] Peer Down Key=192.168.128.132 Reason="notification-received code 6(cease) subcode 3(peer deconfigured)" State=BGP_FSM_ESTABLISHED Topic=Peer
DEBU[0028] state changed Key=192.168.128.132 Topic=Peer new=BGP_FSM_IDLE old=BGP_FSM_ESTABLISHED reason="notification-received code 6(cease) subcode 3(peer deconfigured)"
DEBU[0028] Removing withdrawals Key=192.168.3.0/24 Topic=Table
DEBU[0033] IdleHoldTimer expired Duration=5 Key=192.168.128.132 Topic=Peer
DEBU[0033] state changed Key=192.168.128.132 Topic=Peer new=BGP_FSM_ACTIVE old=BGP_FSM_IDLE reason=idle-hold-timer-expired如果使用kill -9来杀掉132上的bgp server
luzejia@luzejia-virtual-machine:~/bgp$ sudo ./gobgpd -f bgp-graceful.conf -l debug -p -r
INFO[0000] gobgpd started
INFO[0000] Finished reading the config file Topic=Config
INFO[0000] Add Peer Key=192.168.128.134 Topic=config
INFO[0000] Add a peer configuration Key=192.168.128.134 Topic=Peer
DEBU[0000] IdleHoldTimer expired Duration=0 Key=192.168.128.134 Topic=Peer
DEBU[0000] state changed Key=192.168.128.134 Topic=Peer new=BGP_FSM_ACTIVE old=BGP_FSM_IDLE reason=idle-hold-timer-expired
DEBU[0005] try to connect Key=192.168.128.134 Topic=Peer
DEBU[0005] state changed Key=192.168.128.134 Topic=Peer new=BGP_FSM_OPENSENT old=BGP_FSM_ACTIVE reason=new-connection
DEBU[0005] state changed Key=192.168.128.134 Topic=Peer new=BGP_FSM_OPENCONFIRM old=BGP_FSM_OPENSENT reason=open-msg-received
INFO[0005] Peer Up Key=192.168.128.134 State=BGP_FSM_OPENCONFIRM Topic=Peer
DEBU[0005] state changed Key=192.168.128.134 Topic=Peer new=BGP_FSM_ESTABLISHED old=BGP_FSM_OPENCONFIRM reason=open-msg-negotiated
DEBU[0005] Now syncing, suppress sending updates. start deferral timer Duration=360 Key=192.168.128.134 Topic=Server
DEBU[0005] received update Key=192.168.128.134 Topic=Peer attributes="[]" nlri="[]" withdrawals="[]"
DEBU[0005] EOR received AddressFamily=ipv4-unicast Key=192.168.128.134 Topic=Peer
INFO[0005] sync finished Topic=Server
DEBU[0005] sent update Key=192.168.128.134 State=BGP_FSM_ESTABLISHED Topic=Peer attributes="[]" nlri="[]" withdrawals="[]"
DEBU[0012] create Destination Nlri=192.168.3.0/24 Topic=Table
DEBU[0012] sent update Key=192.168.128.134 State=BGP_FSM_ESTABLISHED Topic=Peer attributes="[{Origin: i} {Nexthop: 192.168.128.132} {LocalPref: 100}]" nlri="[192.168.3.0/24]" withdrawals="[]"
已杀死134上观察到132的路由还在，并且有S标志，这个是保留的意思，证明启动了优雅重启，暂时不回撤，等待对端重启：
luzejia@luzejia-virtual-machine:~/Desktop$ ./gobgp neighbor 192.168.128.132 adj-in
ID Network Next Hop AS_PATH Age Attrs
S 0 192.168.3.0/24 192.168.128.132 00:00:21 [{Origin: i} {LocalPref: 100}]134上看到识别出了peer是graceful restart，启动了优雅重启，没有马上回撤路由，但是过了超时时间后还是回撤了路由：
DEBU[0053] From same AS, ignore Key=192.168.128.132 Path="{ 192.168.3.0/24 | src: { 192.168.128.132 | as: 65001, id: 192.168.128.132 }, nh: 192.168.128.132 }" Topic=Peer
INFO[0071] peer graceful restart Key=192.168.128.132 State=BGP_FSM_ESTABLISHED Topic=Peer
INFO[0071] Peer Down Key=192.168.128.132 Reason=graceful-restart State=BGP_FSM_ESTABLISHED Topic=Peer
DEBU[0071] state changed Key=192.168.128.132 Topic=Peer new=BGP_FSM_IDLE old=BGP_FSM_ESTABLISHED reason=graceful-restart
DEBU[0071] Implicit withdrawal of old path, since we have learned new path from the same peer Key=192.168.3.0/24 Path="{ 192.168.3.0/24 | src: { 192.168.128.132 | as: 65001, id: 192.168.128.132 }, nh: 192.168.128.132 }" Topic=Table
DEBU[0076] IdleHoldTimer expired Duration=5 Key=192.168.128.132 Topic=Peer
DEBU[0076] state changed Key=192.168.128.132 Topic=Peer new=BGP_FSM_ACTIVE old=BGP_FSM_IDLE reason=idle-hold-timer-expired
DEBU[0085] try to connect Key=192.168.128.132 Topic=Peer
DEBU[0085] failed to connect Error="dial tcp 0.0.0.0:0->192.168.128.132:179: connect: connection refused" Key=192.168.128.132 Topic=Peer
WARN[0101] graceful restart timer expired Key=192.168.128.132 State=BGP_FSM_ACTIVE Topic=Peer
DEBU[0101] stop connect loop Key=192.168.128.132 Topic=Peer
DEBU[0101] state changed Key=192.168.128.132 Topic=Peer new=BGP_FSM_IDLE old=BGP_FSM_ACTIVE reason=restart-timer-expired
DEBU[0101] Removing withdrawals Key=192.168.3.0/24 Topic=Table
DEBU[0106] IdleHoldTimer expired Duration=5 Key=192.168.128.132 Topic=Peer
DEBU[0106] state changed Key=192.168.128.132 Topic=Peer new=BGP_FSM_ACTIVE old=BGP_FSM_IDLE reason=idle-hold-timer-expired
DEBU[0115] try to connect Key=192.168.128.132 Topic=Peer
DEBU[0115] failed to connect Error="dial tcp 0.0.0.0:0->192.168.128.132:179: connect: connection refused" Key=192.168.128.132 Topic=Peer过了graceful resstart的timeout时间后，看到路由被正常撤回
luzejia@luzejia-virtual-machine:~/Desktop$ ./gobgp neighbor 192.168.128.132 adj-in
neighbor 192.168.128.132's BGP session is not established总结：
- bgp服务非正常退出时，会启动优雅重启特性，此时路由不会马上撤回
- bgp服务是被SIGTERM信号终止的时候，则会马上回撤路由
注意：
- bgp服务是被SIGTERM信号终止的时候，则会马上回撤路由，这部分需要自己实现去捕捉SIGTERM信号，然后调用gobgp server的stop接口，才能实现路由回撤，也就是实际stop接口向对端宣告了一个自己是正常退出的down信息，从而告知对端此时不需要启动优雅重启特性来保持路由，直接回撤即可
捕捉SIGTERM信号并进行处理，参考gobgpd源码，给出一个example：
package main
import (
"fmt"
"io"
"net/http"
_ "net/http/pprof"
"os"
"os/signal"
"runtime"
"syscall"
"github.com/coreos/go-systemd/v22/daemon"
"github.com/jessevdk/go-flags"
"github.com/kr/pretty"
"github.com/sirupsen/logrus"
"golang.org/x/net/context"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials"
"github.com/osrg/gobgp/v3/internal/pkg/version"
"github.com/osrg/gobgp/v3/pkg/config"
"github.com/osrg/gobgp/v3/pkg/server"
)
func main() {
sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, syscall.SIGTERM, syscall.SIGINT)
...
...
logger.Info("gobgpd started")
bgpServer := server.NewBgpServer(server.GrpcListenAddress(opts.GrpcHosts), server.GrpcOption(grpcOpts), server.LoggerOption(&builtinLogger{logger: logger}))
go bgpServer.Serve()
for sig := range sigCh {
if sig != syscall.SIGHUP {
stopServer(bgpServer, opts.UseSdNotify)
return
}
logger.WithFields(logrus.Fields{
"Topic": "Config",
}).Info("Reload the config file")
newConfig, err := config.ReadConfigFile(opts.ConfigFile, opts.ConfigType)
if err != nil {
logger.WithFields(logrus.Fields{
"Topic": "Config",
"Error": err,
}).Warningf("Can't read config file %s", opts.ConfigFile)
continue
}
currentConfig, err = config.UpdateConfig(context.Background(), bgpServer, currentConfig, newConfig)
if err != nil {
logrus.WithFields(logrus.Fields{
"Topic": "Config",
"Error": err,
}).Warningf("Failed to update config %s", opts.ConfigFile)
continue
}
}
}
func stopServer(bgpServer *server.BgpServer, useSdNotify bool) {
logger.Info("stopping gobgpd server")
bgpServer.Stop()
if useSdNotify {
daemon.SdNotify(false, daemon.SdNotifyStopping)
}
}
本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。
