一、描述
mysql 8.0.19版本,三节点mgr架构,zabbix告警member_mode为offline。

二、 处理过程
1、查看日志
查看告警主机数据库的错误日志,梳理报错过程

2、查看MGR集群状态
查看现有集群是否运行正常,因为告警只提示单实例异常,所以可以先看日志。

3、根据分析,进行操作
处理过程很简单,使用start group_replication;等待即可

三、启动组复制告警信息
2021-02-24T03:09:00.813707+08:00 0 [Note] [MY-013471] [Repl] Plugin group_replication reported: 'Distributed recovery will transfer data using: Incremental recovery from a group donor'
2021-02-24T03:09:00.814035+08:00 585 [Note] [MY-011576] [Repl] Plugin group_replication reported: 'Establishing group recovery connection with a possible donor. Attempt 1/10'
2021-02-24T03:09:00.814054+08:00 0 [Note] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to mgr2:3311, mgr1:3311, mgr3:3311 on view 16023164214094886:13.'
2021-02-24T03:09:00.845097+08:00 585 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state master_host='', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='mgr3', master_port= 3311, master_log_file='', master_log_pos= 4, master_bind=''.
2021-02-24T03:09:00.874041+08:00 585 [Note] [MY-011580] [Repl] Plugin group_replication reported: 'Establishing connection to a group replication recovery donor 4a84d2c2-0ac9-11eb-8a31-fa163e1a972a at mgr3 port: 3311.'
2021-02-24T03:09:00.874524+08:00 586 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
2021-02-24T03:09:00.877341+08:00 586 [System] [MY-010562] [Repl] Slave I/O thread for channel 'group_replication_recovery': connected to master 'repl@mgr3:3311',replication started in log 'FIRST' at position 4
2021-02-24T03:09:00.932437+08:00 587 [Note] [MY-010581] [Repl] Slave SQL thread for channel 'group_replication_recovery' initialized, starting replication in log 'FIRST' at position 0, relay log '/zxlog/my3311/log/relay-log-group_replication_recovery.000001' position: 4
2021-02-24T03:11:00.016033+08:00 587 [Note] [MY-010559] [Repl] Multi-threaded slave statistics for channel 'group_replication_recovery': seconds elapsed = 120; events assigned = 211969; worker queues filled over overrun level = 0; waited due a Worker queue full = 0; waited due the total size = 0; waited at clock conflicts = 114749941900 waited (count) when Workers occupied = 0 waited when Workers occupied = 0
2021-02-24T03:13:00.200716+08:00 587 [Note] [MY-010559] [Repl] Multi-threaded slave statistics for channel 'group_replication_recovery': seconds elapsed = 120; events assigned = 459777; worker queues filled over overrun level = 0; waited due a Worker queue full = 0; waited due the total size = 0; waited at clock conflicts = 232870937300 waited (count) when Workers occupied = 0 waited when Workers occupied = 0
2021-02-24T03:15:00.228929+08:00 587 [Note] [MY-010559] [Repl] Multi-threaded slave statistics for channel 'group_replication_recovery': seconds elapsed = 120; events assigned = 735233; worker queues filled over overrun level = 0; waited due a Worker queue full = 0; waited due the total size = 0; waited at clock conflicts = 350703540000 waited (count) when Workers occupied = 0 waited when Workers occupied = 0
2021-02-24T03:17:00.348988+08:00 587 [Note] [MY-010559] [Repl] Multi-threaded slave statistics for channel 'group_replication_recovery': seconds elapsed = 120; events assigned = 1006593; worker queues filled over overrun level = 0; waited due a Worker queue full = 0; waited due the total size = 0; waited at clock conflicts = 468687630500 waited (count) when Workers occupied = 0 waited when Workers occupied = 0
2021-02-24T03:19:00.272722+08:00 587 [Note] [MY-010559] [Repl] Multi-threaded slave statistics for channel 'group_replication_recovery': seconds elapsed = 120; events assigned = 1295361; worker queues filled over overrun level = 0; waited due a Worker queue full = 0; waited due the total size = 0; waited at clock conflicts = 586516461900 waited (count) when Workers occupied = 0 waited when Workers occupied = 0
2021-02-24T03:19:58.442469+08:00 585 [Note] [MY-011585] [Repl] Plugin group_replication reported: 'Terminating existing group replication donor connection and purging the corresponding logs.'
2021-02-24T03:19:58.454104+08:00 587 [Note] [MY-010587] [Repl] Slave SQL thread for channel 'group_replication_recovery' exiting, replication stopped in log 'mysql-bin.001168' at position 346695824
2021-02-24T03:19:58.456004+08:00 586 [Note] [MY-011026] [Repl] Slave I/O thread killed while reading event for channel 'group_replication_recovery'.
2021-02-24T03:19:58.456034+08:00 586 [Note] [MY-010570] [Repl] Slave I/O thread exiting for channel 'group_replication_recovery', read up to log 'mysql-bin.001168', position 364987891
2021-02-24T03:19:58.594325+08:00 585 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state master_host='mgr3', master_port= 3311, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.
2021-02-24T03:19:58.894546+08:00 572 [Note] [MY-010559] [Repl] Multi-threaded slave statistics for channel 'group_replication_applier': seconds elapsed = 662; events assigned = 1025; worker queues filled over overrun level = 0; waited due a Worker queue full = 0; waited due the total size = 0; waited at clock conflicts = 257603800 waited (count) when Workers occupied = 2 waited when Workers occupied = 6565400
2021-02-24T03:20:11.483440+08:00 0 [Note] [MY-011490] [Repl] Plugin group_replication reported: 'This server was declared online within the replication group.'

四、定位
根据错误日志分析,大概率是Bug引起导致,后来又把相关信息提交给官方,收到的回复也是Bug,需要版本升级。目前已经开始在做升级的测试工作。