1、切换过程,注意需要关闭masterha_manager
[root@es3 ~]# masterha_master_switch --master_state=alive --conf=/root/app1.cnf --new_master_host=es3 --orig_master_is_new_slave --running_updates_limit=10000 Tue Aug 20 10:12:02 2019 - [info] MHA::MasterRotate version 0.58. Tue Aug 20 10:12:02 2019 - [info] Starting online master switch.. Tue Aug 20 10:12:02 2019 - [info] Tue Aug 20 10:12:02 2019 - [info] * Phase 1: Configuration Check Phase.. Tue Aug 20 10:12:02 2019 - [info] Tue Aug 20 10:12:02 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Tue Aug 20 10:12:02 2019 - [info] Reading application default configuration from /root/app1.cnf.. Tue Aug 20 10:12:02 2019 - [info] Reading server configuration from /root/app1.cnf.. Tue Aug 20 10:12:03 2019 - [info] GTID failover mode = 1 Tue Aug 20 10:12:03 2019 - [info] Current Alive Master: es1(192.168.56.14:3306) Tue Aug 20 10:12:03 2019 - [info] Alive Slaves: Tue Aug 20 10:12:03 2019 - [info] es2(192.168.56.15:3306) Version=5.7.24-log (oldest major version between slaves) log-bin:enabled Tue Aug 20 10:12:03 2019 - [info] GTID ON Tue Aug 20 10:12:03 2019 - [info] Replicating from 192.168.56.14(192.168.56.14:3306) Tue Aug 20 10:12:03 2019 - [info] es3(192.168.56.16:3306) Version=5.7.24-log (oldest major version between slaves) log-bin:enabled Tue Aug 20 10:12:03 2019 - [info] GTID ON Tue Aug 20 10:12:03 2019 - [info] Replicating from es1(192.168.56.14:3306) It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on es1(192.168.56.14:3306)? (YES/no): yes Tue Aug 20 10:12:06 2019 - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time.. Tue Aug 20 10:12:06 2019 - [info] ok. Tue Aug 20 10:12:06 2019 - [info] Checking MHA is not monitoring or doing failover.. Tue Aug 20 10:12:06 2019 - [info] Checking replication health on es2.. Tue Aug 20 10:12:06 2019 - [info] ok. Tue Aug 20 10:12:06 2019 - [info] Checking replication health on es3.. Tue Aug 20 10:12:06 2019 - [info] ok. Tue Aug 20 10:12:06 2019 - [info] es3 can be new master. Tue Aug 20 10:12:06 2019 - [info] From: es1(192.168.56.14:3306) (current master) +--es2(192.168.56.15:3306) +--es3(192.168.56.16:3306) To: es3(192.168.56.16:3306) (new master) +--es2(192.168.56.15:3306) +--es1(192.168.56.14:3306) Starting master switch from es1(192.168.56.14:3306) to es3(192.168.56.16:3306)? (yes/NO): yes Tue Aug 20 10:12:11 2019 - [info] Checking whether es3(192.168.56.16:3306) is ok for the new master.. Tue Aug 20 10:12:11 2019 - [info] ok. Tue Aug 20 10:12:11 2019 - [info] es1(192.168.56.14:3306): SHOW SLAVE STATUS returned empty result. To check replication filtering rules, temporarily executing CHANGE MASTER to a dummy host. Tue Aug 20 10:12:11 2019 - [info] es1(192.168.56.14:3306): Resetting slave pointing to the dummy host. Tue Aug 20 10:12:11 2019 - [info] ** Phase 1: Configuration Check Phase completed. Tue Aug 20 10:12:11 2019 - [info] Tue Aug 20 10:12:11 2019 - [info] * Phase 2: Rejecting updates Phase.. Tue Aug 20 10:12:11 2019 - [info] Tue Aug 20 10:12:11 2019 - [info] Executing master ip online change script to disable write on the current master: Tue Aug 20 10:12:11 2019 - [info] /usr/local/bin/master_ip_online_change --command=stop --orig_master_host=es1 --orig_master_ip=192.168.56.14 --orig_master_port=3306 --orig_master_user='repl' --new_master_host=es3 --new_master_ip=192.168.56.16 --new_master_port=3306 --new_master_user='repl' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_is_new_slave --orig_master_password=xxx --new_master_password=xxx *************************************************************** Disabling the VIP - 192.168.56.191/24 on old master: es1 *************************************************************** Tue Aug 20 10:12:12 2019 - [info] ok. Tue Aug 20 10:12:12 2019 - [info] Locking all tables on the orig master to reject updates from everybody (including root): Tue Aug 20 10:12:12 2019 - [info] Executing FLUSH TABLES WITH READ LOCK.. Tue Aug 20 10:12:12 2019 - [info] ok. Tue Aug 20 10:12:12 2019 - [info] Orig master binlog:pos is mysqlbin.000005:194. Tue Aug 20 10:12:12 2019 - [info] Waiting to execute all relay logs on es3(192.168.56.16:3306).. Tue Aug 20 10:12:12 2019 - [info] master_pos_wait(mysqlbin.000005:194) completed on es3(192.168.56.16:3306). Executed 0 events. Tue Aug 20 10:12:12 2019 - [info] done. Tue Aug 20 10:12:12 2019 - [info] Getting new master's binlog name and position.. Tue Aug 20 10:12:12 2019 - [info] mysqlbin.000004:154 Tue Aug 20 10:12:12 2019 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='es3 or 192.168.56.16', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='xxx'; Tue Aug 20 10:12:12 2019 - [info] Executing master ip online change script to allow write on the new master: Tue Aug 20 10:12:12 2019 - [info] /usr/local/bin/master_ip_online_change --command=start --orig_master_host=es1 --orig_master_ip=192.168.56.14 --orig_master_port=3306 --orig_master_user='repl' --new_master_host=es3 --new_master_ip=192.168.56.16 --new_master_port=3306 --new_master_user='repl' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_is_new_slave --orig_master_password=xxx --new_master_password=xxx *************************************************************** Enabling the VIP - 192.168.56.191/24 on new master: es3 *************************************************************** Tue Aug 20 10:12:13 2019 - [info] ok. Tue Aug 20 10:12:13 2019 - [info] Tue Aug 20 10:12:13 2019 - [info] * Switching slaves in parallel.. Tue Aug 20 10:12:13 2019 - [info] Tue Aug 20 10:12:13 2019 - [info] -- Slave switch on host es2(192.168.56.15:3306) started, pid: 4076 Tue Aug 20 10:12:13 2019 - [info] Tue Aug 20 10:12:14 2019 - [info] Log messages from es2 ... Tue Aug 20 10:12:14 2019 - [info] Tue Aug 20 10:12:13 2019 - [info] Waiting to execute all relay logs on es2(192.168.56.15:3306).. Tue Aug 20 10:12:13 2019 - [info] master_pos_wait(mysqlbin.000005:194) completed on es2(192.168.56.15:3306). Executed 0 events. Tue Aug 20 10:12:13 2019 - [info] done. Tue Aug 20 10:12:13 2019 - [info] Resetting slave es2(192.168.56.15:3306) and starting replication from the new master es3(192.168.56.16:3306).. Tue Aug 20 10:12:13 2019 - [info] Executed CHANGE MASTER. Tue Aug 20 10:12:13 2019 - [info] Slave started. Tue Aug 20 10:12:14 2019 - [info] End of log messages from es2 ... Tue Aug 20 10:12:14 2019 - [info] Tue Aug 20 10:12:14 2019 - [info] -- Slave switch on host es2(192.168.56.15:3306) succeeded. Tue Aug 20 10:12:14 2019 - [info] Unlocking all tables on the orig master: Tue Aug 20 10:12:14 2019 - [info] Executing UNLOCK TABLES.. Tue Aug 20 10:12:14 2019 - [info] ok. Tue Aug 20 10:12:14 2019 - [info] Starting orig master as a new slave.. Tue Aug 20 10:12:14 2019 - [info] Resetting slave es1(192.168.56.14:3306) and starting replication from the new master es3(192.168.56.16:3306).. Tue Aug 20 10:12:14 2019 - [info] Executed CHANGE MASTER. Tue Aug 20 10:12:14 2019 - [info] Slave started. Tue Aug 20 10:12:14 2019 - [info] All new slave servers switched successfully. Tue Aug 20 10:12:14 2019 - [info] Tue Aug 20 10:12:14 2019 - [info] * Phase 5: New master cleanup phase.. Tue Aug 20 10:12:14 2019 - [info] Tue Aug 20 10:12:14 2019 - [info] es3: Resetting slave info succeeded. Tue Aug 20 10:12:14 2019 - [info] Switching master to es3(192.168.56.16:3306) completed successfully. [root@es3 ~]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:ed:8b:51 brd ff:ff:ff:ff:ff:ff inet 10.0.2.15/24 brd 10.0.2.255 scope global noprefixroute dynamic enp0s3 valid_lft 84197sec preferred_lft 84197sec inet6 fe80::c377:b5dd:2937:f269/64 scope link noprefixroute valid_lft forever preferred_lft forever 3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:1f:f9:c8 brd ff:ff:ff:ff:ff:ff inet 192.168.56.16/24 brd 192.168.56.255 scope global noprefixroute dynamic enp0s8 valid_lft 1030sec preferred_lft 1030sec inet 192.168.56.191/24 brd 192.168.56.255 scope global secondary enp0s8:1 valid_lft forever preferred_lft forever inet6 fe80::90e0:253d:ab32:8ba4/64 scope link noprefixroute valid_lft forever preferred_lft forever [root@es3 ~]#
2、遇到问题,
2.1.1、在主从架构为一主两从的架构中,主为5.6,从1为5.6,从2为5.7,手动往从2切换时,报错(
[error][/usr/share/perl5/vendor_perl/MHA/ServerManager.pm, ln1213] 10.102.2.240 is bad as a new master!
[error][/usr/share/perl5/vendor_perl/MHA/MasterRotate.pm, ln232] Failed to get new master!
[error][/usr/share/perl5/vendor_perl/MHA/ManagerUtil.pm, ln177] Got ERROR: at /bin/masterha_master_switch line 53.
)
[root@mysql02 dba]# masterha_master_switch --conf=/etc/masterha/app_mysql.cnf --master_state=alive --new_master_host=10.102.2.240 --orig_master_is_new_slave Thu Nov 19 17:32:46 2020 - [info] MHA::MasterRotate version 0.57. Thu Nov 19 17:32:46 2020 - [info] Starting online master switch.. Thu Nov 19 17:32:46 2020 - [info] Thu Nov 19 17:32:46 2020 - [info] * Phase 1: Configuration Check Phase.. Thu Nov 19 17:32:46 2020 - [info] Thu Nov 19 17:32:46 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Thu Nov 19 17:32:46 2020 - [info] Reading application default configuration from /etc/masterha/app_mysql.cnf.. Thu Nov 19 17:32:46 2020 - [info] Reading server configuration from /etc/masterha/app_mysql.cnf.. Thu Nov 19 17:32:47 2020 - [info] GTID failover mode = 0 Thu Nov 19 17:32:47 2020 - [info] Current Alive Master: 10.102.2.181(10.102.2.181:3306) Thu Nov 19 17:32:47 2020 - [info] Alive Slaves: Thu Nov 19 17:32:47 2020 - [info] 10.102.2.240(10.102.2.240:3306) Version=5.7.29-log log-bin:enabled Thu Nov 19 17:32:47 2020 - [info] Replicating from 10.102.2.181(10.102.2.181:3306) Thu Nov 19 17:32:47 2020 - [info] Primary candidate for the new Master (candidate_master is set) Thu Nov 19 17:32:47 2020 - [info] 10.102.2.92(10.102.2.92:3306) Version=5.6.39-log (oldest major version between slaves) log-bin:enabled Thu Nov 19 17:32:47 2020 - [info] Replicating from 10.102.2.181(10.102.2.181:3306) It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 10.102.2.181(10.102.2.181:3306)? (YES/no): yes Thu Nov 19 17:33:13 2020 - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time.. Thu Nov 19 17:33:13 2020 - [info] ok. Thu Nov 19 17:33:13 2020 - [info] Checking MHA is not monitoring or doing failover.. Thu Nov 19 17:33:13 2020 - [info] Checking replication health on 10.102.2.240.. Thu Nov 19 17:33:13 2020 - [info] ok. Thu Nov 19 17:33:13 2020 - [info] Checking replication health on 10.102.2.92.. Thu Nov 19 17:33:13 2020 - [info] ok. Thu Nov 19 17:33:13 2020 - [error][/usr/share/perl5/vendor_perl/MHA/ServerManager.pm, ln1213] 10.102.2.240 is bad as a new master! Thu Nov 19 17:33:13 2020 - [error][/usr/share/perl5/vendor_perl/MHA/MasterRotate.pm, ln232] Failed to get new master! Thu Nov 19 17:33:13 2020 - [error][/usr/share/perl5/vendor_perl/MHA/ManagerUtil.pm, ln177] Got ERROR: at /bin/masterha_master_switch line 53. [root@mysql02 dba]#
2.1.2、升级从1位5.7,结构变成:主为5.6,从1为5.7,从2为5.7切换就可以了
[root@mysql2 dba]# masterha_master_switch --conf=/etc/masterha/app_mysql.cnf --master_state=alive --new_master_host=10.102.2.240 --new_master_port=3306 --orig_master_is_new_slave --running_updates_limit=10000 Fri Nov 20 09:20:56 2020 - [info] MHA::MasterRotate version 0.58. Fri Nov 20 09:20:56 2020 - [info] Starting online master switch.. Fri Nov 20 09:20:56 2020 - [info] Fri Nov 20 09:20:56 2020 - [info] * Phase 1: Configuration Check Phase.. Fri Nov 20 09:20:56 2020 - [info] Fri Nov 20 09:20:56 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Fri Nov 20 09:20:56 2020 - [info] Reading application default configuration from /etc/masterha/app_mysql.cnf.. Fri Nov 20 09:20:56 2020 - [info] Reading server configuration from /etc/masterha/app_mysql.cnf.. Fri Nov 20 09:20:57 2020 - [info] GTID failover mode = 0 Fri Nov 20 09:20:57 2020 - [info] Current Alive Master: 10.102.2.181(10.102.2.181:3306) Fri Nov 20 09:20:57 2020 - [info] Alive Slaves: Fri Nov 20 09:20:57 2020 - [info] 10.102.2.240(10.102.2.240:3306) Version=5.7.29-log (oldest major version between slaves) log-bin:enabled Fri Nov 20 09:20:57 2020 - [info] Replicating from 10.102.2.181(10.102.2.181:3306) Fri Nov 20 09:20:57 2020 - [info] Primary candidate for the new Master (candidate_master is set) Fri Nov 20 09:20:57 2020 - [info] 10.102.2.92(10.102.2.92:3306) Version=5.7.29-log (oldest major version between slaves) log-bin:enabled Fri Nov 20 09:20:57 2020 - [info] Replicating from 10.102.2.181(10.102.2.181:3306) It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 10.102.2.181(10.102.2.181:3306)? (YES/no): yes Fri Nov 20 09:20:59 2020 - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time.. Fri Nov 20 09:20:59 2020 - [info] ok. Fri Nov 20 09:20:59 2020 - [info] Checking MHA is not monitoring or doing failover.. Fri Nov 20 09:20:59 2020 - [info] Checking replication health on 10.102.2.240.. Fri Nov 20 09:20:59 2020 - [info] ok. Fri Nov 20 09:20:59 2020 - [info] Checking replication health on 10.102.2.92.. Fri Nov 20 09:20:59 2020 - [info] ok. Fri Nov 20 09:20:59 2020 - [info] 10.102.2.240 can be new master. Fri Nov 20 09:20:59 2020 - [info] From: 10.102.2.181(10.102.2.181:3306) (current master) +--10.102.2.240(10.102.2.240:3306) +--10.102.2.92(10.102.2.92:3306) To: 10.102.2.240(10.102.2.240:3306) (new master) +--10.102.2.92(10.102.2.92:3306) +--10.102.2.181(10.102.2.181:3306) Starting master switch from 10.102.2.181(10.102.2.181:3306) to 10.102.2.240(10.102.2.240:3306)? (yes/NO): yes Fri Nov 20 09:21:10 2020 - [info] Checking whether 10.102.2.240(10.102.2.240:3306) is ok for the new master.. Fri Nov 20 09:21:10 2020 - [info] ok. Fri Nov 20 09:21:10 2020 - [info] 10.102.2.181(10.102.2.181:3306): SHOW SLAVE STATUS returned empty result. To check replication filtering rules, temporarily executing CHANGE MASTER to a dummy host. Fri Nov 20 09:21:10 2020 - [info] 10.102.2.181(10.102.2.181:3306): Resetting slave pointing to the dummy host. Fri Nov 20 09:21:10 2020 - [info] ** Phase 1: Configuration Check Phase completed. Fri Nov 20 09:21:10 2020 - [info] Fri Nov 20 09:21:10 2020 - [info] * Phase 2: Rejecting updates Phase.. Fri Nov 20 09:21:10 2020 - [info] Fri Nov 20 09:21:10 2020 - [info] Executing master ip online change script to disable write on the current master: Fri Nov 20 09:21:10 2020 - [info] /usr/local/bin/master_ip_online_change_mysql --command=stop --orig_master_host=10.102.2.181 --orig_master_ip=10.102.2.181 --orig_master_port=3306 --orig_master_user='chenglong' --new_master_host=10.102.2.240 --new_master_ip=10.102.2.240 --new_master_port=3306 --new_master_user='chenglong' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_is_new_slave --orig_master_password=xxx --new_master_password=xxx Fri Nov 20 09:21:10 2020 559021 Set read_only on the new master.. ok. Fri Nov 20 09:21:10 2020 563807 Drpping app user on the orig master.. Fri Nov 20 09:21:10 2020 564383 Set read_only=1 on the orig master.. ok. disable the VIP on old master: 10.102.2.181 Fri Nov 20 09:21:11 2020 032710 Killing all application threads.. Fri Nov 20 09:21:11 2020 032867 done. Fri Nov 20 09:21:11 2020 - [info] ok. Fri Nov 20 09:21:11 2020 - [info] Locking all tables on the orig master to reject updates from everybody (including root): Fri Nov 20 09:21:11 2020 - [info] Executing FLUSH TABLES WITH READ LOCK.. Fri Nov 20 09:21:11 2020 - [info] ok. Fri Nov 20 09:21:11 2020 - [info] Orig master binlog:pos is mysql-bin.000001:246578113. Fri Nov 20 09:21:11 2020 - [info] Waiting to execute all relay logs on 10.102.2.240(10.102.2.240:3306).. Fri Nov 20 09:21:11 2020 - [info] master_pos_wait(mysql-bin.000001:246578113) completed on 10.102.2.240(10.102.2.240:3306). Executed 0 events. Fri Nov 20 09:21:11 2020 - [info] done. Fri Nov 20 09:21:11 2020 - [info] Getting new master's binlog name and position.. Fri Nov 20 09:21:11 2020 - [info] mysql-bin.000003:332 Fri Nov 20 09:21:11 2020 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.102.2.240', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000003', MASTER_LOG_POS=332, MASTER_USER='repl', MASTER_PASSWORD='xxx'; Fri Nov 20 09:21:11 2020 - [info] Executing master ip online change script to allow write on the new master: Fri Nov 20 09:21:11 2020 - [info] /usr/local/bin/master_ip_online_change_mysql --command=start --orig_master_host=10.102.2.181 --orig_master_ip=10.102.2.181 --orig_master_port=3306 --orig_master_user='chenglong' --new_master_host=10.102.2.240 --new_master_ip=10.102.2.240 --new_master_port=3306 --new_master_user='chenglong' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_is_new_slave --orig_master_password=xxx --new_master_password=xxx Fri Nov 20 09:21:11 2020 170022 Set read_only=0 on the new master. Fri Nov 20 09:21:11 2020 171159 Creating app user on the new master.. enable the VIP: 10.102.2.253/24 on the new master: 10.102.2.240 Fri Nov 20 09:21:11 2020 - [info] ok. Fri Nov 20 09:21:11 2020 - [info] Fri Nov 20 09:21:11 2020 - [info] * Switching slaves in parallel.. Fri Nov 20 09:21:11 2020 - [info] Fri Nov 20 09:21:11 2020 - [info] -- Slave switch on host 10.102.2.92(10.102.2.92:3306) started, pid: 11974 Fri Nov 20 09:21:11 2020 - [info] Fri Nov 20 09:21:12 2020 - [info] Log messages from 10.102.2.92 ... Fri Nov 20 09:21:12 2020 - [info] Fri Nov 20 09:21:11 2020 - [info] Waiting to execute all relay logs on 10.102.2.92(10.102.2.92:3306).. Fri Nov 20 09:21:11 2020 - [info] master_pos_wait(mysql-bin.000001:246578113) completed on 10.102.2.92(10.102.2.92:3306). Executed 0 events. Fri Nov 20 09:21:11 2020 - [info] done. Fri Nov 20 09:21:11 2020 - [info] Resetting slave 10.102.2.92(10.102.2.92:3306) and starting replication from the new master 10.102.2.240(10.102.2.240:3306).. Fri Nov 20 09:21:11 2020 - [info] Executed CHANGE MASTER. Fri Nov 20 09:21:11 2020 - [info] Slave started. Fri Nov 20 09:21:12 2020 - [info] End of log messages from 10.102.2.92 ... Fri Nov 20 09:21:12 2020 - [info] Fri Nov 20 09:21:12 2020 - [info] -- Slave switch on host 10.102.2.92(10.102.2.92:3306) succeeded. Fri Nov 20 09:21:12 2020 - [info] Unlocking all tables on the orig master: Fri Nov 20 09:21:12 2020 - [info] Executing UNLOCK TABLES.. Fri Nov 20 09:21:12 2020 - [info] ok. Fri Nov 20 09:21:12 2020 - [info] Starting orig master as a new slave.. Fri Nov 20 09:21:12 2020 - [info] Resetting slave 10.102.2.181(10.102.2.181:3306) and starting replication from the new master 10.102.2.240(10.102.2.240:3306).. Fri Nov 20 09:21:12 2020 - [info] Executed CHANGE MASTER. Fri Nov 20 09:21:12 2020 - [info] Slave started. Fri Nov 20 09:21:12 2020 - [info] All new slave servers switched successfully. Fri Nov 20 09:21:12 2020 - [info] Fri Nov 20 09:21:12 2020 - [info] * Phase 5: New master cleanup phase.. Fri Nov 20 09:21:12 2020 - [info] Fri Nov 20 09:21:12 2020 - [info] 10.102.2.240: Resetting slave info succeeded. Fri Nov 20 09:21:12 2020 - [info] Switching master to 10.102.2.240(10.102.2.240:3306) completed successfully.