ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移 ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_02

 上面左边是我的个人微 信,如需进一步沟通,请加微 信。  右边是我的公众号“Openstack私有云”,如有兴趣,请关注。

我的哔哩哔哩主页:https://space.bilibili.com/649988837


 

   上个月(2019年7月)proxmox发布了PVE 6.0版本,主要是debian操作系统以及ceph版本、以及其他关键组件进行了相关的升级。PVE自身功能方面,最主要的是能够实现本地磁盘虚拟机实现热迁移。

    引用上个月写的《ProxmoxVE V6.0 新特性介绍》:

    对我来说,V6.0版本有3个地方有重大改进:

    1、逆天新功能,本地磁盘虚拟机实现热迁移

    此次V6.0版本有一个逆天的新功能,宿主机本地盘(local disk)上的虚拟机可以实现热迁移! 

    大家一般的认识,热迁移一般都是使用分布式存储、或者共享磁盘的情况下才能实现,PVE打破了这个限制。这项功能更加丰富了特定应用场景的使用,比如有些人对分布式存储就是不感冒,就是喜欢用本地存储,现在也能使用虚拟机的热迁移功能了。

    2、零命令行,实现集群搭建

    我发现proxmox有一个持续改进的思路,就是零命令行!这一点对于很多开源产品的用户来说,简直是直接戳到了用户的心窝窝(别想歪了),太贴心了。

    通过web管理界面,你不需要配置什么双网卡bond,不需要安装复杂的集群软件和复杂的集群配置,不需要输入哪怕一条命令!

    只需要将所有宿主物理服务器安装好PVE环境,然后在第一个节点创建集群,在第二个节点加入集群(复制粘贴秘钥),第三个节点加入集群(复制粘贴秘钥),第四个节点加入集群(复制粘贴秘钥)。。。。完事。

    3、零命令行,实现ceph集群的搭建和管理

    通过web界面,不需要配置yum源安装ceph集群软件,不需要配置一堆ceph集群参数,直接点ceph,会提示下一步(后台自动安装软件),下一步(ceph参数配置),下一步,完事。

    通过web界面,可以很好的管理ceph集群,比如,查看每个ceph服务所在位置,状态,版本,停止OSD磁盘、mon、mds等,启动OSD磁盘、mon、mds等 。


    其实,以上的第2点和第3点在5.4版本也已经实现了,我是根据官网发布的介绍视频做的一个文字总结而已。

    

    今天抽空验证一下第1点,本地磁盘虚拟机实现热迁移。


    先说结论:

    1、pve6.0版本能够实现本地磁盘虚拟机热迁移;

    2、整个过程没有在pve后端执行一条命令(测试除外);


    具体过程详见下面的操作记录。


设计思路:

基本想法是,3个以上物理节点的集群环境,组成HA,具体设想如下:

    1.网络方面需要内部集群管理(corosync)网络,业务网络,2个网络。由于是测试本地磁盘虚拟机迁移,因此没有ceph集群网络。

    2.存储方面,除了系统盘之外,每台物理机使用1块硬盘作为本地lvm存储,用作虚拟机的磁盘使用,因为我们这里需要测试6.0的本地磁盘虚拟机实现热迁移,因此不建ceph分布式存储集群;

    3.CPU和内存方面根据实际需求确定,这里不赘述。

操作记录:

一.环境准备:

    基于以上考虑,在这台pve环境上面,创建3个虚拟机,3vcpu/12G内存/2块硬盘/2块网卡 ,其中硬盘方面1块硬盘为系统盘31G,1块50G的lvm盘,网卡方面1块网卡作为虚拟机网络(192.168.170.0/24网段)使用,1块网卡给集群使用(192.168.1.0/24网段)。创建虚拟机如下:

ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_03

 剩下2台测试虚拟机,直接克隆,分别为pve-2,pve-3 .

    接下来分别启动pve-1,pve-2,pve-3 进行proxmox VE的安装,如下图:

ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_04

注意,我这里使用PVE创建PVE虚拟机,需要用到虚拟化嵌套,安装PVE的时候提示没有开启kvm虚拟化功能,后面会给出解决办法。如下提示,点OK:

ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_05

安装步骤都很简单,傻瓜式的,其中选择硬盘会默认第一块硬盘安装系统,然后是设置root密码(123456)等等,网络配置步骤:

ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_06

 3台虚拟机同时安装,非常快,不到10分钟,居然就安装完了!感觉比装vmware的ESXI也更快啊。

ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_07

点右下角的Reboot重启。

ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_08


第一阶段搞定,后面就可以建集群,配网络,建存储之类的后续操作了。

    通过输入网址:

    https://192.168.1.111:8006

    https://192.168.1.112:8006

    https://192.168.1.113:8006

    分别可以管理这3台pve。


二.配置网络

    先将第二块网卡的网络配上,系统安装过程中配置了第一块网卡作为内部管理集群网络,网段是192.168.1.0/24 ,登陆系统后可以发现这个地址是配置在网桥上的,网桥上搭一块网卡进行桥接,因此新建一个网桥并配置192.168.170.0/24网段的地址给它,搭另一块网卡进行桥接,如下图:

ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_09ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_10

  注意第一块网卡设置了网关是192.168.1.1,这块网卡就不能设置网关了 。这个比较好理解,其实就是在主机上设置了默认网关,只能有一个。

    增加完网桥之后,需要重启pve节点生效配置。(网络更改都需要重启节点才能生效。)

三.存储设置

    接下来处理存储,进去看pve节点的存储,安装的时候安装程序会默认在安装盘上建立一个vg,名称是pve,在这个vg上建立swap/root/data三个lv ,其中root这个lv上挂载/根目录。 安装完成之后直接默认有两个存储ID,一个是local,对应的是目录型存储,挂载目录是/var/lib/vz ,一个是local-lvm,挂载的是data这个lv,这个lv的类型是lvm-thin ,lvm-thin这种类型很适合虚拟机使用,具体如何创建后面再介绍。另外,pve的各种存储类型主要有目录型/lvm/lvm-thin/zfs/nfs/ceph等等,具体可以参考官网的admin文档。

    我当初的设计是需要将最后一块50G的盘/dev/sdb作为虚拟机磁盘存储,直接在web界面上创建:

ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_11

上传第一个ISO镜像,通过scp程序,我用的是winscp,直接从本机上传到pve-1的/var/lib/vz/templates/iso/目录中:

ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_12


四.创建集群:

    注意:在创建集群前,先确保3个节点的网络配置都已经配置好了,不要忘了重启pve使网络配置生效。    

    在其中一台pve服务器的web管理界面上首先创建一个集群,如下图:

ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_13


ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_14

点击上面的“copy information” ,在另外2个pve节点加入这个集群,如下:

ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_15


ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_16


完成之后,随便登录一台pve的web管理界面,可以看到3台pve节点的集群信息:

ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_17


发现pve-2和pve-3上的第2块硬盘还没有创建成lvm-thin盘,使用与pve-1一样的方法,创建lvm-thin盘,本地盘,用于存放虚拟机镜像,创建好之后如下:

ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_18


五、热迁移功能验证(本地磁盘虚拟机

    万事俱备,只欠东风。我们前面创建了3个节点的集群,集群主机没有使用分布式存储ceph,而是准备使用本地lvm-thin磁盘存放虚拟机镜像,现在我们来创建一个测试虚拟机,比如在pve-1上面创建一个centos7的虚拟机,然后热迁移到pve-2或者pve-3上面。

    虚拟机配置如下:

ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_19

启动虚拟机,报错:

ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_20

上面提到的虚拟化嵌套的问题。原来的解决办法是通过kvm的虚拟化嵌套解决,后面我发现有更简单的办法,就是在创建嵌套虚拟机的时候,将CPU类型选择为host,如下:

ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_21


重新启动上面创建的虚拟机,成功!

ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_22

安装完成,开始进行热迁移测试,从pve-1迁移到pve-2上,注意,现在pve-1、pve-2、pve-3都只有本地磁盘,没有共享存储或者分布式存储ceph,可以看到有提示:

Migration with local disk might take long .

意思是本地磁盘虚拟机迁移会花费很长时间。如下图:

ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_23

报错:

ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_24

需要将虚拟机的cdrom卸载掉:

ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_25

再次测试,可以了:

ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_26

迁移的同时,打开终端做ping测试,看看是否影响虚拟机的正常使用:

ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_27

发现迁移过程不影响使用。迁移的原理应该是后台拷贝虚拟机镜像文件到目标pve节点,保存虚拟机运行状态以及增量更新数据,等拷贝完成后,将增量更新数据拷贝到新节点,同时将虚拟机运行状态转移到目标pve节点,将源pve节点虚拟机关闭,在目标pve节点开启虚拟机。

ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_28

热迁移成功!从上面可以看到,虚拟机已经从pve-1迁移到了pve-2上了。


ProxmoxVE V6.0 新特性验证_proxmoxVE6.0 热迁移_29

观察ping的结果,丢包率为0,说明虚拟机正常运行,未受到迁移的影响。


后面附上迁移过程的日志,有心的同学可以自行研究一下,proxmoxVE是怎么实现本地磁盘虚拟机热迁移的,日志如下:


Task viewer: VM 100 - Migrate (pve-1 ---> pve-2)

OutputStatus

Stop

2019-08-16 11:43:27 starting migration of VM 100 to node 'pve-2' (192.168.1.112)
2019-08-16 11:43:27 found local disk 'local-lvm:vm-100-disk-0' (in current VM config)
2019-08-16 11:43:27 copying disk images
2019-08-16 11:43:27 starting VM 100 on remote node 'pve-2'
2019-08-16 11:43:31 start remote tunnel
2019-08-16 11:43:32 ssh tunnel ver 1
2019-08-16 11:43:32 starting storage migration
2019-08-16 11:43:32 scsi0: start migration to nbd:192.168.1.112:60000:exportname=drive-scsi0
drive mirror is starting for drive-scsi0
drive-scsi0: transferred: 0 bytes remaining: 34359738368 bytes total: 34359738368 bytes progression: 0.00 % busy: 1 ready: 0
drive-scsi0: transferred: 36700160 bytes remaining: 34323038208 bytes total: 34359738368 bytes progression: 0.11 % busy: 1 ready: 0
drive-scsi0: transferred: 94371840 bytes remaining: 34265366528 bytes total: 34359738368 bytes progression: 0.27 % busy: 1 ready: 0
drive-scsi0: transferred: 169869312 bytes remaining: 34189869056 bytes total: 34359738368 bytes progression: 0.49 % busy: 1 ready: 0
drive-scsi0: transferred: 255852544 bytes remaining: 34103885824 bytes total: 34359738368 bytes progression: 0.74 % busy: 1 ready: 0
drive-scsi0: transferred: 372244480 bytes remaining: 33987493888 bytes total: 34359738368 bytes progression: 1.08 % busy: 1 ready: 0
drive-scsi0: transferred: 479199232 bytes remaining: 33880539136 bytes total: 34359738368 bytes progression: 1.39 % busy: 1 ready: 0
drive-scsi0: transferred: 565182464 bytes remaining: 33794555904 bytes total: 34359738368 bytes progression: 1.64 % busy: 1 ready: 0
drive-scsi0: transferred: 670040064 bytes remaining: 33689698304 bytes total: 34359738368 bytes progression: 1.95 % busy: 1 ready: 0
drive-scsi0: transferred: 783286272 bytes remaining: 33576452096 bytes total: 34359738368 bytes progression: 2.28 % busy: 1 ready: 0
drive-scsi0: transferred: 887095296 bytes remaining: 33472643072 bytes total: 34359738368 bytes progression: 2.58 % busy: 1 ready: 0
drive-scsi0: transferred: 995098624 bytes remaining: 33364639744 bytes total: 34359738368 bytes progression: 2.90 % busy: 1 ready: 0
drive-scsi0: transferred: 1110441984 bytes remaining: 33249296384 bytes total: 34359738368 bytes progression: 3.23 % busy: 1 ready: 0
drive-scsi0: transferred: 1233125376 bytes remaining: 33126612992 bytes total: 34359738368 bytes progression: 3.59 % busy: 1 ready: 0
drive-scsi0: transferred: 1333788672 bytes remaining: 33025949696 bytes total: 34359738368 bytes progression: 3.88 % busy: 1 ready: 0
drive-scsi0: transferred: 1453326336 bytes remaining: 32906412032 bytes total: 34359738368 bytes progression: 4.23 % busy: 1 ready: 0
drive-scsi0: transferred: 1555038208 bytes remaining: 32804700160 bytes total: 34359738368 bytes progression: 4.53 % busy: 1 ready: 0
drive-scsi0: transferred: 1723858944 bytes remaining: 32635879424 bytes total: 34359738368 bytes progression: 5.02 % busy: 1 ready: 0
drive-scsi0: transferred: 1804599296 bytes remaining: 32555139072 bytes total: 34359738368 bytes progression: 5.25 % busy: 1 ready: 0
drive-scsi0: transferred: 1892679680 bytes remaining: 32467058688 bytes total: 34359738368 bytes progression: 5.51 % busy: 1 ready: 0
drive-scsi0: transferred: 1993342976 bytes remaining: 32366395392 bytes total: 34359738368 bytes progression: 5.80 % busy: 1 ready: 0
drive-scsi0: transferred: 2077229056 bytes remaining: 32282509312 bytes total: 34359738368 bytes progression: 6.05 % busy: 1 ready: 0
drive-scsi0: transferred: 2147483648 bytes remaining: 32212254720 bytes total: 34359738368 bytes progression: 6.25 % busy: 1 ready: 0
drive-scsi0: transferred: 2283798528 bytes remaining: 32075939840 bytes total: 34359738368 bytes progression: 6.65 % busy: 1 ready: 0
drive-scsi0: transferred: 2405433344 bytes remaining: 31954305024 bytes total: 34359738368 bytes progression: 7.00 % busy: 1 ready: 0
drive-scsi0: transferred: 2529165312 bytes remaining: 31830573056 bytes total: 34359738368 bytes progression: 7.36 % busy: 1 ready: 0
drive-scsi0: transferred: 2632974336 bytes remaining: 31726764032 bytes total: 34359738368 bytes progression: 7.66 % busy: 1 ready: 0
drive-scsi0: transferred: 2750414848 bytes remaining: 31609323520 bytes total: 34359738368 bytes progression: 8.00 % busy: 1 ready: 0
drive-scsi0: transferred: 2867855360 bytes remaining: 31491883008 bytes total: 34359738368 bytes progression: 8.35 % busy: 1 ready: 0
drive-scsi0: transferred: 2955935744 bytes remaining: 31403802624 bytes total: 34359738368 bytes progression: 8.60 % busy: 1 ready: 0
drive-scsi0: transferred: 3070230528 bytes remaining: 31289507840 bytes total: 34359738368 bytes progression: 8.94 % busy: 1 ready: 0
drive-scsi0: transferred: 3196059648 bytes remaining: 31163678720 bytes total: 34359738368 bytes progression: 9.30 % busy: 1 ready: 0
drive-scsi0: transferred: 3246391296 bytes remaining: 31113347072 bytes total: 34359738368 bytes progression: 9.45 % busy: 1 ready: 0
drive-scsi0: transferred: 3297771520 bytes remaining: 31061966848 bytes total: 34359738368 bytes progression: 9.60 % busy: 1 ready: 0
drive-scsi0: transferred: 3342860288 bytes remaining: 31016878080 bytes total: 34359738368 bytes progression: 9.73 % busy: 1 ready: 0
drive-scsi0: transferred: 3388997632 bytes remaining: 30970740736 bytes total: 34359738368 bytes progression: 9.86 % busy: 1 ready: 0
drive-scsi0: transferred: 3435134976 bytes remaining: 30924603392 bytes total: 34359738368 bytes progression: 10.00 % busy: 1 ready: 0
drive-scsi0: transferred: 3498049536 bytes remaining: 30861688832 bytes total: 34359738368 bytes progression: 10.18 % busy: 1 ready: 0
drive-scsi0: transferred: 3579838464 bytes remaining: 30779965440 bytes total: 34359803904 bytes progression: 10.42 % busy: 1 ready: 0
drive-scsi0: transferred: 3670016000 bytes remaining: 30689787904 bytes total: 34359803904 bytes progression: 10.68 % busy: 1 ready: 0
drive-scsi0: transferred: 3802136576 bytes remaining: 30557667328 bytes total: 34359803904 bytes progression: 11.07 % busy: 1 ready: 0
drive-scsi0: transferred: 3905945600 bytes remaining: 30453858304 bytes total: 34359803904 bytes progression: 11.37 % busy: 1 ready: 0
drive-scsi0: transferred: 4005560320 bytes remaining: 30354243584 bytes total: 34359803904 bytes progression: 11.66 % busy: 1 ready: 0
drive-scsi0: transferred: 4124049408 bytes remaining: 30235754496 bytes total: 34359803904 bytes progression: 12.00 % busy: 1 ready: 0
drive-scsi0: transferred: 4248829952 bytes remaining: 30110973952 bytes total: 34359803904 bytes progression: 12.37 % busy: 1 ready: 0
drive-scsi0: transferred: 4366270464 bytes remaining: 29993533440 bytes total: 34359803904 bytes progression: 12.71 % busy: 1 ready: 0
drive-scsi0: transferred: 4501536768 bytes remaining: 29858267136 bytes total: 34359803904 bytes progression: 13.10 % busy: 1 ready: 0
drive-scsi0: transferred: 4588568576 bytes remaining: 29771235328 bytes total: 34359803904 bytes progression: 13.35 % busy: 1 ready: 0
drive-scsi0: transferred: 4688183296 bytes remaining: 29671620608 bytes total: 34359803904 bytes progression: 13.64 % busy: 1 ready: 0
drive-scsi0: transferred: 4782555136 bytes remaining: 29577248768 bytes total: 34359803904 bytes progression: 13.92 % busy: 1 ready: 0
drive-scsi0: transferred: 4894752768 bytes remaining: 29465051136 bytes total: 34359803904 bytes progression: 14.25 % busy: 1 ready: 0
drive-scsi0: transferred: 5006950400 bytes remaining: 29352853504 bytes total: 34359803904 bytes progression: 14.57 % busy: 1 ready: 0
drive-scsi0: transferred: 5107613696 bytes remaining: 29252190208 bytes total: 34359803904 bytes progression: 14.87 % busy: 1 ready: 0
drive-scsi0: transferred: 5214568448 bytes remaining: 29145235456 bytes total: 34359803904 bytes progression: 15.18 % busy: 1 ready: 0
drive-scsi0: transferred: 5314183168 bytes remaining: 29045620736 bytes total: 34359803904 bytes progression: 15.47 % busy: 1 ready: 0
drive-scsi0: transferred: 5428477952 bytes remaining: 28931325952 bytes total: 34359803904 bytes progression: 15.80 % busy: 1 ready: 0
drive-scsi0: transferred: 5535432704 bytes remaining: 28824371200 bytes total: 34359803904 bytes progression: 16.11 % busy: 1 ready: 0
drive-scsi0: transferred: 5649727488 bytes remaining: 28710076416 bytes total: 34359803904 bytes progression: 16.44 % busy: 1 ready: 0
drive-scsi0: transferred: 5754585088 bytes remaining: 28605218816 bytes total: 34359803904 bytes progression: 16.75 % busy: 1 ready: 0
drive-scsi0: transferred: 5874122752 bytes remaining: 28485681152 bytes total: 34359803904 bytes progression: 17.10 % busy: 1 ready: 0
drive-scsi0: transferred: 5994708992 bytes remaining: 28365094912 bytes total: 34359803904 bytes progression: 17.45 % busy: 1 ready: 0
drive-scsi0: transferred: 6098518016 bytes remaining: 28261285888 bytes total: 34359803904 bytes progression: 17.75 % busy: 1 ready: 0
drive-scsi0: transferred: 6235881472 bytes remaining: 28123922432 bytes total: 34359803904 bytes progression: 18.15 % busy: 1 ready: 0
drive-scsi0: transferred: 6340739072 bytes remaining: 28019064832 bytes total: 34359803904 bytes progression: 18.45 % busy: 1 ready: 0
drive-scsi0: transferred: 6458179584 bytes remaining: 27901624320 bytes total: 34359803904 bytes progression: 18.80 % busy: 1 ready: 0
drive-scsi0: transferred: 6591348736 bytes remaining: 27768455168 bytes total: 34359803904 bytes progression: 19.18 % busy: 1 ready: 0
drive-scsi0: transferred: 6698303488 bytes remaining: 27661500416 bytes total: 34359803904 bytes progression: 19.49 % busy: 1 ready: 0
drive-scsi0: transferred: 6828326912 bytes remaining: 27531476992 bytes total: 34359803904 bytes progression: 19.87 % busy: 1 ready: 0
drive-scsi0: transferred: 6936330240 bytes remaining: 27423473664 bytes total: 34359803904 bytes progression: 20.19 % busy: 1 ready: 0
drive-scsi0: transferred: 7057965056 bytes remaining: 27301838848 bytes total: 34359803904 bytes progression: 20.54 % busy: 1 ready: 0
drive-scsi0: transferred: 7176454144 bytes remaining: 27183349760 bytes total: 34359803904 bytes progression: 20.89 % busy: 1 ready: 0
drive-scsi0: transferred: 7279214592 bytes remaining: 27080589312 bytes total: 34359803904 bytes progression: 21.19 % busy: 1 ready: 0
drive-scsi0: transferred: 7386169344 bytes remaining: 26973634560 bytes total: 34359803904 bytes progression: 21.50 % busy: 1 ready: 0
drive-scsi0: transferred: 7486832640 bytes remaining: 26872971264 bytes total: 34359803904 bytes progression: 21.79 % busy: 1 ready: 0
drive-scsi0: transferred: 7587495936 bytes remaining: 26772307968 bytes total: 34359803904 bytes progression: 22.08 % busy: 1 ready: 0
drive-scsi0: transferred: 7703887872 bytes remaining: 26655916032 bytes total: 34359803904 bytes progression: 22.42 % busy: 1 ready: 0
drive-scsi0: transferred: 7810842624 bytes remaining: 26548961280 bytes total: 34359803904 bytes progression: 22.73 % busy: 1 ready: 0
drive-scsi0: transferred: 7921991680 bytes remaining: 26437812224 bytes total: 34359803904 bytes progression: 23.06 % busy: 1 ready: 0
drive-scsi0: transferred: 8024752128 bytes remaining: 26335051776 bytes total: 34359803904 bytes progression: 23.36 % busy: 1 ready: 0
drive-scsi0: transferred: 8136949760 bytes remaining: 26222854144 bytes total: 34359803904 bytes progression: 23.68 % busy: 1 ready: 0
drive-scsi0: transferred: 8218738688 bytes remaining: 26141065216 bytes total: 34359803904 bytes progression: 23.92 % busy: 1 ready: 0
drive-scsi0: transferred: 8324644864 bytes remaining: 26035159040 bytes total: 34359803904 bytes progression: 24.23 % busy: 1 ready: 0
drive-scsi0: transferred: 8419016704 bytes remaining: 25940787200 bytes total: 34359803904 bytes progression: 24.50 % busy: 1 ready: 0
drive-scsi0: transferred: 8534360064 bytes remaining: 25825443840 bytes total: 34359803904 bytes progression: 24.84 % busy: 1 ready: 0
drive-scsi0: transferred: 8632926208 bytes remaining: 25726877696 bytes total: 34359803904 bytes progression: 25.13 % busy: 1 ready: 0
drive-scsi0: transferred: 8751415296 bytes remaining: 25608388608 bytes total: 34359803904 bytes progression: 25.47 % busy: 1 ready: 0
drive-scsi0: transferred: 8869904384 bytes remaining: 25489899520 bytes total: 34359803904 bytes progression: 25.81 % busy: 1 ready: 0
drive-scsi0: transferred: 8977907712 bytes remaining: 25381896192 bytes total: 34359803904 bytes progression: 26.13 % busy: 1 ready: 0
drive-scsi0: transferred: 9102688256 bytes remaining: 25257115648 bytes total: 34359803904 bytes progression: 26.49 % busy: 1 ready: 0
drive-scsi0: transferred: 9206497280 bytes remaining: 25153306624 bytes total: 34359803904 bytes progression: 26.79 % busy: 1 ready: 0
drive-scsi0: transferred: 9326034944 bytes remaining: 25033834496 bytes total: 34359869440 bytes progression: 27.14 % busy: 1 ready: 0
drive-scsi0: transferred: 9432989696 bytes remaining: 24926879744 bytes total: 34359869440 bytes progression: 27.45 % busy: 1 ready: 0
drive-scsi0: transferred: 9534701568 bytes remaining: 24825167872 bytes total: 34359869440 bytes progression: 27.75 % busy: 1 ready: 0
drive-scsi0: transferred: 9629073408 bytes remaining: 24730796032 bytes total: 34359869440 bytes progression: 28.02 % busy: 1 ready: 0
drive-scsi0: transferred: 9719250944 bytes remaining: 24640618496 bytes total: 34359869440 bytes progression: 28.29 % busy: 1 ready: 0
drive-scsi0: transferred: 9822011392 bytes remaining: 24537858048 bytes total: 34359869440 bytes progression: 28.59 % busy: 1 ready: 0
drive-scsi0: transferred: 9939451904 bytes remaining: 24420417536 bytes total: 34359869440 bytes progression: 28.93 % busy: 1 ready: 0
drive-scsi0: transferred: 10043260928 bytes remaining: 24316608512 bytes total: 34359869440 bytes progression: 29.23 % busy: 1 ready: 0
drive-scsi0: transferred: 10176430080 bytes remaining: 24183439360 bytes total: 34359869440 bytes progression: 29.62 % busy: 1 ready: 0
drive-scsi0: transferred: 10295967744 bytes remaining: 24063901696 bytes total: 34359869440 bytes progression: 29.97 % busy: 1 ready: 0
drive-scsi0: transferred: 10408165376 bytes remaining: 23951704064 bytes total: 34359869440 bytes progression: 30.29 % busy: 1 ready: 0
drive-scsi0: transferred: 10504634368 bytes remaining: 23855235072 bytes total: 34359869440 bytes progression: 30.57 % busy: 1 ready: 0
drive-scsi0: transferred: 10626269184 bytes remaining: 23733600256 bytes total: 34359869440 bytes progression: 30.93 % busy: 1 ready: 0
drive-scsi0: transferred: 10719592448 bytes remaining: 23640276992 bytes total: 34359869440 bytes progression: 31.20 % busy: 1 ready: 0
drive-scsi0: transferred: 10827595776 bytes remaining: 23532273664 bytes total: 34359869440 bytes progression: 31.51 % busy: 1 ready: 0
drive-scsi0: transferred: 10942939136 bytes remaining: 23416930304 bytes total: 34359869440 bytes progression: 31.85 % busy: 1 ready: 0
drive-scsi0: transferred: 11027873792 bytes remaining: 23331995648 bytes total: 34359869440 bytes progression: 32.10 % busy: 1 ready: 0
drive-scsi0: transferred: 11080302592 bytes remaining: 23279566848 bytes total: 34359869440 bytes progression: 32.25 % busy: 1 ready: 0
drive-scsi0: transferred: 11132731392 bytes remaining: 23227138048 bytes total: 34359869440 bytes progression: 32.40 % busy: 1 ready: 0
drive-scsi0: transferred: 11191451648 bytes remaining: 23168417792 bytes total: 34359869440 bytes progression: 32.57 % busy: 1 ready: 0
drive-scsi0: transferred: 11249123328 bytes remaining: 23110746112 bytes total: 34359869440 bytes progression: 32.74 % busy: 1 ready: 0
drive-scsi0: transferred: 11304697856 bytes remaining: 23055171584 bytes total: 34359869440 bytes progression: 32.90 % busy: 1 ready: 0
drive-scsi0: transferred: 11355029504 bytes remaining: 23004839936 bytes total: 34359869440 bytes progression: 33.05 % busy: 1 ready: 0
drive-scsi0: transferred: 11396972544 bytes remaining: 22962896896 bytes total: 34359869440 bytes progression: 33.17 % busy: 1 ready: 0
drive-scsi0: transferred: 11480858624 bytes remaining: 22879010816 bytes total: 34359869440 bytes progression: 33.41 % busy: 1 ready: 0
drive-scsi0: transferred: 11580473344 bytes remaining: 22779396096 bytes total: 34359869440 bytes progression: 33.70 % busy: 1 ready: 0
drive-scsi0: transferred: 11727273984 bytes remaining: 22632595456 bytes total: 34359869440 bytes progression: 34.13 % busy: 1 ready: 0
drive-scsi0: transferred: 11821645824 bytes remaining: 22538223616 bytes total: 34359869440 bytes progression: 34.41 % busy: 1 ready: 0
drive-scsi0: transferred: 11957960704 bytes remaining: 22401908736 bytes total: 34359869440 bytes progression: 34.80 % busy: 1 ready: 0
drive-scsi0: transferred: 12069109760 bytes remaining: 22290759680 bytes total: 34359869440 bytes progression: 35.13 % busy: 1 ready: 0
drive-scsi0: transferred: 12160335872 bytes remaining: 22199533568 bytes total: 34359869440 bytes progression: 35.39 % busy: 1 ready: 0
drive-scsi0: transferred: 12268339200 bytes remaining: 22091530240 bytes total: 34359869440 bytes progression: 35.71 % busy: 1 ready: 0
drive-scsi0: transferred: 12383682560 bytes remaining: 21976186880 bytes total: 34359869440 bytes progression: 36.04 % busy: 1 ready: 0
drive-scsi0: transferred: 12499025920 bytes remaining: 21860843520 bytes total: 34359869440 bytes progression: 36.38 % busy: 1 ready: 0
drive-scsi0: transferred: 12607029248 bytes remaining: 21752840192 bytes total: 34359869440 bytes progression: 36.69 % busy: 1 ready: 0
drive-scsi0: transferred: 12724469760 bytes remaining: 21635399680 bytes total: 34359869440 bytes progression: 37.03 % busy: 1 ready: 0
drive-scsi0: transferred: 12811501568 bytes remaining: 21548367872 bytes total: 34359869440 bytes progression: 37.29 % busy: 1 ready: 0
drive-scsi0: transferred: 12910067712 bytes remaining: 21449801728 bytes total: 34359869440 bytes progression: 37.57 % busy: 1 ready: 0
drive-scsi0: transferred: 13011779584 bytes remaining: 21348089856 bytes total: 34359869440 bytes progression: 37.87 % busy: 1 ready: 0
drive-scsi0: transferred: 13117685760 bytes remaining: 21242183680 bytes total: 34359869440 bytes progression: 38.18 % busy: 1 ready: 0
drive-scsi0: transferred: 13228834816 bytes remaining: 21131034624 bytes total: 34359869440 bytes progression: 38.50 % busy: 1 ready: 0
drive-scsi0: transferred: 13336838144 bytes remaining: 21023031296 bytes total: 34359869440 bytes progression: 38.82 % busy: 1 ready: 0
drive-scsi0: transferred: 13458472960 bytes remaining: 20901396480 bytes total: 34359869440 bytes progression: 39.17 % busy: 1 ready: 0
drive-scsi0: transferred: 13566476288 bytes remaining: 20793393152 bytes total: 34359869440 bytes progression: 39.48 % busy: 1 ready: 0
drive-scsi0: transferred: 13667139584 bytes remaining: 20692729856 bytes total: 34359869440 bytes progression: 39.78 % busy: 1 ready: 0
drive-scsi0: transferred: 13782482944 bytes remaining: 20577386496 bytes total: 34359869440 bytes progression: 40.11 % busy: 1 ready: 0
drive-scsi0: transferred: 13895729152 bytes remaining: 20464140288 bytes total: 34359869440 bytes progression: 40.44 % busy: 1 ready: 0
drive-scsi0: transferred: 13993246720 bytes remaining: 20366622720 bytes total: 34359869440 bytes progression: 40.73 % busy: 1 ready: 0
drive-scsi0: transferred: 14106492928 bytes remaining: 20253376512 bytes total: 34359869440 bytes progression: 41.06 % busy: 1 ready: 0
drive-scsi0: transferred: 14206107648 bytes remaining: 20153761792 bytes total: 34359869440 bytes progression: 41.35 % busy: 1 ready: 0
drive-scsi0: transferred: 14309916672 bytes remaining: 20049952768 bytes total: 34359869440 bytes progression: 41.65 % busy: 1 ready: 0
drive-scsi0: transferred: 14426308608 bytes remaining: 19933560832 bytes total: 34359869440 bytes progression: 41.99 % busy: 1 ready: 0
drive-scsi0: transferred: 14553186304 bytes remaining: 19806683136 bytes total: 34359869440 bytes progression: 42.36 % busy: 1 ready: 0
drive-scsi0: transferred: 14668529664 bytes remaining: 19691339776 bytes total: 34359869440 bytes progression: 42.69 % busy: 1 ready: 0
drive-scsi0: transferred: 14796455936 bytes remaining: 19563413504 bytes total: 34359869440 bytes progression: 43.06 % busy: 1 ready: 0
drive-scsi0: transferred: 14903410688 bytes remaining: 19456458752 bytes total: 34359869440 bytes progression: 43.37 % busy: 1 ready: 0
drive-scsi0: transferred: 14979956736 bytes remaining: 19379912704 bytes total: 34359869440 bytes progression: 43.60 % busy: 1 ready: 0
drive-scsi0: transferred: 15065939968 bytes remaining: 19293929472 bytes total: 34359869440 bytes progression: 43.85 % busy: 1 ready: 0
drive-scsi0: transferred: 15156117504 bytes remaining: 19203751936 bytes total: 34359869440 bytes progression: 44.11 % busy: 1 ready: 0
drive-scsi0: transferred: 15270412288 bytes remaining: 19089457152 bytes total: 34359869440 bytes progression: 44.44 % busy: 1 ready: 0
drive-scsi0: transferred: 15394144256 bytes remaining: 18965725184 bytes total: 34359869440 bytes progression: 44.80 % busy: 1 ready: 0
drive-scsi0: transferred: 15514730496 bytes remaining: 18845138944 bytes total: 34359869440 bytes progression: 45.15 % busy: 1 ready: 0
drive-scsi0: transferred: 15643705344 bytes remaining: 18716164096 bytes total: 34359869440 bytes progression: 45.53 % busy: 1 ready: 0
drive-scsi0: transferred: 15747514368 bytes remaining: 18612355072 bytes total: 34359869440 bytes progression: 45.83 % busy: 1 ready: 0
drive-scsi0: transferred: 15871246336 bytes remaining: 18488623104 bytes total: 34359869440 bytes progression: 46.19 % busy: 1 ready: 0
drive-scsi0: transferred: 15988686848 bytes remaining: 18371182592 bytes total: 34359869440 bytes progression: 46.53 % busy: 1 ready: 0
drive-scsi0: transferred: 16089350144 bytes remaining: 18270519296 bytes total: 34359869440 bytes progression: 46.83 % busy: 1 ready: 0
drive-scsi0: transferred: 16200499200 bytes remaining: 18159370240 bytes total: 34359869440 bytes progression: 47.15 % busy: 1 ready: 0
drive-scsi0: transferred: 16320036864 bytes remaining: 18039832576 bytes total: 34359869440 bytes progression: 47.50 % busy: 1 ready: 0
drive-scsi0: transferred: 16416505856 bytes remaining: 17943363584 bytes total: 34359869440 bytes progression: 47.78 % busy: 1 ready: 0
drive-scsi0: transferred: 16521363456 bytes remaining: 17838505984 bytes total: 34359869440 bytes progression: 48.08 % busy: 1 ready: 0
drive-scsi0: transferred: 16656629760 bytes remaining: 17703239680 bytes total: 34359869440 bytes progression: 48.48 % busy: 1 ready: 0
drive-scsi0: transferred: 16735272960 bytes remaining: 17624596480 bytes total: 34359869440 bytes progression: 48.71 % busy: 1 ready: 0
drive-scsi0: transferred: 16854810624 bytes remaining: 17505058816 bytes total: 34359869440 bytes progression: 49.05 % busy: 1 ready: 0
drive-scsi0: transferred: 16977494016 bytes remaining: 17382375424 bytes total: 34359869440 bytes progression: 49.41 % busy: 1 ready: 0
drive-scsi0: transferred: 17123246080 bytes remaining: 17236623360 bytes total: 34359869440 bytes progression: 49.84 % busy: 1 ready: 0
drive-scsi0: transferred: 17248026624 bytes remaining: 17111842816 bytes total: 34359869440 bytes progression: 50.20 % busy: 1 ready: 0
drive-scsi0: transferred: 17373855744 bytes remaining: 16986013696 bytes total: 34359869440 bytes progression: 50.56 % busy: 1 ready: 0
drive-scsi0: transferred: 17488150528 bytes remaining: 16871718912 bytes total: 34359869440 bytes progression: 50.90 % busy: 1 ready: 0
drive-scsi0: transferred: 17586716672 bytes remaining: 16773152768 bytes total: 34359869440 bytes progression: 51.18 % busy: 1 ready: 0
drive-scsi0: transferred: 17693671424 bytes remaining: 16666198016 bytes total: 34359869440 bytes progression: 51.50 % busy: 1 ready: 0
drive-scsi0: transferred: 17804820480 bytes remaining: 16555048960 bytes total: 34359869440 bytes progression: 51.82 % busy: 1 ready: 0
drive-scsi0: transferred: 17937989632 bytes remaining: 16421879808 bytes total: 34359869440 bytes progression: 52.21 % busy: 1 ready: 0
drive-scsi0: transferred: 18060673024 bytes remaining: 16299196416 bytes total: 34359869440 bytes progression: 52.56 % busy: 1 ready: 0
drive-scsi0: transferred: 18179162112 bytes remaining: 16180707328 bytes total: 34359869440 bytes progression: 52.91 % busy: 1 ready: 0
drive-scsi0: transferred: 18276679680 bytes remaining: 16083189760 bytes total: 34359869440 bytes progression: 53.19 % busy: 1 ready: 0
drive-scsi0: transferred: 18395168768 bytes remaining: 15964700672 bytes total: 34359869440 bytes progression: 53.54 % busy: 1 ready: 0
drive-scsi0: transferred: 18475909120 bytes remaining: 15883960320 bytes total: 34359869440 bytes progression: 53.77 % busy: 1 ready: 0
drive-scsi0: transferred: 18579718144 bytes remaining: 15780151296 bytes total: 34359869440 bytes progression: 54.07 % busy: 1 ready: 0
drive-scsi0: transferred: 18688770048 bytes remaining: 15671099392 bytes total: 34359869440 bytes progression: 54.39 % busy: 1 ready: 0
drive-scsi0: transferred: 18796773376 bytes remaining: 15563096064 bytes total: 34359869440 bytes progression: 54.71 % busy: 1 ready: 0
drive-scsi0: transferred: 18853396480 bytes remaining: 15506472960 bytes total: 34359869440 bytes progression: 54.87 % busy: 1 ready: 0
drive-scsi0: transferred: 18904776704 bytes remaining: 15455092736 bytes total: 34359869440 bytes progression: 55.02 % busy: 1 ready: 0
drive-scsi0: transferred: 18949865472 bytes remaining: 15410003968 bytes total: 34359869440 bytes progression: 55.15 % busy: 1 ready: 0
drive-scsi0: transferred: 18993905664 bytes remaining: 15365963776 bytes total: 34359869440 bytes progression: 55.28 % busy: 1 ready: 0
drive-scsi0: transferred: 19048431616 bytes remaining: 15311437824 bytes total: 34359869440 bytes progression: 55.44 % busy: 1 ready: 0
drive-scsi0: transferred: 19108200448 bytes remaining: 15251668992 bytes total: 34359869440 bytes progression: 55.61 % busy: 1 ready: 0
drive-scsi0: transferred: 19176357888 bytes remaining: 15183511552 bytes total: 34359869440 bytes progression: 55.81 % busy: 1 ready: 0
drive-scsi0: transferred: 19277021184 bytes remaining: 15082848256 bytes total: 34359869440 bytes progression: 56.10 % busy: 1 ready: 0
drive-scsi0: transferred: 19391315968 bytes remaining: 14968553472 bytes total: 34359869440 bytes progression: 56.44 % busy: 1 ready: 0
drive-scsi0: transferred: 19500367872 bytes remaining: 14859501568 bytes total: 34359869440 bytes progression: 56.75 % busy: 1 ready: 0
drive-scsi0: transferred: 19618856960 bytes remaining: 14741012480 bytes total: 34359869440 bytes progression: 57.10 % busy: 1 ready: 0
drive-scsi0: transferred: 19713228800 bytes remaining: 14646640640 bytes total: 34359869440 bytes progression: 57.37 % busy: 1 ready: 0
drive-scsi0: transferred: 19818086400 bytes remaining: 14541783040 bytes total: 34359869440 bytes progression: 57.68 % busy: 1 ready: 0
drive-scsi0: transferred: 19919798272 bytes remaining: 14440071168 bytes total: 34359869440 bytes progression: 57.97 % busy: 1 ready: 0
drive-scsi0: transferred: 20033044480 bytes remaining: 14326824960 bytes total: 34359869440 bytes progression: 58.30 % busy: 1 ready: 0
drive-scsi0: transferred: 20134756352 bytes remaining: 14225113088 bytes total: 34359869440 bytes progression: 58.60 % busy: 1 ready: 0
drive-scsi0: transferred: 20223885312 bytes remaining: 14135984128 bytes total: 34359869440 bytes progression: 58.86 % busy: 1 ready: 0
drive-scsi0: transferred: 20325597184 bytes remaining: 14034272256 bytes total: 34359869440 bytes progression: 59.16 % busy: 1 ready: 0
drive-scsi0: transferred: 20466106368 bytes remaining: 13893763072 bytes total: 34359869440 bytes progression: 59.56 % busy: 1 ready: 0
drive-scsi0: transferred: 20559429632 bytes remaining: 13800439808 bytes total: 34359869440 bytes progression: 59.84 % busy: 1 ready: 0
drive-scsi0: transferred: 20683161600 bytes remaining: 13676707840 bytes total: 34359869440 bytes progression: 60.20 % busy: 1 ready: 0
drive-scsi0: transferred: 20781727744 bytes remaining: 13578141696 bytes total: 34359869440 bytes progression: 60.48 % busy: 1 ready: 0
drive-scsi0: transferred: 20876099584 bytes remaining: 13483769856 bytes total: 34359869440 bytes progression: 60.76 % busy: 1 ready: 0
drive-scsi0: transferred: 20974665728 bytes remaining: 13385203712 bytes total: 34359869440 bytes progression: 61.04 % busy: 1 ready: 0
drive-scsi0: transferred: 21110980608 bytes remaining: 13248888832 bytes total: 34359869440 bytes progression: 61.44 % busy: 1 ready: 0
drive-scsi0: transferred: 21216886784 bytes remaining: 13142982656 bytes total: 34359869440 bytes progression: 61.75 % busy: 1 ready: 0
drive-scsi0: transferred: 21298675712 bytes remaining: 13061193728 bytes total: 34359869440 bytes progression: 61.99 % busy: 1 ready: 0
drive-scsi0: transferred: 21418213376 bytes remaining: 12941656064 bytes total: 34359869440 bytes progression: 62.33 % busy: 1 ready: 0
drive-scsi0: transferred: 21540896768 bytes remaining: 12818972672 bytes total: 34359869440 bytes progression: 62.69 % busy: 1 ready: 0
drive-scsi0: transferred: 21653094400 bytes remaining: 12706775040 bytes total: 34359869440 bytes progression: 63.02 % busy: 1 ready: 0
drive-scsi0: transferred: 21771583488 bytes remaining: 12588285952 bytes total: 34359869440 bytes progression: 63.36 % busy: 1 ready: 0
drive-scsi0: transferred: 21905801216 bytes remaining: 12454068224 bytes total: 34359869440 bytes progression: 63.75 % busy: 1 ready: 0
drive-scsi0: transferred: 22025338880 bytes remaining: 12334530560 bytes total: 34359869440 bytes progression: 64.10 % busy: 1 ready: 0
drive-scsi0: transferred: 22141730816 bytes remaining: 12218138624 bytes total: 34359869440 bytes progression: 64.44 % busy: 1 ready: 0
drive-scsi0: transferred: 22256025600 bytes remaining: 12103843840 bytes total: 34359869440 bytes progression: 64.77 % busy: 1 ready: 0
drive-scsi0: transferred: 22353543168 bytes remaining: 12006326272 bytes total: 34359869440 bytes progression: 65.06 % busy: 1 ready: 0
drive-scsi0: transferred: 22466789376 bytes remaining: 11893080064 bytes total: 34359869440 bytes progression: 65.39 % busy: 1 ready: 0
drive-scsi0: transferred: 22588424192 bytes remaining: 11771445248 bytes total: 34359869440 bytes progression: 65.74 % busy: 1 ready: 0
drive-scsi0: transferred: 22694330368 bytes remaining: 11665539072 bytes total: 34359869440 bytes progression: 66.05 % busy: 1 ready: 0
drive-scsi0: transferred: 22811770880 bytes remaining: 11548098560 bytes total: 34359869440 bytes progression: 66.39 % busy: 1 ready: 0
drive-scsi0: transferred: 22905094144 bytes remaining: 11454775296 bytes total: 34359869440 bytes progression: 66.66 % busy: 1 ready: 0
drive-scsi0: transferred: 23021486080 bytes remaining: 11338383360 bytes total: 34359869440 bytes progression: 67.00 % busy: 1 ready: 0
drive-scsi0: transferred: 23133683712 bytes remaining: 11226185728 bytes total: 34359869440 bytes progression: 67.33 % busy: 1 ready: 0
drive-scsi0: transferred: 23220715520 bytes remaining: 11139153920 bytes total: 34359869440 bytes progression: 67.58 % busy: 1 ready: 0
drive-scsi0: transferred: 23306698752 bytes remaining: 11053170688 bytes total: 34359869440 bytes progression: 67.83 % busy: 1 ready: 0
drive-scsi0: transferred: 23418896384 bytes remaining: 10940973056 bytes total: 34359869440 bytes progression: 68.16 % busy: 1 ready: 0
drive-scsi0: transferred: 23528996864 bytes remaining: 10830872576 bytes total: 34359869440 bytes progression: 68.48 % busy: 1 ready: 0
drive-scsi0: transferred: 23640145920 bytes remaining: 10719723520 bytes total: 34359869440 bytes progression: 68.80 % busy: 1 ready: 0
drive-scsi0: transferred: 23743954944 bytes remaining: 10615914496 bytes total: 34359869440 bytes progression: 69.10 % busy: 1 ready: 0
drive-scsi0: transferred: 23865589760 bytes remaining: 10494279680 bytes total: 34359869440 bytes progression: 69.46 % busy: 1 ready: 0
drive-scsi0: transferred: 23970447360 bytes remaining: 10389422080 bytes total: 34359869440 bytes progression: 69.76 % busy: 1 ready: 0
drive-scsi0: transferred: 24088936448 bytes remaining: 10270932992 bytes total: 34359869440 bytes progression: 70.11 % busy: 1 ready: 0
drive-scsi0: transferred: 24216862720 bytes remaining: 10143006720 bytes total: 34359869440 bytes progression: 70.48 % busy: 1 ready: 0
drive-scsi0: transferred: 24331157504 bytes remaining: 10028711936 bytes total: 34359869440 bytes progression: 70.81 % busy: 1 ready: 0
drive-scsi0: transferred: 24445452288 bytes remaining: 9914417152 bytes total: 34359869440 bytes progression: 71.15 % busy: 1 ready: 0
drive-scsi0: transferred: 24529338368 bytes remaining: 9830531072 bytes total: 34359869440 bytes progression: 71.39 % busy: 1 ready: 0
drive-scsi0: transferred: 24621613056 bytes remaining: 9738256384 bytes total: 34359869440 bytes progression: 71.66 % busy: 1 ready: 0
drive-scsi0: transferred: 24727519232 bytes remaining: 9632350208 bytes total: 34359869440 bytes progression: 71.97 % busy: 1 ready: 0
drive-scsi0: transferred: 24861736960 bytes remaining: 9498132480 bytes total: 34359869440 bytes progression: 72.36 % busy: 1 ready: 0
drive-scsi0: transferred: 24978128896 bytes remaining: 9381740544 bytes total: 34359869440 bytes progression: 72.70 % busy: 1 ready: 0
drive-scsi0: transferred: 25093472256 bytes remaining: 9266397184 bytes total: 34359869440 bytes progression: 73.03 % busy: 1 ready: 0
drive-scsi0: transferred: 25227689984 bytes remaining: 9132179456 bytes total: 34359869440 bytes progression: 73.42 % busy: 1 ready: 0
drive-scsi0: transferred: 25321013248 bytes remaining: 9038856192 bytes total: 34359869440 bytes progression: 73.69 % busy: 1 ready: 0
drive-scsi0: transferred: 25432162304 bytes remaining: 8927707136 bytes total: 34359869440 bytes progression: 74.02 % busy: 1 ready: 0
drive-scsi0: transferred: 25540165632 bytes remaining: 8819703808 bytes total: 34359869440 bytes progression: 74.33 % busy: 1 ready: 0
drive-scsi0: transferred: 25637683200 bytes remaining: 8722186240 bytes total: 34359869440 bytes progression: 74.62 % busy: 1 ready: 0
drive-scsi0: transferred: 25738346496 bytes remaining: 8621522944 bytes total: 34359869440 bytes progression: 74.91 % busy: 1 ready: 0
drive-scsi0: transferred: 25847398400 bytes remaining: 8512471040 bytes total: 34359869440 bytes progression: 75.23 % busy: 1 ready: 0
drive-scsi0: transferred: 25951207424 bytes remaining: 8408662016 bytes total: 34359869440 bytes progression: 75.53 % busy: 1 ready: 0
drive-scsi0: transferred: 26079133696 bytes remaining: 8280735744 bytes total: 34359869440 bytes progression: 75.90 % busy: 1 ready: 0
drive-scsi0: transferred: 26163019776 bytes remaining: 8196849664 bytes total: 34359869440 bytes progression: 76.14 % busy: 1 ready: 0
drive-scsi0: transferred: 26264731648 bytes remaining: 8095137792 bytes total: 34359869440 bytes progression: 76.44 % busy: 1 ready: 0
drive-scsi0: transferred: 26381123584 bytes remaining: 7978745856 bytes total: 34359869440 bytes progression: 76.78 % busy: 1 ready: 0
drive-scsi0: transferred: 26494369792 bytes remaining: 7865499648 bytes total: 34359869440 bytes progression: 77.11 % busy: 1 ready: 0
drive-scsi0: transferred: 26590838784 bytes remaining: 7769030656 bytes total: 34359869440 bytes progression: 77.39 % busy: 1 ready: 0
drive-scsi0: transferred: 26623344640 bytes remaining: 7736524800 bytes total: 34359869440 bytes progression: 77.48 % busy: 1 ready: 0
drive-scsi0: transferred: 26685210624 bytes remaining: 7674658816 bytes total: 34359869440 bytes progression: 77.66 % busy: 1 ready: 0
drive-scsi0: transferred: 26741833728 bytes remaining: 7618035712 bytes total: 34359869440 bytes progression: 77.83 % busy: 1 ready: 0
drive-scsi0: transferred: 26785873920 bytes remaining: 7573995520 bytes total: 34359869440 bytes progression: 77.96 % busy: 1 ready: 0
drive-scsi0: transferred: 26870808576 bytes remaining: 7489060864 bytes total: 34359869440 bytes progression: 78.20 % busy: 1 ready: 0
drive-scsi0: transferred: 26977763328 bytes remaining: 7382106112 bytes total: 34359869440 bytes progression: 78.52 % busy: 1 ready: 0
drive-scsi0: transferred: 27081572352 bytes remaining: 7278297088 bytes total: 34359869440 bytes progression: 78.82 % busy: 1 ready: 0
drive-scsi0: transferred: 27191672832 bytes remaining: 7168196608 bytes total: 34359869440 bytes progression: 79.14 % busy: 1 ready: 0
drive-scsi0: transferred: 27283947520 bytes remaining: 7075921920 bytes total: 34359869440 bytes progression: 79.41 % busy: 1 ready: 0
drive-scsi0: transferred: 27385659392 bytes remaining: 6974210048 bytes total: 34359869440 bytes progression: 79.70 % busy: 1 ready: 0
drive-scsi0: transferred: 27486322688 bytes remaining: 6873546752 bytes total: 34359869440 bytes progression: 80.00 % busy: 1 ready: 0
drive-scsi0: transferred: 27605860352 bytes remaining: 6754009088 bytes total: 34359869440 bytes progression: 80.34 % busy: 1 ready: 0
drive-scsi0: transferred: 27705475072 bytes remaining: 6654394368 bytes total: 34359869440 bytes progression: 80.63 % busy: 1 ready: 0
drive-scsi0: transferred: 27793555456 bytes remaining: 6566313984 bytes total: 34359869440 bytes progression: 80.89 % busy: 1 ready: 0
drive-scsi0: transferred: 27869052928 bytes remaining: 6490816512 bytes total: 34359869440 bytes progression: 81.11 % busy: 1 ready: 0
drive-scsi0: transferred: 27965521920 bytes remaining: 6394347520 bytes total: 34359869440 bytes progression: 81.39 % busy: 1 ready: 0
drive-scsi0: transferred: 28073525248 bytes remaining: 6286344192 bytes total: 34359869440 bytes progression: 81.70 % busy: 1 ready: 0
drive-scsi0: transferred: 28187820032 bytes remaining: 6172049408 bytes total: 34359869440 bytes progression: 82.04 % busy: 1 ready: 0
drive-scsi0: transferred: 28282191872 bytes remaining: 6077677568 bytes total: 34359869440 bytes progression: 82.31 % busy: 1 ready: 0
drive-scsi0: transferred: 28400680960 bytes remaining: 5959188480 bytes total: 34359869440 bytes progression: 82.66 % busy: 1 ready: 0
drive-scsi0: transferred: 28487712768 bytes remaining: 5872156672 bytes total: 34359869440 bytes progression: 82.91 % busy: 1 ready: 0
drive-scsi0: transferred: 28587327488 bytes remaining: 5772541952 bytes total: 34359869440 bytes progression: 83.20 % busy: 1 ready: 0
drive-scsi0: transferred: 28693233664 bytes remaining: 5666635776 bytes total: 34359869440 bytes progression: 83.51 % busy: 1 ready: 0
drive-scsi0: transferred: 28797042688 bytes remaining: 5562826752 bytes total: 34359869440 bytes progression: 83.81 % busy: 1 ready: 0
drive-scsi0: transferred: 28896657408 bytes remaining: 5463212032 bytes total: 34359869440 bytes progression: 84.10 % busy: 1 ready: 0
drive-scsi0: transferred: 29025632256 bytes remaining: 5334237184 bytes total: 34359869440 bytes progression: 84.48 % busy: 1 ready: 0
drive-scsi0: transferred: 29124198400 bytes remaining: 5235671040 bytes total: 34359869440 bytes progression: 84.76 % busy: 1 ready: 0
drive-scsi0: transferred: 29234298880 bytes remaining: 5125570560 bytes total: 34359869440 bytes progression: 85.08 % busy: 1 ready: 0
drive-scsi0: transferred: 29351739392 bytes remaining: 5008130048 bytes total: 34359869440 bytes progression: 85.42 % busy: 1 ready: 0
drive-scsi0: transferred: 29440868352 bytes remaining: 4919001088 bytes total: 34359869440 bytes progression: 85.68 % busy: 1 ready: 0
drive-scsi0: transferred: 29538385920 bytes remaining: 4821483520 bytes total: 34359869440 bytes progression: 85.97 % busy: 1 ready: 0
drive-scsi0: transferred: 29647437824 bytes remaining: 4712431616 bytes total: 34359869440 bytes progression: 86.29 % busy: 1 ready: 0
drive-scsi0: transferred: 29763829760 bytes remaining: 4596039680 bytes total: 34359869440 bytes progression: 86.62 % busy: 1 ready: 0
drive-scsi0: transferred: 29887561728 bytes remaining: 4472307712 bytes total: 34359869440 bytes progression: 86.98 % busy: 1 ready: 0
drive-scsi0: transferred: 29978787840 bytes remaining: 4381081600 bytes total: 34359869440 bytes progression: 87.25 % busy: 1 ready: 0
drive-scsi0: transferred: 30096228352 bytes remaining: 4263641088 bytes total: 34359869440 bytes progression: 87.59 % busy: 1 ready: 0
drive-scsi0: transferred: 30212620288 bytes remaining: 4147249152 bytes total: 34359869440 bytes progression: 87.93 % busy: 1 ready: 0
drive-scsi0: transferred: 30329012224 bytes remaining: 4030857216 bytes total: 34359869440 bytes progression: 88.27 % busy: 1 ready: 0
drive-scsi0: transferred: 30433869824 bytes remaining: 3925999616 bytes total: 34359869440 bytes progression: 88.57 % busy: 1 ready: 0
drive-scsi0: transferred: 30519853056 bytes remaining: 3840016384 bytes total: 34359869440 bytes progression: 88.82 % busy: 1 ready: 0
drive-scsi0: transferred: 30643585024 bytes remaining: 3716284416 bytes total: 34359869440 bytes progression: 89.18 % busy: 1 ready: 0
drive-scsi0: transferred: 30755782656 bytes remaining: 3604086784 bytes total: 34359869440 bytes progression: 89.51 % busy: 1 ready: 0
drive-scsi0: transferred: 30874271744 bytes remaining: 3485597696 bytes total: 34359869440 bytes progression: 89.86 % busy: 1 ready: 0
drive-scsi0: transferred: 31005343744 bytes remaining: 3354525696 bytes total: 34359869440 bytes progression: 90.24 % busy: 1 ready: 0
drive-scsi0: transferred: 31132221440 bytes remaining: 3227648000 bytes total: 34359869440 bytes progression: 90.61 % busy: 1 ready: 0
drive-scsi0: transferred: 31253856256 bytes remaining: 3106013184 bytes total: 34359869440 bytes progression: 90.96 % busy: 1 ready: 0
drive-scsi0: transferred: 31363956736 bytes remaining: 2995912704 bytes total: 34359869440 bytes progression: 91.28 % busy: 1 ready: 0
drive-scsi0: transferred: 31452037120 bytes remaining: 2907832320 bytes total: 34359869440 bytes progression: 91.54 % busy: 1 ready: 0
drive-scsi0: transferred: 31553748992 bytes remaining: 2806120448 bytes total: 34359869440 bytes progression: 91.83 % busy: 1 ready: 0
drive-scsi0: transferred: 31652315136 bytes remaining: 2707554304 bytes total: 34359869440 bytes progression: 92.12 % busy: 1 ready: 0
drive-scsi0: transferred: 31765561344 bytes remaining: 2594308096 bytes total: 34359869440 bytes progression: 92.45 % busy: 1 ready: 0
drive-scsi0: transferred: 31854690304 bytes remaining: 2505179136 bytes total: 34359869440 bytes progression: 92.71 % busy: 1 ready: 0
drive-scsi0: transferred: 31971082240 bytes remaining: 2388787200 bytes total: 34359869440 bytes progression: 93.05 % busy: 1 ready: 0
drive-scsi0: transferred: 32085377024 bytes remaining: 2274492416 bytes total: 34359869440 bytes progression: 93.38 % busy: 1 ready: 0
drive-scsi0: transferred: 32173457408 bytes remaining: 2186412032 bytes total: 34359869440 bytes progression: 93.64 % busy: 1 ready: 0
drive-scsi0: transferred: 32258392064 bytes remaining: 2101477376 bytes total: 34359869440 bytes progression: 93.88 % busy: 1 ready: 0
drive-scsi0: transferred: 32366395392 bytes remaining: 1993474048 bytes total: 34359869440 bytes progression: 94.20 % busy: 1 ready: 0
drive-scsi0: transferred: 32467058688 bytes remaining: 1892810752 bytes total: 34359869440 bytes progression: 94.49 % busy: 1 ready: 0
drive-scsi0: transferred: 32585547776 bytes remaining: 1774321664 bytes total: 34359869440 bytes progression: 94.84 % busy: 1 ready: 0
drive-scsi0: transferred: 32656850944 bytes remaining: 1703018496 bytes total: 34359869440 bytes progression: 95.04 % busy: 1 ready: 0
drive-scsi0: transferred: 32752271360 bytes remaining: 1607598080 bytes total: 34359869440 bytes progression: 95.32 % busy: 1 ready: 0
drive-scsi0: transferred: 32848740352 bytes remaining: 1511129088 bytes total: 34359869440 bytes progression: 95.60 % busy: 1 ready: 0
drive-scsi0: transferred: 32950452224 bytes remaining: 1409417216 bytes total: 34359869440 bytes progression: 95.90 % busy: 1 ready: 0
drive-scsi0: transferred: 33065795584 bytes remaining: 1294073856 bytes total: 34359869440 bytes progression: 96.23 % busy: 1 ready: 0
drive-scsi0: transferred: 33171701760 bytes remaining: 1188167680 bytes total: 34359869440 bytes progression: 96.54 % busy: 1 ready: 0
drive-scsi0: transferred: 33304870912 bytes remaining: 1054998528 bytes total: 34359869440 bytes progression: 96.93 % busy: 1 ready: 0
drive-scsi0: transferred: 33407631360 bytes remaining: 952238080 bytes total: 34359869440 bytes progression: 97.23 % busy: 1 ready: 0
drive-scsi0: transferred: 33519828992 bytes remaining: 840040448 bytes total: 34359869440 bytes progression: 97.56 % busy: 1 ready: 0
drive-scsi0: transferred: 33621540864 bytes remaining: 738328576 bytes total: 34359869440 bytes progression: 97.85 % busy: 1 ready: 0
drive-scsi0: transferred: 33741078528 bytes remaining: 618790912 bytes total: 34359869440 bytes progression: 98.20 % busy: 1 ready: 0
drive-scsi0: transferred: 33833353216 bytes remaining: 526516224 bytes total: 34359869440 bytes progression: 98.47 % busy: 1 ready: 0
drive-scsi0: transferred: 33935065088 bytes remaining: 424804352 bytes total: 34359869440 bytes progression: 98.76 % busy: 1 ready: 0
drive-scsi0: transferred: 34019999744 bytes remaining: 339935232 bytes total: 34359934976 bytes progression: 99.01 % busy: 1 ready: 0
drive-scsi0: transferred: 34119614464 bytes remaining: 240320512 bytes total: 34359934976 bytes progression: 99.30 % busy: 1 ready: 0
drive-scsi0: transferred: 34223423488 bytes remaining: 136511488 bytes total: 34359934976 bytes progression: 99.60 % busy: 1 ready: 0
drive-scsi0: transferred: 34352594944 bytes remaining: 7340032 bytes total: 34359934976 bytes progression: 99.98 % busy: 1 ready: 0
drive-scsi0: transferred: 34359934976 bytes remaining: 0 bytes total: 34359934976 bytes progression: 100.00 % busy: 0 ready: 1
all mirroring jobs are ready
2019-08-16 11:49:08 starting online/live migration on unix:/run/qemu-server/100.migrate
2019-08-16 11:49:08 migrate_set_speed: 8589934592
2019-08-16 11:49:08 migrate_set_downtime: 0.1
2019-08-16 11:49:08 set migration_caps
2019-08-16 11:49:08 set cachesize: 268435456
2019-08-16 11:49:08 start migrate command to unix:/run/qemu-server/100.migrate
2019-08-16 11:49:09 migration status: active (transferred 34104177, remaining 2127441920), total 2165121024)
2019-08-16 11:49:09 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:10 migration status: active (transferred 83541717, remaining 2077822976), total 2165121024)
2019-08-16 11:49:10 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:11 migration status: active (transferred 112052530, remaining 2049282048), total 2165121024)
2019-08-16 11:49:11 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:12 migration status: active (transferred 140756294, remaining 2020519936), total 2165121024)
2019-08-16 11:49:12 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:13 migration status: active (transferred 166986156, remaining 1993859072), total 2165121024)
2019-08-16 11:49:13 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:14 migration status: active (transferred 193587196, remaining 1966002176), total 2165121024)
2019-08-16 11:49:14 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:15 migration status: active (transferred 220842221, remaining 1936834560), total 2165121024)
2019-08-16 11:49:15 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:16 migration status: active (transferred 246517558, remaining 1910943744), total 2165121024)
2019-08-16 11:49:16 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:17 migration status: active (transferred 272587166, remaining 1884532736), total 2165121024)
2019-08-16 11:49:17 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:18 migration status: active (transferred 300293873, remaining 1856671744), total 2165121024)
2019-08-16 11:49:18 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:19 migration status: active (transferred 326048751, remaining 1829994496), total 2165121024)
2019-08-16 11:49:19 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:20 migration status: active (transferred 353334106, remaining 1801932800), total 2165121024)
2019-08-16 11:49:20 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:21 migration status: active (transferred 410838877, remaining 1741094912), total 2165121024)
2019-08-16 11:49:21 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:22 migration status: active (transferred 442024688, remaining 1708388352), total 2165121024)
2019-08-16 11:49:22 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:23 migration status: active (transferred 465984534, remaining 1684221952), total 2165121024)
2019-08-16 11:49:23 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:24 migration status: active (transferred 493546773, remaining 1655017472), total 2165121024)
2019-08-16 11:49:24 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:25 migration status: active (transferred 521094792, remaining 1624829952), total 2165121024)
2019-08-16 11:49:25 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:26 migration status: active (transferred 548641435, remaining 1595265024), total 2165121024)
2019-08-16 11:49:26 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:27 migration status: active (transferred 575626856, remaining 1567657984), total 2165121024)
2019-08-16 11:49:27 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:28 migration status: active (transferred 602964096, remaining 1540210688), total 2165121024)
2019-08-16 11:49:28 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:29 migration status: active (transferred 630810575, remaining 1512095744), total 2165121024)
2019-08-16 11:49:29 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:30 migration status: active (transferred 659941992, remaining 1482526720), total 2165121024)
2019-08-16 11:49:30 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:31 migration status: active (transferred 684641891, remaining 1457012736), total 2165121024)
2019-08-16 11:49:31 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:32 migration status: active (transferred 713545059, remaining 1426956288), total 2165121024)
2019-08-16 11:49:32 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:33 migration status: active (transferred 771149459, remaining 1367371776), total 2165121024)
2019-08-16 11:49:33 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:34 migration status: active (transferred 820144722, remaining 1317756928), total 2165121024)
2019-08-16 11:49:34 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:35 migration status: active (transferred 849408628, remaining 1270755328), total 2165121024)
2019-08-16 11:49:35 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:36 migration status: active (transferred 876510087, remaining 1242517504), total 2165121024)
2019-08-16 11:49:36 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:37 migration status: active (transferred 903330006, remaining 1215688704), total 2165121024)
2019-08-16 11:49:37 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:38 migration status: active (transferred 933149922, remaining 1185878016), total 2165121024)
2019-08-16 11:49:38 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:39 migration status: active (transferred 958648310, remaining 1160380416), total 2165121024)
2019-08-16 11:49:39 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:40 migration status: active (transferred 984819734, remaining 1134227456), total 2165121024)
2019-08-16 11:49:40 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:41 migration status: active (transferred 1009653327, remaining 1109372928), total 2165121024)
2019-08-16 11:49:41 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:42 migration status: active (transferred 1034622586, remaining 1084276736), total 2165121024)
2019-08-16 11:49:42 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:43 migration status: active (transferred 1062612821, remaining 1055969280), total 2165121024)
2019-08-16 11:49:43 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:44 migration status: active (transferred 1090808409, remaining 1027387392), total 2165121024)
2019-08-16 11:49:44 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:45 migration status: active (transferred 1119112934, remaining 997679104), total 2165121024)
2019-08-16 11:49:45 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:46 migration status: active (transferred 1148129357, remaining 968265728), total 2165121024)
2019-08-16 11:49:46 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:47 migration status: active (transferred 1207212710, remaining 908378112), total 2165121024)
2019-08-16 11:49:47 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:48 migration status: active (transferred 1234209452, remaining 881213440), total 2165121024)
2019-08-16 11:49:48 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:49 migration status: active (transferred 1282635645, remaining 829673472), total 2165121024)
2019-08-16 11:49:49 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:50 migration status: active (transferred 1339080485, remaining 772259840), total 2165121024)
2019-08-16 11:49:50 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:51 migration status: active (transferred 1378526208, remaining 731963392), total 2165121024)
2019-08-16 11:49:51 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:52 migration status: active (transferred 1405532032, remaining 704389120), total 2165121024)
2019-08-16 11:49:52 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:53 migration status: active (transferred 1434949462, remaining 673243136), total 2165121024)
2019-08-16 11:49:53 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:54 migration status: active (transferred 1461321793, remaining 645107712), total 2165121024)
2019-08-16 11:49:54 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:55 migration status: active (transferred 1489489706, remaining 616075264), total 2165121024)
2019-08-16 11:49:55 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:56 migration status: active (transferred 1516725427, remaining 588234752), total 2165121024)
2019-08-16 11:49:56 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:58 migration status: active (transferred 1544613728, remaining 559726592), total 2165121024)
2019-08-16 11:49:58 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:49:59 migration status: active (transferred 1572437202, remaining 530903040), total 2165121024)
2019-08-16 11:49:59 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:50:00 migration status: active (transferred 1603688137, remaining 498376704), total 2165121024)
2019-08-16 11:50:00 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:50:01 migration status: active (transferred 1641161464, remaining 453722112), total 2165121024)
2019-08-16 11:50:01 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:50:02 migration status: active (transferred 1694574484, remaining 391405568), total 2165121024)
2019-08-16 11:50:02 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:50:03 migration status: active (transferred 1748789557, remaining 336801792), total 2165121024)
2019-08-16 11:50:03 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:50:04 migration status: active (transferred 1776459608, remaining 306982912), total 2165121024)
2019-08-16 11:50:04 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:50:05 migration status: active (transferred 1815021189, remaining 268378112), total 2165121024)
2019-08-16 11:50:05 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:50:06 migration status: active (transferred 1845138214, remaining 235667456), total 2165121024)
2019-08-16 11:50:06 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:50:07 migration status: active (transferred 1877470040, remaining 203227136), total 2165121024)
2019-08-16 11:50:07 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:50:08 migration status: active (transferred 1904196234, remaining 174321664), total 2165121024)
2019-08-16 11:50:08 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:50:09 migration status: active (transferred 1930237141, remaining 146063360), total 2165121024)
2019-08-16 11:50:09 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:50:10 migration status: active (transferred 1955568776, remaining 120049664), total 2165121024)
2019-08-16 11:50:10 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:50:11 migration status: active (transferred 1984565328, remaining 90357760), total 2165121024)
2019-08-16 11:50:11 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:50:12 migration status: active (transferred 2019780241, remaining 46641152), total 2165121024)
2019-08-16 11:50:12 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:50:13 migration status: active (transferred 2045041805, remaining 17121280), total 2165121024)
2019-08-16 11:50:13 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-08-16 11:50:13 migration status: active (transferred 2048340718, remaining 13561856), total 2165121024)
2019-08-16 11:50:13 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 168 overflow 0
2019-08-16 11:50:13 migration status: active (transferred 2051365391, remaining 10539008), total 2165121024)
2019-08-16 11:50:13 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 905 overflow 0
2019-08-16 11:50:13 migration status: active (transferred 2055810040, remaining 6098944), total 2165121024)
2019-08-16 11:50:13 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 1988 overflow 0
2019-08-16 11:50:13 migration status: active (transferred 2057829395, remaining 4005888), total 2165121024)
2019-08-16 11:50:13 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 2480 overflow 0
2019-08-16 11:50:13 migration status: active (transferred 2060814104, remaining 540672), total 2165121024)
2019-08-16 11:50:13 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 3207 overflow 0
2019-08-16 11:50:13 migration speed: 5.11 MB/s - downtime 49 ms
2019-08-16 11:50:13 migration status: completed
drive-scsi0: transferred: 34360262656 bytes remaining: 0 bytes total: 34360262656 bytes progression: 100.00 % busy: 0 ready: 1
all mirroring jobs are ready
drive-scsi0: Completing block job...
drive-scsi0: Completed successfully.
drive-scsi0 : finished
 Logical volume "vm-100-disk-0" successfully removed
2019-08-16 11:50:21 migration finished successfully (duration 00:06:55)
TASK OK