hdfs 如何实现退役节点快速下线（也就是退役节点上的数据块快速迁移）speed up decommission blocks removal
以下是选择复制源节点的代码
代码总结：
A=datanode上要复制block的Queue size与 target datanode没被选出之前待处理复制工作数之和。
1. 优先选择退役中的节点，因为其无写入请求，负载低。
2. 不会选择退役完成的节点。
3. 如果A未达到复制限制（<maxReplicationStreams，conf配置名为dfs.namenode.replication.max-streams）,在nodelist中随机选择节点
4. 如果A达到复制限制（>=maxReplicationStreams）,除非blockQueue是最高优先级的，会随机选一台节点，否者会pass掉（代码中为continue）。
5. 如果A达到复制硬限制（>=replicationStreamsHardLimit,conf配置为dfs.namenode.replication.max-streams-hard-limit），无论满足什么条件都会pass。/**
* Parse the data-nodes the block belongs to and choose one,
* which will be the replication source.
*
* We prefer nodes that are in DECOMMISSION_INPROGRESS state to other nodes
* since the former do not have write traffic and hence are less busy.
* We do not use already decommissioned nodes as a source.
* Otherwise we choose a random node among those that did not reach their
* replication limits. However, if the replication is of the highest priority
* and all nodes have reached their replication limits, we will choose a
* random node despite the replication limit.
*
* In addition form a list of all nodes containing the block
* and calculate its replication numbers.
解析块所属的数据节点并选择一个，这将是复制源。
我们更偏向处于DECOMMISSION_INPROGRESS（退役中）状态的节点到其他节点，因为前者没有写入流量，因此不太忙。
我们不使用已经退役的节点作为源。
除此之外，我们会在未达到复制限制的节点中选择一个随机节点（<maxReplicationStreams）。
但是，如果复制到达最高优先级并且所有节点都已达到其复制限制(>=maxReplicationStreams,当然如果>=replicationStreamsHardLimit最算是最高优先级也是不行的)，我们将随机选择一个节点，尽管有复制限制。
另外，形成包含块的所有节点的列表并计算其复制数。
*
* @param block Block for which a replication source is needed
* @param containingNodes List to be populated with nodes found to contain the
* given block
* @param nodesContainingLiveReplicas List to be populated with nodes found to
* contain live replicas of the given block
* @param numReplicas NumberReplicas instance to be initialized with the
* counts of live, corrupt, excess, and
* decommissioned replicas of the given
* block.
* @param priority integer representing replication priority of the given
* block
* @return the DatanodeDescriptor of the chosen node from which to replicate
* the given block
*/
@VisibleForTesting
DatanodeDescriptor chooseSourceDatanode(Block block,
List<DatanodeDescriptor> containingNodes,
List<DatanodeStorageInfo> nodesContainingLiveReplicas,
NumberReplicas numReplicas,
int priority) {
containingNodes.clear();
nodesContainingLiveReplicas.clear();
DatanodeDescriptor srcNode = null;
int live = 0;
int decommissioned = 0;
int corrupt = 0;
int excess = 0;
Collection<DatanodeDescriptor> nodesCorrupt = corruptReplicas.getNodes(block);
for(DatanodeStorageInfo storage : blocksMap.getStorages(block)) {
final DatanodeDescriptor node = storage.getDatanodeDescriptor();
LightWeightLinkedSet<Block> excessBlocks =
excessReplicateMap.get(node.getDatanodeUuid());
int countableReplica = storage.getState() == State.NORMAL ? 1 : 0;
if ((nodesCorrupt != null) && (nodesCorrupt.contains(node)))
corrupt += countableReplica;
else if (node.isDecommissionInProgress() || node.isDecommissioned())
decommissioned += countableReplica;
else if (excessBlocks != null && excessBlocks.contains(block)) {
excess += countableReplica;
} else {
nodesContainingLiveReplicas.add(storage);
live += countableReplica;
}
containingNodes.add(node);
// Check if this replica is corrupt
// If so, do not select the node as src node
if ((nodesCorrupt != null) && nodesCorrupt.contains(node))
continue;
if(priority != UnderReplicatedBlocks.QUEUE_HIGHEST_PRIORITY
&& node.getNumberOfBlocksToBeReplicated() >= maxReplicationStreams)
{
continue; // already reached replication limit
}
if (node.getNumberOfBlocksToBeReplicated() >= replicationStreamsHardLimit)
{
continue;
}
// the block must not be scheduled for removal on srcNode
if(excessBlocks != null && excessBlocks.contains(block))
continue;
// never use already decommissioned nodes
if(node.isDecommissioned())
continue;
// we prefer nodes that are in DECOMMISSION_INPROGRESS state
// 如果你是退役中的节点会被优先选择并把你赋值给srcNode,后期的判断就基本跳过DFSUtil.getRandom().nextBoolean()，因为在 if(srcNode.isDecommissionInProgress()) continue;直接continue了。
if(node.isDecommissionInProgress() || srcNode == null) {
srcNode = node;
continue;
}
if(srcNode.isDecommissionInProgress())
continue;
// switch to a different node randomly
// this to prevent from deterministically selecting the same node even
// if the node failed to replicate the block on previous iterations
// 这里主要是防止一直选相同节点，如果所遍历的node是退役中节点，不会走这个方法（在上层就continue了），如果某个待复制的block（正常块，不存在损坏超额等问题）所在的三个节点都是普通正常节点，那具体选择哪个node就完全随机了，虽然是个伪随机。
if(DFSUtil.getRandom().nextBoolean())
srcNode = node;
}
if(numReplicas != null)
numReplicas.initialize(live, decommissioned, corrupt, excess, 0);
return srcNode;
}加快退役的三个参数
默认数分别为2，4，2 修改为 10，20，5 后
测试2w+个块（15G左右）迁移，后者比前者(默认参数)快4-5倍左右
但是如果线上不建议开过大，会严重影响线上正常服务
<!-- speed up decommission -->
<property>
<name>dfs.namenode.replication.max-streams</name>
<value>10</value>
</property>
<property>
<name>dfs.namenode.replication.max-streams-hard-limit</name>
<value>20</value>
</property>
<property>
<name>dfs.namenode.replication.work.multiplier.per.iteration</name>
<value>5</value>
</property>前两个参数已经介绍完了，下面来介绍下第三个参数的用途
dfs.namenode.replication.work.multiplier.per.iteration 参数决定了可以从很多under replication blocks中选出多少个block 准备进行复制。
可以选出的block数与集群live 的datadnode 成正比。
int blocksToProcess = numlive * this.blocksReplWorkMultiplier（就是这个参数dfs.namenode.replication.work.multiplier.per.iteration），如果线上有10个in service datanode 那默认就可以选出 10 * 2 = 20个block 准备进行replicate
源码如图：
总结一下这三个参数：
这三个参数其实就是个限流参数
第三个参数相当于入口限流参数，决定了可以从under replication blocks 的集合中选出多少个blocks 加到复制队列中准备进行复制。
前两个参数相当于出口限流，他们是在遍历复制队列中的blocks时，决定是否给当前block一个srcNode（也可能返回null）。如果满足限流条件，srcNode=当前node；如果不满足就contiune,直到找到满足的node，如果所有node都不满足就返回null（如果返回null 该block是不会relicate的）；问题：
1. 队列中的block 最高优先级指什么，是live的副本数远小于默认的副本数吗？
2. "ask: srcNode" 日志中的srcNode 节点不只是退役中的节点，为什么?问题1解答：
block所在队列的优先级有五个，源码如下：
/** The total number of queues : {@value} */
// 共五个等级
static final int LEVEL = 5;
/** The queue with the highest priority: {@value} */
//最高优先级queue：这些blocks会被优先复制。只有一个副本的块或者这些块有0个live副本,1个副本在正在退役的节点.如果这些blocks所在的磁盘或者服务器出现问题，这些blocks有丢失风险。
static final int QUEUE_HIGHEST_PRIORITY = 0;
/** The queue for blocks that are way below their expected value : {@value} */
//第二优先级: 这些blocks的副本远低于他们所期望的值。目前意味着实际的比例时少于1/3,虽然这些blocks可能并没有风险，但是他们清楚地考虑到blocks重要性。
static final int QUEUE_VERY_UNDER_REPLICATED = 1;
/** The queue for "normally" under-replicated blocks: {@value} */
//第三优先级：这些blocks 也低于所期望值(同样处于复制状态的块)，以及 实际：预期的比例 足够好（不少与1/3），以至于它们不需要进入队列
static final int QUEUE_UNDER_REPLICATED = 2;
/** The queue for blocks that have the right number of replicas,
* but which the block manager felt were badly distributed: {@value}
*/
//第四优先级：一个block满足要求的最小副本数，但是由于分布不充分，如果一个机架当机可能导致所有的副本丢失/下线
static final int QUEUE_REPLICAS_BADLY_DISTRIBUTED = 3;
/** The queue for corrupt blocks: {@value} */
//第五优先级：这适用于已损坏的块，并且当前有可用的非损坏副本（当前）。这里的策略是保持这些损坏的块被复制，但是给出不是更高优先级的块。
static final int QUEUE_WITH_CORRUPT_BLOCKS = 4;问题2的解答：（ xx.x.x.34为退役中节点）
1.退役节点的上要replica 的block非常多，有些block还没来得及replica 就已经过期了，所以不能把退役中的节点作为srcNode
2.由于前期以退役节点为srcNode较多，已经有达到了dfs.namenode.replication.max-streams 或 dfs.namenode.replication.max-streams-hard-limit 的上限，所以只能选其他节点作为srcNode
这是复制block日志可以说明问题2， xx.x.x.34 达到上限10（也可能是20）后，就会选择其他节点作为srcNode了(共70个)。
2018-09-25 11:38:01,439 INFO BlockStateChange: BLOCK* ask xx.x.x.34:50010 to replicate blk_1298821823_225481125 to datanode(s) xx.x.x.28:50010
2018-09-25 11:38:01,439 INFO BlockStateChange: BLOCK* ask xx.x.x.34:50010 to replicate blk_1298821896_225481198 to datanode(s) xx.x.x.36:50010
2018-09-25 11:38:01,439 INFO BlockStateChange: BLOCK* ask xx.x.x.34:50010 to replicate blk_1298821588_225480890 to datanode(s) xx.x.x.45:50010
2018-09-25 11:38:01,439 INFO BlockStateChange: BLOCK* ask xx.x.x.34:50010 to replicate blk_1298821581_225480883 to datanode(s) xx.x.x.30:50010
2018-09-25 11:38:01,439 INFO BlockStateChange: BLOCK* ask xx.x.x.34:50010 to replicate blk_1282045832_208688251 to datanode(s) xx.x.x.30:50010
2018-09-25 11:38:01,439 INFO BlockStateChange: BLOCK* ask xx.x.x.34:50010 to replicate blk_1282045861_208688280 to datanode(s) xx.x.x.44:50010
2018-09-25 11:38:01,439 INFO BlockStateChange: BLOCK* ask xx.x.x.34:50010 to replicate blk_1282045812_208688231 to datanode(s) xx.x.x.31:50010
2018-09-25 11:38:01,439 INFO BlockStateChange: BLOCK* ask xx.x.x.34:50010 to replicate blk_1282045643_208688062 to datanode(s) xx.x.x.11:50010
2018-09-25 11:38:01,439 INFO BlockStateChange: BLOCK* ask xx.x.x.34:50010 to replicate blk_1282045478_208687897 to datanode(s) xx.x.x.43:50010
2018-09-25 11:38:01,439 INFO BlockStateChange: BLOCK* ask xx.x.x.34:50010 to replicate blk_1282045530_208687949 to datanode(s) xx.x.x.22:50010
2018-09-25 11:38:01,439 INFO BlockStateChange: BLOCK* ask xx.x.x.41:50010 to replicate blk_1206547507_133106764 to datanode(s) xx.x.x.42:50010
2018-09-25 11:38:01,439 INFO BlockStateChange: BLOCK* ask xx.x.x.41:50010 to replicate blk_1206548077_133107334 to datanode(s) xx.x.x.28:50010
2018-09-25 11:38:01,439 INFO BlockStateChange: BLOCK* ask xx.x.x.11:50010 to replicate blk_1206548222_133107479 to datanode(s) xx.x.x.19:50010
2018-09-25 11:38:01,439 INFO BlockStateChange: BLOCK* ask xx.x.x.11:50010 to replicate blk_1189768077_116313696 to datanode(s) xx.x.x.45:50010
......解释下图字段的含义：
Node ：退役中的节点hostname
Last contact:节点心跳，这里为空，在页面上面会展示心跳信息
Under replicated blocks：当前block的副本 < 所设置的副本(默认就是小于3) 数量
Blocks with no live replicas：没有live 的副本，存在的副本可能都在退役中的节点上
Under Replicated Blocks In files under construction: 当前正在复制中的block个数
可以通过 这个命令看内存的配置key-value
hdfs getconf -confkey xxx
本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。
