命令作用:清空表数据,但是保留表结构,


分析原因:今天定位一个奇怪的现象,当hbase建立4个region的表,然后用truncate命令时,出现1个region在线而有4个region offline,

然后继续put数据,制定split到4个region时,使用truncate_preserve发现出现9个offline,7个failed,并且再put数据时出现表不存在,

问题还在定位中。所以需要分析源码。


1)truncate shell命令入口

module Shell
  module Commands
    class Truncate < Command
      def help
        return <<-EOF
  Disables, drops and recreates the specified table.
EOF
      end

      def command(table)
        format_simple_command do
          puts "Truncating '#{table}' table (it may take a while):"
          admin.truncate(table) { |log| puts " - #{log}" }
        end
      end

    end
  end
end


2)客户端有HBaseAdmin 发起,经MasterRpcServices转接,由HMaster执行,主要代码如下:preserveSplits是命令truncate_preserve时,保留region个数。否则region变为1.

MasterProcedureUtil.submitProcedure(
      new MasterProcedureUtil.NonceProcedureRunnable(this, nonceGroup, nonce) {
      @Override
      protected void run() throws IOException {
        getMaster().getMasterCoprocessorHost().preTruncateTable(tableName);
        LOG.info(getClientIdAuditPrefix() + " truncate " + tableName);
        long procId = submitProcedure(new TruncateTableProcedure(procedureExecutor.getEnvironment(),
 tableName, preserveSplits));
        ProcedureSyncWait.waitForProcedureToComplete(procedureExecutor, procId);
        getMaster().getMasterCoprocessorHost().postTruncateTable(tableName);
      }

这里可以分两步,

第一个submitProcedure怎么运行的。

第二个truncateTableprocedure里的具体流程是什么


1、submitProcedure

这个流程主要如下:

1)从Hmaster获取ProcedureExecutor

2)ProcedureExecutor 是在HMaster 启动时执行的startProcedureExecutor();

3)startProcedureExecutor里面调用了,procedureExecutor.start(numThreads, abortOnCorruption);

4)start里面调用一个循环execLoop(); ----(类ProcedureExecutor中)

5)execLoop里面是循环,不断从集合(ProcedureRunnableSet runnables)里拿到进程然后执行(execLoop(proc);)

6)execLoop(proc)主要执行 execProcedure(procStack, proc); ----(类ProcedureExecutor中)

7)最终会调用 procedure.doExecute(getEnvironment());

8)doExecute 会调用execute(env); (类Procedure中)

9)在execute(env)中会调用 executeFromState (类StateMachineProcedure中)

所以只需要关心procedure的executeFromState方法就行。

2、TruncateTableprocedure

在这个类里面主要关心方法executeFromState:

1)TRUNCATE_TABLE_PRE_OPERATION(0, 1),   准备动作:regions=ProcedureSyncWait.getRegionsFromMeta(env, getTableName());

        这里还做了写协处理器问题。getRegionsFromMeta就只一个查询动作

2)TRUNCATE_TABLE_REMOVE_FROM_META(1, 2),

     在META表中删除数据:DeleteTableProcedure.deleteFromMeta(env, getTableName(), regions);

                这个就是一些delete操作

3)TRUNCATE_TABLE_CLEAR_FS_LAYOUT(2, 3),

DeleteTableProcedure.deleteFromFs(env, getTableName(), regions, true);

                    这里主要注意的是如果设置里归档,会将记录进行归档,否则直接会删除

regions= recreateRegionInfo(regions);

                    这里如果保留regions的分区,采用的这个方法,如果不是,就直接new一个regionInfo startkey 和endkey都为null

4)TRUNCATE_TABLE_CREATE_FS_LAYOUT(3, 4),

  重新创建FS目录,主要根据region建立:regions=CreateTableProcedure.createFsLayout(env, hTableDescriptor, regions);

5)TRUNCATE_TABLE_ADD_TO_META(4, 5),

regions =CreateTableProcedure.addTableToMeta(env, hTableDescriptor, regions);

6)TRUNCATE_TABLE_ASSIGN_REGIONS(5, 6),

CreateTableProcedure.assignRegions(env, getTableName(), regions);

        这个是createTable必须做的事情。

7)TRUNCATE_TABLE_POST_OPERATION(6, 7), 

postTruncate(env);

完整代码如下:

@Override
  protected Flow executeFromState(final MasterProcedureEnv env, TruncateTableState state)
      throws InterruptedException {
  
    try {
      switch (state) {
        case TRUNCATE_TABLE_PRE_OPERATION:
          // Verify if we can truncate the table
          if (!prepareTruncate(env)) {
            assert isFailed() : "the truncate should have an exception here";
            return Flow.NO_MORE_STATE;
          }
          // TODO: Move out... in the acquireLock()
          LOG.debug("waiting for '" + getTableName() + "' regions in transition");
          regions = ProcedureSyncWait.getRegionsFromMeta(env, getTableName());
          assert regions != null && !regions.isEmpty() : "unexpected 0 regions";
          ProcedureSyncWait.waitRegionInTransition(env, regions);
          // Call coprocessors
          preTruncate(env);
          setNextState(TruncateTableState.TRUNCATE_TABLE_REMOVE_FROM_META);
          break;
        case TRUNCATE_TABLE_REMOVE_FROM_META:
          hTableDescriptor = env.getMasterServices().getTableDescriptors().get(tableName);
          DeleteTableProcedure.deleteFromMeta(env, getTableName(), regions);
          DeleteTableProcedure.deleteAssignmentState(env, getTableName());
          setNextState(TruncateTableState.TRUNCATE_TABLE_CLEAR_FS_LAYOUT);
          break;
        case TRUNCATE_TABLE_CLEAR_FS_LAYOUT:
          DeleteTableProcedure.deleteFromFs(env, getTableName(), regions, true);
          if (!preserveSplits) {
            // if we are not preserving splits, generate a new single region
            regions = Arrays.asList(ModifyRegionUtils.createHRegionInfos(hTableDescriptor, null));
          } else {
            regions = recreateRegionInfo(regions);
          }
          setNextState(TruncateTableState.TRUNCATE_TABLE_CREATE_FS_LAYOUT);
          break;
        case TRUNCATE_TABLE_CREATE_FS_LAYOUT:
          regions = CreateTableProcedure.createFsLayout(env, hTableDescriptor, regions);
          CreateTableProcedure.updateTableDescCache(env, getTableName());
          setNextState(TruncateTableState.TRUNCATE_TABLE_ADD_TO_META);
          break;
        case TRUNCATE_TABLE_ADD_TO_META:
          regions = CreateTableProcedure.addTableToMeta(env, hTableDescriptor, regions);
          setNextState(TruncateTableState.TRUNCATE_TABLE_ASSIGN_REGIONS);
          break;
        case TRUNCATE_TABLE_ASSIGN_REGIONS:
          CreateTableProcedure.assignRegions(env, getTableName(), regions);
          setNextState(TruncateTableState.TRUNCATE_TABLE_POST_OPERATION);
          hTableDescriptor = null;
          regions = null;
          break;
        case TRUNCATE_TABLE_POST_OPERATION:
          postTruncate(env);
          LOG.debug("truncate '" + getTableName() + "' completed");
          return Flow.NO_MORE_STATE;
        default:
          throw new UnsupportedOperationException("unhandled state=" + state);
      }
    } catch (HBaseException|IOException e) {
      LOG.warn("Retriable error trying to truncate table=" + getTableName() + " state=" + state, e);
    }
    return Flow.HAS_MORE_STATE;
  }