1. Kettle转换执行流程
Kettle转换执行流程体现在Trans类的execute()方法，代码如下所示：
public void execute( String[] arguments ) throws KettleException {
prepareExecution( arguments );
startThreads();
}1.1 prepareExecution流程分析
prepareExecution方法完成的主要工作是:
- 设置转换参数、变量
- 处理步骤之间数据传递是分发还是复制
- 初始化设置转换日志表、步骤日志表、性能记录表、添加一系列TransListener
- 构造StepMetaDataCombi对象集合，该对象负责传递Step,StepMeta以及StepData对象给RunThread
- 调用每个Step的init方法进行步骤初始化,确保所有Step的初始化都顺利进行,出错就提前结束所有步骤
- 步骤监控的初始化
1.2 startThreads流程分析
startThreads方法完成的主要工作是:
- TransListener和StepListener构造
- 当listener准备好之后就根据转换类型启动线程跑转换里的步骤，通过构造RunThread对象来执行步骤
- 当所有步骤完成会触发StepListener的stepFinished方法里面的fireTransFinishedListeners()，通知转换，步骤已经全部执行结束,转换可以结束了,设置相关参数并清理现场
核心StepListener构造片段代码如下所示:
StepListener stepListener = new StepListener() {
.....
public void stepFinished( Trans trans, StepMeta stepMeta, StepInterface step ) {
synchronized ( Trans.this ) {
nrOfFinishedSteps++;
if ( nrOfFinishedSteps >= steps.size() ) {
// Set the finished flag
//
setFinished( true );
// Grab the performance statistics one last time (if enabled)
//
addStepPerformanceSnapShot();
try {
fireTransFinishedListeners();
} catch ( Exception e ) {
step.setErrors( step.getErrors() + 1L );
log.logError( getName()
+ " : " + BaseMessages.getString( PKG, "Trans.Log.UnexpectedErrorAtTransformationEnd" ), e );
}
}
// If a step fails with an error, we want to kill/stop the others
// too...
//
if ( step.getErrors() > 0 ) {
log.logMinimal( BaseMessages.getString( PKG, "Trans.Log.TransformationDetectedErrors" ) );
log.logMinimal( BaseMessages.getString(
PKG, "Trans.Log.TransformationIsKillingTheOtherSteps" ) );
killAllNoWait();
}
}
}
};
// Make sure this is called first!
//
if ( sid.step instanceof BaseStep ) {
( (BaseStep) sid.step ).getStepListeners().add( 0, stepListener );
} else {
sid.step.addStepListener( stepListener );
}fireTransFinishedListeners方法代码如下所示:
/**
* Make attempt to fire all registered listeners if possible.
*
* @throws KettleException
* if any errors occur during notification
*/
protected void fireTransFinishedListeners() throws KettleException {
// PDI-5229 sync added
synchronized ( transListeners ) {
if ( transListeners.size() == 0 ) {
return;
}
//prevent Exception from one listener to block others execution
List<KettleException> badGuys = new ArrayList<KettleException>( transListeners.size() );
for ( TransListener transListener : transListeners ) {
try {
transListener.transFinished( this );
} catch ( KettleException e ) {
badGuys.add( e );
}
}
// Signal for the the waitUntilFinished blocker...
transFinishedBlockingQueue.add( new Object() );
if ( !badGuys.isEmpty() ) {
//FIFO
throw new KettleException( badGuys.get( 0 ) );
}
}
}fireTransFinishedListeners方法负责通知所有的TransListener转换已经结束，通过调用
transListener.transFinished(this)来完成通知。这个方法还负责解除waitUntilFinished方法调用的阻塞状态。waitUntilFinished方法在execute方法执行后调用可以等待转换结束或者出错才返回。这是因为该方法利用了一个阻塞队列(BlockingQueue)transFinishedBlockingQueue的poll方法来进行阻塞，而只有当上面讲到的fireTransFinishedListeners方法触发了，才会执行
// Signal for the the waitUntilFinished blocker...
transFinishedBlockingQueue.add( new Object() );来解除阻塞队列的阻塞状态。
1.3 RunThread.run()流程分析
一般情况下转换里的每个步骤隔离到单独的线程执行，步骤执行的逻辑代码表现在类RunThread(实现了Runnable接口)的run方法里面，核心逻辑很简单，调用step的processRow方法直到没有输入数据要处理表示该步骤已经结束，结束的时候会调用step.dispose清理现场资源，最后调用step.markStop（）标记步骤已经停止，在markStop方法里面会回调所有StepListener的stepFinished方法(实现在BaseStep类里面)，因此在Trans类的startThreads方法里面构造的核心StepListener方法将被调用。
RunThread的run方法代码如下所示：
public void run() {
try {
step.setRunning( true );
step.getLogChannel().snap( Metrics.METRIC_STEP_EXECUTION_START );
if ( log.isDetailed() ) {
log.logDetailed( BaseMessages.getString( "System.Log.StartingToRun" ) );
}
// Wait
while ( step.processRow( meta, data ) ) {
if ( step.isStopped() ) {
break;
}
}
} catch ( Throwable t ) {
try {
// check for OOME
if ( t instanceof OutOfMemoryError ) {
// Handle this different with as less overhead as possible to get an error message in the log.
// Otherwise it crashes likely with another OOME in Me$$ages.getString() and does not log
// nor call the setErrors() and stopAll() below.
log.logError( "UnexpectedError: ", t );
} else {
t.printStackTrace();
log.logError( BaseMessages.getString( "System.Log.UnexpectedError" ), t );
}
String logChannelId = log.getLogChannelId();
LoggingObjectInterface loggingObject = LoggingRegistry.getInstance().getLoggingObject( logChannelId );
String parentLogChannelId = loggingObject.getParent().getLogChannelId();
List<String> logChannelChildren = LoggingRegistry.getInstance().getLogChannelChildren( parentLogChannelId );
int childIndex = Const.indexOfString( log.getLogChannelId(), logChannelChildren );
System.out.println( "child index = "
+ childIndex + ", logging object : " + loggingObject.toString() + " parent=" + parentLogChannelId );
KettleLogStore.getAppender().getBuffer( "2bcc6b3f-c660-4a8b-8b17-89e8cbd5b29b", false );
// baseStep.logError(Const.getStackTracker(t));
} catch ( OutOfMemoryError e ) {
e.printStackTrace();
} finally {
step.setErrors( 1 );
step.stopAll();
}
} finally {
step.dispose( meta, data );
step.getLogChannel().snap( Metrics.METRIC_STEP_EXECUTION_STOP );
try {
long li = step.getLinesInput();
long lo = step.getLinesOutput();
long lr = step.getLinesRead();
long lw = step.getLinesWritten();
long lu = step.getLinesUpdated();
long lj = step.getLinesRejected();
long e = step.getErrors();
if ( li > 0 || lo > 0 || lr > 0 || lw > 0 || lu > 0 || lj > 0 || e > 0 ) {
log.logBasic( BaseMessages.getString( PKG, "BaseStep.Log.SummaryInfo", String.valueOf( li ),
String.valueOf( lo ), String.valueOf( lr ), String.valueOf( lw ),
String.valueOf( lu ), String.valueOf( e + lj ) ) );
} else {
log.logDetailed( BaseMessages.getString( PKG, "BaseStep.Log.SummaryInfo", String.valueOf( li ),
String.valueOf( lo ), String.valueOf( lr ), String.valueOf( lw ),
String.valueOf( lu ), String.valueOf( e + lj ) ) );
}
} catch ( Throwable t ) {
//
// it's likely an OOME, so we don't want to introduce overhead by using BaseMessages.getString(), see above
//
log.logError( "UnexpectedError: " + Const.getStackTracker( t ) );
} finally {
step.markStop();
}
}
}
本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。
