Expert Consultancy from Yellow Pelican

tCheckpoint

A site about Talend

tCheckpoint Component

Whether you're programming in Talend or using any other language, checkpointing is a useful technique for ensuring that you can restart your Jobs and for them to be able to recover from a previous error.

This custom component, available from the Talend Exchange, will speed up your development and will help you to write robust Jobs that are able to automatically recover from a failure in a predictable manner.

Checkpointing allows you to achieve several important objectives.

  • Record the point of failure
  • Resume from the point of failure - don't re-run successfully executed code
  • Execute custom recovery code
  • Allow normal execution of previously unexecuted code

Talend Enterprise Checkpoints

Talend Enterprise allows checkpoints to be defined for OnSubjobOk connectors; however, this functionality is not available for Talend Open Studio. This enterprise feature does not provide me with the level of functionality that I am looking for, which is the reason that I built tCheckpoint, tCheckpointStart and tCheckpointEnd. These may be used in both Talend Enterprise and Talend Open Studio.

Download tCheckpoint

This download is comprised of 3 components, tCheckpoint, tCheckpointStart and tCheckpointEnd. You may download tCheckpoint now, or directly import these components in to Talend Studio from the Talend Exchange.

Demonstration Job

In the demonstration Job shown below, you'll see that there is a series of two checkpoints. Each checkpoint performs a specific task and also has some logic to recover the task, should it fail.

The two tasks, DoSomeStuff and DoSomeMoreStuff have been designed to periodically fail, as can bee seen from the code fragment, below.

// tJava_1 DoSomeStuff
System.out.println("Executing Checkpoint 1");
java.util.Random random = new java.util.Random();
Integer i = random.nextInt(8);
if(i == 0) throw new Exception("Deliberately Failing Checkpoint 1");

tCheckpointStart

The tCheckpointStart component starts a series of checkpoints; which must be ended with a tCheckpointEnd component. tCheckpointStart provides two settings, Checkpoint Directory and (optionally) Checkpoint File.

Checkpoint Directory

This setting allows you to specify the directory where your checkpoint files should be located.

Checkpoint File

This is an optional setting that allows you to specify the name of your checkpoint file. In keeping with all Talend components, if you do not wish to provide a value, you must enter "" (the default). The default chekpoint file name is constructed from your project name and Job name, for example, MYPROJECT.MyJob.checkpoint.

On completion of tCheckpointStart a single gobalMap object is made available. This can be referenced in your Job, as shown below.

System.out.println((Boolean) globalMap.get("tCheckpoint_inRecovery"));

tCheckpoint

The tCheckpoint component provides a checkpoints within your Job and must be preceded by a tCheckpointStart component. tCheckpoint provides a single setting, for providing custom Code.

Code

This code is used in the same manner as your would when using the tJava component. This code is executed on completion of the checkpoint.

The previous execution status of a checkpoint is made available through two return values, as detailed below.

tCheckpoint_XXX_NEED_RECOVERY

This boolean return value indicates whether or not this checkpoint was the point of failure during the previous execution of the Job. If you have recovery code that needs executing, then use an If connector to execute the code. Remember that the recovery code would normally be executed prior to the logic that is being recovered, so set the connector order to order:1.

tCheckpoint_XXX_NEEDS_EXECUTION

This boolean return value indicates whether or not this checkpoint requires execution. A checkpoint would normally require execution unless it had been previously executed on a failed execution of the Job i.e. it's checkpoint was prior to the checkpoint that failed.

tCheckpointEnd

The tCheckpointEnd component ends a series of checkpoints. It must be preceded by a tCheckpointStart component and one or more tCheckpoint components.

Clear Checkpoint File

This parameter allows you to specify if the checkpoint file should be cleared at the end of a successful checkpoint series. tCheckpointEnd will write an END record to the file; which will always persist between executions of the Job. For debugging purposes, it may be helpful to retain checkpoint records between executions of your Job.

Output from a Recovering Job

The following shows the output from our demonstration job, when it is recovering from a failure during Checkpoint 1.

Starting job CP at 07:17 04/04/2014.

[statistics] connecting to socket on port 3608
[statistics] connected
************************************
*            tCheckpoint           *
*  FINAL CHECKPOINT NOT COMPLETED  *
*        RECOVERY REQUIRED         *
*  CHECKPOINT 001 NEEDS RECOVERING *
************************************
Checkpoint 1 needs recovery
Checkpoint 1 needs execution
Recovering Checkpoint 1
Executing Checkpoint 1
Checkpoint 2 needs execution
Executing Checkpoint 2
[statistics] disconnected
Job CP ended at 07:17 04/04/2014. [exit code=0]

The only output from the checkpoint components, comes from tCheckpointStart. This clearly shows the point of failure.

************************************
*            tCheckpoint           *
*  FINAL CHECKPOINT NOT COMPLETED  *
*        RECOVERY REQUIRED         *
*  CHECKPOINT 001 NEEDS RECOVERING *
************************************

The remaining messages in this demonstration Job, come from user-generated code. This clearly shows the recovery of Checkpoint 1 and the execution of both Checkpoint 1 and Checkpoint 2.

Checkpoint 1 needs recovery
Checkpoint 1 needs execution
Recovering Checkpoint 1
Executing Checkpoint 1
Checkpoint 2 needs execution
Executing Checkpoint 2

A Sample Checkpoint File

The following sample checkpoint file shows several successful executions of our Job, followed by a single failure and then a recovery.

1;START;NeedsRecovery=false, NeedsExecution=true
2;START;NeedsRecovery=false, NeedsExecution=true
0;END;Checkpoint series ended.     
1;START;NeedsRecovery=false, NeedsExecution=true
2;START;NeedsRecovery=false, NeedsExecution=true
0;END;Checkpoint series ended.
1;START;NeedsRecovery=false, NeedsExecution=true
1;START;NeedsRecovery=true, NeedsExecution=true
2;START;NeedsRecovery=false, NeedsExecution=true
0;END;Checkpoint series ended.




Expert Consultancy from Yellow Pelican
comments powered by Disqus

© www.TalendByExample.com