Expert Consultancy from Yellow Pelican

Talend Job Deployment & Scheduling

A site about Talend

Deployment & Scheduling

Now that you've written your Job and executed it within the Talend Designer, it's time to deploy your Job to Test or Production (depending on your working enviroment).

If you're using Talend Open Studio (TOS), then you can export your Job in a number of export types. For the purpose of this tutorial, we will look at deploying your Job as an Autonomous Job. If you're using the Enterprise edition of Talend, then you are more likely to be deploying your Job to the Talend Command Centre (TAC). Enterprise Talend is not currently covered in this tutorial. This tutorial will concentrate on Autonomous Jobs and we will look at the other formats at a later time.

The Talend export types are: -

  • Autonomous Job
  • Axis WebService (WAR)
  • AxisWebService (ZIP)
  • JBoss ESB
  • Petals ESB
  • OSGI Bundle For ESB

For this tutorial, we're going to use a simple Job that does no more than display a message to stdout.

Export Job

To export your Job as an Autonomous Job, right-click your Job in the Talend Repository Browser and select Export Job. The Export Jobs dialog will be displayed.

Archive File

Jobs are exported to a ZIP file archive. Enter a name of your choice for To file archive or leave the default value. You may use Browse to select a location.

Job Version

By default, the latest version of your Job will be exported. If you have multiple versions, you may select an alternative version here.

Export Type

as previously discussed, for the purposes of this tutorial, we'll be exporting the default Export Type, an Autonomous Job.

Options

Options allow you to control different aspects of how your Job is exported. The following sections describe the Options that are available when you export your Job.

Shell Launchers

The Shell launchers option allows you to specify if you want to create shell launch scripts. These are Unix and Windows style scripts for launching your Job. You may choose the style of scripts that you want to export. For more on these scripts, read our Job Shell Launchers Tutorial.

Context Scripts

If you select the Context scripts option, then Talend will export scripts for each of the Context that you have defined in your Job (even if you have defined no Context Variables). By default, a Job has a single Context named Default. You may have created your own Context, for example, Production and you may even have changed the name of your default Context. If you export Context Scripts then Talend will amend the Shell Launch Scripts so that, by default, they read the Context that you specified in the accompanying drop-down list.

It is important to note that if you have Context Variables in your Job, you do not need to export the scripts and, likewise, if you have no Context Variables, you may still export the scripts. The key point to remember here is that if you use Context Variables but do not export the scripts, no default values will be assigned to your Context Variables when you launch your Job.

For more information on Context read our Context Tutorial and for more on Shell launch scripts, read our Job Shell Launchers Tutorial.

Apply Context to children Jobs

The Apply Context to Children Jobs is an option that I have yet to find a purpose for. By default, this option is checked and my suggestion would be to never change it.

From what I can understand, when un-checking this option, it causes a JAR Archive modification so that when a Child Job is called, and you've selected a Context other than the default from the Context scripts accompanying drop-down list, the default Context continues to be passed to Child Jobs rather than the Context that you selected. To be clear, we're talking about the Context Name here, rather than passing Context Variables in the same manner as when you select Transmit whole context, when configuring a tRunJob component. Once your export has dissassociated the Context of the Parent Job from its Child Jobs, there appears to be no ways to alter this behaviour in the Autonomous Job without performing a new export.

Override Parameter's values

By selecting Override Parameter's values, a dialog will be displayed that will alow you to override the value of any Context Variables irrespective of the Context in which your Job executes. This dialog also allows you to specify new parameters that are passed to your Job. When you override a parameter's values, no change is made to your Context Scripts. Parameters are overriden by passing them as command-line arguments to your Job.

The following example shows a Unix style Shell Launchers with two additional parameters. The modification to the value of a parameter named myString and the addition of a new parameter named newString. Remember that this has caused no modification to any Context Script and that you may add, modify or remove these parameters directly in your Shell Launcher scripts. Note that -- indicates that this is a parameter that is passed to your Job rather than a directive to the Java Runtime.

java...--context=Default --context_param myString="modified value" --context_param newString="new value" "$@"

Java Sources

This option allows you to specify if the Java Source Code that has been generated for your Job, should be exported. We'll investigate this further in a later article.

Items

This option allows you to specify if Items should be exported. Items represents the Talend definition of your Job that is used within the Talend Design Tool, which could be thought of as your Talend Source Code. Exporting Items is a convenient and important part of you Source Code Control and backup regime. Exporting Items allows you to migrate your Jobs easily from one Project or Workspace to another. You may export Items independently of your Autonomous Job by selecting Export items rather than Export Job.

Analysis of a Simple Job Export

To getter a better understanding of what export does and how we can then execute our Job, Let's create a simple Job that only prints a message, and then we'll export the Job.

For this first analysis, we'll uncheck all of the options on the Export Jobs Dialog except for two options. We'll check Shell launcher as it seems likely that we'll, at least, want to run our Job and this is a convenient way of doing so. We'll also check Apply Context to Children Jobs as, as I said earlier, I see no usecase for having this unchecked. we'll look at the output from the other options in later article. For now, we just want to be able to run our Job.

Let's look at the files that are created by an export. In our example, we've extracted our fresh export of a Job named SimpleJob and, below, we can see the files that have been created. Remember that our Job is exported to a ZIP file archive. Although we can request that our export automatically extracts the ZIP file archive, my preference is to so this manually.

Generated Files

You'll see from the code snippet below, that I've created a new directory named SimpleJob and I've extracted my exported file to that directory. Some of the files in the export file will be extracted to the current working directory; therefore, by creating a new directory, we're provided with a clean view of what has been extracted.

The first thing that you notice is that some files have been created under a sub-directory named SimpleJob. These are files that are specific to the single Job that we exported; whereas other files created including jobInfo.properties and lib are not Job specific. Had we have exported multiple Jobs (in the same file archive), then we would have seen additional directories being created for each of the other Jobs, containing their Job specific files; while the other files would become cross-Job.

mkdir SimpleJob
cd SimpleJob
unzip ../SimpleJob_0.1.zip

./SimpleJob
   ./SimpleJob/SimpleJob_run.bat
   ./SimpleJob/SimpleJob_run.sh
   ./SimpleJob/items
      ./SimpleJob/items/talendbyexample
      ./SimpleJob/items/talendbyexample/code
      ./SimpleJob/items/talendbyexample/code/routines
      ./SimpleJob/items/talendbyexample/code/routines/system
      ./SimpleJob/items/talendbyexample/code/routines/system/DataOperation_0.1.item
      ./SimpleJob/items/talendbyexample/code/routines/system/DataOperation_0.1.properties
      ...
      ./SimpleJob/items/talendbyexample/code/routines/system/TalendString_0.1.item
      ./SimpleJob/items/talendbyexample/code/routines/system/TalendString_0.1.properties
      ./SimpleJob/items/talendbyexample/talend.project
   ./SimpleJob/simplejob_0_1.jar
   ./jobInfo.properties
   ./lib
      ./lib/dom4j-1.6.1.jar
      ./lib/systemRoutines.jar
      ./lib/userRoutines.jar

Shell Launchers

If you selected the Shell launcher option, then Talend can generate both a Unix Shell Script or a Windows Batch File for launching your Job. The Unix Shell Script is, of course, suitable for Linux and OSX too. If you don't require both styles of script, then you may also select the script type that you want.

For further information, read Shell Launchers and Scheduling with cron.




Expert Consultancy from Yellow Pelican
comments powered by Disqus

© www.TalendByExample.com