Expert Consultancy from Yellow Pelican

Job Shell Launchers

A site about Talend

Job Shell Launchers

If you've Exported your Job as an Autonomous Job, then you will have been given the option of creating Shell Launchers in both Unix and Windows format. For Unix, a Shell Script (.sh) is created that is compatible with Unix, Linux and Apple OSX. The Windows format launcher is a Windows Batch File (.bat).

The two files perform the same task; however there are some syntactical differences. For the purpose of this tutorial, we'll analyse the Unix flavour of this file. You should not have too much difficulty translating this document, for the Windows version, if this is your platform of choice.

The Simplest of Launchers

In our tutorial Talend Job Deployment, we analysed the output from exporting a trivial Job, named SimpleJob. This Job did no more than print a message to standard output.

Let's have a look at the Unix Shell Launcher, SimpleJob_run.sh, that was created.

-rw-r--r--  1 talend  talend    258  5 May 11:13 SimpleJob_run.sh

New to Unix? The first thing to note when looking at this file is that, on Unix (and all variants), a shell script needs to be executable unless it is passed as an argument to a Shell Interpreter. This is easily achieved by running the command chmod u+x SimpleJob_run.sh. You should then see the file attributes change to -rwxr--r--. Provided that the directory that contains this file is in your PATH (This is unikely for a exported Talend Job), you can simply enter the script name to execute it. If it is not in your PATH, then you must enter a fully qualified path, for example, ./SimpleJob_run.sh. Alternatively, you may pass a qualified script name to a Shell Interpreter, for example, bash SimpleJob_run.sh.

What Does a Talend Shell Launcher Script Do?

The simple answer is that the script will execute your Talend Job. Let's take a look at the file content, to see what is going on.

cd `dirname $0`
ROOT_PATH=`pwd`
java -Xms256M -Xmx1024M -cp $ROOT_PATH/../lib/dom4j-1.6.1.jar:$ROOT_PATH:$ROOT_PATH/../lib/systemRoutines.jar::$ROOT_PATH/../lib/userRoutines.jar::.:$ROOT_PATH/simplejob_0_1.jar: talendbyexample.simplejob_0_1.SimpleJob "$@"

Without straying in to a Unix Shell Scripting lesson, the first two lines of our three-line script are fairly straight forward. Regardless of how and from where we called our script, the script will change directory to the directory that contains the script and then set ROOT_PATH to the fully qualified directory name. ROOT_PATH will be used for passing a Java Classpath to Java when the Job is executed.

Calling Java

The next line of our script is where it all happens. Java is called and a number of arguments are passed. The first assumption that we can make here is that a Java Runtime is installed on the computer that you are deploying your Job to, and that the Java executable, java.exe, is in your PATH.

Java Memory Management

The first two parameters that are passed -Xms256M -Xmx1024M are for Java Memory Management. These two parameters, in turn, specify the initial amount of Heap Memory that is allocated to the process, and the maximum. In our example, 256MB will be initially allocated and then memory will be allocated in further 256MB chunks, as needed, until 1024MB has been reached.

For further information on these parameters, read Java -X Command-line Options.

These default values were created by Talend when you exported your Job. As well as being able to amend them within this script, you can also amend the values within the Talend Designer, as you tune your Job. These paramters are available under Advanced Settings of the Run (Job...) tab, when you open your Job.

Image 1

Java Classpath

The next parameter -cp is how the Java Classpath is passed. This is how the Java Runtime is told where to find all of the code that is required to run your Job. Many a seasoned Java developer falls foul of getting a Java Classpath correctly set, so hopefully, Talend will usually get this right!

In our example, we're passing a : delimited list of Jar files: $ROOT_PATH/../lib/dom4j-1.6.1.jar:$ROOT_PATH:$ROOT_PATH/../lib/systemRoutines.jar::$ROOT_PATH/../lib/userRoutines.jar::.:$ROOT_PATH/simplejob_0_1.jar: each of which is qualified with ROOT_PATH.

Class to be Executed

The next argument to be passed to the Java Runtime is the Class that is to be executed, in our case, talendbyexample.simplejob_0_1.SimpleJob - our Job.

Additional Parameters

The final aregument being passed is "$@" (or in the case of the Windows compatible script %*). This allows us to pass additional command-line arguments to our script which are, in-turn, passed to Java and our Job. There is no specific definition of what these parameters are, they are simply parameters that you may want to pass; but are not specified within the Shell Launcher Script.

Moving Beyond our Simple Job

Now that we've looked at a very simple Job, you should have a good understanding of the export process, the files created and how you control and run your Job.

Moving beyond this, you are likely to be using Context Variables in your Job, you may require additional Java Archive Files, and you may want to change your export options such as exporting your Source Code.

Passing Context

You're probably going to want to use Context within your Job. You may have specified Context Variables that are specific to your Job and you may have added Context Groups. You may also have more than one Context, for example, Default, Test and Production. Every Job will have at least one Context, named Default.

If you selected the Context scripts option when you exported your Job, then Talend will export scripts for each of the Context that you have defined in your Job (even if you have defined no Context Variables). By default, a Job has a single Context named Default. You may have created your own Context, for example, Production and you may even have changed the name of your default Context. Talend will also amend the Shell launch scripts so that, by default, they read the Context that you specified in the accompanying drop-down list.

It is important to note that if you have Context Variables in your Job, you do not need to export the scripts and, likewise, if you have no Context Variables, you may still export the scripts. The key point to remember here is that if you use Context Variables but do not export the scripts, no default values will be assigned to your Context Variables when you launch your Job.

For more information on Context read our Context Tutorial and for more on Deployment, read our Talend Job Deployment Tutorial.

Let's have a look at how Talend modified our Launch Script when we chose to export Context Scripts.

java -Xms256M -Xmx1024M -cp $ROOT_PATH/../lib/dom4j-1.6.1.jar:$ROOT_PATH:$ROOT_PATH/../lib/systemRoutines.jar::$ROOT_PATH/../lib/userRoutines.jar::.:$ROOT_PATH/simplejob_0_1.jar: talendbyexample.simplejob_0_1.SimpleJob --context=Default "$@"

You will see that an extra parameter --context=Default has been added. Note that the -- flag indicates that this is a parameter that is being passed to your Job, rather than a directive to Java.

This parameter is instructing your Job to run with the Default Context. You may use other Context in your Job, for example, Production. If you want to run the Job with your Production Context, then you simply amend the parameter, to read --context=Production. You'll also remember that we were given the option to choose the default Context when we exported our Job.

Where is My Context?

for each of the Context that you have defined in your Job, Talend exports a Context Script. In our simple example, we have a single Context named Default. Below, you can see the file that was exported.

.../SimpleJob/talendbyexample/simplejob_0_1/contexts/Default.properties
   #
   #Thu May 09 18:11:25 BST 2013
   myString=Hello World\!

The path to context scripts is fairly straightforward. Your Job name, Project name (always in lowercase), Job name and version (always in lowercase), "context" and then the individual Context Scripts, for example, Default.properties.

Context Best Practice

I always recommend that you store your Context in external files. If you do, in a large part, it renders this Context Passing redundant; however, there are some cases where I still use this practice.

If you are loading your Context from an ex For more information on this, read our Load Context Tutorial.

Next

Now that you've successfully exported your Job, you may want to schedule it to run. Read our article Scheduling Talend Jobs using cron for more information on this subject.




Expert Consultancy from Yellow Pelican
comments powered by Disqus

© www.TalendByExample.com