• • Running TupleFlow

Running TupleFlow

To run TupleFlow on the above parameter file (or on one that you have created), do one of the following:

Local (non-distributed):

mkdir /path/to/my/tmp/dir
java -Xmx900m galago.tupleflow.execution.JobExecutor \
      local paramFile.xml  /path/to/my/tmp/dir

DRMAA (distributed):

mkdir /path/to/my/tmp/dir
java -Xmx900m galago.tupleflow.execution.JobExecutor \
      drmaa paramFile.xml  /path/to/my/tmp/dir

If you are running in local mode, all of the logging messages and any errors will be displayed to the stderr on your screen. However, if you use the distributed mode, these messages are all kept in files within the temporary directory (in the example above, these files are located in: /path/to/my/tmp/dir/stderr/.

Galago keeps track of which stages fail and succeed. A quick way to check is to do:

ls /path/to/my/tmp/dir/jobs/*/

This will list the jobs that each stage was broken into (for each stage, these are numbered from 0 to the max number of jobs a stage could be split into, based on the hashCount specified at the top of the parameter file). Stages that completed successfully will have an addition file with the job number followed by .complete. If the job failed, it will be followed by .error.

If you are able to fix the bug that caused the error, and the changes you made only affect the data flow from where the errors occurred onwards, then if you pass the same temporary directory to TupleFlow, it won't redo the stages that completed successfully, but rather start at the failed stages.

However, if you do change something in your code that affects the flow of data from stages that have already finished successfully, or if you want a clean start, delete the contents of the temporary directory before running TupleFlow.

Galago

• edu.umass (Includes a distributed retrieval framework and custom operators built on top of Galago)