Developer.TACC.cloud

Developer documentation for Agave, Abaco, and other TACC APIs

Job monitoring

Once you submit your job request, the job will be handed off to Agave’s back end execution service. Your job may run right away, or it may wait in a batch queue on the execution system until the required resources are available. Either way, the execution process occurs completely asynchronous to the submission process. To monitor the status of your job, Agave supports two different mechanisms: polling and webhooks.

:information_source: For the sake of brevity, we placed a detailed explanation of the job lifecycle in a separate, aptly title post, The Job Lifecycle. There you will find detailed information about how, when, and why everything moves from place to place and how you can peek behind the curtains.

Polling

If you have ever taken a long road trip with children, you are probably painfully aware of how polling works. Starting several minutes from the time you leave the house, a child asks, “Are we there yet? You reply, “No.” Several minutes later the child again asks, “Are we there yet?” You again reply, “No.” This process continues until you finally arrive at your destination. This is called polling and polling is bad

Polling for your job status works the same way. After submitting your job, you start a while loop that queries the Jobs service for your job status until it detects that the job is in a terminal state. The following two URLs both return the status of your job. The first will result in a list of abbreviated job descriptions, the second will result in a full description of the job with the given $JOB_ID, exactly like that returned when submitting the job. The third will result in a much smaller response object that contains only the $JOB_ID and status being returned.

jobs-list -v
jobs-list -v $JOB_ID
jobs-list $JOB_ID

Show cURL   

curl -sk -H "Authorization: Bearer  $ACCESS_TOKEN" https://agave.iplantc.org/jobs/v2/?pretty=true
curl -sk -H "Authorization: Bearer  $ACCESS_TOKEN" https://agave.iplantc.org/jobs/v2/$JOB_ID
curl -sk -H "Authorization: Bearer  $ACCESS_TOKEN" https://agave.iplantc.org/jobs/v2/$JOB_ID/status

Show response   

{
 "id" : "$JOB_ID",
 "name" : "$USERNAME-$APP_ID",
 "owner" : "$USERNAME",
 "appId" : "$APP_ID",
 "executionSystem" : "$PUBLIC_EXECUTION_SYSTEM",
 "batchQueue": "normal",
 "nodeCount": 1,
 "processorsPerNode": 16,
 "memoryPerNode": 32,
 "maxRunTime": "01:00:00",
 "archive": false,
 "retries": 0,
 "localId": "659413",
 "created": "2018-01-26T15:08:02.000-06:00",
 "lastUpdated": "2018-01-26T15:09:55.000-06:00",
 "outputPath": "$USERNAME/$JOB_ID-$APP_ID",
 "status": "FINISHED",
 "submitTime": "2018-01-26T15:09:45.000-06:00",
 "startTime": "2018-01-26T15:09:53.000-06:00",
 "endTime": "2018-01-26T15:09:55.000-06:00",
 "inputs": {
   "inputBam": [
     "agave://data.iplantcollaborative.org/shared/iplantcollaborative/example_data/Samtools_mpileup/ex1.bam"
   ]
 },
 "parameters": {
   "nameSort": true,
   "maxMemSort": 800000000
 },
 "_links": {
   "self": {
     "href": "https://agave.iplantc.org/jobs/v2/$JOB_ID"
   },
   "app": {
     "href": "https://agave.iplantc.org/apps/v2/$APP_ID"
   },
   "executionSystem": {
     "href": "https://agave.iplantc.org/systems/v2/$PUBLIC_EXECUTION_SYSTEM"
   },
   "archiveSystem": {
     "href": "https://agave.iplantc.org/systems/v2/$PUBLIC_EXECUTION_SYSTEM""
   },
   "archiveData": {
     "href": "https://agave.iplantc.org/jobs/v2/$JOB_ID/outputs/listings"
   },
   "owner": {
     "href": "https://agave.iplantc.org/profiles/v2/$USERNAME"
   },
   "permissions": {
     "href": "https://agave.iplantc.org/jobs/v2/$JOB_ID/pems"
   },
   "history": {
     "href": "https://agave.iplantc.org/jobs/v2/$JOB_ID/history"
   },
   "metadata": {
     "href": "https://agave.iplantc.org/meta/v2/data/?q=%7B%22associationIds%22%3A%22462259152402771480-242ac113-0001-007%22%7D"
   },
   "notifications": {
     "href": "https://agave.iplantc.org/notifications/v2/?associatedUuid=$JOB_ID"
   }
 }
}

The list of all possible job statuses is given in table 2.

Event Description
CREATED The job was updated
UPDATED The job was updated
DELETED The job was deleted
PERMISSION_GRANT User permission was granted
PERMISSION_REVOKE Permission was removed for a user on this job
PENDING Job accepted and queued for submission.
STAGING_INPUTS Transferring job input data to execution system
CLEANING_UP Job completed execution
ARCHIVING Transferring job output to archive system
STAGING_JOB Job inputs staged to execution system
FINISHED Job complete
KILLED Job execution killed at user request
FAILED Job failed
STOPPED Job execution intentionally stopped
RUNNING Job started running
PAUSED Job execution paused by user
QUEUED Job successfully placed into queue
SUBMITTING Preparing job for execution and staging binaries to execution system
STAGED Job inputs staged to execution system
PROCESSING_INPUTS Identifying input files for staging
ARCHIVING_FINISHED Job archiving complete
ARCHIVING_FAILED Job archiving failed
HEARTBEAT Job heartbeat received

Table 2. Job statuses listed in progressive order from job submission to completion.

Polling is an incredibly effective approach, but it is bad practice for two reasons. First, it does not scale well. Querying for one job status every few seconds does not take much effort, but querying for 100 takes quite a bit of time and puts unnecessary load on Agave’s servers. Second, polling provides what is effectively a binary response. It tells you whether a job is done or not done, it does not give you any information on what is actually going on with the job or where it is in the overall execution process.

The job history URL provides much more detailed information on the various state changes, system messages, and progress information associated with data staging. The syntax of the job history URL is as follows:

jobs-history -v $JOB_ID

Show cURL   

curl -sk -H "Authorization: Bearer  $ACCESS_TOKEN" https://agave.iplantc.org/jobs/v2/$JOB_ID/history?pretty=true

Show response   

{
 "status":"success",
 "message":null,
 "version":"2.1.0-r6d11c",
 "result":[
   {
     "created":"2014-10-24T04:47:45.000-05:00",
     "status":"PENDING",
     "description":"Job accepted and queued for submission."
   },
   {
     "created":"2014-10-24T04:47:47.000-05:00",
     "status":"PROCESSING_INPUTS",
     "description":"Attempt 1 to stage job inputs"
   },
   {
     "created":"2014-10-24T04:47:47.000-05:00",
     "status":"PROCESSING_INPUTS",
     "description":"Identifying input files for staging"
   },
   {
     "created":"2014-10-24T04:47:48.000-05:00",
     "status":"STAGING_INPUTS",
     "description":"Staging agave://$PUBLIC_STORAGE_SYSTEM/$API_USERNAME/inputs/pyplot/testdata.csv to remote job directory"
   },
   {
     "progress":{
       "averageRate":0,
       "totalFiles":1,
       "source":"agave://$PUBLIC_STORAGE_SYSTEM/$API_USERNAME/inputs/pyplot/testdata.csv",
       "totalActiveTransfers":0,
       "totalBytes":3212,
       "totalBytesTransferred":3212
     },
     "created":"2014-10-24T04:47:48.000-05:00",
     "status":"STAGING_INPUTS",
     "description":"Copy in progress"
   },
   {
     "created":"2014-10-24T04:47:50.000-05:00",
     "status":"STAGED",
     "description":"Job inputs staged to execution system"
   },
   {
     "created":"2014-10-24T04:47:55.000-05:00",
     "status":"SUBMITTING",
     "description":"Preparing job for submission."
   },
   {
     "created":"2014-10-24T04:47:55.000-05:00",
     "status":"SUBMITTING",
     "description":"Attempt 1 to submit job"
   },
   {
     "created":"2014-10-24T04:48:08.000-05:00",
     "status":"RUNNING",
     "description":"Job started running"
   },
   {
     "created":"2014-10-24T04:48:12.000-05:00",
     "status":"CLEANING_UP"
   },
   {
     "created":"2014-10-24T04:48:15.000-05:00",
     "status":"FINISHED",
     "description":"Job completed. Skipping archiving at user request."
   }
 ]
}

Depending on the nature of your job and the reliability of the underlying systems, the response from this service can grow rather large, so it is important to be aware that this query can be an expensive call for your client application to make. Everything we said before about polling job status applies to polling job history with the additional caveat that you can chew through quite a bit of bandwidth polling this service, so keep that in mind if your application is bandwidth starved.

Often times, however, polling is unavoidable. In these situations, we recommend using an exponential backoff to check job status. An exponential backoff is an alogrithm that increases the time between retries as the number of failures increases.

Webhooks

Webhooks are the alternative, preferred way for your application to monitor the status of asynchronous actions in Agave. If you are a Gang of Four disciple, webhooks are a mechanism for implementing the Observer Pattern. They are widely used across the web and chances are that something you’re using right now is leveraging them. In the context of Agave, a webhook is a URL that you give to Agave in advance of an event which it later POSTs a response to when that event occurs. A webhook can be any web accessible URL.

:information_source: For more information about webhooks, events, and notifications in Agave, please see the Notifications and Events Guides.

The Jobs service provides several template variables for constructing dynamic URLs. Template variables can be included anywhere in your URL by surrounding the variable name in the following manner ${VARIABLE_NAME}. When an event of interest occurs, the variables will be resolved and the resulting URL called. Several example urls are given below.

http://example.com/?job_id=${JOB_ID}&job_status=${EVENT}

http://example.com/trigger/job/${JOB_NAME}/${EVENT}

http://example.com/webhooks/?nonce=sdfkajerouiwe234289fahlkqr&id=${JOB_ID}&status=${EVENT}&start=${JOB_START_TIME}&end=${JOB_END_TIME}&url=${JOB_ARCHIVE_URL}

The full list of template variables are listed in the following table.

Variable Description
UUID The UUID of the job
EVENT The event which occurred
JOB_STATUS The status of the job at the time the event occurs
JOB_URL The url of the job within the API
JOB_ID The unique id used to reference the job within Agave.
JOB_SYSTEM ID of the job execution system (ex. ssh.execute.example.com)
JOB_NAME The user-supplied name of the job
JOB_START_TIME The time when the job started running in ISO8601 format.
JOB_END_TIME The time when the job stopped running in ISO8601 format.
JOB_SUBMIT_TIME The time when the job was submitted to Agave for execution by the user in ISO8601 format.
JOB_ARCHIVE_PATH The path on the archive system where the job output will be staged.
JOB_ARCHIVE_URL The Agave URL for the archived data.
JOB_ERROR The error message explaining why a job failed. Null if completed successfully.

Table 3. Template variables available for use when defining webhooks for your job.

Email

In situations where you do not have a persistent web address, or access to a backend service, you may find it more convenient to subscribe for email notifications rather then providing a webhook. Agave supports email notifications as well. Simply specify a valid email address in the url field in your job submission notification object and an email will be sent to that address when a relevant event occurs. A sample email message is given below.

The status of job 0001414144065563-5056a550b8-0001-007, "demo-pyplot-demo-advanced test-1414139896," has changed to FINISHED.

Name: demo-pyplot-demo-advanced test-1414139896
URL: https://public.tenants.agaveapi.co/jobs/v2/0001414144065563-5056a550b8-0001-007
Message: Job completed successfully.
Submit Time: 2014-10-24T04:48:11.000-05:00
Start Time: 2014-10-24T04:48:08.000-05:0
End Time: 2014-10-24T04:48:15.000-05:00
Output Path: $API_USERNAME/archive/jobs/job-0001414144065563-5056a550b8-0001-007
Output URL: https://public.tenants.agaveapi.co/jobs/v2/0001414144065563-5056a550b8-0001-007/outputs