====== oarextratime ======

Coming with OAR 2.5.8, oarwalltime is a new OAR command.

This page talk about oarextratime, which was the former name for oarwalltime. 

One could just exchange the two words.

===== Usage =====
<code>
$ oarextratime 
Usage:
  /usr/local/lib/oar/oarextratime [<options>] <job_id> [<extra_time>]

Manage extra time requests for a job.
- If extra time is given, the command requests the addition of some extra time
  to the job or update a previous request.
- If no extra time is given, the command shows the current extra time status for
  the job.
Job must be running.
Extra time is to be given as an number of seconds.

Options:
  -D, --delay-next-jobs  allow extra time to delay the next batch jobs
  -V, --version          print OAR version
  -h, --help             print help

</code>

=== Manual page ===
<code>
NAME
       oarextratime - request or query extra time for a job handled by the OAR batch scheduler.

SYNOPSIS
       oarextratime [options] <job_id> [extra_time]

DESCRIPTION
       This command allows users to request or query extra time for a job.

       If extra_time is given, the command requests the addition of some extra time to the job or update a previous request.

       If extra_time is not given, the command shows the current extra time status for the job.

       Job must be running.  Extra time is to be given as an number of seconds.

       Once a request is registered, it will by handled during the next pass of scheduling and granted if it fits with other jobs.

OPTIONS
       -D, --delay-next-jobs
           Allow extra time to delay next batch jobs (but not advance reservations), if needed in order to fit in the resources allocation.

       -h, --help
           Print help command message.

       -V, --version
           Print OAR version.

NOTES
       The extra time functionality allows one to increase the walltime of a job, as an emergency measure when some more time is critically needed to complete a work. There
       is no warranty that extra time can be granted, since it depends on the resources availability (next jobs).

       Please mind the walltime of the job and do not rely too much on extra time.

       Extra time events for a job are listed in the oarstat output.

       As per configuration, the extra time functionality may be disabled in your installation.

       As per configuration, an extra time request may only be possible some time before the predicted end of the job.

       Configuration can be set per queue.

       Read your local documentation or ask your administrator(s) to know the configured settings.

       The extra time functionality does not support the computation of the resources shared allocation in the case of jobs of type timesharing or container/inner or
       placeholder/allowed.

SEE ALSO
       oarstat(1), oarsub(1), oardel(1)

</code>

===== Configuration =====
Related oar.conf snippet.
<code>
###########################################################################
# OAR EXTRA TIME                                                          #
# Allow users to request some extra time to finish a job. Extra time is   #
# only granted if it does not conflict with the next jobs.                #
# WARNING:                                                                #
# - This functionnality does not handle the timesharing, container/inner  #
# and placeholder/Allowed types of jobs.                                  #
# - This functionnality is not compatible with the RUNTIME suspend/resume #
# function (see man oarhold and the USERS_ALLOWED_HOLD_RESUME line above).#
###############################################################################
#
# Extra time duration limit
# -1 means that the extra time functionality is disabled (global setting, cannot be used per queue)
#  0 means that the extra time functionality is allowed for oar and root only (with no limit)
# >0 sets the maximum duration of possible extra time users can request in seconds
#  default is 0
#EXTRA_TIME_DURATION=0

# Set extra time request delay: do not accept requests from users until delay seconds before the predicted end of the job
# Default is 0, i.e. no delay
#EXTRA_TIME_REQUEST_DELAY=0

# Set the minimum walltime of jobs for which extra time can be requested
# Default is 0, i.e. minimum walltime
#EXTRA_TIME_MINIMUM_WALLTIME=0

# List users (* for all) for which extra time can delay the next batch jobs (i.e. only advance reserations cannot be moved)
# Default is "", i.e. only root and the oar user are allowed
#EXTRA_TIME_DELAY_NEXT_JOBS_ALLOWED_USERS=""

# Those 4 settings can also be provided per queue, using the Perl hash syntax with the queue names as the keys.
# Not defined queues get the value associated to the _ key, or if not defined either, the default value.
# e.g.:
# - Allow extra time requests for up to 2 hours for the default queue only (0 for others):
#EXTRA_TIME_DURATION="{default => 7200}"
# - Allow any user to delay next jobs with requesting extra time for a job in all but the besteffort queue:
#EXTRA_TIME_DELAY_NEXT_JOBS_ALLOWED_USERS="{_ => '*', besteffort => ''}"

</code>

====== REST API ======
For ref, the [[rest-api-specs]] page.

Extratime now uses the /oarapi-priv/jobs/<jobid> URL with GET or (POST or PUT <- both do the same...)

  *Extratime syntax in a job query output: GET /oarapi-priv/jobs/<jobid>.json
    * ''possible'': possible requests in seconds, 0 if not request is possible (for any cause)
    * ''pending'': pending requests in seconds to be handled by the scheduler
    * ''granted'': granted extratime in seconds
    * ''granted_with_delaying_next_jobs'': part of ''granted'' (in seconds) that was granted with ''delay_next_jobs'' = ''YES'', does not show up if equal to 0.
    * ''delay_next_jobs'': ''FORBIDDEN'' if not allowed, ''YES'' if allowed and requested and ''pending'' > 0, ''NO'' if not requested and ''pending'' > 0, does not show up si ''pending'' = 0

  * Extratime request syntax in a job update: POST /oarapi-priv/jobs/<jobid>.json -d ''{ ... }''
    * ''duration'': requested duration of extratime in seconds, mandatory
    * ''delay_next_jobs'': ''YES'' or ''NO'' (default if not set)

==== Querying a job that does not exist ===
<code>
$ curl -i -X GET http://docker:docker@localhost:46668/oarapi-priv/jobs/1.json -H'Content-Type: application/json'
HTTP/1.1 404 Not Found
Date: Mon, 07 Nov 2016 17:29:24 GMT
Server: Apache/2.4.10 (Debian)
Transfer-Encoding: chunked
Content-Type: application/json; charset=ISO-8859-1

{
   "message" : "Job not found",
   "code" : 404,
   "title" : "Job not found"
}
</code>
  * request fails
<code>
$ curl -i -X POST http://docker:docker@localhost:46668/oarapi-priv/jobs/1.json -H'Content-Type: application/json' -d '{"method":"extratime", "duration":"300", "delay_next_jobs": "YES" }'
HTTP/1.1 403 Forbidden
Date: Mon, 07 Nov 2016 17:39:59 GMT
Server: Apache/2.4.10 (Debian)
Transfer-Encoding: chunked
Content-Type: application/json; charset=ISO-8859-1

{
   "code" : 403,
   "message" : "Job 1 is not running",
   "title" : "Forbidden"
}
</code>

==== Querying a job that exists ====
  * with default configuration (extratime not activated for regular users)
<code>
$ curl -i -X GET http://docker:docker@localhost:46668/oarapi-priv/jobs/1.json -H'Content-Type: application/json'
HTTP/1.1 200 OK
Date: Mon, 07 Nov 2016 17:29:35 GMT
Server: Apache/2.4.10 (Debian)
Allow: GET, POST, PUT, DELETE
Transfer-Encoding: chunked
Content-Type: application/json; charset=ISO-8859-1

{
   "project" : "default",
   "submission_time" : 1478539772,
   "dependencies" : [],
   "reservation" : "None",
   "command" : "",
   "properties" : "(desktop_computing = 'NO') AND drain='NO'",
   "array_id" : 1,
   "state" : "Running",
   "events" : [],
   "links" : [
      {
         "href" : "/oarapi-priv/jobs/1",
         "rel" : "self"
      },
      {
         "title" : "resources",
         "rel" : "collection",
         "href" : "/oarapi-priv/jobs/1/resources"
      },
      {
         "href" : "/oarapi-priv/jobs/1/nodes",
         "title" : "nodes",
         "rel" : "collection"
      }
   ],
   "id" : 1,
   "exit_code" : null,
   "name" : null,
   "walltime" : 7200,
   "types" : [],
   "stdout_file" : "OAR.1.stdout",
   "cpuset_name" : "docker_1",
   "owner" : "docker",
   "stop_time" : 0,
   "queue" : "default",
   "type" : "INTERACTIVE",
   "api_timestamp" : 1478539775,
   "launching_directory" : "/home/docker",
   "message" : "R=1,W=2:0:0,J=I (Karma=0.000,quota_ok)",
   "scheduled_start" : 1478539773,
   "extratime" : {
      "possible" : 0,
      "granted" : 0,
      "pending" : 0,
      "delay_next_jobs" : "FORBIDDEN"
   },
   "initial_request" : "oarsub -I",
   "start_time" : 1478539773,
   "wanted_resources" : "-l \"{type = 'default'}/resource_id=1,walltime=2:0:0\" ",
   "stderr_file" : "OAR.1.stderr",
   "array_index" : 1,
   "resubmit_job_id" : 0
}
</code>
  * with configuration changed: EXTRA_TIME_DURATION=3600
<code>
   "extratime" : {
      "granted" : 0,
      "delay_next_jobs" : "FORBIDDEN",
      "possible" : "3600",
      "pending" : 0
   },
</code>

  * with request not allowed because of EXTRA_TIME_REQUEST_DELAY or EXTRA_TIME_MINIMUM_WALLTIME, or because the job is not running yet
<code>
   "extratime" : {
      "possible" : 0,
      "delay_next_jobs" : "FORBIDDEN",
      "granted" : 0,
      "pending" : 0
   },
</code>
  * with EXTRA_TIME_DELAY_NEXT_JOBS_ALLOWED_USERS="*"
<code>
   "extratime" : {
      "possible" : 3600,
      "granted" : 0,
      "pending" : 0
   },
</code>
  * with a pending request not yet (fully) granted (a job is blocking)
<code>
   "extratime" : {
      "possible" : 3600,
      "granted" : 100,
      "pending" : 200,
      "delay_next_jobs" : "NO",
  },
</code>
  * once (fully) granted, if ''pending = 0'' (no more blocking job), ''"delay_next_jobs" : "NO"'' is not displayed
<code>
   "extratime" : {
      "possible" : 3600,
      "granted" : 300,
      "pending" : 0,
  },
</code>

==== Doing a valid request ====
With ''delay_next_job'' not allowed in this example
<code>
$ curl -i -X POST http://docker:docker@localhost:46668/oarapi-priv/jobs/2.json -H'Content-Type: application/json' -d '{"method":"extratime", "duration":"300" }'
HTTP/1.1 202 Accepted
Date: Mon, 07 Nov 2016 17:41:12 GMT
Server: Apache/2.4.10 (Debian)
Transfer-Encoding: chunked
Content-Type: application/json; charset=ISO-8859-1

{
   "id" : "2",
   "cmd_output" : "Extra time request registered for job 2. It will be handled shortly",
   "api_timestamp" : 1478540472,
   "status" : "Accepted",
   "links" : [
      {
         "href" : "/oarapi-priv/jobs/2",
         "rel" : "self"
      }
   ]
}
</code>

<code>
$ curl -i -X GET http://docker:docker@localhost:46668/oarapi-priv/jobs/2.json -H'Content-Type: application/json'HTTP/1.1 200 OK
Date: Mon, 07 Nov 2016 17:46:00 GMT
Server: Apache/2.4.10 (Debian)
Allow: GET, POST, PUT, DELETE
Transfer-Encoding: chunked
Content-Type: application/json; charset=ISO-8859-1

{
   "wanted_resources" : "-l \"{type = 'default'}/network_address=ALL,walltime=2:5:0\" ",
   "message" : "R=12,W=2:0:0,J=I (Karma=0.000,quota_ok)",
   "array_id" : 2,
   "api_timestamp" : 1478540760,
   "links" : [
      {
         "rel" : "self",
         "href" : "/oarapi-priv/jobs/2"
      },
      {
         "title" : "resources",
         "href" : "/oarapi-priv/jobs/2/resources",
         "rel" : "collection"
      },
      {
         "href" : "/oarapi-priv/jobs/2/nodes",
         "rel" : "collection",
         "title" : "nodes"
      }
   ],
   "queue" : "default",
   "stop_time" : 0,
   "properties" : "(desktop_computing = 'NO') AND drain='NO'",
   "resubmit_job_id" : 0,
   "project" : "default",
   "scheduled_start" : 1478540292,
   "types" : [],
   "events" : [
      {
         "date" : 1478540473,
         "type" : "EXTRATIME",
         "job_id" : 2,
         "to_check" : "NO",
         "event_id" : 2,
         "description" : " 300 s of extra time added to the job (new walltime = 7500 s)"
      }
   ],
   "name" : null,
   "dependencies" : [],
   "submission_time" : 1478540240,
   "exit_code" : null,
   "type" : "INTERACTIVE",
   "command" : "",
   "initial_request" : "oarsub -l nodes=ALL -I",
   "owner" : "docker",
   "state" : "Running",
   "extratime" : {
      "delay_next_jobs" : "FORBIDDEN",
      "possible" : "3600",
      "granted" : 300,
      "pending" : 0
   },
   "stderr_file" : "OAR.2.stderr",
   "id" : 2,
   "walltime" : 7500,
   "start_time" : 1478540292,
   "cpuset_name" : "docker_2",
   "array_index" : 1,
   "launching_directory" : "/home/docker",
   "reservation" : "None",
   "stdout_file" : "OAR.2.stdout"
}
</code>

Now try to delay next jobs
<code>
$ curl -i -X POST http://docker:docker@localhost:46668/oarapi-priv/jobs/3.json -H'Content-Type: application/json' -d '{"method":"extratime", "duration":"300", "delay_next_jobs":"YES" }'
HTTP/1.1 403 Forbidden
Date: Mon, 07 Nov 2016 18:33:08 GMT
Server: Apache/2.4.10 (Debian)
Transfer-Encoding: chunked
Content-Type: application/json; charset=ISO-8859-1

{
   "code" : 403,
   "message" : "Delaying next jobs is not allowed",
   "title" : "Forbidden"
}
</code>
==== request with delaying next jobs ====
  * submit request
<code>
$ curl -i -X POST http://docker:docker@localhost:46668/oarapi-priv/jobs/2.json -H'Content-Type: application/json' -d '{"method":"extratime", "duration":"300", "delay_next_jobs":"YES" }'
HTTP/1.1 202 Accepted
Date: Mon, 07 Nov 2016 17:58:22 GMT
Server: Apache/2.4.10 (Debian)
Transfer-Encoding: chunked
Content-Type: application/json; charset=ISO-8859-1

{
   "cmd_output" : "Extra time request updated for job 2. It will be handled shortly",
   "links" : [
      {
         "href" : "/oarapi-priv/jobs/2",
         "rel" : "self"
      }
   ],
   "api_timestamp" : 1478541502,
   "status" : "Accepted",
   "id" : "2"
}
</code>

  * request still pending
<code>
$ curl -i -X GET http://docker:docker@localhost:46668/oarapi-priv/jobs/2.json -H'Content-Type: application/json'
HTTP/1.1 200 OK
Date: Mon, 07 Nov 2016 18:00:39 GMT
Server: Apache/2.4.10 (Debian)
Allow: GET, POST, PUT, DELETE
Transfer-Encoding: chunked
Content-Type: application/json; charset=ISO-8859-1

{
   "links" : [
      {
         "href" : "/oarapi-priv/jobs/2",
         "rel" : "self"
      },
      {
         "href" : "/oarapi-priv/jobs/2/resources",
         "title" : "resources",
         "rel" : "collection"
      },
      {
         "href" : "/oarapi-priv/jobs/2/nodes",
         "title" : "nodes",
         "rel" : "collection"
      }
   ],
   "stdout_file" : "OAR.2.stdout",
   "wanted_resources" : "-l \"{type = 'default'}/network_address=ALL,walltime=2:10:0\" ",
   "api_timestamp" : 1478541640,
   "walltime" : 7800,
   "resubmit_job_id" : 0,
   "name" : null,
   "array_index" : 1,
   "state" : "Running",
   "queue" : "default",
   "start_time" : 1478540292,
   "scheduled_start" : 1478540292,
   "types" : [],
   "owner" : "docker",
   "initial_request" : "oarsub -l nodes=ALL -I",
   "exit_code" : null,
   "submission_time" : 1478540240,
   "project" : "default",
   "extratime" : {
      "granted" : 600,
      "pending" : 300,
      "possible" : "3600",
      "delay_next_jobs" : "YES",
      "granted_with_delaying_next_jobs" : 300
   },
   "message" : "R=12,W=2:0:0,J=I (Karma=0.000,quota_ok)",
   "id" : 2,
   "events" : [
      {
         "to_check" : "NO",
         "job_id" : 2,
         "description" : " 300 s of extra time added to the job (new walltime = 7500 s)",
         "date" : 1478540473,
         "event_id" : 2,
         "type" : "EXTRATIME"
      },
      {
         "event_id" : 3,
         "type" : "EXTRATIME",
         "date" : 1478541043,
         "job_id" : 2,
         "description" : " 300 s of extra time added to the job (new walltime = 7800 s)",
         "to_check" : "NO"
      }
   ],
   "reservation" : "None",
   "array_id" : 2,
   "type" : "INTERACTIVE",
   "stop_time" : 0,
   "properties" : "(desktop_computing = 'NO') AND drain='NO'",
   "command" : "",
   "launching_directory" : "/home/docker",
   "stderr_file" : "OAR.2.stderr",
   "dependencies" : [],
   "cpuset_name" : "docker_2"
}
</code>
  * request is accepted
<code>
$ curl -i -X GET http://docker:docker@localhost:46668/oarapi-priv/jobs/2.json -H'Content-Type: application/json'
HTTP/1.1 200 OK
Date: Mon, 07 Nov 2016 18:01:03 GMT
Server: Apache/2.4.10 (Debian)
Allow: GET, POST, PUT, DELETE
Transfer-Encoding: chunked
Content-Type: application/json; charset=ISO-8859-1

{
   "properties" : "(desktop_computing = 'NO') AND drain='NO'",
   "command" : "",
   "reservation" : "None",
   "dependencies" : [],
   "project" : "default",
   "submission_time" : 1478540240,
   "walltime" : 8100,
   "id" : 2,
   "name" : null,
   "exit_code" : null,
   "array_id" : 2,
   "events" : [
      {
         "description" : " 300 s of extra time added to the job (new walltime = 7500 s)",
         "date" : 1478540473,
         "type" : "EXTRATIME",
         "event_id" : 2,
         "to_check" : "NO",
         "job_id" : 2
      },
      {
         "event_id" : 3,
         "to_check" : "NO",
         "job_id" : 2,
         "description" : " 300 s of extra time added to the job (new walltime = 7800 s)",
         "type" : "EXTRATIME",
         "date" : 1478541043
      },
      {
         "date" : 1478541647,
         "type" : "EXTRATIME",
         "description" : " 300s of extra time added to the job (new walltime = 8100s) (with delaying next jobs)",
         "job_id" : 2,
         "event_id" : 4,
         "to_check" : "NO"
      }
   ],
   "state" : "Running",
   "links" : [
      {
         "href" : "/oarapi-priv/jobs/2",
         "rel" : "self"
      },
      {
         "href" : "/oarapi-priv/jobs/2/resources",
         "title" : "resources",
         "rel" : "collection"
      },
      {
         "title" : "nodes",
         "rel" : "collection",
         "href" : "/oarapi-priv/jobs/2/nodes"
      }
   ],
   "api_timestamp" : 1478541663,
   "type" : "INTERACTIVE",
   "queue" : "default",
   "stop_time" : 0,
   "types" : [],
   "stdout_file" : "OAR.2.stdout",
   "cpuset_name" : "docker_2",
   "owner" : "docker",
   "stderr_file" : "OAR.2.stderr",
   "array_index" : 1,
   "resubmit_job_id" : 0,
   "extratime" : {
      "granted" : 900,
      "pending" : 0,
      "possible" : "3600",
      "granted_with_delaying_next_jobs" : 600
   },
   "wanted_resources" : "-l \"{type = 'default'}/network_address=ALL,walltime=2:15:0\" ",
   "start_time" : 1478540292,
   "initial_request" : "oarsub -l nodes=ALL -I",
   "scheduled_start" : 1478540292,
   "launching_directory" : "/home/docker",
   "message" : "R=12,W=2:0:0,J=I (Karma=0.000,quota_ok)"
}
</code>

==== If job is terminated ===
We get a ''possible = 0''
<code>
$ curl -i -X GET http://docker:docker@localhost:46668/oarapi-priv/jobs/2.json -H'Content-Type: application/json'
HTTP/1.1 200 OK
Date: Mon, 07 Nov 2016 18:09:11 GMT
Server: Apache/2.4.10 (Debian)
Allow: GET, POST, PUT, DELETE
Transfer-Encoding: chunked
Content-Type: application/json; charset=ISO-8859-1

{
   "resubmit_job_id" : 0,
   "project" : "default",
   "scheduled_start" : null,
   "types" : [],
   "events" : [
      {
         "to_check" : "NO",
         "job_id" : 2,
         "event_id" : 2,
         "description" : " 300 s of extra time added to the job (new walltime = 7500 s)",
         "type" : "EXTRATIME",
         "date" : 1478540473
      },
      {
         "to_check" : "NO",
         "job_id" : 2,
         "description" : " 300 s of extra time added to the job (new walltime = 7800 s)",
         "event_id" : 3,
         "type" : "EXTRATIME",
         "date" : 1478541043
      },
      {
         "to_check" : "NO",
         "job_id" : 2,
         "description" : " 300s of extra time added to the job (new walltime = 8100s) (with delaying next jobs)",
         "event_id" : 4,
         "date" : 1478541647,
         "type" : "EXTRATIME"
      },
      {
         "to_check" : "NO",
         "job_id" : 2,
         "event_id" : 5,
         "description" : "[bipbip 2] Ask to change the job state",
         "date" : 1478542150,
         "type" : "SWITCH_INTO_TERMINATE_STATE"
      }
   ],
   "name" : null,
   "dependencies" : [],
   "wanted_resources" : "-l \"{type = 'default'}/network_address=ALL,walltime=2:15:0\" ",
   "message" : "R=12,W=2:0:0,J=I (Karma=0.000,quota_ok)",
   "array_id" : 2,
   "api_timestamp" : 1478542152,
   "links" : [
      {
         "href" : "/oarapi-priv/jobs/2",
         "rel" : "self"
      },
      {
         "rel" : "collection",
         "href" : "/oarapi-priv/jobs/2/resources",
         "title" : "resources"
      },
      {
         "title" : "nodes",
         "href" : "/oarapi-priv/jobs/2/nodes",
         "rel" : "collection"
      }
   ],
   "queue" : "default",
   "stop_time" : 1478542150,
   "properties" : "(desktop_computing = 'NO') AND drain='NO'",
   "stderr_file" : "OAR.2.stderr",
   "id" : 2,
   "cpuset_name" : "docker_2",
   "start_time" : 1478540292,
   "walltime" : 8100,
   "array_index" : 1,
   "launching_directory" : "/home/docker",
   "stdout_file" : "OAR.2.stdout",
   "reservation" : "None",
   "submission_time" : 1478540240,
   "exit_code" : null,
   "type" : "INTERACTIVE",
   "command" : "",
   "initial_request" : "oarsub -l nodes=ALL -I",
   "owner" : "docker",
   "state" : "Terminated",
   "extratime" : {
      "possible" : 0,
      "granted" : 900,
      "pending" : 0,
      "granted_with_delaying_next_jobs" : 600
   }
}
</code>
A request fails
<code>
$ curl -i -X POST http://docker:docker@localhost:46668/oarapi-priv/jobs/2.json -H'Content-Type: application/json' -d '{"method":"extratime", "duration":"300" }'
HTTP/1.1 403 Forbidden
Date: Mon, 07 Nov 2016 18:33:02 GMT
Server: Apache/2.4.10 (Debian)
Transfer-Encoding: chunked
Content-Type: application/json; charset=ISO-8859-1

{
   "code" : 403,
   "message" : "Job 2 is not running",
   "title" : "Forbidden"
}
</code>