====== oarextratime ====== Coming with OAR 2.5.8, oarwalltime is a new OAR command. This page talk about oarextratime, which was the former name for oarwalltime. One could just exchange the two words. ===== Usage ===== $ oarextratime Usage: /usr/local/lib/oar/oarextratime [] [] Manage extra time requests for a job. - If extra time is given, the command requests the addition of some extra time to the job or update a previous request. - If no extra time is given, the command shows the current extra time status for the job. Job must be running. Extra time is to be given as an number of seconds. Options: -D, --delay-next-jobs allow extra time to delay the next batch jobs -V, --version print OAR version -h, --help print help === Manual page === NAME oarextratime - request or query extra time for a job handled by the OAR batch scheduler. SYNOPSIS oarextratime [options] [extra_time] DESCRIPTION This command allows users to request or query extra time for a job. If extra_time is given, the command requests the addition of some extra time to the job or update a previous request. If extra_time is not given, the command shows the current extra time status for the job. Job must be running. Extra time is to be given as an number of seconds. Once a request is registered, it will by handled during the next pass of scheduling and granted if it fits with other jobs. OPTIONS -D, --delay-next-jobs Allow extra time to delay next batch jobs (but not advance reservations), if needed in order to fit in the resources allocation. -h, --help Print help command message. -V, --version Print OAR version. NOTES The extra time functionality allows one to increase the walltime of a job, as an emergency measure when some more time is critically needed to complete a work. There is no warranty that extra time can be granted, since it depends on the resources availability (next jobs). Please mind the walltime of the job and do not rely too much on extra time. Extra time events for a job are listed in the oarstat output. As per configuration, the extra time functionality may be disabled in your installation. As per configuration, an extra time request may only be possible some time before the predicted end of the job. Configuration can be set per queue. Read your local documentation or ask your administrator(s) to know the configured settings. The extra time functionality does not support the computation of the resources shared allocation in the case of jobs of type timesharing or container/inner or placeholder/allowed. SEE ALSO oarstat(1), oarsub(1), oardel(1) ===== Configuration ===== Related oar.conf snippet. ########################################################################### # OAR EXTRA TIME # # Allow users to request some extra time to finish a job. Extra time is # # only granted if it does not conflict with the next jobs. # # WARNING: # # - This functionnality does not handle the timesharing, container/inner # # and placeholder/Allowed types of jobs. # # - This functionnality is not compatible with the RUNTIME suspend/resume # # function (see man oarhold and the USERS_ALLOWED_HOLD_RESUME line above).# ############################################################################### # # Extra time duration limit # -1 means that the extra time functionality is disabled (global setting, cannot be used per queue) # 0 means that the extra time functionality is allowed for oar and root only (with no limit) # >0 sets the maximum duration of possible extra time users can request in seconds # default is 0 #EXTRA_TIME_DURATION=0 # Set extra time request delay: do not accept requests from users until delay seconds before the predicted end of the job # Default is 0, i.e. no delay #EXTRA_TIME_REQUEST_DELAY=0 # Set the minimum walltime of jobs for which extra time can be requested # Default is 0, i.e. minimum walltime #EXTRA_TIME_MINIMUM_WALLTIME=0 # List users (* for all) for which extra time can delay the next batch jobs (i.e. only advance reserations cannot be moved) # Default is "", i.e. only root and the oar user are allowed #EXTRA_TIME_DELAY_NEXT_JOBS_ALLOWED_USERS="" # Those 4 settings can also be provided per queue, using the Perl hash syntax with the queue names as the keys. # Not defined queues get the value associated to the _ key, or if not defined either, the default value. # e.g.: # - Allow extra time requests for up to 2 hours for the default queue only (0 for others): #EXTRA_TIME_DURATION="{default => 7200}" # - Allow any user to delay next jobs with requesting extra time for a job in all but the besteffort queue: #EXTRA_TIME_DELAY_NEXT_JOBS_ALLOWED_USERS="{_ => '*', besteffort => ''}" ====== REST API ====== For ref, the [[rest-api-specs]] page. Extratime now uses the /oarapi-priv/jobs/ URL with GET or (POST or PUT <- both do the same...) *Extratime syntax in a job query output: GET /oarapi-priv/jobs/.json * ''possible'': possible requests in seconds, 0 if not request is possible (for any cause) * ''pending'': pending requests in seconds to be handled by the scheduler * ''granted'': granted extratime in seconds * ''granted_with_delaying_next_jobs'': part of ''granted'' (in seconds) that was granted with ''delay_next_jobs'' = ''YES'', does not show up if equal to 0. * ''delay_next_jobs'': ''FORBIDDEN'' if not allowed, ''YES'' if allowed and requested and ''pending'' > 0, ''NO'' if not requested and ''pending'' > 0, does not show up si ''pending'' = 0 * Extratime request syntax in a job update: POST /oarapi-priv/jobs/.json -d ''{ ... }'' * ''duration'': requested duration of extratime in seconds, mandatory * ''delay_next_jobs'': ''YES'' or ''NO'' (default if not set) ==== Querying a job that does not exist === $ curl -i -X GET http://docker:docker@localhost:46668/oarapi-priv/jobs/1.json -H'Content-Type: application/json' HTTP/1.1 404 Not Found Date: Mon, 07 Nov 2016 17:29:24 GMT Server: Apache/2.4.10 (Debian) Transfer-Encoding: chunked Content-Type: application/json; charset=ISO-8859-1 { "message" : "Job not found", "code" : 404, "title" : "Job not found" } * request fails $ curl -i -X POST http://docker:docker@localhost:46668/oarapi-priv/jobs/1.json -H'Content-Type: application/json' -d '{"method":"extratime", "duration":"300", "delay_next_jobs": "YES" }' HTTP/1.1 403 Forbidden Date: Mon, 07 Nov 2016 17:39:59 GMT Server: Apache/2.4.10 (Debian) Transfer-Encoding: chunked Content-Type: application/json; charset=ISO-8859-1 { "code" : 403, "message" : "Job 1 is not running", "title" : "Forbidden" } ==== Querying a job that exists ==== * with default configuration (extratime not activated for regular users) $ curl -i -X GET http://docker:docker@localhost:46668/oarapi-priv/jobs/1.json -H'Content-Type: application/json' HTTP/1.1 200 OK Date: Mon, 07 Nov 2016 17:29:35 GMT Server: Apache/2.4.10 (Debian) Allow: GET, POST, PUT, DELETE Transfer-Encoding: chunked Content-Type: application/json; charset=ISO-8859-1 { "project" : "default", "submission_time" : 1478539772, "dependencies" : [], "reservation" : "None", "command" : "", "properties" : "(desktop_computing = 'NO') AND drain='NO'", "array_id" : 1, "state" : "Running", "events" : [], "links" : [ { "href" : "/oarapi-priv/jobs/1", "rel" : "self" }, { "title" : "resources", "rel" : "collection", "href" : "/oarapi-priv/jobs/1/resources" }, { "href" : "/oarapi-priv/jobs/1/nodes", "title" : "nodes", "rel" : "collection" } ], "id" : 1, "exit_code" : null, "name" : null, "walltime" : 7200, "types" : [], "stdout_file" : "OAR.1.stdout", "cpuset_name" : "docker_1", "owner" : "docker", "stop_time" : 0, "queue" : "default", "type" : "INTERACTIVE", "api_timestamp" : 1478539775, "launching_directory" : "/home/docker", "message" : "R=1,W=2:0:0,J=I (Karma=0.000,quota_ok)", "scheduled_start" : 1478539773, "extratime" : { "possible" : 0, "granted" : 0, "pending" : 0, "delay_next_jobs" : "FORBIDDEN" }, "initial_request" : "oarsub -I", "start_time" : 1478539773, "wanted_resources" : "-l \"{type = 'default'}/resource_id=1,walltime=2:0:0\" ", "stderr_file" : "OAR.1.stderr", "array_index" : 1, "resubmit_job_id" : 0 } * with configuration changed: EXTRA_TIME_DURATION=3600 "extratime" : { "granted" : 0, "delay_next_jobs" : "FORBIDDEN", "possible" : "3600", "pending" : 0 }, * with request not allowed because of EXTRA_TIME_REQUEST_DELAY or EXTRA_TIME_MINIMUM_WALLTIME, or because the job is not running yet "extratime" : { "possible" : 0, "delay_next_jobs" : "FORBIDDEN", "granted" : 0, "pending" : 0 }, * with EXTRA_TIME_DELAY_NEXT_JOBS_ALLOWED_USERS="*" "extratime" : { "possible" : 3600, "granted" : 0, "pending" : 0 }, * with a pending request not yet (fully) granted (a job is blocking) "extratime" : { "possible" : 3600, "granted" : 100, "pending" : 200, "delay_next_jobs" : "NO", }, * once (fully) granted, if ''pending = 0'' (no more blocking job), ''"delay_next_jobs" : "NO"'' is not displayed "extratime" : { "possible" : 3600, "granted" : 300, "pending" : 0, }, ==== Doing a valid request ==== With ''delay_next_job'' not allowed in this example $ curl -i -X POST http://docker:docker@localhost:46668/oarapi-priv/jobs/2.json -H'Content-Type: application/json' -d '{"method":"extratime", "duration":"300" }' HTTP/1.1 202 Accepted Date: Mon, 07 Nov 2016 17:41:12 GMT Server: Apache/2.4.10 (Debian) Transfer-Encoding: chunked Content-Type: application/json; charset=ISO-8859-1 { "id" : "2", "cmd_output" : "Extra time request registered for job 2. It will be handled shortly", "api_timestamp" : 1478540472, "status" : "Accepted", "links" : [ { "href" : "/oarapi-priv/jobs/2", "rel" : "self" } ] } $ curl -i -X GET http://docker:docker@localhost:46668/oarapi-priv/jobs/2.json -H'Content-Type: application/json'HTTP/1.1 200 OK Date: Mon, 07 Nov 2016 17:46:00 GMT Server: Apache/2.4.10 (Debian) Allow: GET, POST, PUT, DELETE Transfer-Encoding: chunked Content-Type: application/json; charset=ISO-8859-1 { "wanted_resources" : "-l \"{type = 'default'}/network_address=ALL,walltime=2:5:0\" ", "message" : "R=12,W=2:0:0,J=I (Karma=0.000,quota_ok)", "array_id" : 2, "api_timestamp" : 1478540760, "links" : [ { "rel" : "self", "href" : "/oarapi-priv/jobs/2" }, { "title" : "resources", "href" : "/oarapi-priv/jobs/2/resources", "rel" : "collection" }, { "href" : "/oarapi-priv/jobs/2/nodes", "rel" : "collection", "title" : "nodes" } ], "queue" : "default", "stop_time" : 0, "properties" : "(desktop_computing = 'NO') AND drain='NO'", "resubmit_job_id" : 0, "project" : "default", "scheduled_start" : 1478540292, "types" : [], "events" : [ { "date" : 1478540473, "type" : "EXTRATIME", "job_id" : 2, "to_check" : "NO", "event_id" : 2, "description" : " 300 s of extra time added to the job (new walltime = 7500 s)" } ], "name" : null, "dependencies" : [], "submission_time" : 1478540240, "exit_code" : null, "type" : "INTERACTIVE", "command" : "", "initial_request" : "oarsub -l nodes=ALL -I", "owner" : "docker", "state" : "Running", "extratime" : { "delay_next_jobs" : "FORBIDDEN", "possible" : "3600", "granted" : 300, "pending" : 0 }, "stderr_file" : "OAR.2.stderr", "id" : 2, "walltime" : 7500, "start_time" : 1478540292, "cpuset_name" : "docker_2", "array_index" : 1, "launching_directory" : "/home/docker", "reservation" : "None", "stdout_file" : "OAR.2.stdout" } Now try to delay next jobs $ curl -i -X POST http://docker:docker@localhost:46668/oarapi-priv/jobs/3.json -H'Content-Type: application/json' -d '{"method":"extratime", "duration":"300", "delay_next_jobs":"YES" }' HTTP/1.1 403 Forbidden Date: Mon, 07 Nov 2016 18:33:08 GMT Server: Apache/2.4.10 (Debian) Transfer-Encoding: chunked Content-Type: application/json; charset=ISO-8859-1 { "code" : 403, "message" : "Delaying next jobs is not allowed", "title" : "Forbidden" } ==== request with delaying next jobs ==== * submit request $ curl -i -X POST http://docker:docker@localhost:46668/oarapi-priv/jobs/2.json -H'Content-Type: application/json' -d '{"method":"extratime", "duration":"300", "delay_next_jobs":"YES" }' HTTP/1.1 202 Accepted Date: Mon, 07 Nov 2016 17:58:22 GMT Server: Apache/2.4.10 (Debian) Transfer-Encoding: chunked Content-Type: application/json; charset=ISO-8859-1 { "cmd_output" : "Extra time request updated for job 2. It will be handled shortly", "links" : [ { "href" : "/oarapi-priv/jobs/2", "rel" : "self" } ], "api_timestamp" : 1478541502, "status" : "Accepted", "id" : "2" } * request still pending $ curl -i -X GET http://docker:docker@localhost:46668/oarapi-priv/jobs/2.json -H'Content-Type: application/json' HTTP/1.1 200 OK Date: Mon, 07 Nov 2016 18:00:39 GMT Server: Apache/2.4.10 (Debian) Allow: GET, POST, PUT, DELETE Transfer-Encoding: chunked Content-Type: application/json; charset=ISO-8859-1 { "links" : [ { "href" : "/oarapi-priv/jobs/2", "rel" : "self" }, { "href" : "/oarapi-priv/jobs/2/resources", "title" : "resources", "rel" : "collection" }, { "href" : "/oarapi-priv/jobs/2/nodes", "title" : "nodes", "rel" : "collection" } ], "stdout_file" : "OAR.2.stdout", "wanted_resources" : "-l \"{type = 'default'}/network_address=ALL,walltime=2:10:0\" ", "api_timestamp" : 1478541640, "walltime" : 7800, "resubmit_job_id" : 0, "name" : null, "array_index" : 1, "state" : "Running", "queue" : "default", "start_time" : 1478540292, "scheduled_start" : 1478540292, "types" : [], "owner" : "docker", "initial_request" : "oarsub -l nodes=ALL -I", "exit_code" : null, "submission_time" : 1478540240, "project" : "default", "extratime" : { "granted" : 600, "pending" : 300, "possible" : "3600", "delay_next_jobs" : "YES", "granted_with_delaying_next_jobs" : 300 }, "message" : "R=12,W=2:0:0,J=I (Karma=0.000,quota_ok)", "id" : 2, "events" : [ { "to_check" : "NO", "job_id" : 2, "description" : " 300 s of extra time added to the job (new walltime = 7500 s)", "date" : 1478540473, "event_id" : 2, "type" : "EXTRATIME" }, { "event_id" : 3, "type" : "EXTRATIME", "date" : 1478541043, "job_id" : 2, "description" : " 300 s of extra time added to the job (new walltime = 7800 s)", "to_check" : "NO" } ], "reservation" : "None", "array_id" : 2, "type" : "INTERACTIVE", "stop_time" : 0, "properties" : "(desktop_computing = 'NO') AND drain='NO'", "command" : "", "launching_directory" : "/home/docker", "stderr_file" : "OAR.2.stderr", "dependencies" : [], "cpuset_name" : "docker_2" } * request is accepted $ curl -i -X GET http://docker:docker@localhost:46668/oarapi-priv/jobs/2.json -H'Content-Type: application/json' HTTP/1.1 200 OK Date: Mon, 07 Nov 2016 18:01:03 GMT Server: Apache/2.4.10 (Debian) Allow: GET, POST, PUT, DELETE Transfer-Encoding: chunked Content-Type: application/json; charset=ISO-8859-1 { "properties" : "(desktop_computing = 'NO') AND drain='NO'", "command" : "", "reservation" : "None", "dependencies" : [], "project" : "default", "submission_time" : 1478540240, "walltime" : 8100, "id" : 2, "name" : null, "exit_code" : null, "array_id" : 2, "events" : [ { "description" : " 300 s of extra time added to the job (new walltime = 7500 s)", "date" : 1478540473, "type" : "EXTRATIME", "event_id" : 2, "to_check" : "NO", "job_id" : 2 }, { "event_id" : 3, "to_check" : "NO", "job_id" : 2, "description" : " 300 s of extra time added to the job (new walltime = 7800 s)", "type" : "EXTRATIME", "date" : 1478541043 }, { "date" : 1478541647, "type" : "EXTRATIME", "description" : " 300s of extra time added to the job (new walltime = 8100s) (with delaying next jobs)", "job_id" : 2, "event_id" : 4, "to_check" : "NO" } ], "state" : "Running", "links" : [ { "href" : "/oarapi-priv/jobs/2", "rel" : "self" }, { "href" : "/oarapi-priv/jobs/2/resources", "title" : "resources", "rel" : "collection" }, { "title" : "nodes", "rel" : "collection", "href" : "/oarapi-priv/jobs/2/nodes" } ], "api_timestamp" : 1478541663, "type" : "INTERACTIVE", "queue" : "default", "stop_time" : 0, "types" : [], "stdout_file" : "OAR.2.stdout", "cpuset_name" : "docker_2", "owner" : "docker", "stderr_file" : "OAR.2.stderr", "array_index" : 1, "resubmit_job_id" : 0, "extratime" : { "granted" : 900, "pending" : 0, "possible" : "3600", "granted_with_delaying_next_jobs" : 600 }, "wanted_resources" : "-l \"{type = 'default'}/network_address=ALL,walltime=2:15:0\" ", "start_time" : 1478540292, "initial_request" : "oarsub -l nodes=ALL -I", "scheduled_start" : 1478540292, "launching_directory" : "/home/docker", "message" : "R=12,W=2:0:0,J=I (Karma=0.000,quota_ok)" } ==== If job is terminated === We get a ''possible = 0'' $ curl -i -X GET http://docker:docker@localhost:46668/oarapi-priv/jobs/2.json -H'Content-Type: application/json' HTTP/1.1 200 OK Date: Mon, 07 Nov 2016 18:09:11 GMT Server: Apache/2.4.10 (Debian) Allow: GET, POST, PUT, DELETE Transfer-Encoding: chunked Content-Type: application/json; charset=ISO-8859-1 { "resubmit_job_id" : 0, "project" : "default", "scheduled_start" : null, "types" : [], "events" : [ { "to_check" : "NO", "job_id" : 2, "event_id" : 2, "description" : " 300 s of extra time added to the job (new walltime = 7500 s)", "type" : "EXTRATIME", "date" : 1478540473 }, { "to_check" : "NO", "job_id" : 2, "description" : " 300 s of extra time added to the job (new walltime = 7800 s)", "event_id" : 3, "type" : "EXTRATIME", "date" : 1478541043 }, { "to_check" : "NO", "job_id" : 2, "description" : " 300s of extra time added to the job (new walltime = 8100s) (with delaying next jobs)", "event_id" : 4, "date" : 1478541647, "type" : "EXTRATIME" }, { "to_check" : "NO", "job_id" : 2, "event_id" : 5, "description" : "[bipbip 2] Ask to change the job state", "date" : 1478542150, "type" : "SWITCH_INTO_TERMINATE_STATE" } ], "name" : null, "dependencies" : [], "wanted_resources" : "-l \"{type = 'default'}/network_address=ALL,walltime=2:15:0\" ", "message" : "R=12,W=2:0:0,J=I (Karma=0.000,quota_ok)", "array_id" : 2, "api_timestamp" : 1478542152, "links" : [ { "href" : "/oarapi-priv/jobs/2", "rel" : "self" }, { "rel" : "collection", "href" : "/oarapi-priv/jobs/2/resources", "title" : "resources" }, { "title" : "nodes", "href" : "/oarapi-priv/jobs/2/nodes", "rel" : "collection" } ], "queue" : "default", "stop_time" : 1478542150, "properties" : "(desktop_computing = 'NO') AND drain='NO'", "stderr_file" : "OAR.2.stderr", "id" : 2, "cpuset_name" : "docker_2", "start_time" : 1478540292, "walltime" : 8100, "array_index" : 1, "launching_directory" : "/home/docker", "stdout_file" : "OAR.2.stdout", "reservation" : "None", "submission_time" : 1478540240, "exit_code" : null, "type" : "INTERACTIVE", "command" : "", "initial_request" : "oarsub -l nodes=ALL -I", "owner" : "docker", "state" : "Terminated", "extratime" : { "possible" : 0, "granted" : 900, "pending" : 0, "granted_with_delaying_next_jobs" : 600 } } A request fails $ curl -i -X POST http://docker:docker@localhost:46668/oarapi-priv/jobs/2.json -H'Content-Type: application/json' -d '{"method":"extratime", "duration":"300" }' HTTP/1.1 403 Forbidden Date: Mon, 07 Nov 2016 18:33:02 GMT Server: Apache/2.4.10 (Debian) Transfer-Encoding: chunked Content-Type: application/json; charset=ISO-8859-1 { "code" : 403, "message" : "Job 2 is not running", "title" : "Forbidden" }