OARGRID API
From WikiOAR
Contents |
Introduction
The OARGRID API is currently a CGI script providing access to oargrid using http requests in the REST way. It is based on the OAR API . The current version, provided with oar 2.4 (not yet released) is partly based on wrappers to oargrid and oar command line tools using ssh. But on the OAR roadmap (OAR 3.0) the API is to be coded deeper into OAR and OARGRID to provide better performances and functionalities.
Concepts
Access
A simple GET query to the API using wget may look like:
# List the sites in the YAML format wget -O - http://www.mydomain.org/oargridapi/sites.yaml?structure=simple
You can also access to the API using a browser. Make it point to http://www.mydomain.org/oargridapi/index.html and you'll see a very simple HTML interface allowing you to browse the grid resources and even post a grid job using a form.
But generally, you'll use a REST client or a REST library provided for your favorite language. You'll see examples using a ruby rest library at the end of this page.
Authentication
Most of the time, you'll make requests that needs you to be authenticated. The API may be configured to use the IDENT protocol for authentication from trusted hosts, like a cluster frontend. In this case, your login is automatically used by the API. This only works for hosts that have been correctly configured (for which the security rules are trusted by the admistrator). If IDENT is not used or not trusted from your host, the API can use the basic HTTP authentication, depending on how it has been configured inside the http server that serves the CGI.
The recommended way is to use ident. But as some only a few hosts may be trusted, you'll probably have to open a tunnel to this host. You may use ssh to do this. For example:
$ ssh -NL 8080:www.grenoble.grid5000.fr:80 login@access.grenoble.grid5000.fr
Then, point your REST client to:
http://localhost:8080
Data structures and formats
The API currently can serve data into YAML, JSON or HTML. Posted data can also be coded into YAML, JSON or x-www-form-urlencoded (for HTML from posts). You may specify the requested format by 2 ways:
- giving an extension to resources: .yaml, .json or .html
- setting the HTTP_ACCEPT header variable to text/yaml, application/json or text/html
For the posted data, you have to correctly set the HTTP_CONTENT_TYPE variable to text/yaml, application/json or application/x-www-form-urlencoded.
Sometimes, the data structures returned (not the coding format, but the contents: array, hashes, array of hashes,...) may be changed. Currently, we have 2 available data structures: simple and oar. The structure is passed through the variable structure that you may pass in the url, for example: ?structure=simple
- The simple data structure tries to be as simple as possible, using simple arrays in place of hashes wherever it is possible
- The oar data structure serves data in the way oar does with the oarnodes/oarstat export options (-Y, -D, -J,...)
By default, we use the simple data structure.
Resources
- GET /sites
- GET /sites/<site>
- GET /sites/<site>/resources
- GET /sites/<site>/resources/all
- GET /sites/<site>/resources/<id>
- GET /sites/<site>/resources/nodes/<node_name>
- GET /grid/jobs
- POST /grid/jobs
- GET /grid/jobs/<id>
- DELETE /grid/jobs/<id>
- GET /grid/jobs/<id>/resources
- GET /grid/jobs/<id>/nodes
Errors and debug
When the API returns an error, it generally uses a standard HTTP return status (404 NOT FOUND, 406 NOT ACCEPTABLE, ...). But it also returns a body containing a hash like the following.
{
"title" : "ERROR 406 - Invalid content type required */*",
"message" : "Valid types are text/yaml, application/json or text/html",
"code" : "200"
}
This error body is formated in the requested format. But if this format was not given, it uses JSON by default.
To allow you to see the error body, you may find it useful to activate the debug=1 variable. It will force the API to always return a 200 OK status, even if there's an error so that I can see the body with a simple browser or a rest client without having to manage the errors. For example:
wget -nv -O - "http://localhost:8080/oargridapi/sites/grenoble?debug=1"
Examples
Examples of output data
- GET /sites.yaml
- site: nancy uri: /sites/nancy - site: sophia uri: /sites/sophia - site: orsay uri: /sites/orsay
- GET /sites/orsay.yaml
-- clusters: - gdx - netgdx frontend: frontend.orsay.grid5000.fr site: orsay uri: /sites/orsay
- GET /sites/orsay.json
"site" : "orsay",
"frontend" : "frontend.orsay.grid5000.fr",
"clusters" : [
"gdx",
"netgdx"
],
"uri" : "/sites/orsay"
- GET /sites/nancy/jobs.yaml?structure=simple
--- - id: 216033 name: session 1 owner: dmargery queue: default state: Waiting submission: 1236762970 uri: /sites/nancy/jobs/216033 - id: 216361 name: Maintenance owner: cconstantin queue: admin state: Waiting submission: 1237824986 uri: /sites/nancy/jobs/216361
- GET /sites/nancy/jobs.yaml?structure=oar
--- 215958: name: Maintenance owner: cconstantin queue: admin state: Waiting submission: 1236238707 uri: /sites/nancy/jobs/215958 216033: name: session 1 owner: dmargery queue: default state: Waiting submission: 1236762970 uri: /sites/nancy/jobs/216033
- POST /grid/jobs.yaml
--- command: "oargridsub --walltime=\"01:00:00\" --program=\"/bin/sleep 300\" grenoble:rdef='/nodes=2',rennes:rdef='/cpu=1'" id: 19017 nodes: '/grid/jobs/19017/resources/nodes.html' resources: '/grid/jobs/19017/resources.html' ssh_key_path: /tmp/oargrid//oargrid_ssh_key_bbzeznik_19017 ssh_private_key_uri: '/grid/jobs/19017/keys/private.html' ssh_public_key_uri: '/grid/jobs/19017/keys/public.html' state: submitted uri: '/grid/jobs/19017.html'
Ruby interactive REST client
One of the easiest way for testing this API is to use the rest-client ruby module:
http://rest-client.heroku.com/rdoc/
It is available as a rubygem, so to install it, simply install rubygems and do "gem install rest-client". Then, you can run the interactive client which is nothing else than irb with shortcuts. Here is an example irb session (useless output have ben removed for readability)
export PATH=$PATH:/var/lib/gems/1.8/gems/rest-client-0.8/bin restclient http://www.grenoble.grid5000.fr/oargridapi
or, if an http auth is needed:
restclient http://www.grenoble.grid5000.fr/oargridapi <login> <password>
example queries
# Getting sites infos
# in JSON
irb(main):004:0> puts get('/sites.json')
# Same thing, asking for the "oar" data structure
irb(main):004:0> puts get('/sites.json?structure=oar')
# Same thing, asking for the "simple" data structure
irb(main):004:0> puts get('/sites.json?structure=simple')
# in YAML
irb(main):005:0> puts get('/sites.yaml')
# Same thing but using the Accept header
irb(main):050:0> puts get('/sites', :accept=>"text/yaml")
# Details about a site
irb(main):008:0> puts get('/sites/grenoble.yaml')
# Resources details on a site or a cluster
irb(main):022:0> puts get('/sites/simpsons/resources.yaml')
# Getting jobs infos of a site
irb(main):006:0> puts get('/sites/grenoble/jobs.yaml')
irb(main):009:0> puts get('/sites/grenoble/jobs/12.yaml')
# Submiting a job on a given site (using YAML format)
irb(main):010:0> require 'yaml'
irb(main):012:0> j={ 'resource' => '/nodes=2/cpu=1', 'script_path' => '/usr/bin/id' }
irb(main):015:0> job=post('/sites/grenoble/jobs' , j.to_yaml , :content_type => 'text/yaml')
irb(main):016:0> puts job
# Getting details about the previously submited job irb(main):035:0> puts get(YAML::load(job)['uri'])
# Submitting a job using JSON format, but requiring the result in YAML
irb(main):010:0> require 'json'
irb(main):037:0> job=post('/sites/grenoble/jobs.yaml' , j.to_json , :content_type => 'application/json')
# Submitting a grid job
irb(main):029:0> j={ 'resources' => 'grenoble:rdef=/cpu=2,lille:rdef=/cpu=2' }
irb(main):030:0> job=post('/grid/jobs.yaml' , j.to_json , :content_type => 'application/json')
irb(main):031:0> puts job
# List the resources I obtained (if the state is ok) irb(main):061:0> job=YAML::load(job) irb(main):062:0> if job['state'] == 'submitted' irb(main):063:1> puts get(job['resources'],:accept => 'text/yaml') irb(main):064:1> else irb(main):065:1* puts "Job was rejected" irb(main):066:1> end
# Get an array of node names inside a grid job
irb(main):063:1> puts get('/grid/jobs/18611/resources/nodes.json')
# List my running grid jobs, in HTML
irb(main):067:0> puts get('/grid/jobs.html')
# Show details about a grid job, in JSON
irb(main):069:0> puts get('/grid/jobs/18611.json')
# Deleting a grid job
irb(main):111:0> delete("/grid/jobs/18611.yaml")
A Ruby script using restclient
#!/usr/bin/ruby
# Example of a ruby script using restclient on the Oargrid RESTfull API.
# It first parses the grid to find 3 sites where 1 resource is free and
# then submits a job and prints the properties of the nodes obtained.
# The job is deleted at the end, as it is just an example :-)
require 'rubygems'
require 'rest_client'
require 'json'
require 'pp'
# Custom variables
APIURI="http://www.grenoble.grid5000.fr/oargridapi"
IGNORE_SITES=['grenoble-obs','grenoble-exp','grenoble-ext','sophia','lille']
N_RESOURCES=3
# Function to get objects from the api
# We use the JSON format and the 'simple' data structure
def get(api,uri)
begin
return JSON.parse(api[uri+'?structure=simple'].get(:accept => 'application/json'))
rescue => e
if e.respond_to?('http_code')
puts "ERROR #{e.http_code}:\n #{e.response.body}"
else
puts "Parse error:"
puts e.inspect
end
exit 1
end
end
# Instanciate an api connection
api = RestClient::Resource.new APIURI
# Parse the sites and find at least N_RESOURCES free resources
puts "Getting sites..."
sites = get(api,'/sites')
puts "Got #{sites.length} sites"
ok_sites=[]
sites.each do |site|
site_name = site['site']
unless IGNORE_SITES.index(site_name)
puts "Checking #{site_name}..."
resources = get(api,"/sites/#{site_name}/resources/all")
resources.each do |resource|
if resource['state'] == "Alive" && resource['jobs'].nil?
ok_sites << site_name
puts " got 1 resource!"
break
end
end
end
break if ok_sites.length >= N_RESOURCES
end
# If we got N_RESOURCES sites, submit the job
if ok_sites.length < N_RESOURCES
puts "Not enough resources found on the grid!"
exit 1
end
puts "OK, we got #{N_RESOURCES} free resources, let's submit a job..."
j={ 'resources' => ok_sites.join(':rdef=/resource_id=1,')+':rdef=/resource_id=1' }
#j={ 'resources' => ok_sites.join(':rdef=/resource_id=1,')+':rdef=/resource_id=1' , 'verbose' => 1 }
begin
job=JSON.parse(api['/grid/jobs'].post(j.to_json,:content_type => 'application/json'))
rescue => e
puts "ERROR #{e.http_code}:\n #{JSON.parse(e.response.body)['message']}"
exit 2
end
# Check the job
if job['state'] == "submitted"
puts "GRID JOB SUCCESSFUL :-)"
puts "-----------------------------------------------"
puts "Id: #{job['id']}"
puts "Waiting for resources to be allocated..."
# Wait for allocated resources
nodes=[]
while nodes.length < ok_sites.length do
nodes=get(api,"/grid/jobs/#{job['id']}/resources/nodes")
sleep(2)
end
# Ok, print nodes
puts "Ok, all nodes are there:"
puts nodes.join(',')
# Fetching properties of the nodes
puts "Properties:"
resources=get(api,"/grid/jobs/#{job['id']}/resources")
resources.each do |r|
site=r['site']
r['jobs'].each_value do |j|
j['nodes'].each do |node|
puts "\n" + node + " : "
prop=get(api,"/sites/#{site}/resources/nodes/#{node}")
pp prop
end
end
end
# Delete the job
begin
puts "Deleting job because it was just for fun..."
api["/grid/jobs/#{job['id']}.json"].delete
puts "Job deleted"
rescue => e
puts "ERROR #{e.http_code} DELETING THE GRID JOB :\n #{JSON.parse(e.response.body)['message']}"
end
# If the job has failed:
else
puts "PROBLEM: Grid job #{job['state']}"
puts "OUTPUT MESSAGE:"
puts job['output']
end
