View source

= Introduction =
The OARGRID API is currently a CGI script providing access to oargrid using http requests in the REST way. It is based on the [[ OAR_API | OAR API ]]. The current version, provided with oar 2.4 (not yet released) is partly based on wrappers to oargrid and oar command line tools using ssh. But on the OAR roadmap (OAR 3.0) the API is to be coded deeper into OAR and OARGRID to provide better performances and functionalities.

= Concepts =
==Access==
A simple GET query to the API using wget may look like:
 # List the sites in the YAML format
 wget -O - http://www.mydomain.org/oargridapi/sites.yaml?structure=simple
You can also access to the API using a browser. Make it point to http://www.mydomain.org/oargridapi/index.html and you'll see a very simple HTML interface allowing you to browse the grid resources and even post a grid job using a form.

But generally, you'll use a REST client or a REST library provided for your favorite language. You'll see examples using a ruby rest library at the end of this page.

==Authentication==
Most of the time, you'll make requests that needs you to be authenticated. The API may be configured to use the [http://en.wikipedia.org/wiki/Ident IDENT] protocol for authentication from trusted hosts, like a cluster frontend. In this case, your login is automatically used by the API. This only works for hosts that have been correctly configured (for which the security rules are trusted by the admistrator). If IDENT is not used or not trusted from your host, the API can use the basic HTTP authentication, depending on how it has been configured inside the http server that serves the CGI.

The recommended way is to use ident. But as some only a few hosts may be trusted, you'll probably have to open a tunnel to this host. You may use ssh to do this. For example:
 $ ssh -NL 8080:www.grenoble.grid5000.fr:80 login@access.grenoble.grid5000.fr
Then, point your REST client to:
 http://localhost:8080

==Data structures and formats==
The API currently can serve data into [http://www.yaml.org YAML], [http://www.json.org JSON] or HTML. Posted data can also be coded into YAML, JSON or x-www-form-urlencoded (for HTML from posts). You may specify the requested format by 2 ways:
* giving an extension to resources: '''.yaml''', '''.json''' or '''.html'''
* setting the HTTP_ACCEPT header variable to '''text/yaml''', '''application/json''' or '''text/html'''

For the posted data, you have to correctly set the HTTP_CONTENT_TYPE variable to text/yaml, application/json or application/x-www-form-urlencoded.

Sometimes, the data structures returned (not the coding format, but the contents: array, hashes, array of hashes,...) may be changed. Currently, we have 2 available data structures: '''simple''' and '''oar'''. The structure is passed through the variable '''structure''' that you may pass in the url, for example: ?structure=simple
* The '''simple''' data structure tries to be as simple as possible, using simple arrays in place of hashes wherever it is possible
* The '''oar''' data structure serves data in the way oar does with the oarnodes/oarstat export options (-Y, -D, -J,...)
By default, we use the simple data structure.

==Resources==
* GET /sites
* GET /sites/<site>
* GET /sites/<site>/resources
* GET /sites/<site>/resources/all
* GET /sites/<site>/resources/<id>
* GET /sites/<site>/resources/nodes/<node_name>
* GET /grid/jobs
* POST /grid/jobs
* GET /grid/jobs/<id>
* DELETE /grid/jobs/<id>
* GET /grid/jobs/<id>/resources
* GET /grid/jobs/<id>/nodes

==Errors and debug==
When the API returns an error, it generally uses a standard HTTP return status (404 NOT FOUND, 406 NOT ACCEPTABLE, ...). But it also returns a body containing a hash like the following.
 {
   "title" : "ERROR 406 - Invalid content type required */*",
   "message" : "Valid types are text/yaml, application/json or text/html",
   "code" : "200"
 }
This error body is formated in the requested format. But if this format was not given, it uses JSON by default.

To allow you to see the error body, you may find it useful to activate the '''debug=1''' variable. It will force the API to always return a 200 OK status, even if there's an error so that I can see the body with a simple browser or a rest client without having to manage the errors. For example:
 wget -nv -O - "<nowiki>http://localhost:8080/oargridapi/sites/grenoble</nowiki>'''?debug=1'''"

= Examples =
== Examples of output data ==

* GET /sites.yaml
 - site: nancy
  uri: /sites/nancy
 - site: sophia
  uri: /sites/sophia
 - site: orsay
  uri: /sites/orsay

* GET /sites/orsay.yaml
 --
 clusters:
   - gdx
   - netgdx
 frontend: frontend.orsay.grid5000.fr
 site: orsay
 uri: /sites/orsay

* GET /sites/orsay.json
  "site" : "orsay",
  "frontend" : "frontend.orsay.grid5000.fr",
  "clusters" : [
      "gdx",
      "netgdx"
  ],
  "uri" : "/sites/orsay"

* GET /sites/nancy/jobs.yaml?structure=simple
 ---
 - id: 216033
   name: session 1
   owner: dmargery
   queue: default
   state: Waiting
   submission: 1236762970
   uri: /sites/nancy/jobs/216033
 - id: 216361
   name: Maintenance
   owner: cconstantin
   queue: admin
   state: Waiting
   submission: 1237824986
   uri: /sites/nancy/jobs/216361

* GET /sites/nancy/jobs.yaml?structure=oar
 ---
 215958:
   name: Maintenance
   owner: cconstantin
   queue: admin
   state: Waiting
   submission: 1236238707
   uri: /sites/nancy/jobs/215958
 216033:
   name: session 1
   owner: dmargery
   queue: default
   state: Waiting
   submission: 1236762970
   uri: /sites/nancy/jobs/216033

* POST /grid/jobs.yaml
 ---
 command: "oargridsub  --walltime=\"01:00:00\" --program=\"/bin/sleep 300\" grenoble:rdef='/nodes=2',rennes:rdef='/cpu=1'"
 id: 19017
 nodes: '/grid/jobs/19017/resources/nodes.html'
 resources: '/grid/jobs/19017/resources.html'
 ssh_key_path: /tmp/oargrid//oargrid_ssh_key_bbzeznik_19017
 ssh_private_key_uri: '/grid/jobs/19017/keys/private.html'
 ssh_public_key_uri: '/grid/jobs/19017/keys/public.html'
 state: submitted
 uri: '/grid/jobs/19017.html'

== Ruby interactive REST client ==
One of the easiest way for testing this API is to use the rest-client ruby module:

http://rest-client.heroku.com/rdoc/

It is available as a rubygem, so to install it, simply install rubygems and do "gem install rest-client". Then, you can run the interactive client which is nothing else than irb with shortcuts. Here is an example irb session (useless output have ben removed for readability)

 export PATH=$PATH:/var/lib/gems/1.8/gems/rest-client-0.8/bin
 restclient http://www.grenoble.grid5000.fr/oargridapi
or, if an http auth is needed:
 restclient http://www.grenoble.grid5000.fr/oargridapi <login> <password>

=== example queries ===

 # Getting sites infos                                                                                               
   # in JSON                                                                                                         
 irb(main):004:0> puts get('/sites.json')
   # Same thing, asking for the "oar" data structure                                                                 
 irb(main):004:0> puts get('/sites.json?structure=oar')
   # Same thing, asking for the "simple" data structure                                                              
 irb(main):004:0> puts get('/sites.json?structure=simple')
   # in YAML                                                                                                         
 irb(main):005:0> puts get('/sites.yaml')                                                                            
   # Same thing but using the Accept header                                                                          
 irb(main):050:0> puts get('/sites', :accept=>"text/yaml")                                                           
   # Details about a site                                                                                            
 irb(main):008:0> puts get('/sites/grenoble.yaml')                                                                   
   # Resources details on a site or a cluster                                                                        
 irb(main):022:0> puts get('/sites/simpsons/resources.yaml')

 # Getting jobs infos of a site                                                                                      
 irb(main):006:0> puts get('/sites/grenoble/jobs.yaml')                                                              
 irb(main):009:0> puts get('/sites/grenoble/jobs/12.yaml')

 # Submiting a job on a given site (using YAML format)                                                               
 irb(main):010:0> require 'yaml'                                                                                     
 irb(main):012:0> j={ 'resource' => '/nodes=2/cpu=1', 'script_path' => '/usr/bin/id' }                               
 irb(main):015:0> job=post('/sites/grenoble/jobs' , j.to_yaml , :content_type => 'text/yaml')
 irb(main):016:0> puts job

 # Getting details about the previously submited job
 irb(main):035:0> puts get(YAML::load(job)['uri'])

 # Submitting a job using JSON format, but requiring the result in YAML
 irb(main):010:0> require 'json'
 irb(main):037:0> job=post('/sites/grenoble/jobs.yaml' , j.to_json , :content_type => 'application/json')

 # Submitting a grid job
 irb(main):029:0> j={ 'resources' => 'grenoble:rdef=/cpu=2,lille:rdef=/cpu=2' }
 irb(main):030:0> job=post('/grid/jobs.yaml' , j.to_json , :content_type => 'application/json')
 irb(main):031:0> puts job

 # List the resources I obtained (if the state is ok)
 irb(main):061:0> job=YAML::load(job)
 irb(main):062:0> if job['state'] == 'submitted'
 irb(main):063:1>   puts get(job['resources'],:accept => 'text/yaml')
 irb(main):064:1> else
 irb(main):065:1*   puts "Job was rejected"
 irb(main):066:1> end

 # Get an array of node names inside a grid job
 irb(main):063:1>   puts get('/grid/jobs/18611/resources/nodes.json')

 # List my running grid jobs, in HTML
 irb(main):067:0> puts get('/grid/jobs.html')

 # Show details about a grid job, in JSON
 irb(main):069:0> puts get('/grid/jobs/18611.json')

 # Deleting a grid job
 irb(main):111:0> delete("/grid/jobs/18611.yaml")

== A Ruby script using restclient ==
 #!/usr/bin/ruby                                                                                               
 # Example of a ruby script using restclient on the Oargrid RESTfull API.                                      
 # It first parses the grid to find 3 sites where 1 resource is free and                                       
 # then submits a job and prints the properties of the nodes obtained.                                         
 # The job is deleted at the end, as it is just an example :-)                                                 
                                                                                                               
 require 'rubygems'                                                                                            
 require 'rest_client'                                                                                         
 require 'json'                                                                                                
 require 'pp'                                                                                                  
                                                                                                               
 # Custom variables                                                                                            
 APIURI="http://www.grenoble.grid5000.fr/oargridapi"                                                           
 IGNORE_SITES=['grenoble-obs','grenoble-exp','grenoble-ext','sophia','lille']                                  
 N_RESOURCES=3                                                                                                 
                                                                                                               
 # Function to get objects from the api                                                                        
 # We use the JSON format and the 'simple' data structure                                                      
 def get(api,uri)                                                                                              
   begin                                                                                                       
     return JSON.parse(api[uri+'?structure=simple'].get(:accept => 'application/json'))                        
   rescue => e                                                                                                 
     if e.respond_to?('http_code')                                                                             
       puts "ERROR #{e.http_code}:\n #{e.response.body}"                                                       
     else                                                                                                      
       puts "Parse error:"                                                                                     
       puts e.inspect                                                                                          
     end                                                                                                       
     exit 1                                                                                                    
   end                                                                                                         
 end                                                                                                           
                                                                                                               
 # Instanciate an api connection                                                                               
 api = RestClient::Resource.new APIURI                                                                         
                                                                                                               
 # Parse the sites and find at least N_RESOURCES free resources                                                
 puts "Getting sites..."                                                                                       
 sites = get(api,'/sites')                                                                                     
 puts "Got #{sites.length} sites"                                                                              
 ok_sites=[]                                                                                                   
 sites.each do |site|                                                                                          
   site_name = site['site']                                                                                    
   unless IGNORE_SITES.index(site_name)                                                                        
     puts "Checking #{site_name}..."                                                                           
     resources = get(api,"/sites/#{site_name}/resources/all")                                                  
     resources.each do |resource|                                                                              
       if resource['state'] == "Alive" && resource['jobs'].nil?                                                
         ok_sites << site_name                                                                                 
        puts "   got 1 resource!"                                                                              
         break                                                                                                 
       end                                                                                                     
     end                                                                                                       
   end                                                                                                         
   break if ok_sites.length >= N_RESOURCES                                                                     
 end                                                                                                           
                                                                                                               
 # If we got N_RESOURCES sites, submit the job                                                                 
 if ok_sites.length < N_RESOURCES                                                                              
   puts "Not enough resources found on the grid!"                                                              
   exit 1                                                                                                      
 end                                                                                                           
 puts "OK, we got #{N_RESOURCES} free resources, let's submit a job..."                                        
 j={ 'resources' => ok_sites.join(':rdef=/resource_id=1,')+':rdef=/resource_id=1' }                            
 #j={ 'resources' => ok_sites.join(':rdef=/resource_id=1,')+':rdef=/resource_id=1' , 'verbose' => 1 }          
 begin                                                                                                         
   job=JSON.parse(api['/grid/jobs'].post(j.to_json,:content_type => 'application/json'))                       
 rescue => e                                                                                                   
     puts "ERROR #{e.http_code}:\n #{JSON.parse(e.response.body)['message']}"                                  
     exit 2                                                                                                    
 end                                                                                                           
                                                                                                               
 # Check the job                                                                                               
 if job['state'] == "submitted"                                                                                
   puts "GRID JOB SUCCESSFUL :-)"                                                                              
   puts "-----------------------------------------------"                                                      
   puts "Id: #{job['id']}"                                                                                     
   puts "Waiting for resources to be allocated..."                                                             
 
 # Wait for allocated resources
   nodes=[]
   while nodes.length < ok_sites.length do
     nodes=get(api,"/grid/jobs/#{job['id']}/resources/nodes")
     sleep(2)
   end
 
 # Ok, print nodes
   puts "Ok, all nodes are there:"
   puts nodes.join(',')
 
 # Fetching properties of the nodes
   puts "Properties:"
   resources=get(api,"/grid/jobs/#{job['id']}/resources")
   resources.each do |r|
     site=r['site']
     r['jobs'].each_value do |j|
       j['nodes'].each do |node|
         puts "\n" + node + " : "
         prop=get(api,"/sites/#{site}/resources/nodes/#{node}")
         pp prop
       end
     end
   end
 
 # Delete the job
   begin
     puts "Deleting job because it was just for fun..."
     api["/grid/jobs/#{job['id']}.json"].delete
     puts "Job deleted"
   rescue => e
     puts "ERROR #{e.http_code} DELETING THE GRID JOB :\n #{JSON.parse(e.response.body)['message']}"
   end
 
 # If the job has failed:
 else
   puts "PROBLEM: Grid job #{job['state']}"
   puts "OUTPUT MESSAGE:"
   puts job['output']
 end