Student and mentor ;), please read carefully this page…
Student: Narayanan K
Mentor: Joseph Emeras
Co-Mentor: Yiannis Georgiou
The main aim of this project is to test the available OAR REST APIs to ensure that they are working as expected without any broken links while calling the APIs for implementing various functionalities.<br /> A list of available OAR REST APIs is present in the Documentation which can be pulled from http://wiki-oar.imag.fr/index.php/OAR_API <br /> Both unit testing as well and functional(Scenario) testing will be done on the OAR REST APIs.<br /> So, we currently make available 2 ways of testing the APIs. They are as follows:<br /> 1. Regular RSpec Testing<br /> 2. Testing using Kameleon Tool, IRB and RSpec<br />
So users of the APIs who are familiar with RSpec Test framework can use the first method of testing, while those developers who are fond of Kameleon tool and love to write recipes and steps can pick up the second option for testing the APIs.<br />
In this Wiki, we cover the basics of how testcases can be written, by using both the methods mentioned above.<br /> Before launching onto the 2 testing methods, let us understand about the RSpec Ruby Testing Framework which is common in both the methodologies. We cover that next.<br />
Rspec is a behavior driven test development framework developed for Ruby.<br /> Behavior Driven Development(BDD) began its journey as an attempt to better understand and explain the process of Test Driven Development.<br /> Behavior driven development is an agile software development technique .<br /> RSpec uses simple plain English like statements to write test cases. <br /> In Testsuites project, we use RSpec to carry out unit/scenario testing of the various OAR REST APIs.<br /> RSpec Documentation with examples of simple scenario testing of OAR APIs can be pulled from gforge scm - branch/gsoc2010/tests/testsuite/Documentation.
This is the most easiest and quickest way to test the OAR APIs.<br /> Once testing using RSpec framework is clear, one can write very efficient testcases (testelements)for scenarios/unit functioning of APIs in a few minutes and run them on the fly using the spec command.<br /> A Testsuite can be made by placing all the test files (must end with _spec.rb) in a spec folder. <br /> By running the command: spec path_to_spec_folder –format specdoc , each test spec file in the directory will get executed one by one displaying the result for each test file after it is run.<br />
The RSpec Documentation I made, clarifies how this method of testing can be done with examples. But however for the sake of completeness, I will describe a small example of testing of APIs using simple RSpec testelements.
Following are testelements to submit a job successfully and check the queue if job has been submitted. Note: jobid is made global so that all testelements can test APIs that require this jobid. Also submit_job and full_job_details are methods of library oarrestapi that calls the POST/GET APIs which are called using the object of the library class. Read RSpec Documentation for more details. Assume the elements are from file testspec.rb.
#Submitting a job
it "should submit a job successfully " do
jhash = { 'resource' => "/nodes=1/core=1" , 'script' => "ls;pwd;whoami;sleep 60" }
begin
@obj.submit_job(jhash)
$jobid = @obj.jobstatus[['id']].to_s
rescue
puts "#{$!}"
exit
end
$!.should == "" #Usually returns NULL or empty string
@obj.jobstatus[['status']].to_s.should == "submitted"
end
#Checking the queue (Can use GET /jobs to check) immediately.
it "should contain jobid in queue of created job" do
begin
@obj.full_job_details
rescue
puts "#{$!}"
exit
end
@obj.jobarray[['items']] do |value|
if value[['job_id']]== $jobid
@c=1
end
end
$!.should == ""
@c.should == 1
end
The testelement passes only if each should or should_not methods inside it are succeeding. If one fails, the testelement fails.<br /> Running using spec : $ spec testspec.rb –format specdoc <br /> OarApis<br /> - should submit a job successfully <br /> - should contain jobid in queue of created job <br /> Finished in 03.062151 seconds <br /> 2 examples, 0 failures<br />
Thus tests can be written and run easily as shown above. <br /> The next method is for Kameleon fans and is described elaborately in the sections that follow.
Main Tools used in this Idea:
Tests will be written in simple yaml file format. Kameleon engine can be used to parse the yaml test files. <br /> Recipe Test file will contain macrosteps and microsteps. <br /> The actual testing is done using the RSpec Testing Framework internally.<br /> An IRB Session is opened up in one of the microsteps. Named FIFO pipes are used for giving input and collecting back output/error to and from the IRB respectively. A library of RSpec test elements of the REST OAR APIs along with few scenario testing are made. This library is loaded in IRB in one of the microsteps and testings mapped with the corresponding macrosteps are run in IRB.<br /> Ruby commands to load RSpec test library and to run the tests are given as input to the IRB via the named FIFO pipe. Similarly, output/errors of these tests run in IRB are collected back in other named pipes. <br />
An overall tentative architecture of the testsuite has been drawn and committed to the Inria SVN Repository (gsoc2010/tests/testsuite/Design).<br /> Testsuite_Design.jpg
A sample steps of test recipe will tentatively look like:
steps:
- irb #Initializes IRB - load_lib_create_object #Load rspectests_lib and creates object of the lib - unit_tests: - test_submit_job #Test whether the post /jobs REST API is submitting job successfully - test_get_job #Test whether the get /jobs API is getting details of job - test_delete_job #Test whether the delete /jobs API is fine - test_get_version #Test whether GET /version is working fine - test_get_timezone #Test whether GET /timezone is working fine # # Scenario Testing #Test Scenario 1: Submit a job, check if submitted, Delete job, Check if deleted - scenario1: - submit_job - test_if_submitted - delete_job - test_if_deleted #- print_output #Only 1 print_output should be there in a recipe. It should be put only after all the tests/scenarios # - scenario2: #Test Scenario 2: Test to Submit job, Hold job, Resume job - submit_job_running - test_if_submitted - hold_job_running - test_if_job_held - resume_job - test_if_job_resumed - print_output #Print output of tests
The irb.yaml macrostep: <br /> irb:
- create_pipes:
- exec_appliance: mkfifo irb_stdout #Named Fifo pipe to store the output produced after testing in IRB
- exec_appliance: mkfifo irb_stdin #Named Fifo pipe which accepts input for IRB
- exec_appliance: mkfifo irb_stderr #Named Fifo pipe to log the errors produced after testing in IRB
- start_irb:
- exec_appliance: irb <irb_stdin >irb_stdout &
- exec_appliance: IRB_PID=$!
- bash_function:
- exec_appliance: function write_irb(){ echo "$1" > irb_stdin; }
- clean:
- exec_appliance: echo "kill $IRB_PID" >> $$workdir/clean.sh
- exec_appliance: echo "rm -f $$workdir/chroot/irb_stdout" >> $$workdir/clean.sh
- exec_appliance: echo "rm -f $$workdir/chroot/irb_stdin" >> $$workdir/clean.sh
- exec_appliance: echo "rm -f $$workdir/chroot/irb_stderr" >> $$workdir/clean.sh
This macrostep creates named pipes for input, output and error and initializes a session of IRB. The clean step adds rm -f commands that delete pipes into clean.sh
The load_lib_create_object step contains the following microsteps:
load_lib_create_object:
- load_lib_create_object:
- load_libs:
- exec_appliance: write_irb "require '$$rpsec_lib_path'"
- create_objects:
- exec_appliance: write_irb "obj = Tests.new($$apiuri)"
Here, the microsteps uses 2 Global variables declared in the Kameleon recipe file which are referenced using $$.They are:<br /> RSpec Library Path and Apiuri Path.<br /> rpsec_lib_path: /home/kameleon/lib/rspectests_lib <br /> apiuri: “http://kameleon:kameleon@localhost/oarapi-priv”
The test_submit_job macrostep will contain the following microsteps:
test_submit_job:
- test_submit_job:
- exec_appliance: write_irb "jhash = { 'resource' => \\"/nodes=1/core=1\\" , 'script' => \\"ls;pwd;whoami\\" }"
- exec_appliance: write_irb "obj.test_submit_job(jhash)"
Here, the load_lib_create_object step imports the RSpec Test library file and creates an object of RSpec class, Test. It uses this object to explicitly call the test_submit_job() ruby method which contains the RSpec test element and runs the test element separately.
The scenario1 macrostep will contain the following microsteps:
scenario1:
- submit_job:
- exec_appliance: write_irb "jhash = { 'resource' => \\"/nodes=1/core=1\\" , 'script' => \\"ls;pwd;whoami;sleep 60\\" }"
- test_if_submitted:
- exec_appliance: write_irb "obj.test_submit_job(jhash)"
- exec_appliance: write_irb "jid = obj.jobid"
- exec_appliance: write_irb "puts \\"Submitted job \\"+jid.to_s"
- exec_appliance: write_irb "obj.test_job_in_queue(jid)"
- delete_job:
- exec_appliance: write_irb "puts \\"Deleting job \\"+jid.to_s"
- exec_appliance: write_irb "puts jid.to_s"
- test_if_deleted:
- exec_appliance: write_irb "obj.test_jobs_delete(jid)"
- exec_appliance: write_irb "obj.test_job_notin_queue(jid)"
- exec_appliance: write_irb "puts \\"Scenario1 successfully tested\\""
The print_output macrostep contains the following microsteps:
print_output:
- print:
- exec_appliance: write_irb "exit"
- exec_appliance: sleep 1
- exec_appliance: cat irb_stdout|tee output_file
- exec_appliance: echo "Results of the Testsuite Run"
- exec_appliance: grep -C 1 "OarApis" output_file
- exec_appliance: grep failure output_file>numexamples
- exec_appliance: cat numexamples|wc -l|sed 's/^/Total Number of examples run:/'
- exec_appliance: awk -F '[[|,]]' '{static s; if($4 == 0) s++}; END { print s }' numexamples|sed 's/^/Number of Successes:/'
- exec_appliance: echo "Time taken to run each tests"
- exec_appliance: grep seconds output_file|tee testtime
- exec_appliance: echo "Total Time taken:"
- exec_appliance: awk '{ static timesum; timesum+=$3 }; END { print timesum }' testtime
Only 1 print_output microstep should be present in the recipe file. If 2 scenario testings needs to be done,place print_output step after 2nd scenario. This will contain the output of tests of both the scenarios.
Notice here that print_output microstep is rather big. The intention was to add more creativity in the output after the test run. It can also be rather small and simple; depends on the requirements of the tester.
The rspectests_lib currently contains the following methods/rspec test elements. As and when more lib methods are required, it can be easily added to the library. The OAR REST APIs are called by this library using the oarrestapi_lib library. For writing tests, one has the flexibility to require either rspectests_lib or oarrestapi_lib based on their specific requirements.
Class: Test
Methods:
1. test_get_version - Testing the GET /version REST API 2. test_get_timezone - Testing the GET /timezone REST API 3. test_get_jobs_details - Testing the GET /jobs/details REST API 4. test_get_running_jobs - Testing the GET /jobs REST API 5. test_get_jobs_id (jid) - Testing the GET /jobs/<ID> REST API 6. test_job_in_queue(jid) - Test to check if job is there in queue GET /jobs/details API 7. test_job_notin_queue(jid) - Test to check if job is deleted from queue using GET /jobs/details API 8. test_get_jobs_table - Testing the GET /jobs/table REST API 9. test_submit_job (jhash) - Testing the POST /jobs REST API 10. test_jobs_delete_post (jid) - Testing the POST /jobs/id/deletions/new REST API 11. test_jobs_delete (jid) - Testing the DELETE /jobs/<id> REST API 12. test_get_resources - Testing the GET /resources REST API 13. test_get_resources_full - Testing the GET /resources/full REST API 14. test_job_rholds (jid) - Testing the POST /jobs/<id>/rholds/new REST API 15. test_job_hold (jid)\t - Testing the POST /jobs/<jobid>/holds/new REST API 16. test_job_resumption (jid) - Testing the POST /jobs/<id>/resumption/new REST API 17. test_job_update (jid, actionhash) - Testing POST /jobs/<id>/ API (deleting use when browsers dont support DELETE) 18. test_if_job_delete_updated (jid) - Testing POST /jobs/<id>/ to see if job is updated with actionhash, Call this after calling above method 19. test_job_checkpoint (jid) - Testing the POST /jobs/<jobid>/checkpoints/new REST API 20. test_job_running (jid) - Testing if job is currently running
The rspectests_lib call the OAR REST APIs through the oarrestapi_lib library.
Class: OarApi
Methods:
1. get(api,uri) - GET REST OAR API; Function to get objects from the api 2. post(api,uri,j)\t\t - POST REST OAR API; Function to create/delete/hold/resume objects through the api 3. delete(api, uri)\t\t - DELETE REST OAR API; Function to Delete objects through the api 4. oar_version\t\t - Gives version info & Timezone about OAR and OAR API/Server. 5. oar_timezone\t\t - Gives the timezone of the OAR API server. 6. full_job_details\t\t - List the current jobs & some details like assigned resources 7. run_job_details - List currently running jobs 8. specific_job_details(jobid) - Get Details of a specific job 9. dump_job_table\t\t - Dump the jobs table (only current jobs) 10. submit_job(jhash)\t\t - Submits job 11. del_job(jobid) - Delete job - POST /jobs/id/deletions/new 12. send_checkpoint(jobid) - Send checkpoint signal to a job 13. hold_waiting_job(jobid)\t - Hold a Waiting job 14. hold_running_job(jobid) - Hold a Running job 15. resume_hold_job(jobid) - Resume a Holded job 16. send_signal_job(jobid, signo) - Send signal to a job with signalno. 17. update_job(jobid, actionhash) - Update a job 18. resource_list_state - Get list of Resources and state 19. list_resource_details - Get list of all the resources and all their details 20. specific_resource_details(jobid) - Get details of resources identified by an ID 21. resource_of_nodes(netaddr) - Get details about the resources belonging to the node identified by network address 22. create_resource(rhash) - Create Resource 23. statechange_resource(jid,harray) - Change the state of resources of a job 24. delete_job(jobid)\t\t - Delete or kill a job using DELETE API 25. delete_resource(resid) - Delete the resource identified by id 26. delete_resource_cpuset(node,cpuid)- Delete the resource corresponding to cpuset id on node node.