Test Suite GSOC 2010
From WikiOAR
Student and mentor ;), please read carefully this page...
Student: Narayanan K
Mentor: Joseph Emeras
Co-Mentor: Yiannis Georgiou
Student: Things to do before starting
-
Subscribe to oar commits mailing list -
Play with Kameleon (generate a OAR/Debian appliance) -
Practice OAR RESTfull API (using the OAR appliance) -
Compute a summary of available tools for functional testing in ruby
Main Milestones
- July 5 - 15: Implement the testsuite engine and recipe to test the basic REST OAR APIs and few scenario testings.
What are we testing and How?
The main aim of this project is to test the available OAR REST APIs to ensure that they are working as expected without any broken links while calling the APIs for implementing various functionalities.
A list of available OAR REST APIs is present in the Documentation which can be pulled from http://oar.imag.fr/archive/wiki-oar/index.php/OAR_API
Both unit testing as well and functional(Scenario) testing will be done on the OAR REST APIs.
So, we currently make available 2 ways of testing the APIs. They are as follows:
1. Regular RSpec Testing
2. Testing using Kameleon Tool, IRB and RSpec
So users of the APIs who are familiar with RSpec Test framework can use the first method of testing, while those developers who are fond of Kameleon tool and love to write recipes and steps can pick up the second option for testing the APIs.
In this Wiki, we cover the basics of how testcases can be written, by using both the methods mentioned above.
Before launching onto the 2 testing methods, let us understand about the RSpec Ruby Testing Framework which is common in both the methodologies. We cover that next.
RSpec Ruby Test Framework
Rspec is a behavior driven test development framework developed for Ruby.
Behavior Driven Development(BDD) began its journey as an attempt to better understand and explain the process of Test Driven Development.
Behavior driven development is an agile software development technique .
RSpec uses simple plain English like statements to write test cases.
In Testsuites project, we use RSpec to carry out unit/scenario testing of the various OAR REST APIs.
RSpec Documentation with examples of simple scenario testing of OAR APIs can be pulled from gforge scm - branch/gsoc2010/tests/testsuite/Documentation.
Regular RSpec Testing Method
This is the most easiest and quickest way to test the OAR APIs.
Once testing using RSpec framework is clear, one can write very efficient testcases (testelements)for scenarios/unit functioning of APIs in a few minutes and run them on the fly using the spec command.
A Testsuite can be made by placing all the test files (must end with _spec.rb) in a spec folder.
By running the command: spec path_to_spec_folder --format specdoc , each test spec file in the directory will get executed one by one displaying the result for each test file after it is run.
The RSpec Documentation I made, clarifies how this method of testing can be done with examples. But however for the sake of completeness, I will describe a small example of testing of APIs using simple RSpec testelements.
Following are testelements to submit a job successfully and check the queue if job has been submitted. Note: jobid is made global so that all testelements can test APIs that require this jobid. Also submit_job and full_job_details are methods of library oarrestapi that calls the POST/GET APIs which are called using the object of the library class. Read RSpec Documentation for more details. Assume the elements are from file testspec.rb.
#Submitting a job
it "should submit a job successfully " do
jhash = { 'resource' => "/nodes=1/core=1" , 'script' => "ls;pwd;whoami;sleep 60" }
begin
@obj.submit_job(jhash)
$jobid = @obj.jobstatus['id'].to_s
rescue
puts "#{$!}"
exit
end
$!.should == "" #Usually returns NULL or empty string
@obj.jobstatus['status'].to_s.should == "submitted"
end
#Checking the queue (Can use GET /jobs to check) immediately.
it "should contain jobid in queue of created job" do
begin
@obj.full_job_details
rescue
puts "#{$!}"
exit
end
@obj.jobarray['items'] do |value|
if value['job_id']== $jobid
@c=1
end
end
$!.should == ""
@c.should == 1
end
The testelement passes only if each should or should_not methods inside it are succeeding. If one fails, the testelement fails.
Running using spec : $ spec testspec.rb --format specdoc
OarApis
- should submit a job successfully
- should contain jobid in queue of created job
Finished in 03.062151 seconds
2 examples, 0 failures
Thus tests can be written and run easily as shown above.
The next method is for Kameleon fans and is described elaborately in the sections that follow.
The Kameleon way of Testing: Idea
Main Tools used in this Idea:
- Kameleon
- Interactive Ruby (IRB)
- RSpec Ruby Gem - A Ruby BDD Testing framework
Tests will be written in simple yaml file format. Kameleon engine can be used to parse the yaml test files.
Recipe Test file will contain macrosteps and microsteps.
The actual testing is done using the RSpec Testing Framework internally.
An IRB Session is opened up in one of the microsteps. Named FIFO pipes are used for giving input and collecting back output/error to and from the IRB respectively.
A library of RSpec test elements of the REST OAR APIs along with few scenario testing are made. This library is loaded in IRB in one of the microsteps and testings mapped with the corresponding macrosteps are run in IRB.
Ruby commands to load RSpec test library and to run the tests are given as input to the IRB via the named FIFO pipe. Similarly, output/errors of these tests run in IRB are collected back in other named pipes.
Testsuite Installation
The Testsuite Engine Design
An overall tentative architecture of the testsuite has been drawn and committed to the Inria SVN Repository (gsoc2010/tests/testsuite/Design).
Advantages
- Test files can be easily written as it is in the standard YAML format.
- Usage of Kameleon engine will make parsing of test files quicker, powerful and efficient. All features of kameleon is derived here.
- Inner test cases made using RSpec, which is easier to learn, code; takes just few minutes to add a new spec test element to the rspec test library.
- Since each test is carried out in IRB through kameleon, debugging is easy at each step.
Sample Test Recipe snippet
A sample steps of test recipe will tentatively look like:
steps:
- irb #Initializes IRB
- load_lib_create_object #Load rspectests_lib and creates object of the lib
- unit_tests:
- test_submit_job #Test whether the post /jobs REST API is submitting job successfully
- test_get_job #Test whether the get /jobs API is getting details of job
- test_delete_job #Test whether the delete /jobs API is fine
- test_get_version #Test whether GET /version is working fine
- test_get_timezone #Test whether GET /timezone is working fine
#
# Scenario Testing #Test Scenario 1: Submit a job, check if submitted, Delete job, Check if deleted
- scenario1:
- submit_job
- test_if_submitted
- delete_job
- test_if_deleted
#- print_output #Only 1 print_output should be there in a recipe. It should be put only after all the tests/scenarios
#
- scenario2: #Test Scenario 2: Test to Submit job, Hold job, Resume job
- submit_job_running
- test_if_submitted
- hold_job_running
- test_if_job_held
- resume_job
- test_if_job_resumed
- print_output #Print output of tests
The irb.yaml macrostep:
irb:
- create_pipes:
- exec_appliance: mkfifo irb_stdout #Named Fifo pipe to store the output produced after testing in IRB
- exec_appliance: mkfifo irb_stdin #Named Fifo pipe which accepts input for IRB
- exec_appliance: mkfifo irb_stderr #Named Fifo pipe to log the errors produced after testing in IRB
- start_irb:
- exec_appliance: irb <irb_stdin >irb_stdout &
- exec_appliance: IRB_PID=$!
- bash_function:
- exec_appliance: function write_irb(){ echo "$1" > irb_stdin; }
- clean:
- exec_appliance: echo "kill $IRB_PID" >> $$workdir/clean.sh
- exec_appliance: echo "rm -f $$workdir/chroot/irb_stdout" >> $$workdir/clean.sh
- exec_appliance: echo "rm -f $$workdir/chroot/irb_stdin" >> $$workdir/clean.sh
- exec_appliance: echo "rm -f $$workdir/chroot/irb_stderr" >> $$workdir/clean.sh
This macrostep creates named pipes for input, output and error and initializes a session of IRB. The clean step adds rm -f commands that delete pipes into clean.sh
The load_lib_create_object step contains the following microsteps:
load_lib_create_object:
- load_lib_create_object:
- load_libs:
- exec_appliance: write_irb "require '$$rpsec_lib_path'"
- create_objects:
- exec_appliance: write_irb "obj = Tests.new($$apiuri)"
Here, the microsteps uses 2 Global variables declared in the Kameleon recipe file which are referenced using $$.They are:
RSpec Library Path and Apiuri Path.
rpsec_lib_path: /home/kameleon/lib/rspectests_lib
apiuri: "http://kameleon:kameleon@localhost/oarapi-priv"
The test_submit_job macrostep will contain the following microsteps:
test_submit_job:
- test_submit_job:
- exec_appliance: write_irb "jhash = { 'resource' => \"/nodes=1/core=1\" , 'script' => \"ls;pwd;whoami\" }"
- exec_appliance: write_irb "obj.test_submit_job(jhash)"
Here, the load_lib_create_object step imports the RSpec Test library file and creates an object of RSpec class, Test. It uses this object to explicitly call the test_submit_job() ruby method which contains the RSpec test element and runs the test element separately.
The scenario1 macrostep will contain the following microsteps:
scenario1:
- submit_job:
- exec_appliance: write_irb "jhash = { 'resource' => \"/nodes=1/core=1\" , 'script' => \"ls;pwd;whoami;sleep 60\" }"
- test_if_submitted:
- exec_appliance: write_irb "obj.test_submit_job(jhash)"
- exec_appliance: write_irb "jid = obj.jobid"
- exec_appliance: write_irb "puts \"Submitted job \"+jid.to_s"
- exec_appliance: write_irb "obj.test_job_in_queue(jid)"
- delete_job:
- exec_appliance: write_irb "puts \"Deleting job \"+jid.to_s"
- exec_appliance: write_irb "puts jid.to_s"
- test_if_deleted:
- exec_appliance: write_irb "obj.test_jobs_delete(jid)"
- exec_appliance: write_irb "obj.test_job_notin_queue(jid)"
- exec_appliance: write_irb "puts \"Scenario1 successfully tested\""
The print_output macrostep contains the following microsteps:
print_output:
- print:
- exec_appliance: write_irb "exit"
- exec_appliance: sleep 1
- exec_appliance: cat irb_stdout|tee output_file
- exec_appliance: echo "Results of the Testsuite Run"
- exec_appliance: grep -C 1 "OarApis" output_file
- exec_appliance: grep failure output_file>numexamples
- exec_appliance: cat numexamples|wc -l|sed 's/^/Total Number of examples run:/'
- exec_appliance: awk -F '[ ,]' '{static s; if($4 == 0) s++}; END { print s }' numexamples|sed 's/^/Number of Successes:/'
- exec_appliance: echo "Time taken to run each tests"
- exec_appliance: grep seconds output_file|tee testtime
- exec_appliance: echo "Total Time taken:"
- exec_appliance: awk '{ static timesum; timesum+=$3 }; END { print timesum }' testtime
Only 1 print_output microstep should be present in the recipe file. If 2 scenario testings needs to be done,place print_output step after 2nd scenario. This will contain the output of tests of both the scenarios.
Notice here that print_output microstep is rather big. The intention was to add more creativity in the output after the test run. It can also be rather small and simple; depends on the requirements of the tester.
Schema
List of rspectests_lib methods
The rspectests_lib currently contains the following methods/rspec test elements. As and when more lib methods are required, it can be easily added to the library. The OAR REST APIs are called by this library using the oarrestapi_lib library. For writing tests, one has the flexibility to require either rspectests_lib or oarrestapi_lib based on their specific requirements.
Class: Test
Methods:
1. test_get_version - Testing the GET /version REST API 2. test_get_timezone - Testing the GET /timezone REST API 3. test_get_jobs_details - Testing the GET /jobs/details REST API 4. test_get_running_jobs - Testing the GET /jobs REST API 5. test_get_jobs_id (jid) - Testing the GET /jobs/<ID> REST API 6. test_job_in_queue(jid) - Test to check if job is there in queue GET /jobs/details API 7. test_job_notin_queue(jid) - Test to check if job is deleted from queue using GET /jobs/details API 8. test_get_jobs_table - Testing the GET /jobs/table REST API 9. test_submit_job (jhash) - Testing the POST /jobs REST API 10. test_jobs_delete_post (jid) - Testing the POST /jobs/id/deletions/new REST API 11. test_jobs_delete (jid) - Testing the DELETE /jobs/<id> REST API 12. test_get_resources - Testing the GET /resources REST API 13. test_get_resources_full - Testing the GET /resources/full REST API 14. test_job_rholds (jid) - Testing the POST /jobs/<id>/rholds/new REST API 15. test_job_hold (jid) - Testing the POST /jobs/<jobid>/holds/new REST API 16. test_job_resumption (jid) - Testing the POST /jobs/<id>/resumption/new REST API 17. test_job_update (jid, actionhash) - Testing POST /jobs/<id>/ API (deleting use when browsers dont support DELETE) 18. test_if_job_delete_updated (jid) - Testing POST /jobs/<id>/ to see if job is updated with actionhash, Call this after calling above method 19. test_job_checkpoint (jid) - Testing the POST /jobs/<jobid>/checkpoints/new REST API 20. test_job_running (jid) - Testing if job is currently running
List of oarrestapi_lib methods
The rspectests_lib call the OAR REST APIs through the oarrestapi_lib library.
Class: OarApi
Methods:
1. get(api,uri) - GET REST OAR API; Function to get objects from the api 2. post(api,uri,j) - POST REST OAR API; Function to create/delete/hold/resume objects through the api 3. delete(api, uri) - DELETE REST OAR API; Function to Delete objects through the api 4. oar_version - Gives version info & Timezone about OAR and OAR API/Server. 5. oar_timezone - Gives the timezone of the OAR API server. 6. full_job_details - List the current jobs & some details like assigned resources 7. run_job_details - List currently running jobs 8. specific_job_details(jobid) - Get Details of a specific job 9. dump_job_table - Dump the jobs table (only current jobs) 10. submit_job(jhash) - Submits job 11. del_job(jobid) - Delete job - POST /jobs/id/deletions/new 12. send_checkpoint(jobid) - Send checkpoint signal to a job 13. hold_waiting_job(jobid) - Hold a Waiting job 14. hold_running_job(jobid) - Hold a Running job 15. resume_hold_job(jobid) - Resume a Holded job 16. send_signal_job(jobid, signo) - Send signal to a job with signalno. 17. update_job(jobid, actionhash) - Update a job 18. resource_list_state - Get list of Resources and state 19. list_resource_details - Get list of all the resources and all their details 20. specific_resource_details(jobid) - Get details of resources identified by an ID 21. resource_of_nodes(netaddr) - Get details about the resources belonging to the node identified by network address 22. create_resource(rhash) - Create Resource 23. statechange_resource(jid,harray) - Change the state of resources of a job 24. delete_job(jobid) - Delete or kill a job using DELETE API 25. delete_resource(resid) - Delete the resource identified by id 26. delete_resource_cpuset(node,cpuid)- Delete the resource corresponding to cpuset id on node node.
TODO list
-
Create a library calling various OAR REST APIs, with Error Management. -
Find out how named pipes can be used to provide inputs and accept output/errors from IRB -
Create a library of RSpec tests with basic and scenario testing of the OAR APIs. -
Change names of some methods and classes to meet OAR Coding conventions. -
Make APIURI as a global variable in kameleon recipe. Also make path of libs used in "require" as a kameleon global variable . -
Split the tests.yaml to 2, lib and unit_test; lib will contain load_lib and create_object. Create a separate print_output step. -
Make more detailed schema to show how the various layers interact to give the test results..(From recipe to IRB and so on) -
Write Testcases in simple RSpec Files and compare the speed of execution via kameleon and from without kameleon. -
Test testsuite codes, recipes and steps in a virgin appliance to see if issue due to non-availability of .irbrc file occurs. If occurs, next ToDo. -
Find out how to set IRBRC environment variable to point to .irbrc configuration file in a generic path under /, overriding the default search in ~/.irbrc.
