This shows you the differences between two versions of the page.
— | wiki:old:gsoc_2010_test_suite [2013/07/10 22:55] (current) – created - external edit 127.0.0.1 | ||
---|---|---|---|
Line 1: | Line 1: | ||
+ | **Student and mentor ;), please read carefully this page...** | ||
+ | |||
+ | Student: **Narayanan K** | ||
+ | |||
+ | Mentor: Joseph Emeras | ||
+ | |||
+ | Co-Mentor: Yiannis Georgiou | ||
+ | |||
+ | ---- | ||
+ | |||
+ | ===== Student: Things to do before starting ===== | ||
+ | * < | ||
+ | * < | ||
+ | * < | ||
+ | * < | ||
+ | |||
+ | ===== Main Milestones ===== | ||
+ | * July 5 - 15: Implement the testsuite engine and recipe to test the basic REST OAR APIs and few scenario testings. | ||
+ | |||
+ | ===== What are we testing and How? ===== | ||
+ | The main aim of this project is to test the available OAR REST APIs to ensure that they are working as expected without any broken links while calling the APIs for implementing various functionalities.< | ||
+ | A list of available OAR REST APIs is present in the Documentation which can be pulled from http:// | ||
+ | Both **unit testing** as well and **functional(Scenario)** testing will be done on the OAR REST APIs.<br /> | ||
+ | So, we currently make available 2 ways of testing the APIs. They are as follows:< | ||
+ | 1. **Regular RSpec Testing**< | ||
+ | 2. **Testing using Kameleon Tool, IRB and RSpec**< | ||
+ | |||
+ | So users of the APIs who are familiar with RSpec Test framework can use the first method of testing, while those developers who are fond of Kameleon tool and love to write recipes and steps can pick up the second option for testing the APIs.<br /> | ||
+ | |||
+ | In this Wiki, we cover the basics of how testcases can be written, by using both the methods mentioned above.< | ||
+ | Before launching onto the 2 testing methods, let us understand about the RSpec Ruby Testing Framework which is common in both the methodologies. We cover that next.<br /> | ||
+ | |||
+ | ===== RSpec Ruby Test Framework ===== | ||
+ | |||
+ | **Rspec** is a behavior driven test development framework developed for Ruby.<br /> | ||
+ | Behavior Driven Development(**BDD**) began its journey as an attempt to better understand and explain the process of Test Driven Development.< | ||
+ | Behavior driven development is an agile software development technique .<br /> | ||
+ | RSpec uses simple plain English like statements to write test cases. <br /> | ||
+ | In Testsuites project, we use RSpec to carry out **unit/ | ||
+ | **RSpec Documentation** with examples of simple scenario testing of OAR APIs can be pulled from gforge scm - branch/ | ||
+ | |||
+ | ===== Regular RSpec Testing Method ===== | ||
+ | This is the most easiest and quickest way to test the OAR APIs.<br /> | ||
+ | Once testing using RSpec framework is clear, one can write very efficient testcases (testelements)for scenarios/ | ||
+ | A Testsuite can be made by placing all the test files (must end with _spec.rb) in a spec folder. <br /> | ||
+ | By running the command: **spec path_to_spec_folder --format specdoc** , each test spec file in the directory will get executed one by one displaying the result for each test file after it is run.<br /> | ||
+ | |||
+ | The RSpec Documentation I made, clarifies how this method of testing can be done with examples. But however for the sake of completeness, | ||
+ | |||
+ | Following are testelements to submit a job successfully and check the queue if job has been submitted. Note: jobid is made global so that all testelements can test APIs that require this jobid. Also submit_job and full_job_details are methods of library oarrestapi that calls the POST/GET APIs which are called using the object of the library class. Read RSpec Documentation for more details. Assume the elements are from file testspec.rb. | ||
+ | |||
+ | < | ||
+ | #Submitting a job | ||
+ | it " | ||
+ | jhash = { ' | ||
+ | begin | ||
+ | @obj.submit_job(jhash) | ||
+ | $jobid = @obj.jobstatus[[' | ||
+ | rescue | ||
+ | puts "# | ||
+ | exit | ||
+ | end | ||
+ | $!.should == "" | ||
+ | @obj.jobstatus[[' | ||
+ | end | ||
+ | #Checking the queue (Can use GET /jobs to check) immediately. | ||
+ | it " | ||
+ | begin | ||
+ | @obj.full_job_details | ||
+ | rescue | ||
+ | puts "# | ||
+ | exit | ||
+ | end | ||
+ | @obj.jobarray[[' | ||
+ | if value[[' | ||
+ | @c=1 | ||
+ | end | ||
+ | end | ||
+ | $!.should == "" | ||
+ | @c.should == 1 | ||
+ | end </ | ||
+ | |||
+ | The testelement passes only if each should or should_not methods inside it are succeeding. If one fails, the testelement fails.< | ||
+ | Running using spec : $ **spec testspec.rb --format specdoc** <br /> | ||
+ | // | ||
+ | - should submit a job successfully <br /> | ||
+ | - should contain jobid in queue of created job <br /> | ||
+ | Finished in 03.062151 seconds <br /> | ||
+ | 2 examples, 0 failures< | ||
+ | |||
+ | Thus tests can be written and run easily as shown above. <br /> | ||
+ | The next method is for Kameleon fans and is described elaborately in the sections that follow. | ||
+ | |||
+ | ===== The Kameleon way of Testing: Idea ===== | ||
+ | |||
+ | Main Tools used in this Idea: | ||
+ | * **Kameleon** | ||
+ | * **Interactive Ruby (IRB)** | ||
+ | * **RSpec Ruby Gem - A Ruby BDD Testing framework** | ||
+ | |||
+ | Tests will be written in simple yaml file format. Kameleon engine can be used to parse the yaml test files. <br /> | ||
+ | Recipe Test file will contain macrosteps and microsteps. <br /> | ||
+ | The actual testing is done using the RSpec Testing Framework internally.< | ||
+ | An IRB Session is opened up in one of the microsteps. Named FIFO pipes are used for giving input and collecting back output/ | ||
+ | A library of RSpec test elements of the REST OAR APIs along with few scenario testing are made. This library is loaded in IRB in one of the microsteps and testings mapped with the corresponding macrosteps are run in IRB.<br /> | ||
+ | Ruby commands to load RSpec test library and to run the tests are given as input to the IRB via the named FIFO pipe. Similarly, output/ | ||
+ | <br /> | ||
+ | |||
+ | ===== Testsuite Installation ===== | ||
+ | |||
+ | [[Image: | ||
+ | |||
+ | ===== The Testsuite Engine Design ===== | ||
+ | |||
+ | An overall tentative architecture of the testsuite has been drawn and committed to the Inria SVN Repository (gsoc2010/ | ||
+ | [[Image: | ||
+ | |||
+ | ===== Advantages ===== | ||
+ | * Test files can be easily written as it is in the standard YAML format. | ||
+ | * Usage of Kameleon engine will make parsing of test files quicker, powerful and efficient. All features of kameleon is derived here. | ||
+ | * Inner test cases made using RSpec, which is easier to learn, code; takes just few minutes to add a new spec test element to the rspec test library. | ||
+ | * Since each test is carried out in IRB through kameleon, debugging is easy at each step. | ||
+ | |||
+ | ===== Sample Test Recipe snippet ===== | ||
+ | |||
+ | A sample steps of test recipe will tentatively look like: | ||
+ | |||
+ | **steps:** | ||
+ | < | ||
+ | - irb # | ||
+ | - load_lib_create_object | ||
+ | - unit_tests: | ||
+ | - test_submit_job | ||
+ | - test_get_job | ||
+ | - test_delete_job | ||
+ | - test_get_version | ||
+ | - test_get_timezone | ||
+ | # | ||
+ | # Scenario Testing | ||
+ | - scenario1: | ||
+ | - submit_job | ||
+ | - test_if_submitted | ||
+ | - delete_job | ||
+ | - test_if_deleted | ||
+ | #- print_output | ||
+ | # | ||
+ | - scenario2: | ||
+ | - submit_job_running | ||
+ | - test_if_submitted | ||
+ | - hold_job_running | ||
+ | - test_if_job_held | ||
+ | - resume_job | ||
+ | - test_if_job_resumed | ||
+ | - print_output | ||
+ | |||
+ | |||
+ | The irb.yaml macrostep: <br /> | ||
+ | **irb:** | ||
+ | < | ||
+ | - create_pipes: | ||
+ | - exec_appliance: | ||
+ | - exec_appliance: | ||
+ | - exec_appliance: | ||
+ | - start_irb: | ||
+ | - exec_appliance: | ||
+ | - exec_appliance: | ||
+ | - bash_function: | ||
+ | - exec_appliance: | ||
+ | - clean: | ||
+ | - exec_appliance: | ||
+ | - exec_appliance: | ||
+ | - exec_appliance: | ||
+ | - exec_appliance: | ||
+ | |||
+ | This macrostep creates named pipes for input, output and error and initializes a session of IRB. The clean step adds rm -f commands that delete pipes into clean.sh | ||
+ | |||
+ | The **load_lib_create_object** step contains the following microsteps: | ||
+ | |||
+ | **load_lib_create_object: | ||
+ | < | ||
+ | - load_lib_create_object: | ||
+ | - load_libs: | ||
+ | - exec_appliance: | ||
+ | - create_objects: | ||
+ | - exec_appliance: | ||
+ | |||
+ | Here, the microsteps uses 2 Global variables declared in the Kameleon recipe file which are referenced using $$.They are:<br /> | ||
+ | RSpec Library Path and Apiuri Path.<br /> | ||
+ | rpsec_lib_path: | ||
+ | apiuri: " | ||
+ | |||
+ | The **test_submit_job** macrostep will contain the following microsteps: | ||
+ | |||
+ | **test_submit_job: | ||
+ | < | ||
+ | - test_submit_job: | ||
+ | - exec_appliance: | ||
+ | - exec_appliance: | ||
+ | |||
+ | Here, the load_lib_create_object step imports the RSpec Test library file and creates an object of RSpec class, Test. It uses this object to explicitly call the test_submit_job() ruby method which contains the RSpec test element and runs the test element separately. | ||
+ | |||
+ | The **scenario1** macrostep will contain the following microsteps: | ||
+ | |||
+ | **scenario1: | ||
+ | < | ||
+ | - submit_job: | ||
+ | - exec_appliance: | ||
+ | - test_if_submitted: | ||
+ | - exec_appliance: | ||
+ | - exec_appliance: | ||
+ | - exec_appliance: | ||
+ | - exec_appliance: | ||
+ | - delete_job: | ||
+ | - exec_appliance: | ||
+ | - exec_appliance: | ||
+ | - test_if_deleted: | ||
+ | - exec_appliance: | ||
+ | - exec_appliance: | ||
+ | - exec_appliance: | ||
+ | |||
+ | The **print_output** macrostep contains the following microsteps: | ||
+ | |||
+ | **print_output: | ||
+ | < | ||
+ | - print: | ||
+ | - exec_appliance: | ||
+ | - exec_appliance: | ||
+ | - exec_appliance: | ||
+ | - exec_appliance: | ||
+ | - exec_appliance: | ||
+ | - exec_appliance: | ||
+ | - exec_appliance: | ||
+ | - exec_appliance: | ||
+ | - exec_appliance: | ||
+ | - exec_appliance: | ||
+ | - exec_appliance: | ||
+ | - exec_appliance: | ||
+ | |||
+ | Only 1 print_output microstep should be present in the recipe file. If 2 scenario testings needs to be done,place print_output step after 2nd scenario. This will contain the output of tests of both the scenarios. | ||
+ | |||
+ | Notice here that print_output microstep is rather big. The intention was to add more creativity in the output after the test run. It can also be rather small and simple; depends on the requirements of the tester. | ||
+ | |||
+ | ===== Schema ===== | ||
+ | |||
+ | [[Image: | ||
+ | |||
+ | [[Image: | ||
+ | |||
+ | [[Image: | ||
+ | |||
+ | [[Image: | ||
+ | |||
+ | ===== List of rspectests_lib methods ===== | ||
+ | |||
+ | The **rspectests_lib** currently contains the following methods/ | ||
+ | The OAR REST APIs are called by this library using the oarrestapi_lib library. | ||
+ | For writing tests, one has the flexibility to require either rspectests_lib or oarrestapi_lib based on their specific requirements. | ||
+ | |||
+ | **Class: Test** | ||
+ | |||
+ | **Methods: | ||
+ | |||
+ | < | ||
+ | 1. test_get_version | ||
+ | 2. test_get_timezone | ||
+ | 3. test_get_jobs_details | ||
+ | 4. test_get_running_jobs | ||
+ | 5. test_get_jobs_id (jid) - Testing the GET / | ||
+ | 6. test_job_in_queue(jid) | ||
+ | 7. test_job_notin_queue(jid) | ||
+ | 8. test_get_jobs_table | ||
+ | 9. test_submit_job (jhash) | ||
+ | 10. test_jobs_delete_post (jid) - Testing the POST / | ||
+ | 11. test_jobs_delete (jid) - Testing the DELETE / | ||
+ | 12. test_get_resources | ||
+ | 13. test_get_resources_full | ||
+ | 14. test_job_rholds (jid) - Testing the POST / | ||
+ | 15. test_job_hold (jid)\t | ||
+ | 16. test_job_resumption (jid) - Testing the POST / | ||
+ | 17. test_job_update (jid, actionhash) - Testing POST / | ||
+ | 18. test_if_job_delete_updated (jid) - Testing POST / | ||
+ | 19. test_job_checkpoint (jid) - Testing the POST / | ||
+ | 20. test_job_running (jid) - Testing if job is currently running | ||
+ | |||
+ | ===== List of oarrestapi_lib methods ===== | ||
+ | |||
+ | The rspectests_lib call the OAR REST APIs through the **oarrestapi_lib** library. | ||
+ | |||
+ | **Class: OarApi** | ||
+ | |||
+ | **Methods: | ||
+ | |||
+ | < | ||
+ | 1. get(api, | ||
+ | 2. post(api, | ||
+ | 3. delete(api, uri)\t\t | ||
+ | 4. oar_version\t\t | ||
+ | 5. oar_timezone\t\t | ||
+ | 6. full_job_details\t\t | ||
+ | 7. run_job_details | ||
+ | 8. specific_job_details(jobid) | ||
+ | 9. dump_job_table\t\t | ||
+ | 10. submit_job(jhash)\t\t | ||
+ | 11. del_job(jobid) | ||
+ | 12. send_checkpoint(jobid) | ||
+ | 13. hold_waiting_job(jobid)\t | ||
+ | 14. hold_running_job(jobid) | ||
+ | 15. resume_hold_job(jobid) | ||
+ | 16. send_signal_job(jobid, | ||
+ | 17. update_job(jobid, | ||
+ | 18. resource_list_state | ||
+ | 19. list_resource_details | ||
+ | 20. specific_resource_details(jobid) | ||
+ | 21. resource_of_nodes(netaddr) | ||
+ | 22. create_resource(rhash) | ||
+ | 23. statechange_resource(jid, | ||
+ | 24. delete_job(jobid)\t\t | ||
+ | 25. delete_resource(resid) | ||
+ | 26. delete_resource_cpuset(node, | ||
+ | |||
+ | ===== TODO list ===== | ||
+ | * < | ||
+ | * < | ||
+ | * < | ||
+ | * < | ||
+ | * < | ||
+ | * < | ||
+ | * < | ||
+ | * < | ||
+ | * < | ||
+ | * < | ||