This shows you the differences between two versions of the page.
| — | wiki:old:gsoc_kameleon [2013/07/10 22:55] (current) – created - external edit 127.0.0.1 | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| + | This is the **Kameleon GSOC 2009** page. Also check the main page here: http:// | ||
| + | |||
| + | [[GSoC_Proposal_OAR_LiveCD| See original proposal here]] | ||
| + | |||
| + | |||
| + | **Student, please read carefully this page...** | ||
| + | |||
| + | |||
| + | Student: Darko Ilic | ||
| + | |||
| + | Mentor: Yiannis Georgiou | ||
| + | |||
| + | Co-Mentors: Bruno Bzeznik, Yiannis Georgiou, Joseph Emeras | ||
| + | |||
| + | ---- | ||
| + | |||
| + | |||
| + | ===== Student: Things to do before starting ===== | ||
| + | |||
| + | * < | ||
| + | * Create a valid user report for grid5000 platform | ||
| + | |||
| + | ===== How to start the project ===== | ||
| + | |||
| + | * play with installing and configuring OAR using DEB / RPM packages upon grid5000 image | ||
| + | * start using OAR live CD on laptop or grid5000 | ||
| + | * modify script for construction of OAR live CD to support: | ||
| + | * the fetch of a debian image using debootstrap (for the moment it uses a minimal debian customized image stored locally) | ||
| + | * the activation of cpusets | ||
| + | * the automatic detection of resources using the script oar_resources_init and detect_resources that we use for the configuration upon grid5000 | ||
| + | * to check the initialization script used in SysrescueCD and see how they choose the keyboard language... it has to continue with a default value after a specific | ||
| + | |||
| + | ===== May's 19 Brainstorming ===== | ||
| + | |||
| + | Meeting between Olivier, Bruno and Yiannis | ||
| + | |||
| + | * Tried to conceptualize the project | ||
| + | * Has to be general enough so that it can be modular and extensible. | ||
| + | * We have to define the different steps of the procedure. | ||
| + | * Initial Idea based on a main Kameleon Engine along with description files composed by Macrosteps and microsteps, with the engine implemented in Ruby and the description files in Yaml. | ||
| + | * The user will be able to provide the MacroStep he wants to start from. | ||
| + | * Each macrostep will be composed by various microsteps. | ||
| + | * We have to anticipate for debugging options. | ||
| + | * The macrosteps and microsteps have to collect the error codes of the commands executed and treat them. | ||
| + | * ... | ||
| + | |||
| + | |||
| + | |||
| + | ===== Project' | ||
| + | |||
| + | In the context of the latest trends related with ' | ||
| + | * to test/debug OAR latest version or bugfix on your laptop. | ||
| + | * to try and play with OAR resource management system and its features, before actually installing it. | ||
| + | * to create a virtual cluster and use it | ||
| + | * ... | ||
| + | |||
| + | The goal of the project is to create the mechanism of the automatic construction of an iso image and appliances with OAR installed. The created iso image/ | ||
| + | We would like to give as input values like: | ||
| + | * distribution (debian lenny, centos 5,...) | ||
| + | * kernel | ||
| + | * output type (iso, cow, tgz,...) | ||
| + | * packages (libraries(openmpi, | ||
| + | * ... | ||
| + | |||
| + | and taking as an output the specified type of image ready for installation/ | ||
| + | |||
| + | |||
| + | |||
| + | ==== Implementation Ideas ==== | ||
| + | http:// | ||
| + | |||
| + | According to our Brainstorming we tried to define an implementation description. | ||
| + | * The main Kameleon engine will be programmed in Ruby | ||
| + | * The architecture will be based on Yaml description/ | ||
| + | * The description files will be composed by macrosteps and microsteps | ||
| + | |||
| + | The architecture can look like this: | ||
| + | |||
| + | [[Image: | ||
| + | |||
| + | === Directory structure === | ||
| + | All the configuration files are in a data directory, for example / | ||
| + | < | ||
| + | / | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | |||
| + | === The recipe file === | ||
| + | This file is a configuration file. It has a **global** part configuring some variables and a **steps** part listing all the steps (macrosteps composed of microsteps) that have to be executed in the given order. | ||
| + | In the global part, some variables are mandatory and others may be custom variables used into microsteps. In the steps part, if no microsteps are given, then it means that all the microsteps are executed in the order they have been defined into the corresponding macrostep file. | ||
| + | |||
| + | Example: | ||
| + | |||
| + | < | ||
| + | #/ | ||
| + | # | ||
| + | | ||
| + | distrib: debian-lenny-x86_64 | ||
| + | type: iso | ||
| + | initial_appliance: | ||
| + | workdir_base: | ||
| + | distrib-repository: | ||
| + | packages: file:// | ||
| + | | ||
| + | - bootstrap | ||
| + | - minimal_packages_installation | ||
| + | - system_configuration | ||
| + | - oar_install | ||
| + | - oar_init: | ||
| + | - config_oar_key | ||
| + | - stop_host_mysql | ||
| + | - stop_oar_server | ||
| + | - start_appliance_mysql | ||
| + | - init_mysql_db | ||
| + | - init_oar_db | ||
| + | - create_oar_resources | ||
| + | - stop_appliance_mysql | ||
| + | - kernel_install | ||
| + | - image_create | ||
| + | - clean</ | ||
| + | |||
| + | === The macrosteps files === | ||
| + | Example: | ||
| + | |||
| + | < | ||
| + | #/ | ||
| + | # | ||
| + | | ||
| + | - init_repo: | ||
| + | - check_cmd: | ||
| + | - exec_chroot: | ||
| + | - exec_appliance: | ||
| + | - exec_appliance: | ||
| + | - config_apt_local: | ||
| + | - append_file: | ||
| + | - / | ||
| + | - deb file:/var debs/ | ||
| + | - exec_chroot: | ||
| + | - oar_install: | ||
| + | - exec_chroot: | ||
| + | | ||
| + | | ||
| + | - oar_add_online_repo: | ||
| + | - append_file: | ||
| + | - / | ||
| + | - | | ||
| + | deb http:// | ||
| + | deb http:// | ||
| + | deb http:// | ||
| + | #/ | ||
| + | - config_oar_key: | ||
| + | - exec_chroot: | ||
| + | - stop_host_mysql: | ||
| + | - exec_current: | ||
| + | RC=0 | ||
| + | / | ||
| + | if [[|" | ||
| + | then | ||
| + | / | ||
| + | MYSQL_STARTED=1 | ||
| + | fi | ||
| + | - init_mysql_db: | ||
| + | - write_file: | ||
| + | - / | ||
| + | - | | ||
| + | CREATE DATABASE IF NOT EXISTS oar; | ||
| + | CONNECT mysql; | ||
| + | INSERT INTO user (Host, | ||
| + | INSERT INTO user (Host, | ||
| + | INSERT INTO db (Host, | ||
| + | VALUES | ||
| + | (' | ||
| + | INSERT INTO db (Host, | ||
| + | VALUES | ||
| + | (' | ||
| + | FLUSH PRIVILEGES; | ||
| + | GRANT ALL ON oar.* TO oar@localhost; | ||
| + | GRANT ALL ON oar.* TO oar@" | ||
| + | GRANT SELECT ON oar.* TO oarreader@localhost; | ||
| + | GRANT SELECT ON oar.* TO oarreader@" | ||
| + | FLUSH PRIVILEGES; | ||
| + | - exec_chroot: | ||
| + | - init_oar_db: | ||
| + | - exec_chroot: | ||
| + | mysql oar < / | ||
| + | mysql oar < / | ||
| + | mysql oar < / | ||
| + | - create_oar_resources: | ||
| + | - exec_chroot: | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | #/ | ||
| + | | ||
| + | - include: / | ||
| + | - start_appliance_mysql: | ||
| + | - exec_chroot: | ||
| + | - stop_oar_server: | ||
| + | - exec_chroot: | ||
| + | / | ||
| + | / | ||
| + | - stop_appliance_mysql: | ||
| + | - exec_chroot: | ||
| + | |||
| + | * Each macrostep will produce a bash script that will be executed | ||
| + | * Each bash script has to store the environment variables somewhere so that the next macrostep can collect them | ||
| + | * Each microstep definition is an array of commands. Each command is a {key => value} pair. The key is the name of the command, and the value is the argument. It will result in a part of the bash script, automatically generated by the kameleon engine. | ||
| + | * The value may contain macro variables: they start with **$$** and are filled by the global section of the recipe file. | ||
| + | * A special macro variable **$$workdir** gives the path of the temporary directory used for the currently built appliance | ||
| + | * When there' | ||
| + | |||
| + | === Microsteps commands === | ||
| + | * **check_cmd** | ||
| + | Raise an error if the specified command does not exist on the host system | ||
| + | * **check_appliance_cmd** | ||
| + | Raise an error if the specified command does not exist inside a chroot in tha appliance system | ||
| + | * **exec_current** | ||
| + | Execute the given bash scriptlet on the host system from the current directory (the one from which kameleon has been run) | ||
| + | * **exec_appliance** | ||
| + | Execute the given bash scriptlet on the host system from the root of the appliance directory | ||
| + | * **exec_chroot** | ||
| + | Execute the given bash scriptlet inside a chroot of the appliance directory | ||
| + | * **append_file** | ||
| + | Append a content to a file of the appliance (relative to the root of the appliance). | ||
| + | This commands takes an array of 2 elements as an argument: the first element is the name of the file and the second one is the content to be appended to the file. | ||
| + | * **write_file** | ||
| + | Write a content into a file of the appliance. If the file already exists, erase it. Else, create it. | ||
| + | This commands takes an array of 2 elements as an argument: the first element is the name of the file and the second one is the content to be writed into the file. | ||
| + | * **set_var** | ||
| + | Set up a bash variable that may be used by latter microsteps. The argument is a 2 element array (name of the variable, value of the variable) | ||
| + | * **include** | ||
| + | Not a microstep command, but may be found in place of a microstep name. It means that the given argument is a name of a file to be included | ||
| + | * ... | ||
| + | |||
| + | === Breakpoints and error handling === | ||
| + | * **Breakpoints** | ||
| + | The kameleon engine should provide a way to specify macrostep/ | ||
| + | * **Error handling** | ||
| + | We want the engine to raise formated errors when a bash scriptlet crashes, with the name of the macrostep and microstep where the error occured. But, we don't want the engine to execute each scriptlet separatelly and return to ruby between each microstep as it may result in performance drawbacks. The engine should generate a script for each macrostep and execute it as a whole. So, to know from which microstep an error occured, it may be necessary to had some kind of labels inside the generated script, for example a trap between each microstep that will force the script to exit with a given status code corresponding to a microstep (limiting the number of microsteps to 255 per macrosteps, by the way...) | ||
| + | |||
| + | ==== Changelog/ | ||
| + | * Not finished...more details will come...--[[User: | ||
| + | * Details added. Still need to continue examples and definitions of microsteps commands --[[User: | ||
| + | * YAML examples have been syntax checked with ruby. I hope the specifications are ok for coding the engine now! --[[User: | ||
| + | * Maybe missing: examples of what scriptlet should produce the microsteps commands (resulting script of the provided oar_install and oar_init examples) --[[User: | ||
| + | |||
| + | ==== Initial Workplan ==== | ||
| + | |||
| + | ===== Roadmap (and Timeline) ===== | ||
| + | |||
| + | ===== TODO list ===== | ||
| + | |||
| + | ==== Mentor ==== | ||
| + | |||
| + | * < | ||
| + | * Write a wiki page for explanations on how to configure an OAR cluster upon Grid5000 ...the configuration process used upon grid5000 will be quite similar (but simpler) with that used for the case of cluster of liveCDs or appliances... | ||
| + | * check about the lvm partitions used in the case of images with xen | ||
| + | * finish the construction of the lenny-xen image upon Grid5000 | ||
| + | * install OAR upon it | ||
| + | * configure it to work with lvm partitions | ||
| + | * construct the fedora liveCD from RPMs (first simple construction from Bruno) ... < | ||
| + | --[[User: | ||
| + | |||
| + | |||
| + | ===== Links ===== | ||
| + | * **Febootstrap**, | ||
| + | * **rpmstrap**, | ||
| + | * CenOS bootstraping not very complicated: | ||
| + | * **Rinse** PROMISING!!! http:// | ||
| + | Note: amd64 for x64_64: rinse --distribution centos-5 --directory ./ | ||
| + | * **CAOS Linux** a linux distribution specialized for clusters with SLURM or Torque as options for resource management, http:// | ||
| + | * **kpartx**: for hdd image creation (Qemu appliance): http:// | ||
| + | * **kiwi**: Image system : | ||
| + | * **Fedora appliance tools**: | ||
| + | * **Ubuntu JeOS**: | ||
| + | * **Chef**: http:// | ||
| + | * **KVM-NBD**: | ||
| + | |||
| + | ===== Videos ===== | ||
| + | * kameleon video !!! : | ||
| + | * Waou, with such a chameleon, our kameleon should be able to make a mid-debian/ | ||