Table of Contents

This is the Kameleon GSOC 2009 page. Also check the main page here: http://wiki-oar.imag.fr/index.php/Kameleon

See original proposal here

Student, please read carefully this page…

Student: Darko Ilic

Mentor: Yiannis Georgiou

Co-Mentors: Bruno Bzeznik, Yiannis Georgiou, Joseph Emeras

Student: Things to do before starting

How to start the project

May's 19 Brainstorming

Meeting between Olivier, Bruno and Yiannis

Project's specifications

In the context of the latest trends related with 'virtualization', 'cluster in a box' and 'cloud computing' in this project we are interested in providing a tool (named Kameleon) that will be able to construct an OAR iso image or appliance (VMWare, Xen, Grid5000) which will be used for automatic installation and configuration of a virtual OAR cluster. The created iso image/appliance will be useful in various cases like:

The goal of the project is to create the mechanism of the automatic construction of an iso image and appliances with OAR installed. The created iso image/appliance, once it is deployed, it will be able to autoconfigure a virtual cluster according to the initial specifications of the user. We would like to give as input values like:

and taking as an output the specified type of image ready for installation/autoconfiguration

Implementation Ideas

http://wiki-oar.imag.fr/index.php/Image:WhiteBoardKameleonSnapshot.jpg

According to our Brainstorming we tried to define an implementation description.

The architecture can look like this:

spec.png

Directory structure

All the configuration files are in a data directory, for example /var/lib/kameleon. We can have: 

 /var/lib/kameleon
             |_/recipes
             |     |_/debian-lenny-x86_64-oar-2.4-iso.yaml
             |     |_/centos-5.2-x86_64-oar-2.4-xen.yaml
             |     |_/...
             |_/steps
                   |_/common    #(includes)
                         |_/oar_resources_setup.yaml
                         |_/...
                   |_/debian-lenny-x86_64
                         |_/bootstrap.yaml
                         |_/...
                   |_/...

The recipe file

This file is a configuration file. It has a global part configuring some variables and a steps part listing all the steps (macrosteps composed of microsteps) that have to be executed in the given order. In the global part, some variables are mandatory and others may be custom variables used into microsteps. In the steps part, if no microsteps are given, then it means that all the microsteps are executed in the order they have been defined into the corresponding macrostep file.

Example: 

 #/var/lib/kameleon/recipes/debian-lenny-oar-2.4.yaml
 #example of a recipe
 global:
  distrib: debian-lenny-x86_64
  type: iso
  initial_appliance: file://var/tmp/base_debian-lenny.tgz
  workdir_base: /var/tmp/
  distrib-repository: ftp://debian.ftp.fr/deb/
  packages: file://./svn/build-area/
 steps:
  - bootstrap
  - minimal_packages_installation
  - system_configuration
  - oar_install
  - oar_init:
    - config_oar_key
    - stop_host_mysql
    - stop_oar_server
    - start_appliance_mysql
    - init_mysql_db
    - init_oar_db
    - create_oar_resources
    - stop_appliance_mysql
  - kernel_install
  - image_create
  - clean

The macrosteps files

Example: 

 #/var/lib/kameleon/steps/debian-lenny-x86_64/oar_install.yaml
 #macrostep oar_install for debian lenny 64 bits
 oar_install:
  - init_repo:
    - check_cmd:       dpkg-scanpackages
    - exec_chroot:     mkdir -p /var/lib/debs
    - exec_appliance:  cp $$packages/*.deb ./var/lib/debs/
    - exec_appliance:  cd ./var/lib && dpkg-scanpackages debs /dev/null | gzip > debs/Packages.gz
  - config_apt_local:
    - append_file:
      - /etc/apt/sources.lists
      - deb file:/var debs/
    - exec_chroot:     apt-get update -q
  - oar_install:
    - exec_chroot: |
                   DEBIAN_FRONTEND=noninteractive apt-get -q install -y --force-yes oar-server oar-user  oar-node oar-doc oar-admin oar-web-status
                   DEBIAN_FRONTEND=noninteractive apt-get -q install -y --force-yes oar-api || true
  - oar_add_online_repo:
    - append_file:
      - /etc/apt/sources.lists
      - |
        deb http://ftp.fr.debian.org/debian lenny main
        deb http://security.debian.org/ lenny/updates main
        deb http://oar.imag.fr/debian/unstable/2.4/ ./
 #/var/lib/kameleon/steps/common/oar_init.yaml                                                 
  - config_oar_key:                                                                            
    - exec_chroot:      perl -pi -e 's/^/environment="OAR_KEY=1" /' ~oar/.ssh/authorized_keys  
  - stop_host_mysql:                                                                           
    - exec_current: |                                                                          
                    RC=0                                                                       
                    /etc/init.d/mysql status > /dev/null || RC=$?                              
                    if [[|"$RC" = "0" ]]                                                         
                    then
                      /etc/init.d/mysql stop
                      MYSQL_STARTED=1
                    fi
  - init_mysql_db:
    - write_file:
      - /tmp/init.sql
      - |
        CREATE DATABASE IF NOT EXISTS oar;
        CONNECT mysql;
        INSERT INTO user (Host,User,Password) VALUES('localhost','oar',PASSWORD('oar'));
        INSERT INTO user (Host,User,Password) VALUES('%','oar',PASSWORD('oar'));
        INSERT INTO db  (Host,Db,User,Select_priv,Insert_priv,Update_priv,Delete_priv, Create_priv,Drop_priv)
          VALUES
          ('localhost','oar','oar','Y','Y','Y','Y','Y','Y');
        INSERT INTO db  (Host,Db,User,Select_priv,Insert_priv,Update_priv,Delete_priv, Create_priv,Drop_priv)
          VALUES
          ('%','oar','oar','Y','Y','Y','Y','Y','Y');
        FLUSH PRIVILEGES;
        GRANT ALL ON oar.* TO oar@localhost;
        GRANT ALL ON oar.* TO oar@"%";
        GRANT SELECT ON oar.* TO oarreader@localhost;
        GRANT SELECT ON oar.* TO oarreader@"%";
        FLUSH PRIVILEGES;
    - exec_chroot: mysql < /tmp/init.sql
  - init_oar_db:
    - exec_chroot: |
                   mysql oar < /usr/lib/oar/mysql_structure.sql
                   mysql oar < /usr/lib/oar/mysql_default_admission_rules.sql
                   mysql oar < /usr/lib/oar/default_data.sql || true
  - create_oar_resources:
    - exec_chroot: |
                   oarproperty -a core
                   oarproperty -a cpu
                   oarnodesetting -a -h node1 -p cpu=0 =p core=0
                   oarnodesetting -a -h node1 -p cpu=0 =p core=1
                   oarnodesetting -a -h node1 -p cpu=1 =p core=0
                   oarnodesetting -a -h node1 -p cpu=1 =p core=1
                   oarnodesetting -a -h node2 -p cpu=0 =p core=0
                   oarnodesetting -a -h node2 -p cpu=0 =p core=1
                   oarnodesetting -a -h node2 -p cpu=1 =p core=0
                   oarnodesetting -a -h node2 -p cpu=1 =p core=1
 #/var/lib/kameleon/steps/debian-lenny-x86_64/oar_init.yaml
 oar_init:
  - include: /var/lib/kameleon/steps/common/oar_init.yaml
  - start_appliance_mysql:
    - exec_chroot: /etc/init.d/mysql start
  - stop_oar_server:
    - exec_chroot: |
                   /etc/init.d/oar-server stop || true
                   /etc/init.d/oar-node stop || true
  - stop_appliance_mysql:
    - exec_chroot: sleep 2; /etc/init.d/mysql stop

Microsteps commands

Raise an error if the specified command does not exist on the host system

Raise an error if the specified command does not exist inside a chroot in tha appliance system

Execute the given bash scriptlet on the host system from the current directory (the one from which kameleon has been run)

Execute the given bash scriptlet on the host system from the root of the appliance directory

Execute the given bash scriptlet inside a chroot of the appliance directory

Append a content to a file of the appliance (relative to the root of the appliance). This commands takes an array of 2 elements as an argument: the first element is the name of the file and the second one is the content to be appended to the file.

Write a content into a file of the appliance. If the file already exists, erase it. Else, create it. This commands takes an array of 2 elements as an argument: the first element is the name of the file and the second one is the content to be writed into the file.

Set up a bash variable that may be used by latter microsteps. The argument is a 2 element array (name of the variable, value of the variable)

Not a microstep command, but may be found in place of a microstep name. It means that the given argument is a name of a file to be included

Breakpoints and error handling

The kameleon engine should provide a way to specify macrostep/microstep breakpoints. For example, the user may want to stop kameleon just before the “oar_init/init_oar_db” step. Similarily, we may want to start at a specified step. For this last feature, it may be necessary to find a way for the user to provide the name of the working directory, maybe given by a previous breakpoint.

We want the engine to raise formated errors when a bash scriptlet crashes, with the name of the macrostep and microstep where the error occured. But, we don't want the engine to execute each scriptlet separatelly and return to ruby between each microstep as it may result in performance drawbacks. The engine should generate a script for each macrostep and execute it as a whole. So, to know from which microstep an error occured, it may be necessary to had some kind of labels inside the generated script, for example a trap between each microstep that will force the script to exit with a given status code corresponding to a microstep (limiting the number of microsteps to 255 per macrosteps, by the way…)

Changelog/todo

Initial Workplan

Roadmap (and Timeline)

TODO list

Mentor

–Ygeorgiou 15:57, 29 April 2009 (UTC)

Note: amd64 for x64_64: rinse –distribution centos-5 –directory ./centos-oar-appliance –arch amd64 –Bzizou 12:33, 30 April 2009 (UTC)

Videos