Differences

This shows you the differences between two versions of the page.

Link to this comparison view

wiki:old:kameleon [2013/07/10 23:06] (current)
Line 1: Line 1:
 +====== Yet another tool to build VM images! ======
 +
 +Kameleon is a very simple tool to create appliances, vm or cd images, or what you want... Actually, it is a bash script generator using **YAML** as an imput format. But it has been designed mainly to create initial images from scratch as we haven'​t found a tool for doing this and only this, for whatever linux distribution and without a client/​server architecture,​ without features to maintain a living system or other "​heavy"​ features.
 +It is written in Ruby and have a very few dependencies.
 +
 +With Kameleon, you can use provided or self-written **recipes** using provided or self-written **macrosteps** and **microsteps**. Provided examples are self-explanatory so that you can quickly start to create your own recipes to generate customized images. Our provided recipes are part of Kameleon and allow you to obtain, for example, a bootable Debian Linux system (etch, lenny, squeeze,​...) in qcow2 format in less than 3 minutes!
 +
 +The use of the YAML format makes the bash pieces of code very easy to read and maintain. As an example, when you write into a file using "​cat"​ and "<<​EOF"​ you generally break the indentation of your meta-code and it becomes uggly and hard to maintain. But kameleon files are always clean with a nice YAML indentation :-)
 +
 +====== Quickstart ======
 +**Warning:​** kameleon must be run as root and as it constructs a system image into a chroot directory, it may start/stop some services. this may have unpredictable effects on your running system. so, don't use it on a production server. if you want to experiment kameleon with no risk, we recommend you to use the kameleon appliance.
 +
 +<​code>​
 + > sudo su -
 + > mkdir -p kameleon/​recipes
 + > mkdir kameleon/​steps
 + > cp /​usr/​share/​kameleon/​recipes/​debian.yaml kameleon/​recipes/​my_debian.yaml
 + > # or if you installed from sources:
 + > # cp /​usr/​local/​share/​kameleon/​recipes/​debian.yaml kameleon/​recipes/​my_debian.yaml
 + > vi kameleon/​recipes/​my_debian.yaml
 +   # customize the recipe if needed
 + > cd kameleon
 + > kameleon my_debian
 +   # wait...
 + > sudo kvm -hda /​var/​tmp/​kameleon/​2010-04-22-16-27-27/​image.qcow
 +</​code>​
 +
 +====== How it works ======
 +=====  Overview =====
 +<​code>​
 + ​Recipe |
 +        |---> Kameleon Engine ---> Appliance (tgz, qcow, iso,...)
 + ​Steps  ​
 +</​code>​
 +
 +  *  **Recipe**: it is a YAML file that describes an appliance. It has 2 parts:
 +    *  //The global variables definition//:​ the variables defined in this section are usable into the steps files (used as //​$$variable_name//​)
 +    *  //The steps listing//: the recipe lists all the Macrosteps to execute in the appropriate order. The Microsteps that compose the Macrosteps can be explicitly listed in order to filter/​reorder them.
 +  *  **Steps**: Steps are Macrosteps and Microsteps. Macrosteps are YAML files that define Microsteps. Microsteps are simple kameleon commands that expand to pieces of bash script. So, a Macrostep expands to a bash script that is executed by kameleon.
 +
 +At any microstep, kameleon can escape to a shell, either by a specific "​breakpoint"​ microstep command or when an error occurs.
 +
 +===== Microstep principal commands =====
 +  *  **exec_appliance <​cmd>​**:​ Execute the given bash scriptlet on the host system from the root of the appliance directory
 +  *  **exec_chroot <​cmd>​**:​ Execute the given bash scriptlet inside a chroot of the appliance directory
 +  *  **append_file**:​ Append a content to a file of the appliance (relative to the root of the appliance). This commands takes an array of 2 elements as an argument: the first element is the name of the file and the second one is the content to be appended to the file.
 +  *  **write_file**:​ Write a content into a file of the appliance. If the file already exists, erase it. Else, create it.
 +For more commands, see the Documentation.rst file of the kameleon distribution.
 +
 +===== Samples =====
 +Sample recipe (debian.yaml):​
 +<​code>​
 + ​global:​
 +  workdir_base:​ /​var/​tmp/​kameleon
 +  distrib: debian
 +  debian_version_name:​ squeeze
 +  distrib_repository:​ http://​ftp.fr.debian.org/​debian/​
 +  arch: amd64
 +  kernel_arch:​ "​amd64"​
 + ​steps:​
 +  - debian_check_deps
 +  - check_deps:
 +    - rsync
 +    - building_appliance
 +    - building_kvm_images
 + # - oar/​oar_precheck
 + # - checkpoint_resume
 +  - bootstrap
 +  - system_config
 + # - network_config_static
 +  - root_passwd
 +  - mount_proc
 +  - software_install:​
 +    - extra_packages
 +  - kernel_install
 + # - checkpoint
 + # - oar/​oar_debian_install
 + # - oar/​oar_system_config
 +  - strip
 +  - umount_proc
 + # - xen_domu
 + # - oar/​oar_build_tgz
 +  - build_appliance:​
 +    - clean_udev
 +    - save_as_tgz
 +    - create_raw_image
 +    - create_nbd_device
 +    - mkfs
 +    - mount_image
 +    - copy_system_tree
 + # ​  - grub_197_workaround
 +    - install_grub ​
 +    - umount_image ​
 +    - save_as_raw
 + # ​  - save_as_vmdk
 +    - save_as_qcow2
 + # ​  - save_as_vdi
 +  - clea
 +</​code>​
 +
 +Sample macrostep file (debian/​kernel_install.yaml):​
 +<​code>​
 + ​kernel_install:​
 +  - kernel-img_conf:​
 +    - write_file:
 +      - /​etc/​kernel-img.conf
 +      - |
 +        do_symlinks = yes
 +        relative_links = yes
 +        do_bootloader = yes
 +        do_bootfloppy = no
 +        do_initrd = yes
 +        link_in_boot = no
 +  - kernel_install:​
 +    - exec_chroot:​ apt-get -y --force-yes install linux-image-$$kernel_arc
 +</​code>​
 +
 +====== Download ======
 +Kameleon is distributed under the GPL licence, copyright LIG <​http://​lig.imag.fr>​ and can be downloaded from the INRIA sourceforge here: https://​gforge.inria.fr/​projects/​kameleon.
 +
 +The development sources are hosted by the OAR project:
 +https://​gforge.inria.fr/​scm/?​group_id=2308
 +
 +====== Install ======
 +
 +===== Debian packages =====
 +
 +The debian packages are build with //​squeeze//​. Currently, they will also work for //lenny// and //wheezy//. According to your needs, put on of these repositories into your /​etc/​apt/​sources.list:​
 +
 +<​code>​
 + # snapshots (automatically builded)
 + deb http://​oar-ftp.imag.fr/​kameleon/​debian squeeze-snapshots main
 + # unstable/​testing
 + deb http://​oar-ftp.imag.fr/​kameleon/​debian squeeze-proposed-updates main
 + # stable
 + deb http://​oar-ftp.imag.fr/​kameleon/​debian squeeze mai
 +</​code>​
 +
 +
 +Add the following gpg key to the apt keyring:
 +<​code>​
 +  curl http://​oar-ftp.imag.fr/​kameleon/​debian/​oar-archive-keyring.asc | sudo apt-key add 
 +</​code>​
 +
 +
 +And do:
 +<​code>​
 + ​apt-get update
 + ​apt-get install kameleon rubygems
 + gem install sessio
 +</​code>​
 +
 +===== Sources =====
 +
 +Download the lastest .tgz source files:
 +
 +<​code>​
 + # last sources
 + ​http://​oar-ftp.imag.fr/​kameleon/​sources/​
 + # old sources
 + ​https://​gforge.inria.fr/​frs/?​group_id=230
 +</​code>​
 +
 +Untar and do "make install"​
 +
 +===== You'll need the "​session"​ ruby module =====
 +In any case, it can be easily installed by Rubygems:
 +<​code>​
 + gem install sessio
 +</​code>​
 +
 +
 +====== The kameleon appliance ======
 +A KVM/QEMU appliance is available as a Debian/​squeeze system with Kameleon already installed. This allows you to test kameleon without risks. You just have to download the following compressed qcow2 image file:
 +<​code>​
 + # 64bits:
 +   # stable
 + ​http://​oar-ftp.imag.fr/​kameleon/​appliances/​kameleon-appliance_1.2.8-1_squeeze_amd64.qcow2
 + # 32bits:
 +   # stable:
 + ​http://​oar-ftp.imag.fr/​kameleon/​appliances/​kameleon-appliance_1.2.8-1_squeeze_i386.qcow
 +</​code>​
 +and start KVM:
 +<​code>​
 + # kvm -m 512 -hda kameleon-appliance_1.2.8-1_squeeze_amd64.qcow
 +</​code>​
 +To make a basic debian system, you can directly type into your running appliance:
 +<​code>​
 + # kameleon debia
 +</​code>​
 +Of course, as building an image requires a lot of I/Os, this appliance runs slower than a real system. Also be sure that you have enough free disk space on the partition where you store the image as it will grow (up to 5GB).
 +
 +The appliance is also available as a vmdk (Vmware) image, but not tested (feedback welcome!):
 +<​code>​
 + # 64bits:
 + ​http://​oar-ftp.imag.fr/​kameleon/​appliances/​kameleon-appliance_1.2.8-1_squeeze_amd64.vmdk.gz
 + ​http://​oar-ftp.imag.fr/​kameleon/​appliances/​kameleon-appliance_1.2.8-1_squeeze_amd64.vmx
 + # 32bits:
 + ​http://​oar-ftp.imag.fr/​kameleon/​appliances/​kameleon-appliance_1.2.8-1_squeeze_i386.vmdk.gz
 + ​http://​oar-ftp.imag.fr/​kameleon/​appliances/​kameleon-appliance_1.2.8-1_squeeze_i386.vm
 +</​code>​
 +
 +====== FAQ ======
 +===== Where are the recipes and steps files? =====
 +You can find recipes and steps into 3 locations:
 +  *  The first is where you find the provided recipes and steps: **/​usr/​share/​kameleon** or **/​usr/​local/​share/​kameleon**. You should not edit those files, they might be erased on an upgrade. But you are strongly encouraged to use them and to copy/paste from them.
 +  *  The second location is where you can put custom recipes and steps for your system: **/​var/​lib/​kameleon**. You can simply put a custom recipe into /​var/​lib/​kameleon/​recipes that uses provided steps, but you can also put customized steps into /​var/​lib/​kameleon/​steps. Note that if you create a custom file of the same name as a provided file, the customized one will be used. You can mix custom and provided steps into recipes.
 +  *  The third location is the current **./​recipes** and **./steps** directories. Files are searched firstly into this location. As of /var/lib, you can mix custom and provided steps into recipes.
 +Finally, in every of those 3 locations, the steps directory contains subdirectories in which kameleon will search for the macrosteps yaml definitions. If you specify a **my_step** macrostep into a recipe, it is firstly searched into the **steps/<​distrib>​** subdirectory (where <​distrib>​ is the value of the **distrib** global variable , then, if not found, into the **steps/​default** subdirectory. You can use the **dir/​my_step** notation into recipes to load a macrostep from another distrib.
 +
 +===== I want to write my own steps. How should I do? =====
 +The simplest way is to take a look at all the steps of the debian.yaml recipe. This recipe (including commented steps) uses almost every capability of kameleon and the code is self explanatory. You can also find a documentation into **/​usr/​share/​doc/​kameleon/​Documentation.rst**.
 +===== What is the default root password of the default appliances? =====
 +"​kameleon"​. There may also be a "​kameleon"​ user, with "​kameleon"​ password that is in the sudo group.
 +
 +===== The Kameleon appliance freezes with kernel messages as "xxx blocked for more than 120 seconds"​ =====
 +Add more memory to your KVM instance ("kvm -m 512" should be enough)
 +===== How to get the image generated inside the kameleon appliance? =====
 +We have 2 methods:
 +  *  You can shutdown the kameleon appliance and then access to the filesystem directly from your host:
 +<​code>​
 + sudo modprobe nbd max_part=8
 + sudo qemu-nbd --connect=/​dev/​nbd0 kameleon1.0b-appliance-0.1-squeeze.qcow2 ​
 + mount /dev/nbd0p1 /mnt
 + sudo kvm -hda /​mnt/​var/​tmp/​kameleon/​2010-xx-xx-xx-xx-xx/​debian.qcow
 +</​code>​
 +  *  You can copy the file from the running kameleon appliance to the host via ssh:
 +<​code>​
 + # from the appliance:
 + scp /​var/​tmp/​kameleon/​2010-xx-xx-xx-xx-xx/​debian.qcow2 <​user>​@`ip route show|grep default|cut -d " " -f3`
 +</​code>​
 +
 +===== Should I care about sfdisk complaints like "​sfdisk:​ ERROR: sector 0 does not have an msdos signature"​ or "​BLKRRPART:​ Inappropriate ioctl for device"?​ =====
 +No. Don't worry. We are not going to destroy your system :-) This is just because we run sfdisk on an image file that is not a block device and we haven'​t find a way to prevent sfdisk to output such errors in spite it does the job.
 +===== Image is build, but it does not boot =====
 +The provided recipes require Grub2 version 1.98 or above on your host system (the one that you run kameleon on). We tried a fix for Grub2 1.97 (activate the grub_197_workaround microstep of the build_appliance macrostep) but it was not efficient for us. Tell us if you have more luck!
 +===== I don't want to restart from the bootstrap at each time I run Kameleon to test a new step. How can I do? =====
 +You can use our checkpoint/​checkpoint_resume macrosteps. Do a first recipe that runs a **checkpoint** step where you want (you can uncomment the one we have already set into our provided recipes). It will create the file referenced by the global variable **checkpoint_file**. Then, on the next run, you can add the **checkpoint_resume** step in place of the **bootstrap** step and comment every step that should already have been done into your checkpoint file, except **mount_proc** and such steps that are required for other steps started later into your recipe. Here is an example of the step part of the debian.yaml recipe when using a checkpoint:
 +
 +<​code>​
 + ​steps:​
 +  - debian_check_deps
 +  - check_deps:
 +    - rsync
 +    - building_appliance
 +    - building_kvm_images
 + # - oar/​oar_precheck
 +  - checkpoint_resume
 + # - bootstrap
 + # - system_config
 + # - network_config_static
 + # - root_passwd
 +  - mount_proc
 + # - software_install:​
 + # ​  - extra_packages
 + # - kernel_install
 + # - checkpoint
 + # - oar/​oar_debian_install
 + # - oar/​oar_system_config
 +  - kameleon/​kameleon_debian_install
 +  - strip
 +  - umount_proc ​
 + [...]
 +</​code>​
 +
 +===== I get the following error: W: Failure trying to run: chroot /​var/​tmp/​kameleon/​[...]/​chroot mount -t proc proc /proc  =====
 +It means that you are using a 32 bit system and your recipe is for 64 bit. Unfortunately,​ you can't build a 64 bits system from a 32 bits one. You probably have to set **arch: i386** and **kernel_arch:​ "​686"​** into your recipe.
 +
 +===== How clean is my host system after a kameleon run? =====
 +Some kameleon steps can mount or create special devices that may be left in an inconsistent state in case of an error or a break in the middle of an appliance generation. To prevent as much as possible from this, there'​s a special cleaning script that is automatically executed on exit. This script is automatically generated by using the **exec_on_clean** microstep commands. For example, if you have a microstep that creates a loopback device using:
 +<​code>​
 + - exec_chroot:​ losetup /dev/loop10 image.raw
 +</​code>​
 +you can add just after, the following microstep command:
 +<​code>​
 + - exec_on_clean:​ losetup -d /dev/loop1
 +</​code>​
 +
 +On exit (normal exit, or on a failure), kameleon will execute all the exec_on_clean commands in the reverse order.
 +
 +===== How to connect to my appliance using ssh or any protocol from a KVM host? =====
 +You can use the **-redir** option of KVM. Example:
 +<​code>​
 + kvm /​var/​tmp/​kameleon/​2010-05-03-13-23-52/​debian.raw -redir tcp:​2222::​22 -redir tcp:8080::8
 +</​code>​
 +Then you can:
 +<​code>​
 + $ ssh -p 2222 kameleon@localhost
 + # Or:
 + $ wget -O - http://​localhost:​8080/​oarapi/​resources.yam
 +</​code>​
 +===== How to put my image on an LVM device? =====
 +You just have to make a **raw** appliance (//​save_as_raw//​ microstep of the //​build_appliance//​ macrostep) and copy it to your LVM device:
 +<​code>​
 + dd bs=1M if=/​var/​tmp/​kameleon/​2010-05-04-14-15-06/​debian.raw of=/​dev/​vg-noraid/​kvm-idjant
 +</​code>​
 +Then, if your LVM device is bigger than the raw image (it should at least be the same size) boot your appliance, and from it, use **fdisk** to recreate the partitions, keeping the same order. Don't be afraid, your system should still boot. Reboot it, and from the appliance again, now, make:
 +<​code>​
 + ​resize2fs /dev/hda
 +</​code>​
 +so that the appliance filesystem will use all the space you gave to him in the new partitions scheme.
 +===== What are contexts and how to use them? =====
 +A context allows you to extend the microsteps commands. It may be used, for example, if you want to develop the steps from the inside of a running appliance, and then be able to switch to the appliance generation from a host just by changing your context.
 +To define a new context, you have to create a **contexts** hash into the recipe. Here is an example:
 +<​code>​
 + ​contexts:​
 +  root:
 +   cmd: chroot $$chroot %
 +</​code>​
 +This defines one context named "​root"​. The **%%** symbol will be replaced by the argument from the microstep at execution time. The context is used, in a microstep, as follow:
 +<​code>​
 + ​test_context:​
 + - pwd:
 +   - **exec_context:​ root** pw
 +</​code>​
 +For this example, it will exactly have the same effect as the following microstep:
 +<​code>​
 + ​test_context:​
 + - pwd:
 +   - **exec_chroot** pw
 +</​code>​
 +Both result in the following bash example command:
 +<​code>​
 + ​chroot /​var/​tmp/​kameleon/​2011-02-24-09-20-11/​chroot pw
 +</​code>​
 +So, now, you can change for example, the root context with:
 +<​code>​
 + root:
 +  cmd: ssh -T -n my-host "​%%"​
 +  escape: \\\\
 +</​code>​
 +The pwd microstep will now result into:
 +<​code>​
 + ssh -T -n my-host "pwd
 +</​code>​
 +The **escape** field allows you to define a specific character to be escaped. Here, it will allow such microstep to work:
 +<​code>​
 +   - **exec_context:​ root** echo "Hello World
 +</​code>​
 +You can define several contexts into the **contexts** hash of the recipe. More examples may be found into the debian.yaml default recipe.
 +
 +===== I need to associate some data files (software config, etc.) to the recipe I've written. How can I embed them ? =====
 +since the version 1.2.9, kameleon provides a two variables $$step_dir and $$recipe_dir. These variables contains the folder associated to the step/​recipe. ​
 +
 +For example, if you need to embed data associated the step '/​home/​user/​kameleon/​steps/​config_my_software.yaml',​ you can put you data in the folder '/​home/​user/​kameleon/​steps/​config_my_software/'​. During the kameleon execution the variable $$step_dir will contain the path '/​home/​user/​kameleon/​steps/​config_my_software/'​.
 +
 +You can process in the same way for the recipe: the data content associated to the recipe '/​home/​user/​kameleon/​recipes/​my_recipe.yaml'​ can be put in the folder '/​home/​user/​kameleon/​recipes/​my_recipe/'​ and during the kameleon execution the variuable $$recipe_dir will contain the path '/​home/​user/​kameleon/​recipes/​my_recipe/'​.
 +
 +This mechanism is needed because, since kameleon supports potentially different source folder for recipe and step, the associated data path depends on the source folder path which is unknown before running kameleon.
 +
 +===== Where to report bugs or feature requests? =====
 +Here: https://​gforge.inria.fr/​tracker/?​group_id=2308
 +
 +===== I have questions not listed here, where can I ask for support? =====
 +Subscribe to the kameleon-users mailing list and feel free to ask your questions:
 +http://​lists.gforge.inria.fr/​mailman/​listinfo/​kameleon-users
 +
 +====== Acknowledgement ======
 +Initially developped by Darko Ilic during 2009 Google Summer of Code mentored by Yiannis Georgiou and Bruno Bzeznik (see http://​wiki-oar.imag.fr/​index.php/​Kameleon_Page),​ it is now maintained by Bruno Bzeznik and the OAR team in general.
  
wiki/old/kameleon.txt ยท Last modified: 2013/07/10 23:06 (external edit)
Recent changes RSS feed GNU Free Documentation License 1.3 Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki