Mechanism

Secured update procedure

At the very first boot, a virtual node gets the server GPG public key and never get it again. Then, periodically (at boot and from cron), the virtual node makes an http query to get an update script from the server. The server signs it and the node checks the signature of the script before executing it locally:

idhal_updates.png

The update script

The script looks like:

 UPDATES="/var/lib/idhal-updates"
 # Update #000
 if [! -f $UPDATES/000 ]
 then
    #
    # do update 000
    #
    touch $UPDATES/000
 fi
 # Update #001
 if [! -f $UPDATES/001 ]
 then
    #
    # do update 001
    #
    touch $UPDATES/001
 fi
 # and so on...
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.6 (GNU/Linux)
 iD8DBQFL6V1yWciA6gGguMwRAqU+AJ93IOh+lxAsuYFUUVlrsM6HGorUJACff4ko
 s/bE76z4S4MzZZ1EkN+UyGk ======
====== wfbT
 -----END PGP SIGNATURE-----

The key is that an update (a scriptlet) is executed only once.

The first update sets up the tunnel

Update

 if [! -f $UPDATES/000 ]
 then
   wget -q -O /tmp/gen_certif http://129.88.70.253:8080/cgi-bin/gen_certif?idhaltype=vnode || exit 1
   chmod 700 /tmp/gen_certif
   /tmp/gen_certif || exit 1
   rm -f /tmp/gen_certif
   /etc/init.d/openvpn stop
   /etc/init.d/openvpn start
   sleep 10
   touch $UPDATES/000
 fi

gen_certif cgi-bin script

This script generates a unique ID and a certificate for the virtual node. Then, it outputs a bash script that makes an Openvpn configuration and copies the certificate.

 #!/usr/bin/ruby
 require 'fileutils'
 require "cgi"
 cgi = CGI.new
 idhaltype = cgi['idhaltype'] 
 EASY_RSA_DIR="/etc/openvpn/easy-rsa" 
 def get_next_id
   id=0
   keys_dir=Dir.new(EASY_RSA_DIR + "/keys")
   keys_dir.each do |file|
     if client_id=file.scan(/client(\\d+)\\.crt/)[0]
       if client_id[0].to_i > id
         id=client_id[0].to_i
       end
     end
   end
   return id+1
 end
 Dir::chdir(EASY_RSA_DIR) 
 id=get_next_id
 system(". ./vars > /dev/null && ./pkitool client#{id} > /dev/null 2>&1")
 f = File.new("#{easy_rsa_dir}/keys/client#{id}.type",  "w")
 f.puts idhaltype
 f.close
 puts "Content-type: text/plain"
 puts
 puts "#!/bin/bash"
 puts "if [-d /etc/openvpn ] ; then"
 puts "cat > /etc/openvpn/vpns.conf << EOF
 client
 dev tun
 proto tcp
 remote 129.88.70.253 4242
 ca ca.crt
 cert client.crt
 key client.key
 keepalive 10 60
 cipher none
 up /etc/openvpn/update-resolv-conf
 down /etc/openvpn/update-resolv-conf
 EOF"
 puts "cat > /etc/openvpn/client.crt << EOF"
 IO.foreach("#{easy_rsa_dir}/keys/client#{id}.crt") { |line| puts line }
 puts "EOF" 
 puts "cat > /etc/openvpn/client.key << EOF"
 IO.foreach("#{easy_rsa_dir}/keys/client#{id}.key") { |line| puts line }
 puts "EOF" 
 puts "chmod 600 /etc/openvpn/client.key" 
 puts "fi"

The VPN server side configuration

Openvpn

 port 4242
 proto tcp
 dev tun0
 ca ca.crt
 cert server.crt
 key server.key
 dh dh1024.pem
 server 10.134.0.0 255.255.0.0
 #ifconfig-pool-persist ipp.txt
 keepalive 10 120
 max-clients 1000
 persist-key
 persist-tun
 status /var/log/openvpn-status.log
 # Script where we create/activate OAR nodes
 client-connect /etc/openvpn/client-up.bash
 client-disconnect /etc/openvpn/client-down.bash
 # G5K Routes pushing
 push "route 129.88.70.0 255.255.255.192"
 [[...]]
 # DNS pushing
 push "dhcp-option DNS 129.88.70.61"
 push "dhcp-option DNS grenoble.grid5000.fr"

DNS

Every virtual ip address is statically declared into DNS like this example:

 vnode-7-38              A       10.134.7.38

OAR server

Resources are added automatically when new tunnels come up. Resources are disabled (but not removed) when tunel goes down. This is made inside the client-up/down openvpn scripts:

/etc/openvpn/client-up.bash:

 #!/bin/bash
 set -e
 NODE_NAME=`host -t A $ifconfig_pool_remote_ip |awk -F": " '{if ($1=="name") print $2}'`
 CLIENT_NAME=`host -t A $trusted_ip |awk -F": " '{if ($1=="name") print $2}'`
 NODE=`oarnodes --sql "ip='$ifconfig_pool_remote_ip'"`
 if [[|"$NODE" = "" ]]
 then
   oarnodesetting -a -h "$NODE_NAME" -p "ip=$ifconfig_pool_remote_ip"
 fi
 HIDHAL_TYPE=`cat /etc/openvpn/easy-rsa/keys/$common_name.type 2>/dev/null || true`
 if ["$HIDHAL_TYPE" != ""]
 then
   oarnodesetting -p "idhaltype=$HIDHAL_TYPE" \\
                  -p "cluster=$HIDHAL_TYPE" -h "$NODE_NAME"
 fi
 oarnodesetting -p "idhalcn=$common_name" \\
                -p "idhalclientip=$trusted_ip" \\
               -p "idhalconnectedsince=`date`" \\
               -p "idhalclientname=$CLIENT_NAME" \\
               -h $NODE_NAME
 echo "/usr/local/sbin/send_root_key.sh $NODE_NAME" | at now + 1 minute
 oarnodesetting -s Alive -h "$NODE_NAME"

/etc/openvpn/client-down.bash

 #!/bin/bash
 set -e
 NODE_NAME=`host -t A $ifconfig_pool_remote_ip |awk -F": " '{if ($1=="name") print $2}'` >> /tmp/down
 oarnodesetting -s Absent -h "$NODE_NAME" >> /tmp/down
 echo "$NODE_NAME" >> /tmp/down

Here is a sample resource:

 168
        network_address : vnode-2-98.grenoble.grid5000.fr
        properties : besteffort=YES,cluster=vnode,cpuset=0,deploy=NO,desktop_computing=NO,idhalclientip=152.77.57.112,idhalclientname=browalle.ujf-grenoble.fr,idhalcn=client317,idhalconnectedsince=Thu Jul  3 20:32:35 CEST 2008,idhaltype=vnode,ip=10.134.2.98,network_address=vnode-2-98.grenoble.grid5000.fr,type=default
        state : Absent

Caveats

The main problem with this solution is that every communication between the nodes is made through the VPN. Actually, we want that the nodes that are on a same LAN or routed network, use their local interface to communicate directly. The solution might be on DNS side…

Table of Contents