At the very first boot, a virtual node gets the server GPG public key and never get it again. Then, periodically (at boot and from cron), the virtual node makes an http query to get an update script from the server. The server signs it and the node checks the signature of the script before executing it locally:
The script looks like:
UPDATES="/var/lib/idhal-updates"
# Update #000
if [! -f $UPDATES/000 ]
then
#
# do update 000
#
touch $UPDATES/000
fi
# Update #001
if [! -f $UPDATES/001 ]
then
#
# do update 001
#
touch $UPDATES/001
fi
# and so on...
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFL6V1yWciA6gGguMwRAqU+AJ93IOh+lxAsuYFUUVlrsM6HGorUJACff4ko
s/bE76z4S4MzZZ1EkN+UyGk ======
====== wfbT
-----END PGP SIGNATURE-----
The key is that an update (a scriptlet) is executed only once.
if [! -f $UPDATES/000 ] then wget -q -O /tmp/gen_certif http://129.88.70.253:8080/cgi-bin/gen_certif?idhaltype=vnode || exit 1 chmod 700 /tmp/gen_certif /tmp/gen_certif || exit 1 rm -f /tmp/gen_certif /etc/init.d/openvpn stop /etc/init.d/openvpn start sleep 10 touch $UPDATES/000 fi
This script generates a unique ID and a certificate for the virtual node. Then, it outputs a bash script that makes an Openvpn configuration and copies the certificate.
#!/usr/bin/ruby
require 'fileutils'
require "cgi"
cgi = CGI.new
idhaltype = cgi['idhaltype']
EASY_RSA_DIR="/etc/openvpn/easy-rsa"
def get_next_id
id=0
keys_dir=Dir.new(EASY_RSA_DIR + "/keys")
keys_dir.each do |file|
if client_id=file.scan(/client(\\d+)\\.crt/)[0]
if client_id[0].to_i > id
id=client_id[0].to_i
end
end
end
return id+1
end
Dir::chdir(EASY_RSA_DIR)
id=get_next_id
system(". ./vars > /dev/null && ./pkitool client#{id} > /dev/null 2>&1")
f = File.new("#{easy_rsa_dir}/keys/client#{id}.type", "w")
f.puts idhaltype
f.close
puts "Content-type: text/plain"
puts
puts "#!/bin/bash"
puts "if [-d /etc/openvpn ] ; then"
puts "cat > /etc/openvpn/vpns.conf << EOF
client
dev tun
proto tcp
remote 129.88.70.253 4242
ca ca.crt
cert client.crt
key client.key
keepalive 10 60
cipher none
up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf
EOF"
puts "cat > /etc/openvpn/client.crt << EOF"
IO.foreach("#{easy_rsa_dir}/keys/client#{id}.crt") { |line| puts line }
puts "EOF"
puts "cat > /etc/openvpn/client.key << EOF"
IO.foreach("#{easy_rsa_dir}/keys/client#{id}.key") { |line| puts line }
puts "EOF"
puts "chmod 600 /etc/openvpn/client.key"
puts "fi"
port 4242 proto tcp dev tun0 ca ca.crt cert server.crt key server.key dh dh1024.pem server 10.134.0.0 255.255.0.0 #ifconfig-pool-persist ipp.txt keepalive 10 120 max-clients 1000 persist-key persist-tun status /var/log/openvpn-status.log # Script where we create/activate OAR nodes client-connect /etc/openvpn/client-up.bash client-disconnect /etc/openvpn/client-down.bash # G5K Routes pushing push "route 129.88.70.0 255.255.255.192" [[...]] # DNS pushing push "dhcp-option DNS 129.88.70.61" push "dhcp-option DNS grenoble.grid5000.fr"
Every virtual ip address is statically declared into DNS like this example:
vnode-7-38 A 10.134.7.38
Resources are added automatically when new tunnels come up. Resources are disabled (but not removed) when tunel goes down. This is made inside the client-up/down openvpn scripts:
#!/bin/bash
set -e
NODE_NAME=`host -t A $ifconfig_pool_remote_ip |awk -F": " '{if ($1=="name") print $2}'`
CLIENT_NAME=`host -t A $trusted_ip |awk -F": " '{if ($1=="name") print $2}'`
NODE=`oarnodes --sql "ip='$ifconfig_pool_remote_ip'"`
if [[|"$NODE" = "" ]]
then
oarnodesetting -a -h "$NODE_NAME" -p "ip=$ifconfig_pool_remote_ip"
fi
HIDHAL_TYPE=`cat /etc/openvpn/easy-rsa/keys/$common_name.type 2>/dev/null || true`
if ["$HIDHAL_TYPE" != ""]
then
oarnodesetting -p "idhaltype=$HIDHAL_TYPE" \\
-p "cluster=$HIDHAL_TYPE" -h "$NODE_NAME"
fi
oarnodesetting -p "idhalcn=$common_name" \\
-p "idhalclientip=$trusted_ip" \\
-p "idhalconnectedsince=`date`" \\
-p "idhalclientname=$CLIENT_NAME" \\
-h $NODE_NAME
echo "/usr/local/sbin/send_root_key.sh $NODE_NAME" | at now + 1 minute
oarnodesetting -s Alive -h "$NODE_NAME"
#!/bin/bash
set -e
NODE_NAME=`host -t A $ifconfig_pool_remote_ip |awk -F": " '{if ($1=="name") print $2}'` >> /tmp/down
oarnodesetting -s Absent -h "$NODE_NAME" >> /tmp/down
echo "$NODE_NAME" >> /tmp/down
Here is a sample resource:
168
network_address : vnode-2-98.grenoble.grid5000.fr
properties : besteffort=YES,cluster=vnode,cpuset=0,deploy=NO,desktop_computing=NO,idhalclientip=152.77.57.112,idhalclientname=browalle.ujf-grenoble.fr,idhalcn=client317,idhalconnectedsince=Thu Jul 3 20:32:35 CEST 2008,idhaltype=vnode,ip=10.134.2.98,network_address=vnode-2-98.grenoble.grid5000.fr,type=default
state : Absent
The main problem with this solution is that every communication between the nodes is made through the VPN. Actually, we want that the nodes that are on a same LAN or routed network, use their local interface to communicate directly. The solution might be on DNS side…