Idhal virtual cluster prototype
From WikiOAR
Contents |
Mechanism
Secured update procedure
At the very first boot, a virtual node gets the server GPG public key and never get it again. Then, periodically (at boot and from cron), the virtual node makes an http query to get an update script from the server. The server signs it and the node checks the signature of the script before executing it locally:
The update script
The script looks like:
UPDATES="/var/lib/idhal-updates" # Update #000 if [ ! -f $UPDATES/000 ] then # # do update 000 # touch $UPDATES/000 fi # Update #001 if [ ! -f $UPDATES/001 ] then # # do update 001 # touch $UPDATES/001 fi # and so on... -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFL6V1yWciA6gGguMwRAqU+AJ93IOh+lxAsuYFUUVlrsM6HGorUJACff4ko s/bE76z4S4MzZZ1EkN+UyGk= =wfbT -----END PGP SIGNATURE-----
The key is that an update (a scriptlet) is executed only once.
The first update sets up the tunnel
Update
if [ ! -f $UPDATES/000 ] then wget -q -O /tmp/gen_certif http://129.88.70.253:8080/cgi-bin/gen_certif?idhaltype=vnode || exit 1 chmod 700 /tmp/gen_certif /tmp/gen_certif || exit 1 rm -f /tmp/gen_certif /etc/init.d/openvpn stop /etc/init.d/openvpn start sleep 10 touch $UPDATES/000 fi
gen_certif cgi-bin script
This script generates a unique ID and a certificate for the virtual node. Then, it outputs a bash script that makes an Openvpn configuration and copies the certificate.
#!/usr/bin/ruby
require 'fileutils'
require "cgi"
cgi = CGI.new
idhaltype = cgi['idhaltype']
EASY_RSA_DIR="/etc/openvpn/easy-rsa"
def get_next_id
id=0
keys_dir=Dir.new(EASY_RSA_DIR + "/keys")
keys_dir.each do |file|
if client_id=file.scan(/client(\d+)\.crt/)[0]
if client_id[0].to_i > id
id=client_id[0].to_i
end
end
end
return id+1
end
Dir::chdir(EASY_RSA_DIR)
id=get_next_id
system(". ./vars > /dev/null && ./pkitool client#{id} > /dev/null 2>&1")
f = File.new("#{EASY_RSA_DIR}/keys/client#{id}.type", "w")
f.puts idhaltype
f.close
puts "Content-type: text/plain"
puts
puts "#!/bin/bash"
puts "if [ -d /etc/openvpn ] ; then"
puts "cat > /etc/openvpn/vpns.conf << EOF
client
dev tun
proto tcp
remote 129.88.70.253 4242
ca ca.crt
cert client.crt
key client.key
keepalive 10 60
cipher none
up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf
EOF"
puts "cat > /etc/openvpn/client.crt << EOF"
IO.foreach("#{EASY_RSA_DIR}/keys/client#{id}.crt") { |line| puts line }
puts "EOF"
puts "cat > /etc/openvpn/client.key << EOF"
IO.foreach("#{EASY_RSA_DIR}/keys/client#{id}.key") { |line| puts line }
puts "EOF"
puts "chmod 600 /etc/openvpn/client.key"
puts "fi"
The VPN server side configuration
Openvpn
port 4242 proto tcp dev tun0 ca ca.crt cert server.crt key server.key dh dh1024.pem server 10.134.0.0 255.255.0.0 #ifconfig-pool-persist ipp.txt keepalive 10 120 max-clients 1000 persist-key persist-tun status /var/log/openvpn-status.log # Script where we create/activate OAR nodes client-connect /etc/openvpn/client-up.bash client-disconnect /etc/openvpn/client-down.bash # G5K Routes pushing push "route 129.88.70.0 255.255.255.192" [...] # DNS pushing push "dhcp-option DNS 129.88.70.61" push "dhcp-option DNS grenoble.grid5000.fr"
DNS
Every virtual ip address is statically declared into DNS like this example:
vnode-7-38 A 10.134.7.38
OAR server
Resources are added automatically when new tunnels come up. Resources are disabled (but not removed) when tunel goes down. This is made inside the client-up/down openvpn scripts:
- /etc/openvpn/client-up.bash:
#!/bin/bash
set -e
NODE_NAME=`host -t A $ifconfig_pool_remote_ip |awk -F": " '{if ($1=="Name") print $2}'`
CLIENT_NAME=`host -t A $trusted_ip |awk -F": " '{if ($1=="Name") print $2}'`
NODE=`oarnodes --sql "ip='$ifconfig_pool_remote_ip'"`
if [ "$NODE" = "" ]
then
oarnodesetting -a -h "$NODE_NAME" -p "ip=$ifconfig_pool_remote_ip"
fi
HIDHAL_TYPE=`cat /etc/openvpn/easy-rsa/keys/$common_name.type 2>/dev/null || true`
if [ "$HIDHAL_TYPE" != "" ]
then
oarnodesetting -p "idhaltype=$HIDHAL_TYPE" \
-p "cluster=$HIDHAL_TYPE" -h "$NODE_NAME"
fi
oarnodesetting -p "idhalcn=$common_name" \
-p "idhalclientip=$trusted_ip" \
-p "idhalconnectedsince=`date`" \
-p "idhalclientname=$CLIENT_NAME" \
-h $NODE_NAME
echo "/usr/local/sbin/send_root_key.sh $NODE_NAME" | at now + 1 minute
oarnodesetting -s Alive -h "$NODE_NAME"
- /etc/openvpn/client-down.bash
#!/bin/bash
set -e
NODE_NAME=`host -t A $ifconfig_pool_remote_ip |awk -F": " '{if ($1=="Name") print $2}'` >> /tmp/down
oarnodesetting -s Absent -h "$NODE_NAME" >> /tmp/down
echo "$NODE_NAME" >> /tmp/down
Here is a sample resource:
168
network_address : vnode-2-98.grenoble.grid5000.fr
properties : besteffort=YES,cluster=vnode,cpuset=0,deploy=NO,desktop_computing=NO,idhalclientip=152.77.57.112,idhalclientname=browalle.ujf-grenoble.fr,idhalcn=client317,idhalconnectedsince=Thu Jul 3 20:32:35 CEST 2008,idhaltype=vnode,ip=10.134.2.98,network_address=vnode-2-98.grenoble.grid5000.fr,type=default
state : Absent
Caveats
The main problem with this solution is that every communication between the nodes is made through the VPN. Actually, we want that the nodes that are on a same LAN or routed network, use their local interface to communicate directly. The solution might be on DNS side...
