===== OAR's Debian repositories setup ===== For a list of the existing management tools for Debian packages, see: * https://wiki.debian.org/HowToSetupADebianRepository Til August 2015, reprepro was used to manage OAR's Debian packages (thx plb). But: * it was difficult to use * it was difficult to maintain * it did not support multiple versions of a package in a distribution * snapshot support was limited. We switched to Aplty. ==== Installing aptly ==== * Support for dput/incoming -> deb changes file is available since version 0.9.6 of aptly * https://github.com/smira/aptly/issues/71 * http://beta.aptly.info/doc/aptly/repo/include/ * -> for some time, we used the [[http://www.aptly.info/download/|nightly]] package builds: ''deb http://repo.aptly.info/ nightly main'' * -> now we use the stable repo, which bizarrely is still named squeeze... : ''deb http://repo.aptly.info/ squeeze main'' ==== Repositories setup ==== OAR's Debian repos are managed with the following script: https://github.com/oar-team/oar-aptly/blob/master/oar-aptly.sh Which performs the following tasks: * ''manage-debian-mirrors'': create/update mirrors of the official Debian repositories + snaptshots * ''create-testing-repos'': create the testing local repositories for the target official Debian distributions * ''handle-incoming'': Handle incoming packages (dput -> inoticoming) The ''handle-incomming'' task is run by inoticoming, setup as a service started by crontab: ftpmaster@lig-oar:~$ crontab -l @reboot inoticoming --logfile $HOME/logs/inoticoming.log $HOME/incoming/ --chdir $HOME/incoming/ --stdout-to-log --stderr-to-log --suffix .changes $HOME/bin/oar-aptly.sh handle-incoming {} \; f This allows us to place packages in a local repo for testing/local distrubtions before considering uploading them to the official Debian repositories: * sid -> sid_beta -> deb http://oar-ftp.imag.fr/oar/2.5/debian/ sid_beta main * jessie-backports -> jessie-backports_beta -> deb http://oar-ftp.imag.fr/oar/2.5/debian/ jessie-backports_beta main * wheezy-backports -> whezzy-backports_beta -> deb http://oar-ftp.imag.fr/oar/2.5/debian/ wheezy-backports_beta main * aso Also we eventually keep mirrors or snapshots repos of the Debian official repositories, with OAR's packages inside only thanks to filters. === Issues to follow === * Backports import harmless error: https://github.com/smira/aptly/issues/278 ===== Some other notes about the use of aptly ===== These are examples of aptly commands, if manual actions are required. # Create a mirror for Debian sid $ aptly mirror create -architectures=amd64 -with-sources=true -dep-follow-all-variants=true -filter='Name (oar) | $Source (oar)' -filter-with-deps=false sid http://ftp.fr.debian.org/debian sid $ aptly mirror update sid # Create a snaptshot for the current version of oar in sid $ aptly aptly snapshot create sid_oar-2.5.4-2 from mirror sid $ aptly publish snapshot -distribution="sid_oar-2.5.4-2" sid_oar-2.5.4-2 # Create a local repository for testing oar packages $ aptly repo create -architectures=amd64 sid_testing $ aptly repo import sid sid_testing '$Source (oar)' $ aptly repo include -repo=sid_testing $PWD/build-area/oar_2.5.5~rc1-1_amd64.changes $ aptly publish repo -distribution="sid_testing" sid_testing Here we mirror Vincent's repo for the bpo8 packages, waiting for them to appear in Debian's official backports (https://ftp-master.debian.org/backports-new.html). ftpmaster@lig-oar:~$ aptly mirror create -architectures=amd64 -with-sources=true -dep-follow-source=true -dep-follow-all-variants=true -filter-with-deps=false -filter='$Source (oar)| Name (oar)' vdanjean-jessie https://people.debian.org/~vdanjean/debian jessie main Downloading https://people.debian.org/~vdanjean/debian/dists/jessie/InRelease... gpgv: Signature made Thu 25 Feb 2016 11:22:17 AM CET using RSA key ID 7EC7B751 [...] Mirror [vdanjean-jessie]: https://people.debian.org/~vdanjean/debian/ jessie [src] successfully added. You can run 'aptly mirror update vdanjean-jessie' to download repository contents. ftpmaster@lig-oar:~$ aptly mirror update vdanjean-jessie Downloading https://people.debian.org/~vdanjean/debian/dists/jessie/InRelease... gpgv: Signature made Thu 25 Feb 2016 11:22:17 AM CET using RSA key ID 7EC7B751 [...] Downloading & parsing package files... Downloading https://people.debian.org/~vdanjean/debian/dists/jessie/main/binary-amd64/Packages.bz2... Downloading https://people.debian.org/~vdanjean/debian/dists/jessie/main/binary-amd64/Packages.gz... Downloading https://people.debian.org/~vdanjean/debian/dists/jessie/main/source/Sources.bz2... Downloading https://people.debian.org/~vdanjean/debian/dists/jessie/main/source/Sources.gz... Applying filter... Packages filtered: 98 -> 13. Building download queue... Download queue: 15 items (7.73 MiB) Downloading https://people.debian.org/~vdanjean/debian/pool/main/o/oar/oar-server-mysql_2.5.6-2~bpo8+1_amd64.deb... Downloading https://people.debian.org/~vdanjean/debian/pool/main/o/oar/oar-common_2.5.6-2~bpo8+1_amd64.deb... Downloading https://people.debian.org/~vdanjean/debian/pool/main/o/oar/oar-doc_2.5.6-2~bpo8+1_all.deb... Downloading https://people.debian.org/~vdanjean/debian/pool/main/o/oar/oar-user-mysql_2.5.6-2~bpo8+1_amd64.deb... Downloading https://people.debian.org/~vdanjean/debian/pool/main/o/oar/oar-web-status_2.5.6-2~bpo8+1_all.deb... Downloading https://people.debian.org/~vdanjean/debian/pool/main/o/oar/oar-server_2.5.6-2~bpo8+1_amd64.deb... Downloading https://people.debian.org/~vdanjean/debian/pool/main/o/oar/oar-node_2.5.6-2~bpo8+1_amd64.deb... Downloading https://people.debian.org/~vdanjean/debian/pool/main/o/oar/oar-user_2.5.6-2~bpo8+1_amd64.deb... Downloading https://people.debian.org/~vdanjean/debian/pool/main/o/oar/oar-server-pgsql_2.5.6-2~bpo8+1_amd64.deb... Downloading https://people.debian.org/~vdanjean/debian/pool/main/o/oar/liboar-perl_2.5.6-2~bpo8+1_amd64.deb... Downloading https://people.debian.org/~vdanjean/debian/pool/main/o/oar/oar-user-pgsql_2.5.6-2~bpo8+1_amd64.deb... Downloading https://people.debian.org/~vdanjean/debian/pool/main/o/oar/oar-restful-api_2.5.6-2~bpo8+1_amd64.deb... Downloading https://people.debian.org/~vdanjean/debian/pool/main/o/oar/oar_2.5.6-2~bpo8+1.debian.tar.xz... Downloading https://people.debian.org/~vdanjean/debian/pool/main/o/oar/oar_2.5.6-2~bpo8+1.dsc... Downloading https://people.debian.org/~vdanjean/debian/pool/main/o/oar/oar_2.5.6.orig.tar.gz... Mirror `vdanjean-jessie` has been successfully updated. ftpmaster@lig-oar:~$ aptly snapshot create jessie-backports_waiting from mirror vdanjean-jessie Snapshot jessie-backports_waiting successfully created. You can run 'aptly publish snapshot jessie-backports_waiting' to publish snapshot as Debian repository. ftpmaster@lig-oar:~$ aptly publish -batch=false -gpg-key=D90D0568 -distribution=jessie-backports_waiting snapshot jessie-backports_waiting Loading packages... Generating metadata files and linking package files... Finalizing metadata files... Signing file 'Release' with gpg, please enter your passphrase when prompted: Clearsigning file 'Release' with gpg, please enter your passphrase when prompted: Snapshot jessie-backports_waiting has been successfully published. Please setup your webserver to serve directory '/home/ftpmaster/.aptly/public' with autoindexing. Now you can add following line to apt sources: deb http://your-server/ jessie-backports_waiting main deb-src http://your-server/ jessie-backports_waiting main Don't forget to add your GPG key to apt with apt-key. You can also use `aptly serve` to publish your repositories over HTTP quickly. Removing a version of the OAR packages: ftpmaster@lig-oar:~$ aptly repo remove stretch-backports_beta "Version (=2.5.8~rc4-1~bpo9+1)" Loading packages... [-] oar-user_2.5.8~rc4-1~bpo9+1_amd64 removed [-] oar-doc_2.5.8~rc4-1~bpo9+1_all removed [-] oar-server-dbgsym_2.5.8~rc4-1~bpo9+1_amd64 removed [-] oar-restful-api_2.5.8~rc4-1~bpo9+1_amd64 removed [-] oar-web-status_2.5.8~rc4-1~bpo9+1_all removed [-] oar-server-pgsql_2.5.8~rc4-1~bpo9+1_amd64 removed [-] oar-node_2.5.8~rc4-1~bpo9+1_amd64 removed [-] liboar-perl_2.5.8~rc4-1~bpo9+1_amd64 removed [-] oar-common_2.5.8~rc4-1~bpo9+1_amd64 removed [-] oar-user-dbgsym_2.5.8~rc4-1~bpo9+1_amd64 removed [-] oar-user-pgsql_2.5.8~rc4-1~bpo9+1_amd64 removed [-] oar-user-mysql_2.5.8~rc4-1~bpo9+1_amd64 removed [-] oar-common-dbgsym_2.5.8~rc4-1~bpo9+1_amd64 removed [-] oar-restful-api-dbgsym_2.5.8~rc4-1~bpo9+1_amd64 removed [-] oar-server_2.5.8~rc4-1~bpo9+1_amd64 removed [-] oar-server-mysql_2.5.8~rc4-1~bpo9+1_amd64 removed And then publish the modified repo: $ aptly publish -force-overwrite=true update jessie-backports_beta WARNING: force overwrite mode enabled, aptly might corrupt other published repositories sharing the same package pool. Loading packages... Generating metadata files and linking package files... Finalizing metadata files... Signing file 'Release' with gpg, please enter your passphrase when prompted: Clearsigning file 'Release' with gpg, please enter your passphrase when prompted: Cleaning up prefix "." components main... Publish for local repo ./jessie-backports_beta [amd64] publishes {main: [jessie-backports_beta]} has been successfully updated.