Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/telent/goldleaf

Repeatable Debian installs for virtual hosts
https://github.com/telent/goldleaf

Last synced: 23 days ago
JSON representation

Repeatable Debian installs for virtual hosts

Awesome Lists containing this project

README 

        
h1. What is this for?



Goldleaf is a tool for creating consistent and repeatable KVM Debian system images in which the packages installed, the versions of each package, and the answers to "debconf" configuration questions are specified precisely.



The package manifest and configuration files are text-based and are intended to be kept under Git version control.



The name 'goldleaf' comes from the article "Golden Image or Foil Ball":http://madstop.com/2009/02/04/golden-image-or-foil-ball/ by Luke Kanies.  On which note: it is unclear to me whether he would see this script as a good thing or as a bad thing, but I would argue that even if you reduce the number of images under your control to a single "stem cell" image, being able to recreate (any current or previous version of) that image on-demand is just as valuable as being able to recreate (any current or previous version of) any other application.



h2. What does it do?



# it makes a base system hierarchy using "debootstrap":http://packages.debian.org/search?keywords=debootstrap

# it copies all your package selection data and config files into @/usr/local/master/template@ on the target system, along with a  Makefile that will install all the packages and files so specified into their rightful places.

# it makes a disk image from that with @dd@ and @kpartx@ and @mkfs@ and loopback mounts and stuff

# it makes the image bootable with "extlinux":http://syslinux.zytor.com/wiki/index.php/EXTLINUX.  I'm not going to say anything nasty about Grub, because the problem might just be that I'm too stupid to understand it.  But I had no such trouble with extlinux,which is small, wonderful, and exactly perfect for the task of finding a kernel and initrd on an ext{2,3,4}fs disk and making it boot.  And it has documentation.

# when you boot the image for the first time, a script in @/etc/rc.local@ does the magic referred to above to get everything installed and all your files from @/usr/local/master/template@ to @/@.  We defer this operation to first boot instead of doing it when the image is created, because it makes the image itself smaller - if you create the image locally and upload to a cloud host, better to have the package downloads happen directly on the cloud host than to do it locally then upload a much fatter image over the slow side of a consumer adsl line.



h2. Some assumptions and caveats



* that you're using Git for version control.   If you prefer not to, you could use this @Makefile@ just for the initial image creation, but if you have your config in Git you can make changes on the running target system and push them back into version control to be used on future iterations of image creation.  The Git setup is regrettably not as fancy as "etckeeper":http://kitenet.net/~joey/code/etckeeper/ but etckeeper wasn't available - or at least, wasn't known to me - when I started down this road.  This is a much simpler approach: instead of editing files in @/etc@ directly, make a git-controlled copy in @/usr/local/master@ then run @make sync@ to copy into @/etc@ when done.



* that you have some reasonable way to determine which are the packages you need.  Either you can copy a list from an existing host that does more or less what you want to do, or you can manually install stuff on an "empty" goldfoil host and then take the package list when you're happy with it.  If you're working from an existing system, I'm going to refer to it henceforth as the donor system.



* the package version pinning depends on aptitude to get things right, and has not been tested by me extensively.  It is possible that in some situations you might find yourself running aptitude install a few times until things settle down and it's happy that all its ducks line up, just as you do on an manually installed Debian system. Once you've got it to a stable state, you can snapshot that state again.



* you have an apt proxy.  The internet appreciates careful caching.



h2. How to use it



# @git clone git://github.com/telent/goldleaf.git@

# @(cd goldleaf && make install)@ 

# @git clone git://github.com/telent/goldleaf-example.git@

# @cd goldleaf-example@

# @git remote rm origin@ # better safe than sorry

# This git repository will be replicated onto the target system, so set up a remote now (and push to it, if necessary) that you can push to from the target.  @git remote add host ssh://`hostname``pwd`@ will probably do the trick, if you're running sshd and you don't have some other git setup that you'd prefer to use.

# edit the @Makefile@ - you will at minimum want to change @PROXY@; for other variables you can play with, see @../goldleaf/goldleaf.mk@

# The directory @template@ is where you put the files you want to install on the target system - if you have an existing system to take config from, copy interesting bits of its @/etc/@ into @template/etc@ here. Do include: config files that you've changed from the defaults that Debian installed.  Don't include: anything that came out of a package and that you haven't changed since. If you leave this empty, you will get the Debian default @passwd@, @group@, and @shadow@ files (i.e. without any valid user accounts - note that *this means you have no way to log in*), fairly rudimentary @hosts@ and @fstab@ files, and an @apt/sources.list@ which has @PROXY@ written into it - fine if you are deploying on the same network as your proxy service, less so otherwise.

# If you have a donor system, use it to seed the package list.  Run

@debconf-get-selections > debconf-selections.list ; aptitude -q -F "%?p=%?V %M" --disable-columns search \~i  > installed-packages.list@

on it and copy the files resulting to @template/etc/@.

# run @make disk.img@.  This creates a qemu-compatible disk image of a Debian base image, plus the gubbins in @/usr/local/master@ that make it install the full system when it boots up

# boot the resulting image in kvm: either locally (there's a @make kvm@ target which may or may not fit your needs), or by copying it to The Cloud(tm) and starting it there.

# wait for it to install everything.  It won't need much interaction, but may need supervision (note: you'll need console access) as some packages insist on stopping and asking questions interactively despite my best efforts.

# reboot



You're done.  More or less.  But you probably want to fine-tune the package list.  Log into the freshly booted image again (you did install a valid @/etc/passwd@ somehow, didn't you?) and run the aptitude commands to install the packages you want, then run @# aptitude update && aptitude upgrade@ until the package state settles down, then do



@# cd /usr/local/master && make show-upgraded save-upgraded@



This saves the package state (don't forget to @git commit@ and @git push@) so that future image creation runs start from where you are now.



This time you're definitely done.  But I do further commend to you, unless you already have a better way to manage @/etc@, the practice of continuing to use @/usr/local/master/template/etc@ as a version-controlled repository for it.  There are two rules



* to make changes to files in @/etc@, first copy them (if necessary)  to @/usr/local/master/template/etc@, edit the copy, @git commit@ the  change, and run @make sync@ when done.  Files in  @/usr/local/master/template/@ will overwrite corresponding files in  @/@ whenever you run @make sync@



* after upgrading or installing Debian packages, run 

@cd /usr/local/master && make show-upgraded save-upgraded@

to record the new package state.  Installed package versions and debconf configuration will be overridden by the data in @template/etc/*.list@ whenever you run @make sync@



Caution: @make sync@ is a *one-way sync* - it will overwrite files in the target, and without warning, if they have been changed more recently than their counterparts in the repository.



h2. Further customization



There are a bunch of tweakables.  If you want to create or mount filesystems (other than the root) on the target system, write a file @firstboot-mkfs.sh@ which does that.  If you want to do other one-time installation after package installation, you can put that in @firstboot-postinstall.sh@.  To change the hostname, base Debian release, or disk layout, add definitions at the top of the Makefile.  If you have files which need particular ownership or permissions, add them to @permissions.sh@



The @template@ directory is not limited to use with @/etc@ - you can add stuff in @/usr/local/src@, or put user dotfiles in @template/home/foo/@, or whatever other uses come to mind, provided you (and the user) are happy that they will overwrite whatever changes are made on the filesystem each time you @make sync@



It's only been tested with Debian squeeze.  It should work with other versions, if they are new enough.  It may work with Ubuntu.