[LRUG] Idempotency vs the cloud

[Frederick Cheung]

That was definitely an ah-ha moment when I decided that synchronising state wasn't needed if servers are disposable entities

We build custom AMIs for the various server roles, and we put some time into making sure those can be built automatically but that's relatively simple - it's never anything beyond installing the right packages, occasionally compiling the odd stuff that isn't available as a package and copying over some configuration. I could never quite get into puppet/chef for this so we've got a slightly cobbled together capistrano setup for preparing our images.

Servers themselves are entirely disposable, I just boot one from the custom AMI and it's ready to go within 2-3 minutes with all our bits and bobs installed and ready to run our apps. 

You do need the right tools in place to be able to do this, for example being able to replace an instance without scrapping all the storage associated to it. If you end up rewriting configuration state and so on then you need to be able to introspect your setup and work out (for example) what needs to be added to the load balancer. Some tools/services are more difficult to automate like this, some are really easy (my favourite example of this is elasticsearch - give it a set of aws credentials, tags to filter on etc and it will go out and find the cluster it should join all by itself, synchronise its data and get going)

For things with no shared state (like app servers) this feels very right. It's a bit more of a pain with something like a database server, although even this can be made easier, e.g. if you have your data on a separate volume then you can create a new instance and attach the old data volume to the new instance without too much upheaval.


Fred


 
On 17 Jul 2013, at 12:13, Tom Stuart <tom at codon.com> wrote:

> Hi LRUG,
> 
> I enjoyed Simon's talk about Chef on Monday. I'm an occasional user of both Puppet and Chef, and I like the idea of configuration management in principle, but I've never achieved full proficiency with either and I'm interested in why I find it such a struggle to properly integrate them into my life.
> 
> One recurring thought I've had, which the talk reinforced, is that some of the complexity (and concomitant notional benefit) of these tools is their ability to convert a declarative description of a desirable state into an idempotent series of imperative actions. And okay, that's great, but is it a bit of a historical accident?
> 
> The assumption is that we have persistent servers whose state needs to be carefully massaged over the course of geological time, and that's usually true, but would we have even bothered evolving these tools if some constraint or trend had made us more accustomed to deploying updates by spinning up a fresh EC2 instance instead of frobbing an existing one? If things begin to move more in that direction, can we stop caring about idempotent configuration management, and can it therefore become drastically simpler?
> 
> This question appeals to me because of its connection with the imperative vs functional programming situation. Imperative programming is easier (in the Hickey[0] sense) because it more closely matches our intuitive stateful view of the world, but as any fule kno we eventually get into trouble because of the complexity which emerges from over-reliance on state. Functional programming offers an appealing remedy: stop worrying about state, treat everything as a mapping instead of a mutation, and concentrate on how to turn inputs into outputs. Running a one-shot shell script to provision a fresh instance feels simpler than getting Chef or Puppet to massage an existing one; it's not quite functional programming, but it's a gesture in that direction, if only because it doesn't concern itself with how to keep the plates spinning.
> 
> tl;dr: If you don't care about running your recipes/manifests/whatever repeatedly, do Chef and Puppet still add enough value to justify their complexity?
> 
> Cheers,
> -Tom
> 
> [0] http://www.infoq.com/presentations/Simple-Made-Easy
> _______________________________________________
> Chat mailing list
> Chat at lists.lrug.org
> http://lists.lrug.org/listinfo.cgi/chat-lrug.org