[LRUG] Idempotency vs the cloud

[Paul Robinson]

On 17 Jul 2013, at 12:13, Tom Stuart <tom at codon.com> wrote:

> The assumption is that we have persistent servers whose state needs to be carefully massaged over the course of geological time, and that's usually true, but would we have even bothered evolving these tools if some constraint or trend had made us more accustomed to deploying updates by spinning up a fresh EC2 instance instead of frobbing an existing one? If things begin to move more in that direction, can we stop caring about idempotent configuration management, and can it therefore become drastically simpler?


A couple of thoughts:

- Amazon OpsWorks which is their new funky stack/layer configuration management tool is basically Chef. Chef recipes allow you to say "This is what nginx looks like" and then get OpsWorks to use that recipe and pass in parameters. If you get your Chef repo in the same format OpsWorks uses, you can basically use Amazon's and your own in a point-and-click manner. Quite nice, I'm using it in anger properly for the first time this week.

- A good friend of mine used to work as a dev ops/sysadmin for Amazon in Dublin, and is shortly going back but this time to Seattle HQ. He has two telling attitudes that I think are the future of server management:

1. Assume everything fails all of the time and be fine with that
2. If you have to ssh into a server (or get a tool to ssh in once it has completed booting), you've taken a wrong turn somewhere

That last point is important. To be clear: if he has to ssh into a service, it's because he thing the configuration management is wrong and he's checking. He won't fix the server, he will change the configuration management tool and kill a thousand servers and spin them up again if he has to.

In that light then, I'd say think of Chef as evolving into being your tool to instruct the cloud providers toolchain on how you want specific services and environments to be configured.

You *could* use these to produce AMIs (or their equivalents in your preferred environment), and use those as your building blocks, but why bother if you can avoid it? If you can change a recipe, check it into git and click in a console to get new servers spinning up using that recipe, that seems more in the spirit of "release early, release often" and has much lower friction.

I think getting into the  mindset of seeing a "server" as being a tool that you will temporarily use to get a job done is a good thing. It's a weird shift, but I think within a few months we'll probably finish the day with a completely different set of physical server instances to the ones we started using in the morning. Every working day.

Once you get your head around that ideal, you can see where Chef can help you manage the boot process, but you don't need to go much further...