[LRUG] Idempotency vs the cloud

[Paul Mucur]

Having spoken about Puppet at LRUG before[0] and being currently in
the process of familiarising myself with Chef at a new company, I
initially found configuration management useful as a means of
unambiguous communication. My original need was driven by the pain of
a very error-prone, manual process of server setup (in the form of
gargantuan JIRA tickets to be performed in a different timezone) but
this could equally have been solved with the likes of AMIs
particularly with the emergence of tools like Packer[1].

I was particularly interested in Adrian Holovaty's comments on his new
set up for SoundSlice[2] which seem relevant to this discussion:

"Another approach would be to use Chef or Puppet to automatically
install the necessary packages on each new server at instantiation
time, instead of 'baking' the packages into the AMI itself. We opted
not to do this, because it was unnecessary complexity. The app is
simple enough that the baked-AMI approach works nicely."

The idea of disposable servers is very appealing, serving to reduce
the cost of outages but I do wonder if there is an exception to be
made with long-running, persistent services such as databases.

[0]: http://mudge.name/2011/08/11/managing-web-application-servers-with-puppet.html
[1]: http://www.packer.io/
[2]: http://www.holovaty.com/writing/aws-notes/

On Wed, Jul 17, 2013 at 12:55 PM, Paul Robinson <paul at 32moves.com> wrote:
> On 17 Jul 2013, at 12:40, Paul Battley <pbattley at gmail.com> wrote:
>
>> Except - whilst you can provision and terminate web servers with
>> reckless abandon, you can't quite do the same with data storage.
>> Networks are slow; well, they're never fast enough, anyway. I think
>> I'd rather have to fight Puppet than clone databases. Moving data
>> around seems like a problem that's going to be with us for a long
>> time, at least until we all move to magic self-healing distributed
>> databases, i.e. never.
>
>
> On this point I think right now you have a choice:
>
> 1. I'm too busy to think about this, I'll just pay for RDS from AWS and let them figure it out for me
>
> 2. I really care about this and not too busy, so I'll figure it out myself and get really annoyed when I realise MySQL's clustering is something my niece could have designed better and Postgres is not much better...
>
> We're moving from the latter to the former simply because we have bigger problems to work on than cloning databases and figuring out how to swap over to a new DB instance in a new AZ when the one your primary DB is in falls into misconfigured oblivion.
>
> Amazon worked it all out, charge a small premium, so we're happy to go with that soon.
>
> There is however some work to be done in the idea of storage (whether that be SQL, NoSQL, files or something else), when you do care about its state persisting beyond a server's lifetime.
>
> Right now we seem to say "I don't care if these servers fail! Look, I kill everything in this ASG and it bounces right back again! Hahaha! Except that one. Don't kill that one. That one is special. Please don't go near that one otherwise we're in trouble."
>
> And that's mainly to do with ultimately keeping bytes in RAM and on disc somewhere that is the intrinsic business value of what we're doing beyond the code.
>
> It's basically all the stuff our apps use that aren't in version control.
>
> And that might be the clue for some classes of app... if you're in a heavy-read very low-write environment, then using some kind of distributed version control might actually be a clever solution.
> _______________________________________________
> Chat mailing list
> Chat at lists.lrug.org
> http://lists.lrug.org/listinfo.cgi/chat-lrug.org