[LRUG] Deployment approach to rolling out huge changes

[Paul Robinson]

On 6 Sep 2012, at 18:02, "Ed James (Alt)" <ed.james.spam at gmail.com> wrote:

> We have several background tasks which also need to be changed. These jobs cannot be run in both the Production and Beta environments simultaneously, and cannot be shut down in our production environment. How can we test them?

This smells bad. 

First, I always say we must assume everything fails all the time - words of Amazon's CTO which I suggest are apt for any setup. 

What would *actually* happen if those scripts stopped running? What happens when the server gets patched? You are regularly patching your production servers, yes? If not, you have bigger problems - deal with them first. :-)

Secondly, what is the purpose of beta? What is it buying you that staging (with its fully propagated snapshot - something else that smells bad to my nose) and a CI server is not?

I always get a slightly bad feeling in my spidey-senses when I see real life production data heading into a test environment. It normally means somebody, somewhere, doesn't trust the tests or the test data. It normally comes down to somebody being just a tiny bit... lazy? I'd love to know the use case for that setup here.

> Payments in our Production environment cannot be tested before a full release.

But your testing environment should be good enough that you don't need to. Or at least not in the way you think about it.

I know that's not a very helpful answer, but it's true. Payment testing is a dark art, but you should be able to do it without any problems in a test environment if you want to. You might say "no, that's not possible - callbacks!", but ask what is it you're testing? Your code, or the payment processors?

If it's the payment processors' code, you need to obviously do something a little different (and start by asking why you're doing that), but if it's *your* code and how you would respond to various callback results, and you want the test coverage around that (and if there's one bit you do, it's that!), then mocking is the way forward. Yes, it's a bit expensive, but it's worth it. Honestly.

> Once we deploy this to our users, there is no turning back i.e. we cannot bail out and revert to a previous release.

There's always a way to rollback with a little thought. Heck, snapshotting everything an hour before the deploy and then rolling back to that point is an option if you need it! :-)

Paul

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20120906/c51f7b6f/attachment-0003.html>