[LRUG] Deployment approach to rolling out huge changes

[Ed James (Alt)]

Thanks Paul, I appreciate your comments. Here's some more info in response to your feedback…

> What would *actually* happen if those scripts stopped running? What happens when the server gets patched? You are regularly patching your production servers, yes? If not, you have bigger problems - deal with them first. :-)
The scripts do sometimes fail, but they are restarted by a monitoring daemon. The issue is not that they sometimes fail and restart (which they do), but that once our changes are deployed the way these jobs process data will change.


> Secondly, what is the purpose of beta? What is it buying you that staging (with its fully propagated snapshot - something else that smells bad to my nose) and a CI server is not?
The Beta environment allows us to see application changes in a production environment using live data. It means we don't always have to put new beta features behind things like beta flags or introduce conditions into the code. This is quite common practice as far as I know (Facebook do something similar).  

  
> I'd love to know the use case for that setup here.

Using production snapshots in a staging environment gives us real data to play with. Test data can only take you so far, because you have to create that test data. There are also parts of our system (mainly the UI) that are not covered by automated tests (I know…). Our application is several years old and the database snapshot is over 2GB. Using snapshots also means when we deploy to our staging environment we are testing the deployment itself.


> There's always a way to rollback with a little thought. Heck, snapshotting everything an hour before the deploy and then rolling back to that point is an option if you need it! :-)
Yes, while it would be technically possible to roll back, it's just not feasible. We would lose data (i.e. money) and our users would also lose statistical data. We also have a large number of tables and a fairly complex schema, so restoring data integrity post-rollback would also be a very tricky task, and something we want to avoid having to do.



--  
Ed James (Spam)
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)


On Thursday, 6 September 2012 at 18:26, Paul Robinson wrote:

> On 6 Sep 2012, at 18:02, "Ed James (Alt)" <ed.james.spam at gmail.com (mailto:ed.james.spam at gmail.com)> wrote:
>  
> > We have several background tasks which also need to be changed. These jobs cannot be run in both the Production and Beta environments simultaneously, and cannot be shut down in our production environment. How can we test them?
>  
> This smells bad.  
>  
> First, I always say we must assume everything fails all the time - words of Amazon's CTO which I suggest are apt for any setup.  
>  
> What would *actually* happen if those scripts stopped running? What happens when the server gets patched? You are regularly patching your production servers, yes? If not, you have bigger problems - deal with them first. :-)
>  
> Secondly, what is the purpose of beta? What is it buying you that staging (with its fully propagated snapshot - something else that smells bad to my nose) and a CI server is not?
>  
> I always get a slightly bad feeling in my spidey-senses when I see real life production data heading into a test environment. It normally means somebody, somewhere, doesn't trust the tests or the test data. It normally comes down to somebody being just a tiny bit... lazy? I'd love to know the use case for that setup here.
> > Payments in our Production environment cannot be tested before a full release.
>  
> But your testing environment should be good enough that you don't need to. Or at least not in the way you think about it.
>  
> I know that's not a very helpful answer, but it's true. Payment testing is a dark art, but you should be able to do it without any problems in a test environment if you want to. You might say "no, that's not possible - callbacks!", but ask what is it you're testing? Your code, or the payment processors?
>  
> If it's the payment processors' code, you need to obviously do something a little different (and start by asking why you're doing that), but if it's *your* code and how you would respond to various callback results, and you want the test coverage around that (and if there's one bit you do, it's that!), then mocking is the way forward. Yes, it's a bit expensive, but it's worth it. Honestly.
> > Once we deploy this to our users, there is no turning back i.e. we cannot bail out and revert to a previous release.
> There's always a way to rollback with a little thought. Heck, snapshotting everything an hour before the deploy and then rolling back to that point is an option if you need it! :-)
> Paul  
> _______________________________________________
> Chat mailing list
> Chat at lists.lrug.org (mailto:Chat at lists.lrug.org)
> http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>  
>  


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20120907/2ae6d0c3/attachment-0003.html>