[LRUG] Deployment approach to rolling out huge changes

[Ed James (Alt)]

Hi all 

I'd really appreciate anyone with some experience of deploying big changes to production systems to offer their thoughts/advice.

I am some way through making a fundamental change to our application. I originally planned to deploy this feature incrementally, but as I've delved deeper and deeper into the work, I'm realising that this deployment approach is just not feasible. I was hoping to get some feedback on how others have rolled out these kinds of fundamental changes to production systems without downtime. We have several thousand paying customers so downtime is a definite no-no.

We have 3 environments available:
Staging: typical staging environment. Completely independent of production and fully-functional, including a working payment system. We populate our staging database with production snapshots daily.
Beta: a restricted production environment which uses the production database. It's just a separate application environment for testing beta features. We cannot test payments here because the 3rd-party payment provider's callbacks hit out production domain.
Production: as you'd expect.


All of my concerns lie at the application and data levels. The database schema changes are already in place in the production database, with the handful of new columns that we need just lying dormant for now. Now columns have been removed or renamed - only new columns have been added. (There will be some deletions, but we can leave the destructive db changes for later).

The approach I currently have in mind is as follows:

We create a scheduled task to run in our production environment, which will maintain database integrity with regards to keeping foreign keys correctly populated. We then deploy the feature branch to our Beta environment for some extended testing. Payments will have to be tested in our Staging environment. We can then deploy the feature to our production environment once testing has completed and shut off the scheduled task.

Some issues we are unsure about right now are:
We have several background tasks which also need to be changed. These jobs cannot be run in both the Production and Beta environments simultaneously, and cannot be shut down in our production environment. How can we test them?
Payments in our Production environment cannot be tested before a full release.
Once we deploy this to our users, there is no turning back i.e. we cannot bail out and revert to a previous release.


If anyone has any thoughts/advice, I'd love to hear from you.

Thanks!

-- 
Ed James
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20120906/089ef969/attachment-0003.html>