[LRUG] Continuous * (Happy New Year!)
Samuel Joseph
tansaku at gmail.com
Tue Jul 9 07:41:31 PDT 2019
Hi Everyone,
I was just reading the devops handbook which had the following to say on
the topic of continuous delivery and continuous deployment in practice,
and reminded me of our discussion earlier in the year:
> In Continuous Delivery, Jez Humble and David Farley define the term
> continuous delivery. The term continuous deployment was first
> mentioned by Tim Fitz in his blog post “Continuous Deployment at IMVU:
> Doing the impossible fifty times a day.” However, in 2015, during the
> construction of The DevOps Handbook, Jez Humble commented, “In the
> last five years, there has been confusion around the terms continuous
> delivery versus continuous deployment—and, indeed, my own thinking and
> definitions have changed since we wrote the book. *Every organization
> should create their variations, based on what they need*. The key
> thing we should care about is not the form, but the outcomes:
> deployments should be low-risk, push-button events we can perform on
> demand.”
>
> His updated definitions of continuous delivery and continuous
> deployment are as follows:
> /
> //When all developers are working in small batches on trunk, or
> everyone is working off trunk in short-lived feature branches that get
> merged to trunk regularly, and when trunk is always kept in a
> releasable state, and when we can release on demand at the push of a
> button during normal business hours, we are doing continuous delivery.
> Developers get fast feedback when they introduce any regression
> errors, which include defects, performance issues, security issues,
> usability issues, etc. When these issues are found, they are fixed
> immediately so that trunk is always deployable.
>
> In addition to the above, when we are deploying good builds into
> production on a regular basis through self-service (being deployed by
> Dev or by Ops)—which typically means that we are deploying to
> production at least once per day per developer, or perhaps even
> automatically deploying every change a developer commits—this is when
> we are engaging in continuous deployment. /
>
> Defined this way, continuous delivery is the prerequisite for
> continuous deployment—just as continuous integration is a prerequisite
> prerequisite for continuous delivery. Continuous deployment is likely
> applicable in the context of web services that are delivered online.
> However, continuous delivery is applicable in almost every context
> where we desire deployments and releases that have high quality, fast
> lead times and have highly predictable, low-risk outcomes, including
> for embedded systems, COTS products, and mobile apps.
>
> Kim, Gene. The DevOps Handbook:: How to Create World-Class Agility,
> Reliability, and Security in Technology Organizations . IT Revolution
> Press. Kindle Edition.
Which is interesting. In my mind what's described above as "continuous
delivery" is what I think of as "continuous integration", but I like
their emphasis:
> *Every organization should create their variations, based on what they
> need*. The key thing we should care about is not the form, but the
> outcomes: deployments should be low-risk, push-button events we can
> perform on demand.
Best, Sam
On 16/01/2019 09:45, Samuel Joseph wrote:
>
> Big thanks to everyone for their input on this discussion. In an
> effort to try and wrap up, let me pull together threads from the last
> few emails, rather than replying to each individually.
>
> Gareth, okay, so "deploy master to staging and then to production",
> and "don't merge develop into master until you're sure there will be
> no more hotfixes, and you also merge hotfixes back into develop as
> soon as possible".
>
> I don't know there's any mechanism to ever ensure there would never be
> a hotfix, in that staging can't perfectly production in terms of
> sending real emails, social media messages to real users etc., but yes
> of course, merge hotfixes back ASAP - as we currently do from
> production -> staging -> develop branches in our setup.
>
> Gareth wrote:
>
>> the original core of your question, this is a simple answer. Once
>> your manual tests are complete, this is exactly what the "promote"
>> button (or equivalent cli command) does. It reduces the manual part
>> of the deploy process down to a single button click, which I think is
>> what you wanted from the start of this conversation.
> I'm not sure that's what I wanted. I can do `git rebase staging`,
> `git push origin master` on my command line to achieve the same
> effect, and I could make that a script or hook it up to a button if I
> wanted, but that's not my problem. The problem is that I have to
> initiate a production deploy at all; and I particularly don't want to
> have to remember to do it and them remember to come back and check the
> results later on. I guess the heroku "promote" button is useful in
> this regard in that it does not kick off any tests, or do a lengthy
> deploy process - it gets the code up much faster (but assumes staging
> server on master branch, as you suggest) - I'll need to measure that
> and see ...
>
> What I really want is to not have to remember to come and do a
> production deploy every week, and I also want it to be atomic, i.e.
> not kick it off in the morning and come back in an hour or two to see
> if it worked. I'm trying to deal with my unreliable memory, emotional
> exhaustion etc. I'd like to know that a production deploy will happen
> every week (on wednesdays say) and that I will have to get my act
> together on Tuesday to check for any reasons to stop the deploy.
>
> So I wonder if heroku (and other CI systems) allow automated promotion
> on particular dates ... which leads me to Sasha's email:
>
> Sasha wrote:
>
>> If you haven't already seen
>> https://www.spinnaker.io/guides/user/pipeline/managing-pipelines/
>> then I highly recommend looking over Spinnaker as I believe it will
>> help cover the steps you've outlined as missing in providers like
>> AWS, Heroku, etc.
>
> thanks for the recommendation Sasha, reading through I see pipeline
> triggers can be cron jobs, so indeed one could set things up to
> automatically promote to production on Wednesdays, or similar. And
> indeed I could stick with our current develop/staging/master branch
> model and put the `git rebase` operations into the cron jobs. It
> sounds like we might be alone if we do something like that - from the
> replies on the list is sounds like the common pattern is using
> develop/master and the staging server is on master as well, and it
> sounds like most folks have to remember to take some manual step to
> trigger a production deploy at some frequency.
>
> I wonder if anyone anywhere in the world is doing automated deploys to
> production on particular days of the week ...?
>
> Mark wrote:
>
>> You probably also want a strategy for giving you back the confidence
>> of being able to know what is on prod/staging at any point.
> At the moment we have full confidence of what is on prod/staging at
> any point. I'd argue that with the staging server/branch and
> production/master server/branch model anyone can instantly browse
> exactly what we have on prod/staging via GitHub, facilitating a much
> more open development than if the only way to see what's on those
> servers is through an interface like Herokus, which is not visible to
> just anyone.
>
> https://github.com/AgileVentures/WebsiteOne/tree/staging
>
> https://github.com/AgileVentures/WebsiteOne/tree/master
>
> but does sound like we're the only folks in the LRUG sphere with this
> model ...
>
> Gareth wrote:
>
>> I don't think the environment branches are needed in your setup, and
>> they're a legacy of a different workflow.
> I think they offer some advantages that I've outlined, and I was the
> one that originally implemented that workflow based on what some folks
> in the states were recommending - I wonder if there are books that get
> into the nitty/gritty of all this, or if this is still all just shared
> informal practices spread across different groups around the world ...
>
> Huge thanks to everyone for participating in this discussion. I've
> learnt a great deal, although I'm still not sure if I'll be able to
> use the terms "continuous deployment" and "continuous delivery"
> without causing confusion or being confused :-) but I've got lots to
> experiment with and that was my key objective.
>
> Best, Sam
>
>
> On 09/01/2019 10:42, Gareth Adams wrote:
>>
>>
>> On Wed, 9 Jan 2019, 10:08 Samuel Joseph <tansaku at gmail.com
>> <mailto:tansaku at gmail.com> wrote:
>>
>> I guess we could be deploying to both staging and production from
>> the master branch, but by having the separate develop, staging
>> and production branches, that each auto-deploy to the three
>> servers, it means that we've got a clear place to go (locally and
>> in GitHub) to see a running version of whatever is on each of
>> those three branches
>>
>> In the workflow I think you're describing, the "clear place to go"
>> should be your CD tool - in this case the Heroku pipeline - rather
>> than your source code repository. But there are definitely nuances,
>> and if getting that info back into your repo is important then maybe
>> I'm simplifying too much.
>>
>> and if for some reason production has a weird issues and we have
>> to hot fix (fairly rare) then we have the staging branch to do it
>> on, safe in the knowledge that a hot-fix applied to staging can
>> be auto-deployed to the staging server, and won't accidentally
>> make thngs worse on production (which is of course auto-deployed
>> off master).
>>
>> I'll rephrase what I wrote in my last email to:
>>
>> …you instead think "deploy master to staging *and then* to production".
>>
>> In this workflow, there's no such thing as a "hotfix to staging",
>> just a new commit to master, which follows the same "master ➡️
>> staging ➡️ production" process. Note this means you don't merge
>> develop into master until you're sure there will be no more hotfixes,
>> and you also merge hotfixes back into develop as soon as possible.
>>
>> But then, does no one do any manual tests on staging before
>> there's a deploy to production? What's the trigger that says it's
>> okay to deploy to production following a staging deploy?
>>
>> And now we've circled around to the original core of your question,
>> this is a simple answer. Once your manual tests are complete, this is
>> exactly what the "promote" button (or equivalent cli command) does.
>> It reduces the manual part of the deploy process down to a single
>> button click, which I think is what you wanted from the start of this
>> conversation.
>>
>>
>> On 08/01/2019 11:19, Matthias Berth wrote:
>>> Hi Sam,
>>>
>>> Why are you rebasing in the first place?
>>> Can't you just make a feature branch off master, then merge it
>>> back into master? And deploy to staging and production from the
>>> master branch?
>>>
>>> I also wonder why your 20 minutes sanity checks cannot be
>>> automated. Are you doing something new / creative every time you
>>> do these tests?
>>> Great discussion!
>>>
>>> Cheers
>>>
>>> Matthias
>>>
>>> On Tue, Jan 8, 2019 at 11:48 AM Gareth Adams <g at rethada.ms
>>> <mailto:g at rethada.ms>> wrote:
>>>
>>> Sam, I don't have a grand answer to your whole question, but
>>> a phrase leapt out at me and I wanted to flag it:
>>>
>>> > rebasing along the pipeline
>>>
>>> To me, this suggests the code on your staging branch is not
>>> the same as the code you end up deploying to production (and
>>> it might not be the same as the code in your master branch)
>>>
>>> I guess there are a few reasons that could be: you're
>>> storing some environment-specific configuration on your
>>> environment branches, and can't merge them all together, or
>>> maybe your environment branches contain a different
>>> combination of feature branches that you're trying to keep
>>> control of?
>>>
>>> Either way, I'd consider how (or if) you could change your
>>> workflow to make sure you deploy the same code everywhere
>>> (or at least deploy the same code to production that you
>>> deploy to staging). That's the basis behind e.g. Heroku
>>> pipelines' "Promote" button and it's the pattern I commonly
>>> see now
>>>
>>> On Tue, 8 Jan 2019, 10:13 Samuel Joseph <tansaku at gmail.com
>>> <mailto:tansaku at gmail.com> wrote:
>>>
>>> Hi Gerhard,
>>>
>>> On 07/01/2019 12:00, Gerhard Lazu wrote:
>>>> Hi Sam,
>>>>
>>>> What determines that a build can go from your
>>>> development environment into staging?
>>> Good question, the answer is:
>>>
>>> 1) that all the unit, integration and acceptance tests pass
>>> 2) that there are no merge conflicts
>>> 3) that the manual sanity checks on develop are coming
>>> back okay
>>>> And from staging into production? If you can capture
>>>> this in code, you can put it into a pipeline.
>>>>
>>> I don't think there's any way we can remove the manual
>>> sanity checks as the acceptance tests are just not that
>>> reliable, and although we've poured 1000's of hours into
>>> them and ultimately I can't see any way of making them
>>> perfect.
>>>
>>> I didn't think the presence of a manual step would
>>> prevent us using a pipeline, in as much as I thought of
>>> a pipeline as just being a series of servers with
>>> matching branches and code is them moved along them
>>> whether manually or automatically. Heroku calls such
>>> things pipelines and seems to have no support for
>>> automatically moving code along them, it's purely manual
>>> from what I can see.
>>>>
>>>> Why do you have 3 pipelines?
>>>
>>> I don't think we do. As I understand it, we have one
>>> pipeline:
>>>
>>> develop branch + develop server ---> staging branch +
>>> staging server ---> master branch + production server
>>>
>>> That's three paired branches/servers in one pipeline.
>>> Here's a screenshot of how Heroku presents our pipeline
>>> in their pipeline interface. Note the button "Promote
>>> to staging" which allows you to manually move the code
>>> on the develop server to the staging server, but doesn't
>>> actually do a rebase of the code from develop branch to
>>> staging:
>>>
>>>> Based on the questions that you're asking, I believe
>>>> that it would help if you had a single pipeline.
>>> I agree - I think we do, but perhaps I'm wrong ...
>>>> The question that I would focus on is /What would it
>>>> take to have a single pipeline that has an end-goal of
>>>> creating production builds/? Here is a pipeline example
>>>> which stops after it publishes a Docker image:
>>>> changelog.com, CircleCI
>>>> <https://circleci.com/workflow-run/065467ef-87c0-4f5e-a2ab-5e11be12403f>.
>>> Ooh, thanks for sharing! I had to log in to CircleCI to
>>> see that:
>>>
>>> but that looks really interesting.
>>>> If you are using something like Docker Swarm or
>>>> Kubernetes, the platform/ecosystem has all the
>>>> necessary tools to keep deployment concerns
>>>> self-contained. In the changelog.com
>>>> <http://changelog.com> case, the Docker stack that
>>>> captures the entire deployment has an update component
>>>> that is responsible for app updates
>>>> <https://github.com/thechangelog/changelog.com/blob/cf2ebe0de0f35c96bf664b8bc9183bd1f3468565/docker/changelog.stack.yml#L15-L31>.
>>>> In this specific case, if the new app version starts
>>>> and is healthy for 30 seconds, it gets automatically
>>>> promoted to live. We have been using a similar approach
>>>> since October 2016 <http://changelog.com/podcast/254>,
>>>> a Docker stack just makes it easier.
>>>>
>>>> I want to spur your imagination by sharing the pipeline
>>>> that is responsible for RabbitMQ v3.7.x
>>>> <https://ci.rabbitmq.com/teams/main/pipelines/server-release:v3.7.x>.
>>>> This pipeline captures what is possible if imagination
>>>> is set free:
>>> Wow, that RabbitMQ pipeline looks amazing - and I've
>>> only captured part of it in the screenshot:
>>>
>>>>
>>>> * tests & builds 30+ apps...
>>>> * on all supported major runtime version...
>>>> * and all supported OSes
>>>> * tests upgrades
>>>> * tests client support
>>>> * releases alphas, betas, RCs & GAs
>>>> * and publishes to all supported distribution channels
>>>>
>>>> I hope this helps, Gerhard.
>>>
>>> That's extremely helpful - thankyou!
>>>
>>> But so just to be clear, there is something in these
>>> pipelines that you're sharing that regularly moves code
>>> from one branch to another? And that's something that
>>> CircleCI and RabbitMQ provide? Or these are pipelines
>>> where the same code in the same branch is being moved
>>> through a series of servers, based on tests and checks
>>> passing at each server?
>>>
>>> Many thanks in advance
>>> Best, Sam
>>>
>>> _______________________________________________
>>> Chat mailing list
>>> Chat at lists.lrug.org <mailto:Chat at lists.lrug.org>
>>> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
>>> Manage your subscription:
>>> http://lists.lrug.org/options.cgi/chat-lrug.org
>>> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>>
>>> _______________________________________________
>>> Chat mailing list
>>> Chat at lists.lrug.org <mailto:Chat at lists.lrug.org>
>>> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
>>> Manage your subscription:
>>> http://lists.lrug.org/options.cgi/chat-lrug.org
>>> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>>
>>>
>>> _______________________________________________
>>> Chat mailing list
>>> Chat at lists.lrug.org <mailto:Chat at lists.lrug.org>
>>> Archives:http://lists.lrug.org/pipermail/chat-lrug.org
>>> Manage your subscription:http://lists.lrug.org/options.cgi/chat-lrug.org
>>> List info:http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>> _______________________________________________
>> Chat mailing list
>> Chat at lists.lrug.org <mailto:Chat at lists.lrug.org>
>> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
>> Manage your subscription:
>> http://lists.lrug.org/options.cgi/chat-lrug.org
>> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>
>>
>> _______________________________________________
>> Chat mailing list
>> Chat at lists.lrug.org
>> Archives:http://lists.lrug.org/pipermail/chat-lrug.org
>> Manage your subscription:http://lists.lrug.org/options.cgi/chat-lrug.org
>> List info:http://lists.lrug.org/listinfo.cgi/chat-lrug.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20190709/c5133368/attachment.html>
More information about the Chat
mailing list