[LRUG] Continuous * (Happy New Year!)

Wed Jan 9 02:42:48 PST 2019

On Wed, 9 Jan 2019, 10:08 Samuel Joseph <tansaku at gmail.com wrote:

> I guess we could be deploying to both staging and production from the
> master branch, but by having the separate develop, staging and production
> branches, that each auto-deploy to the three servers, it means that we've
> got a clear place to go (locally and in GitHub) to see a running version of
> whatever is on each of those three branches
>
In the workflow I think you're describing, the "clear place to go" should
be your CD tool - in this case the Heroku pipeline - rather than your
source code repository. But there are definitely nuances, and if getting
that info back into your repo is important then maybe I'm simplifying too
much.

> and if for some reason production has a weird issues and we have to hot
> fix (fairly rare) then we have the staging branch to do it on, safe in the
> knowledge that a hot-fix applied to staging can be auto-deployed to the
> staging server, and won't accidentally make thngs worse on production
> (which is of course auto-deployed off master).
>
I'll rephrase what I wrote in my last email to:

…you instead think "deploy master to staging *and then* to production".

In this workflow, there's no such thing as a "hotfix to staging", just a
new commit to master, which follows the same "master ➡️ staging ➡️
production" process. Note this means you don't merge develop into master
until you're sure there will be no more hotfixes, and you also merge
hotfixes back into develop as soon as possible.

> But then, does no one do any manual tests on staging before there's a
> deploy to production?  What's the trigger that says it's okay to deploy to
> production following a staging deploy?
>
And now we've circled around to the original core of your question, this is
a simple answer. Once your manual tests are complete, this is exactly what
the "promote" button (or equivalent cli command) does. It reduces the
manual part of the deploy process down to a single button click, which I
think is what you wanted from the start of this conversation.

On 08/01/2019 11:19, Matthias Berth wrote:
>
> Hi Sam,
>
> Why are you rebasing in the first place?
> Can't you just make a feature branch off master, then merge it back into
> master? And deploy to staging and production from the master branch?
>
> I also wonder why your 20 minutes sanity checks cannot be automated. Are
> you doing something new / creative every time you do these tests?
> Great discussion!
>
> Cheers
>
> Matthias
>
> On Tue, Jan 8, 2019 at 11:48 AM Gareth Adams <g at rethada.ms> wrote:
>
>> Sam, I don't have a grand answer to your whole question, but a phrase
>> leapt out at me and I wanted to flag it:
>>
>> > rebasing along the pipeline
>>
>> To me, this suggests the code on your staging branch is not the same as
>> the code you end up deploying to production (and it might not be the same
>> as the code in your master branch)
>>
>> I guess there are a few reasons that could be: you're storing some
>> environment-specific configuration on your environment branches, and can't
>> merge them all together, or maybe your environment branches contain a
>> different combination of feature branches that you're trying to keep
>> control of?
>>
>> Either way, I'd consider how (or if) you could change your workflow to
>> make sure you deploy the same code everywhere (or at least deploy the same
>> code to production that you deploy to staging). That's the basis behind
>> e.g. Heroku pipelines' "Promote" button and it's the pattern I commonly see
>> now
>>
>> On Tue, 8 Jan 2019, 10:13 Samuel Joseph <tansaku at gmail.com wrote:
>>
>>> Hi Gerhard,
>>> On 07/01/2019 12:00, Gerhard Lazu wrote:
>>>
>>> Hi Sam,
>>>
>>> What determines that a build can go from your development environment
>>> into staging?
>>>
>>> Good question, the answer is:
>>>
>>> 1) that all the unit, integration and acceptance tests pass
>>> 2) that there are no merge conflicts
>>> 3) that the manual sanity checks on develop are coming back okay
>>>
>>> And from staging into production? If you can capture this in code, you
>>> can put it into a pipeline.
>>>
>>> I don't think there's any way we can remove the manual sanity checks as
>>> the acceptance tests are just not that reliable, and although we've poured
>>> 1000's of hours into them and ultimately I can't see any way of making them
>>> perfect.
>>>
>>> I didn't think the presence of a manual step would prevent us using a
>>> pipeline, in as much as I thought of a pipeline as just being a series of
>>> servers with matching branches and code is them moved along them whether
>>> manually or automatically.  Heroku calls such things pipelines and seems to
>>> have no support for automatically moving code along them, it's purely
>>> manual from what I can see.
>>>
>>>
>>> Why do you have 3 pipelines?
>>>
>>> I don't think we do.  As I understand it, we have one pipeline:
>>>
>>> develop branch + develop server ---> staging branch + staging server
>>> ---> master branch + production server
>>>
>>> That's three paired branches/servers in one pipeline.  Here's a
>>> screenshot of how Heroku presents our pipeline in their pipeline
>>> interface.  Note the button "Promote to staging" which allows you to
>>> manually move the code on the develop server to the staging server, but
>>> doesn't actually do a rebase of the code from develop branch to staging:
>>>
>>> Based on the questions that you're asking, I believe that it would help
>>> if you had a single pipeline.
>>>
>>> I agree - I think we do, but perhaps I'm wrong ...
>>>
>>> The question that I would focus on is *What would it take to have a
>>> single pipeline that has an end-goal of creating production builds*?
>>> Here is a pipeline example which stops after it publishes a Docker image: changelog.com,
>>> CircleCI
>>> <https://circleci.com/workflow-run/065467ef-87c0-4f5e-a2ab-5e11be12403f>
>>> .
>>>
>>> Ooh, thanks for sharing! I had to log in to CircleCI to see that:
>>>
>>> but that looks really interesting.
>>>
>>> If you are using something like Docker Swarm or Kubernetes, the
>>> platform/ecosystem has all the necessary tools to keep deployment concerns
>>> self-contained. In the changelog.com case, the Docker stack that
>>> captures the entire deployment has an update component that is
>>> responsible for app updates
>>> <https://github.com/thechangelog/changelog.com/blob/cf2ebe0de0f35c96bf664b8bc9183bd1f3468565/docker/changelog.stack.yml#L15-L31>.
>>> In this specific case, if the new app version starts and is healthy for 30
>>> seconds, it gets automatically promoted to live. We have been using a
>>> similar approach since October 2016 <http://changelog.com/podcast/254>,
>>> a Docker stack just makes it easier.
>>>
>>> I want to spur your imagination by sharing the pipeline that is
>>> responsible for RabbitMQ v3.7.x
>>> <https://ci.rabbitmq.com/teams/main/pipelines/server-release:v3.7.x>.
>>> This pipeline captures what is possible if imagination is set free:
>>>
>>> Wow, that RabbitMQ pipeline looks amazing - and I've only captured part
>>> of it in the screenshot:
>>>
>>>
>>> * tests & builds 30+ apps...
>>> * on all supported major runtime version...
>>> * and all supported OSes
>>> * tests upgrades
>>> * tests client support
>>> * releases alphas, betas, RCs & GAs
>>> * and publishes to all supported distribution channels
>>>
>>> I hope this helps, Gerhard.
>>>
>>> That's extremely helpful - thankyou!
>>>
>>> But so just to be clear, there is something in these pipelines that
>>> you're sharing that regularly moves code from one branch to another?  And
>>> that's something that CircleCI and RabbitMQ provide?  Or these are
>>> pipelines where the same code in the same branch is being moved through a
>>> series of servers, based on tests and checks passing at each server?
>>>
>>> Many thanks in advance
>>> Best, Sam
>>> _______________________________________________
>>> Chat mailing list
>>> Chat at lists.lrug.org
>>> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
>>> Manage your subscription:
>>> http://lists.lrug.org/options.cgi/chat-lrug.org
>>> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>>
>> _______________________________________________
>> Chat mailing list
>> Chat at lists.lrug.org
>> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
>> Manage your subscription: http://lists.lrug.org/options.cgi/chat-lrug.org
>> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>
>
> _______________________________________________
> Chat mailing listChat at lists.lrug.org
> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
> Manage your subscription: http://lists.lrug.org/options.cgi/chat-lrug.org
> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>
> _______________________________________________
> Chat mailing list
> Chat at lists.lrug.org
> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
> Manage your subscription: http://lists.lrug.org/options.cgi/chat-lrug.org
> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20190109/aa99cc50/attachment-0002.html>