<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Big thanks to everyone for their input on this discussion. In an
effort to try and wrap up, let me pull together threads from the
last few emails, rather than replying to each individually.</p>
<p>Gareth, okay, so "deploy master to staging and then to
production", and "don't merge develop into master until you're
sure there will be no more hotfixes, and you also merge hotfixes
back into develop as soon as possible".</p>
<p>I don't know there's any mechanism to ever ensure there would
never be a hotfix, in that staging can't perfectly production in
terms of sending real emails, social media messages to real users
etc., but yes of course, merge hotfixes back ASAP - as we
currently do from production -> staging -> develop branches
in our setup.</p>
<p>Gareth wrote:<br>
<blockquote type="cite">the original core of your question, this
is a simple answer. Once your manual tests are complete, this is
exactly what the "promote" button (or equivalent cli command)
does. It reduces the manual part of the deploy process down to a
single button click, which I think is what you wanted from the
start of this conversation.</blockquote>
I'm not sure that's what I wanted. I can do `git rebase staging`,
`git push origin master` on my command line to achieve the same
effect, and I could make that a script or hook it up to a button
if I wanted, but that's not my problem. The problem is that I
have to initiate a production deploy at all; and I particularly
don't want to have to remember to do it and them remember to come
back and check the results later on. I guess the heroku "promote"
button is useful in this regard in that it does not kick off any
tests, or do a lengthy deploy process - it gets the code up much
faster (but assumes staging server on master branch, as you
suggest) - I'll need to measure that and see ... <br>
</p>
<p>What I really want is to not have to remember to come and do a
production deploy every week, and I also want it to be atomic,
i.e. not kick it off in the morning and come back in an hour or
two to see if it worked. I'm trying to deal with my unreliable
memory, emotional exhaustion etc. I'd like to know that a
production deploy will happen every week (on wednesdays say) and
that I will have to get my act together on Tuesday to check for
any reasons to stop the deploy. <br>
</p>
<p>So I wonder if heroku (and other CI systems) allow automated
promotion on particular dates ... which leads me to Sasha's email:<br>
<br>
Sasha wrote:<br>
</p>
<blockquote type="cite">If you haven't already seen <a
href="https://www.spinnaker.io/guides/user/pipeline/managing-pipelines/">https://www.spinnaker.io/guides/user/pipeline/managing-pipelines/</a>
then I highly recommend looking over Spinnaker as I believe it
will help cover the steps you've outlined as missing in providers
like AWS, Heroku, etc.</blockquote>
<p>
thanks for the recommendation Sasha, reading through I see
pipeline triggers can be cron jobs, so indeed one could set things
up to automatically promote to production on Wednesdays, or
similar. And indeed I could stick with our current
develop/staging/master branch model and put the `git rebase`
operations into the cron jobs. It sounds like we might be alone
if we do something like that - from the replies on the list is
sounds like the common pattern is using develop/master and the
staging server is on master as well, and it sounds like most folks
have to remember to take some manual step to trigger a production
deploy at some frequency.<br>
</p>
<p> I wonder if anyone anywhere in the world is doing automated
deploys to production on particular days of the week ...?</p>
<p>Mark wrote:<br>
</p>
<p>
<blockquote type="cite">You probably also want a strategy for
giving you back the confidence of being able to know what is on
prod/staging at any point.</blockquote>
At the moment we have full confidence of what is on prod/staging
at any point. I'd argue that with the staging server/branch and
production/master server/branch model anyone can instantly browse
exactly what we have on prod/staging via GitHub, facilitating a
much more open development than if the only way to see what's on
those servers is through an interface like Herokus, which is not
visible to just anyone.<br>
</p>
<p><a class="moz-txt-link-freetext" href="https://github.com/AgileVentures/WebsiteOne/tree/staging">https://github.com/AgileVentures/WebsiteOne/tree/staging</a></p>
<p><a class="moz-txt-link-freetext" href="https://github.com/AgileVentures/WebsiteOne/tree/master">https://github.com/AgileVentures/WebsiteOne/tree/master</a></p>
<p>but does sound like we're the only folks in the LRUG sphere with
this model ...</p>
<p>Gareth wrote:<br>
<blockquote type="cite">I don't think the environment branches are
needed in your setup, and they're a legacy of a different
workflow.</blockquote>
I think they offer some advantages that I've outlined, and I was
the one that originally implemented that workflow based on what
some folks in the states were recommending - I wonder if there are
books that get into the nitty/gritty of all this, or if this is
still all just shared informal practices spread across different
groups around the world ...</p>
<p>Huge thanks to everyone for participating in this discussion.
I've learnt a great deal, although I'm still not sure if I'll be
able to use the terms "continuous deployment" and "continuous
delivery" without causing confusion or being confused :-) but
I've got lots to experiment with and that was my key objective. <br>
</p>
<p>Best, Sam<br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 09/01/2019 10:42, Gareth Adams
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAJ=YDWvbs2SVeF5bV-uoF+uZJHm7G1Jv3m_5rc7-M+b6BtGcdQ@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="auto">
<div><br>
<br>
<div class="gmail_quote">
<div dir="ltr">On Wed, 9 Jan 2019, 10:08 Samuel Joseph <<a
href="mailto:tansaku@gmail.com" rel="noreferrer
noreferrer noreferrer" target="_blank"
moz-do-not-send="true">tansaku@gmail.com</a> wrote:</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>I guess we could be deploying to both staging and
production from the master branch, but by having the
separate develop, staging and production branches,
that each auto-deploy to the three servers, it means
that we've got a clear place to go (locally and in
GitHub) to see a running version of whatever is on
each of those three branches</p>
</div>
</blockquote>
</div>
</div>
<div dir="auto">In the workflow I think you're describing, the
"clear place to go" should be your CD tool - in this case the
Heroku pipeline - <span style="font-family:sans-serif">rather
than your source code repository.</span> But there are
definitely nuances, and if getting that info back into your
repo is important then maybe I'm simplifying too much.</div>
<div dir="auto">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>and if for some reason production has a weird issues
and we have to hot fix (fairly rare) then we have the
staging branch to do it on, safe in the knowledge that
a hot-fix applied to staging can be auto-deployed to
the staging server, and won't accidentally make thngs
worse on production (which is of course auto-deployed
off master).</p>
</div>
</blockquote>
</div>
</div>
<div dir="auto">I'll rephrase what I wrote in my last email to:</div>
<div dir="auto"><br>
</div>
<div dir="auto">…you instead think "deploy master to staging <b>and
then</b> to production".<br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">In this workflow, there's no such thing as a
"hotfix to staging", just a new commit to master, which
follows the same "master ➡️ staging ➡️ production" process.
Note this means you don't merge develop into master until
you're sure there will be no more hotfixes, and you also merge
hotfixes back into develop as soon as possible.</div>
<div dir="auto">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>But then, does no one do any manual tests on staging
before there's a deploy to production? What's the
trigger that says it's okay to deploy to production
following a staging deploy?</p>
</div>
</blockquote>
</div>
</div>
<div dir="auto">And now we've circled around to the original
core of your question, this is a simple answer. Once your
manual tests are complete, this is exactly what the "promote"
button (or equivalent cli command) does. It reduces the manual
part of the deploy process down to a single button click,
which I think is what you wanted from the start of this
conversation.</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div
class="m_7990572999333527681m_8506084164337888487m_-8428862481596492984m_-6735578697607866647moz-cite-prefix">On
08/01/2019 11:19, Matthias Berth wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hi Sam, <br>
</div>
<div><br>
</div>
<div>Why are you rebasing in the first place? <br>
</div>
<div>Can't you just make a feature branch off
master, then merge it back into master? And deploy
to staging and production from the master branch?</div>
<div><br>
</div>
<div>I also wonder why your 20 minutes sanity checks
cannot be automated. Are you doing something new /
creative every time you do these tests?</div>
<div>Great discussion!</div>
<div><br>
</div>
<div>Cheers</div>
<div><br>
</div>
<div>Matthias<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Tue, Jan 8, 2019 at 11:48 AM
Gareth Adams <<a href="mailto:g@rethada.ms"
rel="noreferrer noreferrer noreferrer
noreferrer" target="_blank"
moz-do-not-send="true">g@rethada.ms</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="auto">Sam, I don't have a grand answer
to your whole question, but a phrase leapt out
at me and I wanted to flag it:
<div dir="auto"><br>
</div>
<div dir="auto">> rebasing along the pipeline</div>
<div dir="auto"><br>
</div>
<div dir="auto">To me, this suggests the code on
your staging branch is not the same as the
code you end up deploying to production (and
it might not be the same as the code in your
master branch)</div>
<div dir="auto"><br>
</div>
<div dir="auto">I guess there are a few reasons
that could be: you're storing some
environment-specific configuration on your
environment branches, and can't merge them all
together, or maybe your environment branches
contain a different combination of feature
branches that you're trying to keep control
of?</div>
<div dir="auto"><br>
</div>
<div dir="auto">Either way, I'd consider how (or
if) you could change your workflow to make
sure you deploy the same code everywhere (or
at least deploy the same code to production
that you deploy to staging). That's the basis
behind e.g. Heroku pipelines' "Promote" button
and it's the pattern I commonly see now</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Tue, 8 Jan 2019, 10:13 Samuel
Joseph <<a href="mailto:tansaku@gmail.com"
rel="noreferrer noreferrer noreferrer
noreferrer noreferrer noreferrer"
target="_blank" moz-do-not-send="true">tansaku@gmail.com</a>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Gerhard,<br>
</p>
<div
class="m_7990572999333527681m_8506084164337888487m_-8428862481596492984m_-6735578697607866647gmail-m_-4091885444582072218m_3868980048267634368m_-2707529618717350415m_-2020117851195068980moz-cite-prefix">On
07/01/2019 12:00, Gerhard Lazu wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">Hi Sam,
<div><br>
</div>
<div>What determines that a build can
go from your development environment
into staging? </div>
</div>
</div>
</blockquote>
Good question, the answer is:<br>
<br>
1) that all the unit, integration and
acceptance tests pass<br>
2) that there are no merge conflicts<br>
3) that the manual sanity checks on develop
are coming back okay<br>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div>And from staging into production?
If you can capture this in code, you
can put it into a pipeline.<br>
<br>
</div>
</div>
</div>
</blockquote>
I don't think there's any way we can remove
the manual sanity checks as the acceptance
tests are just not that reliable, and
although we've poured 1000's of hours into
them and ultimately I can't see any way of
making them perfect.<br>
<br>
I didn't think the presence of a manual step
would prevent us using a pipeline, in as
much as I thought of a pipeline as just
being a series of servers with matching
branches and code is them moved along them
whether manually or automatically. Heroku
calls such things pipelines and seems to
have no support for automatically moving
code along them, it's purely manual from
what I can see.<br>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div><br>
</div>
<div>Why do you have 3 pipelines? </div>
</div>
</div>
</blockquote>
<p>I don't think we do. As I understand it,
we have one pipeline:</p>
<p>develop branch + develop server --->
staging branch + staging server --->
master branch + production server</p>
<p>That's three paired branches/servers in
one pipeline. Here's a screenshot of how
Heroku presents our pipeline in their
pipeline interface. Note the button
"Promote to staging" which allows you to
manually move the code on the develop
server to the staging server, but doesn't
actually do a rebase of the code from
develop branch to staging:<br>
</p>
<p><img alt="" moz-do-not-send="true"
width="790" height="157"></p>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div>Based on the questions that
you're asking, I believe that it
would help if you had a single
pipeline. </div>
</div>
</div>
</blockquote>
I agree - I think we do, but perhaps I'm
wrong ...<br>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div>The question that I would focus
on is <i>What would it take to have
a single pipeline that has an
end-goal of creating production
builds</i>? Here is a pipeline
example which stops after it
publishes a Docker image: <a
href="https://circleci.com/workflow-run/065467ef-87c0-4f5e-a2ab-5e11be12403f"
rel="noreferrer noreferrer
noreferrer noreferrer noreferrer
noreferrer noreferrer"
target="_blank"
moz-do-not-send="true">changelog.com,
CircleCI</a>.</div>
</div>
</div>
</blockquote>
Ooh, thanks for sharing! I had to log in to
CircleCI to see that:<br>
<br>
<p><img alt="" moz-do-not-send="true"
width="470" height="208"></p>
but that looks really interesting.<br>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div> If you are using something like
Docker Swarm or Kubernetes, the
platform/ecosystem has all the
necessary tools to keep deployment
concerns self-contained. In the <a
href="http://changelog.com"
rel="noreferrer noreferrer
noreferrer noreferrer noreferrer
noreferrer noreferrer"
target="_blank"
moz-do-not-send="true">changelog.com</a>
case, the Docker stack that captures
the entire deployment has <a
href="https://github.com/thechangelog/changelog.com/blob/cf2ebe0de0f35c96bf664b8bc9183bd1f3468565/docker/changelog.stack.yml#L15-L31"
rel="noreferrer noreferrer
noreferrer noreferrer noreferrer
noreferrer noreferrer"
target="_blank"
moz-do-not-send="true">an update
component that is responsible for
app updates</a>. In this specific
case, if the new app version starts
and is healthy for 30 seconds, it
gets automatically promoted to live.
<a
href="http://changelog.com/podcast/254"
rel="noreferrer noreferrer
noreferrer noreferrer noreferrer
noreferrer noreferrer"
target="_blank"
moz-do-not-send="true">We have
been using a similar approach
since October 2016</a>, a Docker
stack just makes it easier.</div>
<div><br>
</div>
<div>I want to spur your imagination
by sharing <a
href="https://ci.rabbitmq.com/teams/main/pipelines/server-release:v3.7.x"
rel="noreferrer noreferrer
noreferrer noreferrer noreferrer
noreferrer noreferrer"
target="_blank"
moz-do-not-send="true">the
pipeline that is responsible for
RabbitMQ v3.7.x</a>. This pipeline
captures what is possible if
imagination is set free:</div>
</div>
</div>
</blockquote>
Wow, that RabbitMQ pipeline looks amazing -
and I've only captured part of it in the
screenshot:<br>
<br>
<img alt="" moz-do-not-send="true"
width="593" height="375">
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div><br>
</div>
<div>* tests & builds 30+ apps...</div>
<div>* on all supported major runtime
version...</div>
<div>* and all supported OSes</div>
<div>* tests upgrades</div>
<div>* tests client support</div>
<div>* releases alphas, betas, RCs
& GAs</div>
<div>* and publishes to all supported
distribution channels</div>
<div><br>
</div>
<div>I hope this helps, Gerhard.</div>
</div>
</div>
</blockquote>
<p>That's extremely helpful - thankyou!</p>
<p>But so just to be clear, there is
something in these pipelines that you're
sharing that regularly moves code from one
branch to another? And that's something
that CircleCI and RabbitMQ provide? Or
these are pipelines where the same code in
the same branch is being moved through a
series of servers, based on tests and
checks passing at each server?</p>
<p>Many thanks in advance<br>
Best, Sam<br>
</p>
</div>
_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org"
rel="noreferrer noreferrer noreferrer
noreferrer noreferrer noreferrer noreferrer"
target="_blank" moz-do-not-send="true">Chat@lists.lrug.org</a><br>
Archives: <a
href="http://lists.lrug.org/pipermail/chat-lrug.org"
rel="noreferrer noreferrer noreferrer
noreferrer noreferrer noreferrer noreferrer
noreferrer" target="_blank"
moz-do-not-send="true">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a
href="http://lists.lrug.org/options.cgi/chat-lrug.org"
rel="noreferrer noreferrer noreferrer
noreferrer noreferrer noreferrer noreferrer
noreferrer" target="_blank"
moz-do-not-send="true">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a
href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org"
rel="noreferrer noreferrer noreferrer
noreferrer noreferrer noreferrer noreferrer
noreferrer" target="_blank"
moz-do-not-send="true">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
</blockquote>
</div>
_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org"
rel="noreferrer noreferrer noreferrer
noreferrer" target="_blank"
moz-do-not-send="true">Chat@lists.lrug.org</a><br>
Archives: <a
href="http://lists.lrug.org/pipermail/chat-lrug.org"
rel="noreferrer noreferrer noreferrer noreferrer
noreferrer" target="_blank"
moz-do-not-send="true">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a
href="http://lists.lrug.org/options.cgi/chat-lrug.org"
rel="noreferrer noreferrer noreferrer noreferrer
noreferrer" target="_blank"
moz-do-not-send="true">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a
href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org"
rel="noreferrer noreferrer noreferrer noreferrer
noreferrer" target="_blank"
moz-do-not-send="true">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
</blockquote>
</div>
<br>
<fieldset
class="m_7990572999333527681m_8506084164337888487m_-8428862481596492984m_-6735578697607866647mimeAttachmentHeader"></fieldset>
<pre class="m_7990572999333527681m_8506084164337888487m_-8428862481596492984m_-6735578697607866647moz-quote-pre">_______________________________________________
Chat mailing list
<a class="m_7990572999333527681m_8506084164337888487m_-8428862481596492984m_-6735578697607866647moz-txt-link-abbreviated" href="mailto:Chat@lists.lrug.org" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank" moz-do-not-send="true">Chat@lists.lrug.org</a>
Archives: <a class="m_7990572999333527681m_8506084164337888487m_-8428862481596492984m_-6735578697607866647moz-txt-link-freetext" href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank" moz-do-not-send="true">http://lists.lrug.org/pipermail/chat-lrug.org</a>
Manage your subscription: <a class="m_7990572999333527681m_8506084164337888487m_-8428862481596492984m_-6735578697607866647moz-txt-link-freetext" href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank" moz-do-not-send="true">http://lists.lrug.org/options.cgi/chat-lrug.org</a>
List info: <a class="m_7990572999333527681m_8506084164337888487m_-8428862481596492984m_-6735578697607866647moz-txt-link-freetext" href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank" moz-do-not-send="true">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a>
</pre>
</blockquote>
</div>
_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" rel="noreferrer
noreferrer noreferrer noreferrer" target="_blank"
moz-do-not-send="true">Chat@lists.lrug.org</a><br>
Archives: <a
href="http://lists.lrug.org/pipermail/chat-lrug.org"
rel="noreferrer noreferrer noreferrer noreferrer
noreferrer" target="_blank" moz-do-not-send="true">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a
href="http://lists.lrug.org/options.cgi/chat-lrug.org"
rel="noreferrer noreferrer noreferrer noreferrer
noreferrer" target="_blank" moz-do-not-send="true">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a
href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org"
rel="noreferrer noreferrer noreferrer noreferrer
noreferrer" target="_blank" moz-do-not-send="true">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
</blockquote>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Chat mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Chat@lists.lrug.org">Chat@lists.lrug.org</a>
Archives: <a class="moz-txt-link-freetext" href="http://lists.lrug.org/pipermail/chat-lrug.org">http://lists.lrug.org/pipermail/chat-lrug.org</a>
Manage your subscription: <a class="moz-txt-link-freetext" href="http://lists.lrug.org/options.cgi/chat-lrug.org">http://lists.lrug.org/options.cgi/chat-lrug.org</a>
List info: <a class="moz-txt-link-freetext" href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a>
</pre>
</blockquote>
</body>
</html>