<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi Matthias,</p>
<p>Great to hear from you. See the email reply I just sent to
Gareth about why we are rebasing probably answers your first
question - it's to create a nice clean single threaded git
history.</p>
<p>To address your other questions we could be feature branching off
master, but we feature branch off develop - all our developers
work off the develop branch, and features get rebased into develop
squashing all their commits into a single easy to read summary -
see <a class="moz-txt-link-freetext" href="https://github.com/AgileVentures/WebsiteOne">https://github.com/AgileVentures/WebsiteOne</a> (we've 100% open
source and open development). Then when we are ready to move to a
deploy the changes in develop get rebased to staging and
eventually to master.<br>
</p>
<p>I guess we could be deploying to both staging and production from
the master branch, but by having the separate develop, staging and
production branches, that each auto-deploy to the three servers,
it means that we've got a clear place to go (locally and in
GitHub) to see a running version of whatever is on each of those
three branches, and if for some reason production has a weird
issues and we have to hot fix (fairly rare) then we have the
staging branch to do it on, safe in the knowledge that a hot-fix
applied to staging can be auto-deployed to the staging server, and
won't accidentally make thngs worse on production (which is of
course auto-deployed off master).</p>
<p>The sanity checks I mention are automated to an extent, so, for
example we have some pages involving some complex javascript and
we have acceptance tests (that run end to end browser based
checks) for them, integration, front end unit tests, back end unit
tests, etc. but painful experience has shown that even when all
the tests are working and the functionality is working manually on
developer machines, we can still end up with things breaking here
and there due to the vagaries of the settings and architecture in
our legacy codebase. I'm sure our codebase is much smaller and
less complicated that what many on the list manage every day, but
I definitely get caught out once every six months or so.</p>
<p>Now maybe the solution should be fixing those underlying
configuration issues, or just a better overall design of the
codebase - more precision here there or somewhere - I'd love to
have more resources to put into that. Over the years we've
invested chunks of time in, say, overhauling all of the acceptance
tests to try and make them more reliable, careful picking apart of
certain sub-systems, but I still don't feel 100% confident without
several manual checks - as my confidence increases between
incidences that 20 minute check drops to 10 and then sub 5mins,
but then we have an incident and it spikes again.<br>
</p>
<p>There's certainly not anything new and creative being done each
time - it's just a reflection of my lack of absolute faith in the
tests and codebase :-)</p>
<p>What I am starting to sense is that other folks have their
staging and production servers both deploying from the master
branch, and that following a successful deploy to staging, that's
followed by a deployment of master to production. But then, does
no one do any manual tests on staging before there's a deploy to
production? What's the trigger that says it's okay to deploy to
production following a staging deploy?</p>
<p>Thanks for your great questions!<br>
</p>
<p>Best, Sam<br>
</p>
<div class="moz-cite-prefix">On 08/01/2019 11:19, Matthias Berth
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CACkU=L=-naPXQ9O4Fi2QWUK6EL_x=2XQf4U1aFeOpk7Wc41d0Q@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>Hi Sam, <br>
</div>
<div><br>
</div>
<div>Why are you rebasing in the first place? <br>
</div>
<div>Can't you just make a feature branch off master, then merge
it back into master? And deploy to staging and production from
the master branch?</div>
<div><br>
</div>
<div>I also wonder why your 20 minutes sanity checks cannot be
automated. Are you doing something new / creative every time
you do these tests?</div>
<div>Great discussion!</div>
<div><br>
</div>
<div>Cheers</div>
<div><br>
</div>
<div>Matthias<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Tue, Jan 8, 2019 at 11:48 AM Gareth Adams <<a
href="mailto:g@rethada.ms" moz-do-not-send="true">g@rethada.ms</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="auto">Sam, I don't have a grand answer to your whole
question, but a phrase leapt out at me and I wanted to flag
it:
<div dir="auto"><br>
</div>
<div dir="auto">> rebasing along the pipeline</div>
<div dir="auto"><br>
</div>
<div dir="auto">To me, this suggests the code on your
staging branch is not the same as the code you end up
deploying to production (and it might not be the same as
the code in your master branch)</div>
<div dir="auto"><br>
</div>
<div dir="auto">I guess there are a few reasons that could
be: you're storing some environment-specific configuration
on your environment branches, and can't merge them all
together, or maybe your environment branches contain a
different combination of feature branches that you're
trying to keep control of?</div>
<div dir="auto"><br>
</div>
<div dir="auto">Either way, I'd consider how (or if) you
could change your workflow to make sure you deploy the
same code everywhere (or at least deploy the same code to
production that you deploy to staging). That's the basis
behind e.g. Heroku pipelines' "Promote" button and it's
the pattern I commonly see now</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Tue, 8 Jan 2019, 10:13 Samuel Joseph <<a
href="mailto:tansaku@gmail.com" rel="noreferrer
noreferrer" target="_blank" moz-do-not-send="true">tansaku@gmail.com</a>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Gerhard,<br>
</p>
<div
class="gmail-m_-4091885444582072218m_3868980048267634368m_-2707529618717350415m_-2020117851195068980moz-cite-prefix">On
07/01/2019 12:00, Gerhard Lazu wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">Hi Sam,
<div><br>
</div>
<div>What determines that a build can go from your
development environment into staging? </div>
</div>
</div>
</blockquote>
Good question, the answer is:<br>
<br>
1) that all the unit, integration and acceptance tests
pass<br>
2) that there are no merge conflicts<br>
3) that the manual sanity checks on develop are coming
back okay<br>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div>And from staging into production? If you can
capture this in code, you can put it into a
pipeline.<br>
<br>
</div>
</div>
</div>
</blockquote>
I don't think there's any way we can remove the manual
sanity checks as the acceptance tests are just not that
reliable, and although we've poured 1000's of hours into
them and ultimately I can't see any way of making them
perfect.<br>
<br>
I didn't think the presence of a manual step would
prevent us using a pipeline, in as much as I thought of
a pipeline as just being a series of servers with
matching branches and code is them moved along them
whether manually or automatically. Heroku calls such
things pipelines and seems to have no support for
automatically moving code along them, it's purely manual
from what I can see.<br>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div><br>
</div>
<div>Why do you have 3 pipelines? </div>
</div>
</div>
</blockquote>
<p>I don't think we do. As I understand it, we have one
pipeline:</p>
<p>develop branch + develop server ---> staging
branch + staging server ---> master branch +
production server</p>
<p>That's three paired branches/servers in one
pipeline. Here's a screenshot of how Heroku presents
our pipeline in their pipeline interface. Note the
button "Promote to staging" which allows you to
manually move the code on the develop server to the
staging server, but doesn't actually do a rebase of
the code from develop branch to staging:<br>
</p>
<p><img alt="" moz-do-not-send="true" width="790"
height="157"></p>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div>Based on the questions that you're asking, I
believe that it would help if you had a single
pipeline. </div>
</div>
</div>
</blockquote>
I agree - I think we do, but perhaps I'm wrong ...<br>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div>The question that I would focus on is <i>What
would it take to have a single pipeline that
has an end-goal of creating production builds</i>?
Here is a pipeline example which stops after it
publishes a Docker image: <a
href="https://circleci.com/workflow-run/065467ef-87c0-4f5e-a2ab-5e11be12403f"
rel="noreferrer noreferrer noreferrer"
target="_blank" moz-do-not-send="true">changelog.com,
CircleCI</a>.</div>
</div>
</div>
</blockquote>
Ooh, thanks for sharing! I had to log in to CircleCI to
see that:<br>
<br>
<p><img alt="" moz-do-not-send="true" width="470"
height="208"></p>
but that looks really interesting.<br>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div> If you are using something like Docker Swarm
or Kubernetes, the platform/ecosystem has all
the necessary tools to keep deployment concerns
self-contained. In the <a
href="http://changelog.com" rel="noreferrer
noreferrer noreferrer" target="_blank"
moz-do-not-send="true">changelog.com</a> case,
the Docker stack that captures the entire
deployment has <a
href="https://github.com/thechangelog/changelog.com/blob/cf2ebe0de0f35c96bf664b8bc9183bd1f3468565/docker/changelog.stack.yml#L15-L31"
rel="noreferrer noreferrer noreferrer"
target="_blank" moz-do-not-send="true">an
update component that is responsible for app
updates</a>. In this specific case, if the new
app version starts and is healthy for 30
seconds, it gets automatically promoted to live.
<a href="http://changelog.com/podcast/254"
rel="noreferrer noreferrer noreferrer"
target="_blank" moz-do-not-send="true">We have
been using a similar approach since October
2016</a>, a Docker stack just makes it easier.</div>
<div><br>
</div>
<div>I want to spur your imagination by sharing <a
href="https://ci.rabbitmq.com/teams/main/pipelines/server-release:v3.7.x"
rel="noreferrer noreferrer noreferrer"
target="_blank" moz-do-not-send="true">the
pipeline that is responsible for RabbitMQ
v3.7.x</a>. This pipeline captures what is
possible if imagination is set free:</div>
</div>
</div>
</blockquote>
Wow, that RabbitMQ pipeline looks amazing - and I've
only captured part of it in the screenshot:<br>
<br>
<img alt="" moz-do-not-send="true" width="593"
height="375">
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div><br>
</div>
<div>* tests & builds 30+ apps...</div>
<div>* on all supported major runtime version...</div>
<div>* and all supported OSes</div>
<div>* tests upgrades</div>
<div>* tests client support</div>
<div>* releases alphas, betas, RCs & GAs</div>
<div>* and publishes to all supported distribution
channels</div>
<div><br>
</div>
<div>I hope this helps, Gerhard.</div>
</div>
</div>
</blockquote>
<p>That's extremely helpful - thankyou!</p>
<p>But so just to be clear, there is something in these
pipelines that you're sharing that regularly moves
code from one branch to another? And that's something
that CircleCI and RabbitMQ provide? Or these are
pipelines where the same code in the same branch is
being moved through a series of servers, based on
tests and checks passing at each server?</p>
<p>Many thanks in advance<br>
Best, Sam<br>
</p>
</div>
_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" rel="noreferrer
noreferrer noreferrer" target="_blank"
moz-do-not-send="true">Chat@lists.lrug.org</a><br>
Archives: <a
href="http://lists.lrug.org/pipermail/chat-lrug.org"
rel="noreferrer noreferrer noreferrer noreferrer"
target="_blank" moz-do-not-send="true">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a
href="http://lists.lrug.org/options.cgi/chat-lrug.org"
rel="noreferrer noreferrer noreferrer noreferrer"
target="_blank" moz-do-not-send="true">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a
href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org"
rel="noreferrer noreferrer noreferrer noreferrer"
target="_blank" moz-do-not-send="true">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
</blockquote>
</div>
_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank"
moz-do-not-send="true">Chat@lists.lrug.org</a><br>
Archives: <a
href="http://lists.lrug.org/pipermail/chat-lrug.org"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a
href="http://lists.lrug.org/options.cgi/chat-lrug.org"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a
href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
</blockquote>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Chat mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Chat@lists.lrug.org">Chat@lists.lrug.org</a>
Archives: <a class="moz-txt-link-freetext" href="http://lists.lrug.org/pipermail/chat-lrug.org">http://lists.lrug.org/pipermail/chat-lrug.org</a>
Manage your subscription: <a class="moz-txt-link-freetext" href="http://lists.lrug.org/options.cgi/chat-lrug.org">http://lists.lrug.org/options.cgi/chat-lrug.org</a>
List info: <a class="moz-txt-link-freetext" href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a>
</pre>
</blockquote>
</body>
</html>