<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi Sam,</div><br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div>What determines that a build can go from your development
environment into staging? </div>
</div>
</div>
</blockquote>
Good question, the answer is:<br>
<br>
1) that all the unit, integration and acceptance tests pass<br>
2) that there are no merge conflicts<br>
3) that the manual sanity checks on develop are coming back okay<br>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div>And from staging into production? If you can capture this
in code, you can put it into a pipeline.<br>
<br>
</div>
</div>
</div>
</blockquote>
I don't think there's any way we can remove the manual sanity checks
as the acceptance tests are just not that reliable, and although
we've poured 1000's of hours into them and ultimately I can't see
any way of making them perfect.<br></div></blockquote><div><br></div><div>You made 2 points here which hint to a larger problem: merge conflicts and manual checks.</div><div><br></div><div>1. Merge conflicts are a symptom of branches. If all commits go straight to master, and there are no branches, merge conflicts will be small, local & dealt with instantly by humans. This approach will result in many small commits, continuously flowing through your pipeline. Small slices of functionality and value delivered continuously into production is the trick to hundreds of updates daily. 100 merges add a lot of overhead, the total effort is great. Alternatively, 100 commits to master are simple, we all do it all the time.</div><div><br></div><div>2. Manual sanity checks are a symptom of poor observability and instrumentation. Think Service Level Indicators (SLIs) & Service Level Objectives (SLOs) that capture the health of your app / system / platform as a whole. Your pipeline should continue or abort based on SLIs emitted directly, in real-time & continuously. This concept is key to rolling upgrades, canary deploys and progressive roll-outs.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">I didn't think the presence of a manual step would prevent us using
a pipeline, in as much as I thought of a pipeline as just being a
series of servers with matching branches and code is them moved
along them whether manually or automatically. Heroku calls such
things pipelines and seems to have no support for automatically
moving code along them, it's purely manual from what I can see.<br></div></blockquote><div><br></div><div><a href="https://circleci.com/docs/2.0/workflows/#holding-a-workflow-for-a-manual-approval">CircleCI supports manual approval in workflows</a>, which maps to the manual step that you are referring to. From my perspective, this is an anti-pattern, but it might be necessary as a transition step towards SLIs, which is my recommendation.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
<p><img src="cid:1682f535c99aec3d6ac1" alt="" width="790" height="157"></p></div></blockquote><div>This helps a lot to visualise your current state of the world. Notice how you can only promote builds, which in the above screenshot map to a git commit sha. This is correct from all perspectives. What I believe that you are missing in Heroku pipelines is the functionality to define rules for auto-promotion. I am not familiar with Heroku pipelines, but CircleCI, Concourse & GoCD have the functionality which I mentioned. I am sure that there are other CIs which support this, I'm only mentioning the ones that I have done this with.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><p></p>
<div dir="ltr">
<div dir="ltr">
<div>But so just to be clear, there is something in these pipelines
that you're sharing that regularly moves code from one branch to
another?</div></div></div></div></blockquote><div><br></div><div>No, not in the pipelines which I gave as examples. Even if this would be possible, it would be wrong. The only branch that should be deployed and promoted to various environments, is the master branch. The reason why a human should be responsible for all commits (this includes merges, which are a complication), is so that mistakes are small, easy to spot, and easy to fix. Mistakes are an opportunity to learn, not something to avoid. Good systems contain mistakes, make them visible to engineers and preserve themselves at all costs.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><div dir="ltr"><div dir="ltr"><div>And that's something that CircleCI and RabbitMQ
provide?</div></div></div></div></blockquote><div><br></div><div>CircleCI exposes primitives that anyone can use to build the pipeline that they need. The same is true for Concourse, the CI system that we use for RabbitMQ. The RabbitMQ pipeline that I shared is nothing more than one of the many Concourse pipelines that have been built. Many other Concourse pipelines are publicly available, good for inspiration. Something related that I did for a past Ruby Unconference <a href="http://pipeline.gerhard.io/">http://pipeline.gerhard.io/</a></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><div dir="ltr"><div dir="ltr"><div>Or these are pipelines where the same code in the same
branch is being moved through a series of servers, based on tests
and checks passing at each server?<br></div></div></div></div></blockquote><div><br></div><div>The <a href="http://changelog.com">changelog.com</a> pipeline is the same code in the same branch, single repo.</div><div><br></div><div>The RabbitMQ pipeline is a combination of many repositories, always master branch, combined with many dependencies, and built into many artefacts - 38 artefacts according to <a href="https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.7.10">https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.7.10</a></div><div><br></div><div>The challenge that you have on your hands is interesting for me, please reach out offlist if you want to work on this together.</div><div><br></div><div>Gerhard</div></div></div></div></div>