[LRUG] Continuous * (Happy New Year!)

[Gerhard Lazu]

Hi Sam,

What determines that a build can go from your development environment into
> staging?
>
> Good question, the answer is:
>
> 1) that all the unit, integration and acceptance tests pass
> 2) that there are no merge conflicts
> 3) that the manual sanity checks on develop are coming back okay
>
> And from staging into production? If you can capture this in code, you can
> put it into a pipeline.
>
> I don't think there's any way we can remove the manual sanity checks as
> the acceptance tests are just not that reliable, and although we've poured
> 1000's of hours into them and ultimately I can't see any way of making them
> perfect.
>

You made 2 points here which hint to a larger problem: merge conflicts and
manual checks.

1. Merge conflicts are a symptom of branches. If all commits go straight to
master, and there are no branches, merge conflicts will be small, local &
dealt with instantly by humans. This approach will result in many small
commits, continuously flowing through your pipeline. Small slices of
functionality and value delivered continuously into production is the trick
to hundreds of updates daily. 100 merges add a lot of overhead, the total
effort is great. Alternatively, 100 commits to master are simple, we all do
it all the time.

2. Manual sanity checks are a symptom of poor observability and
instrumentation. Think Service Level Indicators (SLIs) & Service Level
Objectives (SLOs) that capture the health of your app / system / platform
as a whole. Your pipeline should continue or abort based on SLIs emitted
directly, in real-time & continuously. This concept is key to rolling
upgrades, canary deploys and progressive roll-outs.


> I didn't think the presence of a manual step would prevent us using a
> pipeline, in as much as I thought of a pipeline as just being a series of
> servers with matching branches and code is them moved along them whether
> manually or automatically.  Heroku calls such things pipelines and seems to
> have no support for automatically moving code along them, it's purely
> manual from what I can see.
>

CircleCI supports manual approval in workflows
<https://circleci.com/docs/2.0/workflows/#holding-a-workflow-for-a-manual-approval>,
which maps to the manual step that you are referring to. From my
perspective, this is an anti-pattern, but it might be necessary as a
transition step towards SLIs, which is my recommendation.


> This helps a lot to visualise your current state of the world. Notice how
you can only promote builds, which in the above screenshot map to a git
commit sha. This is correct from all perspectives. What I believe that you
are missing in Heroku pipelines is the functionality to define rules for
auto-promotion. I am not familiar with Heroku pipelines, but CircleCI,
Concourse & GoCD have the functionality which I mentioned. I am sure that
there are other CIs which support this, I'm only mentioning the ones that I
have done this with.

> But so just to be clear, there is something in these pipelines that you're
> sharing that regularly moves code from one branch to another?
>

No, not in the pipelines which I gave as examples. Even if this would be
possible, it would be wrong. The only branch that should be deployed and
promoted to various environments, is the master branch. The reason why  a
human should be responsible for all commits (this includes merges, which
are a complication), is so that mistakes are small, easy to spot, and easy
to fix. Mistakes are an opportunity to learn, not something to avoid. Good
systems contain mistakes, make them visible to engineers and preserve
themselves at all costs.


> And that's something that CircleCI and RabbitMQ provide?
>

CircleCI exposes primitives that anyone can use to build the pipeline that
they need. The same is true for Concourse, the CI system that we use for
RabbitMQ. The RabbitMQ pipeline that I shared is nothing more than one of
the many Concourse pipelines that have been built. Many other Concourse
pipelines are publicly available, good for inspiration. Something related
that I did for a past Ruby Unconference http://pipeline.gerhard.io/


> Or these are pipelines where the same code in the same branch is being
> moved through a series of servers, based on tests and checks passing at
> each server?
>

The changelog.com pipeline is the same code in the same branch, single repo.

The RabbitMQ pipeline is a combination of many repositories, always master
branch, combined with many dependencies, and built into many artefacts - 38
artefacts according to
https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.7.10

The challenge that you have on your hands is interesting for me, please
reach out offlist if you want to work on this together.

Gerhard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20190108/576c1e32/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nmfedjpijhmiddke.png
Type: image/png
Size: 113143 bytes
Desc: not available
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20190108/576c1e32/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: geelgcfnnoegpbnp.png
Type: image/png
Size: 56951 bytes
Desc: not available
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20190108/576c1e32/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pdkbkpfocdjfmoki.png
Type: image/png
Size: 236620 bytes
Desc: not available
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20190108/576c1e32/attachment-0005.png>