<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi Everyone,</p>
<p>I was just reading the devops handbook which had the following to
say on the topic of continuous delivery and continuous deployment
in practice, and reminded me of our discussion earlier in the
year:<br>
<blockquote type="cite">In Continuous Delivery, Jez Humble and
David Farley define the term continuous delivery. The term
continuous deployment was first mentioned by Tim Fitz in his
blog post “Continuous Deployment at IMVU: Doing the impossible
fifty times a day.” However, in 2015, during the construction of
The DevOps Handbook, Jez Humble commented, “In the last five
years, there has been confusion around the terms continuous
delivery versus continuous deployment—and, indeed, my own
thinking and definitions have changed since we wrote the book. <b>Every
organization should create their variations, based on what
they need</b>. The key thing we should care about is not the
form, but the outcomes: deployments should be low-risk,
push-button events we can perform on demand.” <br>
<br>
His updated definitions of continuous delivery and continuous
deployment are as follows: <br>
<i><br>
</i><i>When all developers are working in small batches on
trunk, or everyone is working off trunk in short-lived feature
branches that get merged to trunk regularly, and when trunk is
always kept in a releasable state, and when we can release on
demand at the push of a button during normal business hours,
we are doing continuous delivery. Developers get fast feedback
when they introduce any regression errors, which include
defects, performance issues, security issues, usability
issues, etc. When these issues are found, they are fixed
immediately so that trunk is always deployable. <br>
<br>
In addition to the above, when we are deploying good builds
into production on a regular basis through self-service (being
deployed by Dev or by Ops)—which typically means that we are
deploying to production at least once per day per developer,
or perhaps even automatically deploying every change a
developer commits—this is when we are engaging in continuous
deployment. </i><br>
<br>
Defined this way, continuous delivery is the prerequisite for
continuous deployment—just as continuous integration is a
prerequisite<br>
prerequisite for continuous delivery. Continuous deployment is
likely applicable in the context of web services that are
delivered online. However, continuous delivery is applicable in
almost every context where we desire deployments and releases
that have high quality, fast lead times and have highly
predictable, low-risk outcomes, including for embedded systems,
COTS products, and mobile apps.<br>
<br>
Kim, Gene. The DevOps Handbook:: How to Create World-Class
Agility, Reliability, and Security in Technology Organizations .
IT Revolution Press. Kindle Edition. <br>
</blockquote>
Which is interesting. In my mind what's described above as
"continuous delivery" is what I think of as "continuous
integration", but I like their emphasis:<br>
<blockquote type="cite"><b>Every organization should create their
variations, based on what they need</b>. The key thing we
should care about is not the form, but the outcomes: deployments
should be low-risk, push-button events we can perform on demand.</blockquote>
Best, Sam<br>
<br>
</p>
<div class="moz-cite-prefix">On 16/01/2019 09:45, Samuel Joseph
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:b1fc2a41-0a04-6c86-129d-670012e63f4a@gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p>Big thanks to everyone for their input on this discussion. In
an effort to try and wrap up, let me pull together threads from
the last few emails, rather than replying to each individually.</p>
<p>Gareth, okay, so "deploy master to staging and then to
production", and "don't merge develop into master until you're
sure there will be no more hotfixes, and you also merge hotfixes
back into develop as soon as possible".</p>
<p>I don't know there's any mechanism to ever ensure there would
never be a hotfix, in that staging can't perfectly production in
terms of sending real emails, social media messages to real
users etc., but yes of course, merge hotfixes back ASAP - as we
currently do from production -> staging -> develop
branches in our setup.</p>
<p>Gareth wrote:<br>
</p>
<blockquote type="cite">the original core of your question, this
is a simple answer. Once your manual tests are complete, this is
exactly what the "promote" button (or equivalent cli command)
does. It reduces the manual part of the deploy process down to a
single button click, which I think is what you wanted from the
start of this conversation.</blockquote>
I'm not sure that's what I wanted. I can do `git rebase staging`,
`git push origin master` on my command line to achieve the same
effect, and I could make that a script or hook it up to a button
if I wanted, but that's not my problem. The problem is that I
have to initiate a production deploy at all; and I particularly
don't want to have to remember to do it and them remember to come
back and check the results later on. I guess the heroku "promote"
button is useful in this regard in that it does not kick off any
tests, or do a lengthy deploy process - it gets the code up much
faster (but assumes staging server on master branch, as you
suggest) - I'll need to measure that and see ... <br>
<p>What I really want is to not have to remember to come and do a
production deploy every week, and I also want it to be atomic,
i.e. not kick it off in the morning and come back in an hour or
two to see if it worked. I'm trying to deal with my unreliable
memory, emotional exhaustion etc. I'd like to know that a
production deploy will happen every week (on wednesdays say) and
that I will have to get my act together on Tuesday to check for
any reasons to stop the deploy. <br>
</p>
<p>So I wonder if heroku (and other CI systems) allow automated
promotion on particular dates ... which leads me to Sasha's
email:<br>
<br>
Sasha wrote:<br>
</p>
<blockquote type="cite">If you haven't already seen <a
href="https://www.spinnaker.io/guides/user/pipeline/managing-pipelines/"
moz-do-not-send="true">https://www.spinnaker.io/guides/user/pipeline/managing-pipelines/</a>
then I highly recommend looking over Spinnaker as I believe it
will help cover the steps you've outlined as missing in
providers like AWS, Heroku, etc.</blockquote>
<p> thanks for the recommendation Sasha, reading through I see
pipeline triggers can be cron jobs, so indeed one could set
things up to automatically promote to production on Wednesdays,
or similar. And indeed I could stick with our current
develop/staging/master branch model and put the `git rebase`
operations into the cron jobs. It sounds like we might be alone
if we do something like that - from the replies on the list is
sounds like the common pattern is using develop/master and the
staging server is on master as well, and it sounds like most
folks have to remember to take some manual step to trigger a
production deploy at some frequency.<br>
</p>
<p> I wonder if anyone anywhere in the world is doing automated
deploys to production on particular days of the week ...?</p>
<p>Mark wrote:<br>
</p>
<p> </p>
<blockquote type="cite">You probably also want a strategy for
giving you back the confidence of being able to know what is on
prod/staging at any point.</blockquote>
At the moment we have full confidence of what is on prod/staging
at any point. I'd argue that with the staging server/branch and
production/master server/branch model anyone can instantly browse
exactly what we have on prod/staging via GitHub, facilitating a
much more open development than if the only way to see what's on
those servers is through an interface like Herokus, which is not
visible to just anyone.<br>
<p><a class="moz-txt-link-freetext"
href="https://github.com/AgileVentures/WebsiteOne/tree/staging"
moz-do-not-send="true">https://github.com/AgileVentures/WebsiteOne/tree/staging</a></p>
<p><a class="moz-txt-link-freetext"
href="https://github.com/AgileVentures/WebsiteOne/tree/master"
moz-do-not-send="true">https://github.com/AgileVentures/WebsiteOne/tree/master</a></p>
<p>but does sound like we're the only folks in the LRUG sphere
with this model ...</p>
<p>Gareth wrote:<br>
</p>
<blockquote type="cite">I don't think the environment branches are
needed in your setup, and they're a legacy of a different
workflow.</blockquote>
I think they offer some advantages that I've outlined, and I was
the one that originally implemented that workflow based on what
some folks in the states were recommending - I wonder if there are
books that get into the nitty/gritty of all this, or if this is
still all just shared informal practices spread across different
groups around the world ...
<p>Huge thanks to everyone for participating in this discussion.
I've learnt a great deal, although I'm still not sure if I'll be
able to use the terms "continuous deployment" and "continuous
delivery" without causing confusion or being confused :-) but
I've got lots to experiment with and that was my key objective.
<br>
</p>
<p>Best, Sam<br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 09/01/2019 10:42, Gareth Adams
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAJ=YDWvbs2SVeF5bV-uoF+uZJHm7G1Jv3m_5rc7-M+b6BtGcdQ@mail.gmail.com">
<meta http-equiv="content-type" content="text/html;
charset=UTF-8">
<div dir="auto">
<div><br>
<br>
<div class="gmail_quote">
<div dir="ltr">On Wed, 9 Jan 2019, 10:08 Samuel Joseph
<<a href="mailto:tansaku@gmail.com" rel="noreferrer
noreferrer noreferrer" target="_blank"
moz-do-not-send="true">tansaku@gmail.com</a> wrote:</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>I guess we could be deploying to both staging and
production from the master branch, but by having the
separate develop, staging and production branches,
that each auto-deploy to the three servers, it means
that we've got a clear place to go (locally and in
GitHub) to see a running version of whatever is on
each of those three branches</p>
</div>
</blockquote>
</div>
</div>
<div dir="auto">In the workflow I think you're describing, the
"clear place to go" should be your CD tool - in this case
the Heroku pipeline - <span style="font-family:sans-serif">rather
than your source code repository.</span> But there are
definitely nuances, and if getting that info back into your
repo is important then maybe I'm simplifying too much.</div>
<div dir="auto">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>and if for some reason production has a weird
issues and we have to hot fix (fairly rare) then we
have the staging branch to do it on, safe in the
knowledge that a hot-fix applied to staging can be
auto-deployed to the staging server, and won't
accidentally make thngs worse on production (which
is of course auto-deployed off master).</p>
</div>
</blockquote>
</div>
</div>
<div dir="auto">I'll rephrase what I wrote in my last email
to:</div>
<div dir="auto"><br>
</div>
<div dir="auto">…you instead think "deploy master to staging <b>and
then</b> to production".<br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">In this workflow, there's no such thing as a
"hotfix to staging", just a new commit to master, which
follows the same "master ➡️ staging ➡️ production" process.
Note this means you don't merge develop into master until
you're sure there will be no more hotfixes, and you also
merge hotfixes back into develop as soon as possible.</div>
<div dir="auto">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>But then, does no one do any manual tests on
staging before there's a deploy to production?
What's the trigger that says it's okay to deploy to
production following a staging deploy?</p>
</div>
</blockquote>
</div>
</div>
<div dir="auto">And now we've circled around to the original
core of your question, this is a simple answer. Once your
manual tests are complete, this is exactly what the
"promote" button (or equivalent cli command) does. It
reduces the manual part of the deploy process down to a
single button click, which I think is what you wanted from
the start of this conversation.</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div
class="m_7990572999333527681m_8506084164337888487m_-8428862481596492984m_-6735578697607866647moz-cite-prefix">On
08/01/2019 11:19, Matthias Berth wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hi Sam, <br>
</div>
<div><br>
</div>
<div>Why are you rebasing in the first place? <br>
</div>
<div>Can't you just make a feature branch off
master, then merge it back into master? And
deploy to staging and production from the master
branch?</div>
<div><br>
</div>
<div>I also wonder why your 20 minutes sanity
checks cannot be automated. Are you doing
something new / creative every time you do these
tests?</div>
<div>Great discussion!</div>
<div><br>
</div>
<div>Cheers</div>
<div><br>
</div>
<div>Matthias<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Tue, Jan 8, 2019 at 11:48 AM
Gareth Adams <<a href="mailto:g@rethada.ms"
rel="noreferrer noreferrer noreferrer
noreferrer" target="_blank"
moz-do-not-send="true">g@rethada.ms</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="auto">Sam, I don't have a grand answer
to your whole question, but a phrase leapt out
at me and I wanted to flag it:
<div dir="auto"><br>
</div>
<div dir="auto">> rebasing along the
pipeline</div>
<div dir="auto"><br>
</div>
<div dir="auto">To me, this suggests the code
on your staging branch is not the same as
the code you end up deploying to production
(and it might not be the same as the code in
your master branch)</div>
<div dir="auto"><br>
</div>
<div dir="auto">I guess there are a few
reasons that could be: you're storing some
environment-specific configuration on your
environment branches, and can't merge them
all together, or maybe your environment
branches contain a different combination of
feature branches that you're trying to keep
control of?</div>
<div dir="auto"><br>
</div>
<div dir="auto">Either way, I'd consider how
(or if) you could change your workflow to
make sure you deploy the same code
everywhere (or at least deploy the same code
to production that you deploy to staging).
That's the basis behind e.g. Heroku
pipelines' "Promote" button and it's the
pattern I commonly see now</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Tue, 8 Jan 2019, 10:13
Samuel Joseph <<a
href="mailto:tansaku@gmail.com"
rel="noreferrer noreferrer noreferrer
noreferrer noreferrer noreferrer"
target="_blank" moz-do-not-send="true">tansaku@gmail.com</a>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Gerhard,<br>
</p>
<div
class="m_7990572999333527681m_8506084164337888487m_-8428862481596492984m_-6735578697607866647gmail-m_-4091885444582072218m_3868980048267634368m_-2707529618717350415m_-2020117851195068980moz-cite-prefix">On
07/01/2019 12:00, Gerhard Lazu wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">Hi Sam,
<div><br>
</div>
<div>What determines that a build
can go from your development
environment into staging? </div>
</div>
</div>
</blockquote>
Good question, the answer is:<br>
<br>
1) that all the unit, integration and
acceptance tests pass<br>
2) that there are no merge conflicts<br>
3) that the manual sanity checks on
develop are coming back okay<br>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div>And from staging into
production? If you can capture
this in code, you can put it into
a pipeline.<br>
<br>
</div>
</div>
</div>
</blockquote>
I don't think there's any way we can
remove the manual sanity checks as the
acceptance tests are just not that
reliable, and although we've poured 1000's
of hours into them and ultimately I can't
see any way of making them perfect.<br>
<br>
I didn't think the presence of a manual
step would prevent us using a pipeline, in
as much as I thought of a pipeline as just
being a series of servers with matching
branches and code is them moved along them
whether manually or automatically. Heroku
calls such things pipelines and seems to
have no support for automatically moving
code along them, it's purely manual from
what I can see.<br>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div><br>
</div>
<div>Why do you have 3 pipelines? </div>
</div>
</div>
</blockquote>
<p>I don't think we do. As I understand
it, we have one pipeline:</p>
<p>develop branch + develop server --->
staging branch + staging server --->
master branch + production server</p>
<p>That's three paired branches/servers in
one pipeline. Here's a screenshot of
how Heroku presents our pipeline in
their pipeline interface. Note the
button "Promote to staging" which allows
you to manually move the code on the
develop server to the staging server,
but doesn't actually do a rebase of the
code from develop branch to staging:<br>
</p>
<p><img alt="" moz-do-not-send="true"
width="790" height="157"></p>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div>Based on the questions that
you're asking, I believe that it
would help if you had a single
pipeline. </div>
</div>
</div>
</blockquote>
I agree - I think we do, but perhaps I'm
wrong ...<br>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div>The question that I would focus
on is <i>What would it take to
have a single pipeline that has
an end-goal of creating
production builds</i>? Here is a
pipeline example which stops after
it publishes a Docker image: <a
href="https://circleci.com/workflow-run/065467ef-87c0-4f5e-a2ab-5e11be12403f"
rel="noreferrer noreferrer
noreferrer noreferrer noreferrer
noreferrer noreferrer"
target="_blank"
moz-do-not-send="true">changelog.com,
CircleCI</a>.</div>
</div>
</div>
</blockquote>
Ooh, thanks for sharing! I had to log in
to CircleCI to see that:<br>
<br>
<p><img alt="" moz-do-not-send="true"
width="470" height="208"></p>
but that looks really interesting.<br>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div> If you are using something
like Docker Swarm or Kubernetes,
the platform/ecosystem has all the
necessary tools to keep deployment
concerns self-contained. In the <a
href="http://changelog.com"
rel="noreferrer noreferrer
noreferrer noreferrer noreferrer
noreferrer noreferrer"
target="_blank"
moz-do-not-send="true">changelog.com</a>
case, the Docker stack that
captures the entire deployment has
<a
href="https://github.com/thechangelog/changelog.com/blob/cf2ebe0de0f35c96bf664b8bc9183bd1f3468565/docker/changelog.stack.yml#L15-L31"
rel="noreferrer noreferrer
noreferrer noreferrer noreferrer
noreferrer noreferrer"
target="_blank"
moz-do-not-send="true">an update
component that is responsible
for app updates</a>. In this
specific case, if the new app
version starts and is healthy for
30 seconds, it gets automatically
promoted to live. <a
href="http://changelog.com/podcast/254"
rel="noreferrer noreferrer
noreferrer noreferrer noreferrer
noreferrer noreferrer"
target="_blank"
moz-do-not-send="true">We have
been using a similar approach
since October 2016</a>, a Docker
stack just makes it easier.</div>
<div><br>
</div>
<div>I want to spur your imagination
by sharing <a
href="https://ci.rabbitmq.com/teams/main/pipelines/server-release:v3.7.x"
rel="noreferrer noreferrer
noreferrer noreferrer noreferrer
noreferrer noreferrer"
target="_blank"
moz-do-not-send="true">the
pipeline that is responsible for
RabbitMQ v3.7.x</a>. This
pipeline captures what is possible
if imagination is set free:</div>
</div>
</div>
</blockquote>
Wow, that RabbitMQ pipeline looks amazing
- and I've only captured part of it in the
screenshot:<br>
<br>
<img alt="" moz-do-not-send="true"
width="593" height="375">
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div><br>
</div>
<div>* tests & builds 30+
apps...</div>
<div>* on all supported major
runtime version...</div>
<div>* and all supported OSes</div>
<div>* tests upgrades</div>
<div>* tests client support</div>
<div>* releases alphas, betas, RCs
& GAs</div>
<div>* and publishes to all
supported distribution channels</div>
<div><br>
</div>
<div>I hope this helps, Gerhard.</div>
</div>
</div>
</blockquote>
<p>That's extremely helpful - thankyou!</p>
<p>But so just to be clear, there is
something in these pipelines that you're
sharing that regularly moves code from
one branch to another? And that's
something that CircleCI and RabbitMQ
provide? Or these are pipelines where
the same code in the same branch is
being moved through a series of servers,
based on tests and checks passing at
each server?</p>
<p>Many thanks in advance<br>
Best, Sam<br>
</p>
</div>
_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org"
rel="noreferrer noreferrer noreferrer
noreferrer noreferrer noreferrer
noreferrer" target="_blank"
moz-do-not-send="true">Chat@lists.lrug.org</a><br>
Archives: <a
href="http://lists.lrug.org/pipermail/chat-lrug.org"
rel="noreferrer noreferrer noreferrer
noreferrer noreferrer noreferrer
noreferrer noreferrer" target="_blank"
moz-do-not-send="true">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a
href="http://lists.lrug.org/options.cgi/chat-lrug.org"
rel="noreferrer noreferrer noreferrer
noreferrer noreferrer noreferrer
noreferrer noreferrer" target="_blank"
moz-do-not-send="true">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a
href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org"
rel="noreferrer noreferrer noreferrer
noreferrer noreferrer noreferrer
noreferrer noreferrer" target="_blank"
moz-do-not-send="true">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
</blockquote>
</div>
_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org"
rel="noreferrer noreferrer noreferrer
noreferrer" target="_blank"
moz-do-not-send="true">Chat@lists.lrug.org</a><br>
Archives: <a
href="http://lists.lrug.org/pipermail/chat-lrug.org"
rel="noreferrer noreferrer noreferrer
noreferrer noreferrer" target="_blank"
moz-do-not-send="true">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a
href="http://lists.lrug.org/options.cgi/chat-lrug.org"
rel="noreferrer noreferrer noreferrer
noreferrer noreferrer" target="_blank"
moz-do-not-send="true">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a
href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org"
rel="noreferrer noreferrer noreferrer
noreferrer noreferrer" target="_blank"
moz-do-not-send="true">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
</blockquote>
</div>
<br>
<fieldset
class="m_7990572999333527681m_8506084164337888487m_-8428862481596492984m_-6735578697607866647mimeAttachmentHeader"></fieldset>
<pre class="m_7990572999333527681m_8506084164337888487m_-8428862481596492984m_-6735578697607866647moz-quote-pre">_______________________________________________
Chat mailing list
<a class="m_7990572999333527681m_8506084164337888487m_-8428862481596492984m_-6735578697607866647moz-txt-link-abbreviated" href="mailto:Chat@lists.lrug.org" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank" moz-do-not-send="true">Chat@lists.lrug.org</a>
Archives: <a class="m_7990572999333527681m_8506084164337888487m_-8428862481596492984m_-6735578697607866647moz-txt-link-freetext" href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank" moz-do-not-send="true">http://lists.lrug.org/pipermail/chat-lrug.org</a>
Manage your subscription: <a class="m_7990572999333527681m_8506084164337888487m_-8428862481596492984m_-6735578697607866647moz-txt-link-freetext" href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank" moz-do-not-send="true">http://lists.lrug.org/options.cgi/chat-lrug.org</a>
List info: <a class="m_7990572999333527681m_8506084164337888487m_-8428862481596492984m_-6735578697607866647moz-txt-link-freetext" href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank" moz-do-not-send="true">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a>
</pre>
</blockquote>
</div>
_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" rel="noreferrer
noreferrer noreferrer noreferrer" target="_blank"
moz-do-not-send="true">Chat@lists.lrug.org</a><br>
Archives: <a
href="http://lists.lrug.org/pipermail/chat-lrug.org"
rel="noreferrer noreferrer noreferrer noreferrer
noreferrer" target="_blank" moz-do-not-send="true">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a
href="http://lists.lrug.org/options.cgi/chat-lrug.org"
rel="noreferrer noreferrer noreferrer noreferrer
noreferrer" target="_blank" moz-do-not-send="true">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a
href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org"
rel="noreferrer noreferrer noreferrer noreferrer
noreferrer" target="_blank" moz-do-not-send="true">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
</blockquote>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Chat mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Chat@lists.lrug.org" moz-do-not-send="true">Chat@lists.lrug.org</a>
Archives: <a class="moz-txt-link-freetext" href="http://lists.lrug.org/pipermail/chat-lrug.org" moz-do-not-send="true">http://lists.lrug.org/pipermail/chat-lrug.org</a>
Manage your subscription: <a class="moz-txt-link-freetext" href="http://lists.lrug.org/options.cgi/chat-lrug.org" moz-do-not-send="true">http://lists.lrug.org/options.cgi/chat-lrug.org</a>
List info: <a class="moz-txt-link-freetext" href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" moz-do-not-send="true">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a>
</pre>
</blockquote>
</blockquote>
</body>
</html>