[LRUG] Puma, CLOSE_WAIT. Arg.

Simon Morley simon at polkaspots.com
Thu Feb 18 08:01:32 PST 2016 

As an interim update, adding the recommended
ActiveRecord::Base.connection_pool.disconnect! doesn't seem to help :(

We're seeing far fewer timeouts but they're still happening on that db
connection. And when they happen, it's carnage for 15s.



Simon Morley

Big Chief | PolkaSpots Supafly Wi-Fi
Bigger Chief | Cucumber Tony

Got an unlicensed Meraki? Set it free with Cucumber
cucumberwifi.io/meraki


On 18 February 2016 at 14:15, Riccardo Tacconi <rtacconi at gmail.com> wrote:

> Sorry I meant you could use background jobs to query the DB, puma could
> just create a job without connection to the DB, that each job will have a
> connection. The main problem seems that you are using one process (worker)
> and one thread (because MRI uses one thread), which is the most inefficient
> thing to do. I think if you will put the database pool to one it will not
> change anything, I meant it would not be worser than now. I would try to:
>
>
>    - use real-threads, rubinius or JRuby,
>    - ActiveRecord::Base.connection.disconnect! before forking
>    - ActiveRecord::Base.establish_connection after forking
>    - use more than one puma worker, say 1 : 50 connections, you will find
>    the right ratio
>    - not really sure about the connection pool, may be should be equal to
>    puma workers times puma threads (real threads)
>
> Your endpoint cannot connected and do what is supposed to do in time,
> that's way it does not have time to close connections. So you need to have
> more processing power, which comes from parallel processing, by using
> processes and / or threads, this power should be used, in part to connect
> to the DB, and having more connections will help too, but this is in the
> end of the line.
>
> If you will use background job, you will have to think about how to send
> back the processing, but it is usually the most scalable option. However
> first you have you Ruby, Puma, DB config setup properly :-)
>
> On 18 February 2016 at 13:30, Simon Morley <simon at polkaspots.com> wrote:
>
>> I'm afraid I have no virtually no idea what you mean! :(
>>
>> We can't swap from cruby to jruby - mainly because that's like inviting
>> someone else to the miserable party we're already hosting.
>>
>>
>>
>>
>>
>> Simon Morley
>>
>> Big Chief | PolkaSpots Supafly Wi-Fi
>> Bigger Chief | Cucumber Tony
>>
>> Got an unlicensed Meraki? Set it free with Cucumber
>> cucumberwifi.io/meraki
>>
>>
>> On 18 February 2016 at 12:43, Riccardo Tacconi <rtacconi at gmail.com>
>> wrote:
>>
>>> Ruby MRI? If yes what's the point of using Puma? With MRI you have one
>>> worker and one thread, which is very inefficient. Would be possible to
>>> split HTTP requests handling from querying the DB? From Puma you could send
>>> requests to a topic (MOM), and multiple workers could process requests and
>>> each worker will have a DB connection. This could work with MRI, although
>>> you will need more RAM. However I would try rubinius of jruby first.
>>>
>>> Sorry if I misunderstood, I did not follow the whole thread.
>>>
>>>
>>> On 18 February 2016 at 12:28, Simon Morley <simon at polkaspots.com> wrote:
>>>
>>>> Ruby 2.2.2
>>>> Rails 4.2.5.1
>>>> mysql2 0.4.2 (tried a few)
>>>> Puma 2.16.0
>>>>
>>>>
>>>>
>>>> Simon Morley
>>>>
>>>> Big Chief | PolkaSpots Supafly Wi-Fi
>>>> Bigger Chief | Cucumber Tony
>>>>
>>>> Got an unlicensed Meraki? Set it free with Cucumber
>>>> cucumberwifi.io/meraki
>>>>
>>>>
>>>> On 18 February 2016 at 12:24, Riccardo Tacconi <rtacconi at gmail.com>
>>>> wrote:
>>>>
>>>>> Which version of Ruby are you using?
>>>>>
>>>>> On 18 February 2016 at 12:17, Simon Morley <simon at polkaspots.com>
>>>>> wrote:
>>>>>
>>>>>> Actually puma docs suggest doing that when using preload_app and
>>>>>> ActiveRecord...
>>>>>>
>>>>>> https://github.com/puma/puma#clustered-mode
>>>>>>
>>>>>>
>>>>>>
>>>>>> Simon Morley
>>>>>>
>>>>>> Big Chief | PolkaSpots Supafly Wi-Fi
>>>>>> Bigger Chief | Cucumber Tony
>>>>>>
>>>>>> Got an unlicensed Meraki? Set it free with Cucumber
>>>>>> cucumberwifi.io/meraki
>>>>>>
>>>>>>
>>>>>> On 18 February 2016 at 12:05, Frederick Cheung <
>>>>>> frederick.cheung at gmail.com> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 18 February 2016 at 11:17:34, Simon Morley (simon at polkaspots.com)
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> class RadiusDatabase
>>>>>>>   self.abstract_class = true
>>>>>>>   establish_connection "radius_#{Rails.env}".to_sym
>>>>>>> end
>>>>>>>
>>>>>>> class Radacct < RadiusDatabase
>>>>>>> end
>>>>>>>
>>>>>>> Then I decreased our database pool from 20 to 5 and added a
>>>>>>> wait_timeout of 5 (since there seems to be some discrepancies with this).
>>>>>>> Things got much better (but weren't fixed).
>>>>>>>
>>>>>>> I tried querying differently, including using
>>>>>>> connection_pool.with_connection. I've tried closing the connections
>>>>>>> manually and also used ActiveRecord::Base.clear_active_connections!
>>>>>>> periodically. No joy.
>>>>>>>
>>>>>>> By this point, we were running 2-4 instances - handling around very
>>>>>>> little traffic in total (about 50rpm). Every few hours, they'd block, all
>>>>>>> of them. At the same time, we'd see a load of rack timeouts - same DB. I've
>>>>>>> checked the connections - they were each opening only a few to MySQL and
>>>>>>> MySQL was looking good.
>>>>>>>
>>>>>>> One day, by chance, I reduced the 4 instances to 1. *And the
>>>>>>> problem is solved!!! WHAT*? Obviously the problem isn't solved, we
>>>>>>> can only use a single server.
>>>>>>>
>>>>>>>
>>>>>>> Are you using puma in the mode where it forks workers? if so, then
>>>>>>> you want to reconnect post fork or multiple processes will share the same
>>>>>>> file descriptor and really weird shit will happen.
>>>>>>>
>>>>>>> The puma readme advises to do this:
>>>>>>>
>>>>>>> before_fork do
>>>>>>>   ActiveRecord::Base.connection_pool.disconnect!
>>>>>>> end
>>>>>>>
>>>>>>> I don't know off the top of my head whether that  will do the job
>>>>>>> for classes that have established a connection to a different db -
>>>>>>> presumably they have a separate connection pool
>>>>>>>
>>>>>>> Fred
>>>>>>>
>>>>>>> I don't know what's going on here. Have I been staring at this for
>>>>>>> too long (yes)?
>>>>>>>
>>>>>>> Our other servers are chugging along happily now, using a connection
>>>>>>> pool of 20, no errors, no timeouts (different db though).
>>>>>>>
>>>>>>> Has anyone got any suggestions / seen this? Is there something
>>>>>>> fundamentally wrong with the way we're establishing a connection to the
>>>>>>> external dbs? Surely this is MySQL related
>>>>>>>
>>>>>>> Thanks for listening,
>>>>>>>
>>>>>>> S
>>>>>>>
>>>>>>>
>>>>>>> Simon Morley
>>>>>>>
>>>>>>> Got an unlicensed Meraki? Set it free with Cucumber
>>>>>>> cucumberwifi.io/meraki
>>>>>>>
>>>>>>>
>>>>>>> On 15 January 2016 at 13:58, Gerhard Lazu <gerhard at lazu.co.uk>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> The understanding of difficult problems/bugs and the learning that
>>>>>>>> comes with it cannot be rushed. Each and every one of us has his / her own
>>>>>>>> pace, and all "speeds" are perfectly fine. The only question that really
>>>>>>>> matters is whether it's worth it (a.k.a. the cost of lost opportunity). If
>>>>>>>> the answer is yes, plough on. If not, look for alternatives.
>>>>>>>>
>>>>>>>> Not everyone likes or wants to run their own infrastructure. The
>>>>>>>> monthly savings on the PaaS, IaaS advertised costs are undisputed, but few
>>>>>>>> like to think - never mind talk - about how many hours / days / weeks have
>>>>>>>> been spent debugging obscure problems which "solve themselves" on a managed
>>>>>>>> environment. Don't get me started on those that are building their own
>>>>>>>> Docker-based PaaS-es without even realising it...
>>>>>>>>
>>>>>>>> As a side-note, I've been dealing with a similar TCP-related
>>>>>>>> problem for a while now, so I could empathise with your struggles the
>>>>>>>> second I've seen your post. One of us is bound to solve it first, and I
>>>>>>>> hope it will be you ; )
>>>>>>>>
>>>>>>>> Have a good one, Gerhard.
>>>>>>>>
>>>>>>>> On Fri, Jan 15, 2016 at 10:01 AM, Simon Morley <
>>>>>>>> simon at polkaspots.com> wrote:
>>>>>>>>
>>>>>>>>> You must be more patient that I am. It's been a long month -
>>>>>>>>> having said that, I'm excited to find the cause.
>>>>>>>>>
>>>>>>>>> I misunderstood you re. file descriptors. We checked the kernel
>>>>>>>>> limits / files open on the systems before and during and there's nothing
>>>>>>>>> untoward.
>>>>>>>>>
>>>>>>>>> Since writing in, it's not happened as before - no doubt it'll
>>>>>>>>> take place during our forthcoming office move today.
>>>>>>>>>
>>>>>>>>> I ran a strace (thanks for that suggestion John) on a couple of
>>>>>>>>> processes yesterday and saw redis blocking. Restarted a few redis servers
>>>>>>>>> to see if that helped. Can't be certain yet.
>>>>>>>>>
>>>>>>>>> As soon as it's on, I'll run a tcpdump. How I'd not thought about
>>>>>>>>> that I don't know...
>>>>>>>>>
>>>>>>>>> Actually, this is one thing I dislike about Rails - it's so nice
>>>>>>>>> and easy to do everything, one forgets we're dealing with the real servers
>>>>>>>>> / components / connections. It's too abstract in ways, but that's a whole
>>>>>>>>> other debate :)
>>>>>>>>>
>>>>>>>>> S
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Simon Morley
>>>>>>>>>
>>>>>>>>> Big Chief | PolkaSpots Supafly Wi-Fi
>>>>>>>>> Bigger Chief | Cucumber Tony
>>>>>>>>>
>>>>>>>>> simon at PolkaSpots.com
>>>>>>>>> Linkedin: I'm on it again and it still sucks
>>>>>>>>> 020 7183 1471 <020%207183%201471>
>>>>>>>>>
>>>>>>>>> 🚀💥
>>>>>>>>>
>>>>>>>>> On 15 January 2016 at 06:53, Gerhard Lazu <gerhard at lazu.co.uk>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> File descriptors, for traditional reasons, include TCP
>>>>>>>>>> connections.
>>>>>>>>>>
>>>>>>>>>> Are you logging all requests to a central location? When the
>>>>>>>>>> problem occurs, it might help taking a closer look at the type of requests
>>>>>>>>>> you're receiving.
>>>>>>>>>>
>>>>>>>>>> Depending on how long the mischief lasts, a tcpdump to pcap, then
>>>>>>>>>> wireshark might help. Same for an strace on the Puma processes, similar to
>>>>>>>>>> what John suggested . Those are low level tools though, verbose, complex
>>>>>>>>>> and complete, it's easy to get lost unless you know what you're looking for.
>>>>>>>>>>
>>>>>>>>>> In summary, CLOSE_WAITs piling up from haproxy (client role) to
>>>>>>>>>> Puma (server role) indicates the app not closing connections in time (or
>>>>>>>>>> maybe ever) - why? It's a fun one to troubleshoot ; )
>>>>>>>>>>
>>>>>>>>>> On Thu, Jan 14, 2016 at 11:35 PM, Simon Morley <
>>>>>>>>>> simon at polkaspots.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Right now, none of the servers have any issues. No close_waits.
>>>>>>>>>>>
>>>>>>>>>>> All is well. Seemingly.
>>>>>>>>>>>
>>>>>>>>>>> When it occurs ALL the servers end up going. Sometimes real
>>>>>>>>>>> fast. That's why I thought we had a db bottleneck. It happens pretty
>>>>>>>>>>> quickly, randomly, no particular times.
>>>>>>>>>>>
>>>>>>>>>>> We don't ever really get spikes of traffic, there's an even load
>>>>>>>>>>> inbound throughout.
>>>>>>>>>>>
>>>>>>>>>>> I thought we had someone running a slow loris style attack on
>>>>>>>>>>> us. So I added some rules to HA Proxy and Cloudflare ain't seen nofin
>>>>>>>>>>> honest guv.
>>>>>>>>>>>
>>>>>>>>>>> Will find a way to chart it and send a link over.
>>>>>>>>>>>
>>>>>>>>>>> Will see if we're not closing any files - not much of that going
>>>>>>>>>>> on. There's some manual gzipping happening - we've had that in place for
>>>>>>>>>>> over a year though - not sure why it'd start playing up now. Memory usage
>>>>>>>>>>> is high but consistent and doesn't increase.
>>>>>>>>>>>
>>>>>>>>>>> S
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Simon Morley
>>>>>>>>>>>
>>>>>>>>>>> Big Chief | PolkaSpots Supafly Wi-Fi
>>>>>>>>>>> Bigger Chief | Cucumber Tony
>>>>>>>>>>>
>>>>>>>>>>> simon at PolkaSpots.com
>>>>>>>>>>> Linkedin: I'm on it again and it still sucks
>>>>>>>>>>> 020 7183 1471 <020%207183%201471>
>>>>>>>>>>>
>>>>>>>>>>> 🚀💥
>>>>>>>>>>>
>>>>>>>>>>> On 14 January 2016 at 22:14, Gerhard Lazu <gerhard at lazu.co.uk>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> That sounds like a file descriptor leak. Are the CLOSE_WAITs
>>>>>>>>>>>> growing over time?
>>>>>>>>>>>>
>>>>>>>>>>>> You're right, New Relic is too high level, this is a layer 4-5
>>>>>>>>>>>> issue.
>>>>>>>>>>>>
>>>>>>>>>>>> The simplest thing that can plot some graphs will work. Throw
>>>>>>>>>>>> the dirtiest script together that curls the data out if it comes easy, it
>>>>>>>>>>>> doesn't matter how you get those metrics as long as you have them.
>>>>>>>>>>>>
>>>>>>>>>>>> This is a great blog post opportunity ; )
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Jan 14, 2016 at 8:40 PM, Simon Morley <
>>>>>>>>>>>> simon at polkaspots.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I would ordinarily agree with you about the connection however
>>>>>>>>>>>>> they hang around for hours sometimes.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The 500 in the hyproxy config was actually left over from a
>>>>>>>>>>>>> previous experiment. Realistically I know they won't cope with that.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Using another server was to find any issues with puma. I'm
>>>>>>>>>>>>> still going to try unicorn just in case.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Will up the numbers too - thanks for that suggestion.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'll look at a better monitoring tool too. So far new relic
>>>>>>>>>>>>> hasn't helped much.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>> S
>>>>>>>>>>>>>
>>>>>>>>>>>>> Simon Morley
>>>>>>>>>>>>> Big Chief | PolkaSpots Supafly Wi-Fi
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm doing it with Cucumber Tony. Are you?
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 14 Jan 2016, at 20:30, Gerhard Lazu <gerhard at lazu.co.uk>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>>>
>>>>>>>>>>>>> CLOSE_WAIT suggests that Puma is not closing connections fast
>>>>>>>>>>>>> enough. The client has asked for the connection to be closed, but Puma is
>>>>>>>>>>>>> busy.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Quickest win would be to increase your Puma instances. Unicorn
>>>>>>>>>>>>> won't help - or any other Rack web server for the matter.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Based on your numbers, start with 10 Puma instances. Anything
>>>>>>>>>>>>> more than 100 connections for a Rails instance is not realistic. I would
>>>>>>>>>>>>> personally go with 50, just to be safe. I think I saw 500 conns in your
>>>>>>>>>>>>> haproxy config, which is way too optimistic.
>>>>>>>>>>>>>
>>>>>>>>>>>>> You want metrics for detailed CPU usage by process,
>>>>>>>>>>>>> connections open with state by process, and memory usage, by process.
>>>>>>>>>>>>> Without these, you're flying blind. Any suggestions anyone makes without
>>>>>>>>>>>>> real metrics - including myself - are just guesses. You'll get there, but
>>>>>>>>>>>>> you're making it far too difficult for yourself.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Let me know how it goes, Gerhard.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Jan 14, 2016 at 3:16 PM, Simon Morley <
>>>>>>>>>>>>> simon at polkaspots.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello All
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We've been battling with Puma for a long while now, I'm
>>>>>>>>>>>>>> looking for some help / love / attention / advice / anything to prevent
>>>>>>>>>>>>>> further hair loss.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We're using it in a reasonably typical Rails 4 application
>>>>>>>>>>>>>> behind Nginx.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Over the last 3 months, our requests have gone from 500 rpm
>>>>>>>>>>>>>> to a little over 1000 depending on the hour. Over this period, we've been
>>>>>>>>>>>>>> seeing weird CLOSE_WAIT conns appearing in netstat, which eventually kill
>>>>>>>>>>>>>> the servers.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We have 3 Rails servers behind Haproxy running things. Load
>>>>>>>>>>>>>> is generally even.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Running netstat on the servers shows a pile of connections in
>>>>>>>>>>>>>> the CLOSE_WAIT state with varying recv-q values as so:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> tcp      2784    0 localhost:58786         localhost:5100
>>>>>>>>>>>>>>      CLOSE_WAIT
>>>>>>>>>>>>>> tcp      717      0 localhost:35794         localhost:5100
>>>>>>>>>>>>>>        CLOSE_WAIT
>>>>>>>>>>>>>> tcp      784      0 localhost:55712         localhost:5100
>>>>>>>>>>>>>>        CLOSE_WAIT
>>>>>>>>>>>>>> tcp        0        0 localhost:38639         localhost:5100
>>>>>>>>>>>>>>          CLOSE_WAIT
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That's just a snippet. A wc reveals over 400 of these on each
>>>>>>>>>>>>>> server.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Puma is running on port 5100 btw. We've tried puma with
>>>>>>>>>>>>>> multiple threads and a single one - same result. Latest version as of today.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've checked haproxy and don't see much lingering around.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Only a kill -9 can stop Puma - otherwise, it says something
>>>>>>>>>>>>>> like 'waiting for requests to finish'
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I ran GDB to see if I could debug the process however I can't
>>>>>>>>>>>>>> claim I knew what I was looking at. The processes that seemed apparent were
>>>>>>>>>>>>>> event machine and mongo.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We then ditched EM (we were using the AMQP gem) in favour of
>>>>>>>>>>>>>> Bunny. That made zero difference.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So we upgraded Mongo and Mongoid to the latest versions,
>>>>>>>>>>>>>> neither of which helped.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I thought we might have a bottleneck somewhere - Mongo, ES or
>>>>>>>>>>>>>> MySQL. But, none of those services seem to have any issues / latencies.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It's also 100% random. Might happen 10 times in an hour, then
>>>>>>>>>>>>>> not at all for a week.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The puma issues on github don't shed much light.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I don't really know where to turn at the moment or what to do
>>>>>>>>>>>>>> next? I was going to resort back to Unicorn but I don't think the issue is
>>>>>>>>>>>>>> that side and I wanted to fix the problem, not just patch it up.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It's starting to look like a nasty in my code somewhere but I
>>>>>>>>>>>>>> don't want to go down that route just yet...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sorry for the long email, thanks in advance. Stuff.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I hope someone can help!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> S
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Simon Morley
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Big Chief | PolkaSpots Supafly Wi-Fi
>>>>>>>>>>>>>> Bigger Chief | Cucumber Tony
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> simon at PolkaSpots.com <simon at polkaspots.com>
>>>>>>>>>>>>>> Linkedin: I'm on it again and it still sucks
>>>>>>>>>>>>>> 020 7183 1471 <020%207183%201471>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 🚀💥
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> Chat mailing list
>>>>>>>>>>>>>> Chat at lists.lrug.org
>>>>>>>>>>>>>> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
>>>>>>>>>>>>>> Manage your subscription:
>>>>>>>>>>>>>> http://lists.lrug.org/options.cgi/chat-lrug.org
>>>>>>>>>>>>>> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Chat mailing list
>>>>>>>>>>>>> Chat at lists.lrug.org
>>>>>>>>>>>>> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
>>>>>>>>>>>>> Manage your subscription:
>>>>>>>>>>>>> http://lists.lrug.org/options.cgi/chat-lrug.org
>>>>>>>>>>>>> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Chat mailing list
>>>>>>>>>>>>> Chat at lists.lrug.org
>>>>>>>>>>>>> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
>>>>>>>>>>>>> Manage your subscription:
>>>>>>>>>>>>> http://lists.lrug.org/options.cgi/chat-lrug.org
>>>>>>>>>>>>> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Chat mailing list
>>>>>>>>>>>> Chat at lists.lrug.org
>>>>>>>>>>>> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
>>>>>>>>>>>> Manage your subscription:
>>>>>>>>>>>> http://lists.lrug.org/options.cgi/chat-lrug.org
>>>>>>>>>>>> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Chat mailing list
>>>>>>>>>>> Chat at lists.lrug.org
>>>>>>>>>>> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
>>>>>>>>>>> Manage your subscription:
>>>>>>>>>>> http://lists.lrug.org/options.cgi/chat-lrug.org
>>>>>>>>>>> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Chat mailing list
>>>>>>>>>> Chat at lists.lrug.org
>>>>>>>>>> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
>>>>>>>>>> Manage your subscription:
>>>>>>>>>> http://lists.lrug.org/options.cgi/chat-lrug.org
>>>>>>>>>> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Chat mailing list
>>>>>>>>> Chat at lists.lrug.org
>>>>>>>>> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
>>>>>>>>> Manage your subscription:
>>>>>>>>> http://lists.lrug.org/options.cgi/chat-lrug.org
>>>>>>>>> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Chat mailing list
>>>>>>>> Chat at lists.lrug.org
>>>>>>>> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
>>>>>>>> Manage your subscription:
>>>>>>>> http://lists.lrug.org/options.cgi/chat-lrug.org
>>>>>>>> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>>>>>>>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Chat mailing list
>>>>>>> Chat at lists.lrug.org
>>>>>>> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
>>>>>>> Manage your subscription:
>>>>>>> http://lists.lrug.org/options.cgi/chat-lrug.org
>>>>>>> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Chat mailing list
>>>>>> Chat at lists.lrug.org
>>>>>> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
>>>>>> Manage your subscription:
>>>>>> http://lists.lrug.org/options.cgi/chat-lrug.org
>>>>>> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Riccardo Tacconi
>>>>>
>>>>> http://github.com/rtacconi
>>>>> http://twitter.com/rtacconi
>>>>>
>>>>> _______________________________________________
>>>>> Chat mailing list
>>>>> Chat at lists.lrug.org
>>>>> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
>>>>> Manage your subscription:
>>>>> http://lists.lrug.org/options.cgi/chat-lrug.org
>>>>> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Chat mailing list
>>>> Chat at lists.lrug.org
>>>> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
>>>> Manage your subscription:
>>>> http://lists.lrug.org/options.cgi/chat-lrug.org
>>>> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>>>
>>>>
>>>
>>>
>>> --
>>> Riccardo Tacconi
>>>
>>> http://github.com/rtacconi
>>> http://twitter.com/rtacconi
>>>
>>> _______________________________________________
>>> Chat mailing list
>>> Chat at lists.lrug.org
>>> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
>>> Manage your subscription:
>>> http://lists.lrug.org/options.cgi/chat-lrug.org
>>> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>>
>>>
>>
>> _______________________________________________
>> Chat mailing list
>> Chat at lists.lrug.org
>> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
>> Manage your subscription: http://lists.lrug.org/options.cgi/chat-lrug.org
>> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>
>>
>
>
> --
> Riccardo Tacconi
>
> http://github.com/rtacconi
> http://twitter.com/rtacconi
>
> _______________________________________________
> Chat mailing list
> Chat at lists.lrug.org
> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
> Manage your subscription: http://lists.lrug.org/options.cgi/chat-lrug.org
> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20160218/9bba4c39/attachment.html>

More information about the Chat mailing list