<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>Cheers. Il take a look. </div><div id="AppleMailSignature"><br></div><div id="AppleMailSignature">The problem still remains but were handling it... Can't wait to find the root. <br><br><div><div><span style="background-color:rgba(255,255,255,0)">Simon Morley</span></div><div><span style="background-color:rgba(255,255,255,0)">Big Chief | PolkaSpots Supafly Wi-Fi</span></div><div><span style="background-color:rgba(255,255,255,0)"><br></span></div><div><span style="background-color:rgba(255,255,255,0)"><br></span></div><div><span style="background-color:rgba(255,255,255,0)">Got an unlicensed Meraki? Set it free with Cucumber. </span></div><div><span style="background-color:rgba(255,255,255,0)"><br></span></div><div><span style="background-color:rgba(255,255,255,0)"><a href="http://cucumberwifi.io/meraki">cucumberwifi.io/meraki</a></span></div></div></div><div><br>On 16 Mar 2016, at 19:25, Riccardo Tacconi <<a href="mailto:rtacconi@gmail.com">rtacconi@gmail.com</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr">This book is quite informative and it is now available on Safari: <a href="https://pragprog.com/book/adrpo/ruby-performance-optimization">https://pragprog.com/book/adrpo/ruby-performance-optimization</a>. It has a chapter on <a href="https://github.com/ruby-prof/ruby-prof">https://github.com/ruby-prof/ruby-prof</a></div><div class="gmail_extra"><br><div class="gmail_quote">On 19 February 2016 at 11:25, Cam Allen <span dir="ltr"><<a href="mailto:campbell.allen@gmail.com" target="_blank">campbell.allen@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>I too have experienced stuck puma threads (via rails app in docker) when connecting to a postgresql db though. Turns out it was a networking error that we ended up fixing by using "net: host" for the rails app docker container.</div><div><br></div><div>I also tried to debug using gdb but had no luck debugging in the docker container as i could never get gdb to attach to the process correctly.</div><div><br></div><div>What version of docker and what docker networking strategy is the rails app container using?</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>Cam</div><div><br></div><div><br></div></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 19, 2016 at 10:52 AM, Simon Morley <span dir="ltr"><<a href="mailto:simon@polkaspots.com" target="_blank">simon@polkaspots.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">And there's this one, which I think the tb one is based off.<div><br></div><div><a href="http://www.rrn.dk/running-ruby-process-callstack/" target="_blank">http://www.rrn.dk/running-ruby-process-callstack/</a><br></div><div><br></div><div class="gmail_extra">I've tried using GDB but the servers in question are docker containers and seem to be missing something. I'm just left with the error:</div><div class="gmail_extra"><br></div><div class="gmail_extra">warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.</div><div class="gmail_extra"><br></div><div class="gmail_extra">Which I couldn't solve with Googling. Tried adding many suggestions to .gdbinit unsuccessfully and put down to docker.I had more success with a non-docker vm but did not 100% understand what was happening. If you do a talk, I'll be there for sure. Plus, the online material is really lacking. </div><div class="gmail_extra"><div><div><div dir="ltr"><div dir="ltr"><div><br></div><div>The changes we made yesterday (adjusting os limits and adding puma config) didn't sadly work. It's like the reconnect isn't happening if the servers have been idle for a while...</div><div><br></div><div>If there's anyone using multiple dbs with Rails, what versions of the mysql2 gem are you using. Down to my last ideas now.</div><span><font color="#888888"><div><br></div><div>S</div><div><br></div><div><div><br></div></div></font></span></div></div></div></div><div><div><div class="gmail_quote">On 19 February 2016 at 10:21, Ben Lovell <span dir="ltr"><<a href="mailto:benjamin.lovell@gmail.com" target="_blank">benjamin.lovell@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
On Feb 19 2016, at 9:20 am, Jon Wood <<a href="mailto:jon@ninjagiraffes.co.uk" target="_blank">jon@ninjagiraffes.co.uk</a>> wrote:
<br>
<p dir="ltr">Is there some documentation on using rb_backtrace and gdb for this? We occasionally see similar issues and it would be great to know how to properly debug them rather than the gut feel approach we've been using so far.</p></blockquote><div><br></div></span><div>Some quick googling threw up both [0] and [1] which at first glance seem like good introductory texts. Perhaps I should give a talk on these things...</div><div><br></div><div>[0] <a href="https://blog.newrelic.com/2013/04/29/debugging-stuck-ruby-processes-what-to-do-before-you-kill-9/" target="_blank">https://blog.newrelic.com/2013/04/29/debugging-stuck-ruby-processes-what-to-do-before-you-kill-9/</a></div><div>[1] <a href="https://robots.thoughtbot.com/using-gdb-to-inspect-a-running-ruby-process" target="_blank">https://robots.thoughtbot.com/using-gdb-to-inspect-a-running-ruby-process</a></div><div><div><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br><div><div dir="ltr">On Fri, 19 Feb 2016 08:52 Ben Lovell <<a href="mailto:benjamin.lovell@gmail.com" target="_blank">benjamin.lovell@gmail.com</a>> wrote:<br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="auto"><div><br></div><div><div><br></div><div><div><br></div><br>Sent from my iPhone</div></div></div><div dir="auto"><div>On 19 Feb 2016, at 08:33, Riccardo Tacconi <<a href="mailto:rtacconi@gmail.com" target="_blank">rtacconi@gmail.com</a>> wrote:<br><br></div></div><div dir="auto"><blockquote type="cite"><div><div dir="ltr">Yes there are, you move from blocking to non-blocking, but with JRuby you have parallel processing, so why sticking with MRI?</div><div><br></div></div></blockquote><div><br></div></div><div dir="auto"><div>I'm one of JRuby's greatest fans and would always recommend it, but you're saying this like it's a trivial change. It isn't. This is also probably *the* worst time to be chucking semi-random tech into a stack for some kind of trial-and-error approach. </div><div><br></div><div>I'd recommend you attach a gdb to one of the stuck processes and rb_backtrace() your way out. Once attached you can (depending on the state of the stuck process) also execute arbitrary Ruby code to help with you investigations by using ruby_eval(...).</div><div><br></div><div>There should be plenty written about this, I'd recommend some googling. Otherwise get in touch, I'll happily help you out (for a fee)</div><div><br></div><div>Good luck,</div><div>Ben</div></div><div dir="auto"><div><br></div><br><blockquote type="cite"><div><div><div>On 18 February 2016 at 23:36, Glenn @ Ruby Pond Ltd <span dir="ltr"><<a href="mailto:glenn@rubypond.com" target="_blank">glenn@rubypond.com</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">There's still a lot of benefit to using Puma, even if you're on MRI. Admittedly not as much benefit as using it with JRuby.<div><br></div><div>This comment from when Heroku recommended customer switch to Puma goes through a basic example: <a href="https://www.reddit.com/r/ruby/comments/2vjoxe/puma_is_now_the_recommended_ruby_webserver_on/coiypgp" target="_blank">https://www.reddit.com/r/ruby/comments/2vjoxe/puma_is_now_the_recommended_ruby_webserver_on/coiypgp</a></div></div><div><div><div><div><br></div><div>On 18 February 2016 at 23:43, Riccardo Tacconi <span dir="ltr"><<a href="mailto:rtacconi@gmail.com" target="_blank">rtacconi@gmail.com</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Ruby MRI? If yes what's the point of using Puma? With MRI you have one worker and one thread, which is very inefficient. Would be possible to split HTTP requests handling from querying the DB? From Puma you could send requests to a topic (MOM), and multiple workers could process requests and each worker will have a DB connection. This could work with MRI, although you will need more RAM. However I would try rubinius of jruby first.<div><br></div><div>Sorry if I misunderstood, I did not follow the whole thread.<div><div><div><br></div><div><div><br></div><div>On 18 February 2016 at 12:28, Simon Morley <span dir="ltr"><<a href="mailto:simon@polkaspots.com" target="_blank">simon@polkaspots.com</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Ruby 2.2.2 <div>Rails 4.2.5.1</div><div>mysql2 0.4.2 (tried a few)</div><div>Puma 2.16.0</div></div><div><span><div><br></div><div><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><br></div><div><br></div><div><div>Simon Morley<br><span style="font-family:arial;font-size:small"><br></span></div><div><span style="font-family:arial;font-size:small">Big Chief</span> | PolkaSpots Supafly Wi-Fi<div>Bigger Chief | Cucumber Tony</div><div><div><br></div><div>Got an unlicensed Meraki? Set it free with Cucumber</div></div></div></div><div><a href="http://cucumberwifi.io/meraki" target="_blank">cucumberwifi.io/meraki</a></div><div><br></div></div></div></div></div></div></div></div>
<br></span><div><div><div>On 18 February 2016 at 12:24, Riccardo Tacconi <span dir="ltr"><<a href="mailto:rtacconi@gmail.com" target="_blank">rtacconi@gmail.com</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Which version of Ruby are you using?</div><div><div><div><div><br></div><div>On 18 February 2016 at 12:17, Simon Morley <span dir="ltr"><<a href="mailto:simon@polkaspots.com" target="_blank">simon@polkaspots.com</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Actually puma docs suggest doing that when using preload_app and ActiveRecord...<div><br></div><div><a href="https://github.com/puma/puma#clustered-mode" target="_blank">https://github.com/puma/puma#clustered-mode</a><br></div></div><div><div><br></div><div><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><br></div><div><br></div><div><div>Simon Morley<br><span style="font-family:arial;font-size:small"><br></span></div><div><span><span style="font-family:arial;font-size:small">Big Chief</span> | PolkaSpots Supafly Wi-Fi<div>Bigger Chief | Cucumber Tony</div></span><div><div><br></div><span><div>Got an unlicensed Meraki? Set it free with Cucumber</div></span></div></div></div><div><a href="http://cucumberwifi.io/meraki" target="_blank">cucumberwifi.io/meraki</a></div><div><br></div></div></div></div></div></div></div></div>
<br><div><span>On 18 February 2016 at 12:05, Frederick Cheung <span dir="ltr"><<a href="mailto:frederick.cheung@gmail.com" target="_blank">frederick.cheung@gmail.com</a>></span> wrote:<br></span><div><div><blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><span><div style="font-family:Helvetica,Arial;font-size:13px;color:rgb(0,0,0);margin:0px"><div><br></div><br></div> <br><p>On 18 February 2016 at 11:17:34, Simon Morley (<a href="mailto:simon@polkaspots.com" target="_blank">simon@polkaspots.com</a>) wrote:</p> <blockquote type="cite"><span><div><div></div><div>
<div dir="ltr">
<div><br></div><div><div><div><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr">
<div>class RadiusDatabase </div>
<div>
<div> self.abstract_class = true</div>
<div> establish_connection "radius_#{Rails.env}".to_sym</div>
</div>
<div>end</div>
<div><br></div>
<div>class Radacct < RadiusDatabase</div>
<div>end</div>
<div><br></div>
<div>Then I decreased our database pool from 20 to 5 and added a
wait_timeout of 5 (since there seems to be some discrepancies with
this). Things got much better (but weren't fixed).</div>
<div><br></div>
<div>I tried querying differently, including
using <span style="color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;font-size:12px;line-height:22.4px;white-space:pre-wrap">connection_pool.with_connection.</span>
I've tried closing the connections manually and also used
ActiveRecord::Base.clear_active_connections! periodically. No
joy.</div>
<div><br></div>
<div>By this point, we were running 2-4 instances - handling around
very little traffic in total (about 50rpm). Every few hours, they'd
block, all of them. At the same time, we'd see a load of rack
timeouts - same DB. I've checked the connections - they were each
opening only a few to MySQL and MySQL was looking good.</div>
<div><br></div>
<div>One day, by chance, I reduced the 4 instances to 1. <b>And the
problem is solved!!! WHAT</b>? Obviously the problem isn't solved,
we can only use a single server. </div>
<div><br></div></div></div></div></div></div></div></div></div></div></div></div></span></blockquote><div><br></div></span>Are you using puma in the mode where it forks workers? if so, then you want to reconnect post fork or multiple processes will share the same file descriptor and really weird shit will happen.<div><br></div><div>The puma readme advises to do this:</div><div><br></div><div><div>before_fork do</div><div> ActiveRecord::Base.connection_pool.disconnect!</div><div>end</div><div><br></div><div>I don't know off the top of my head whether that will do the job for classes that have established a connection to a different db - presumably they have a separate connection pool</div><div><br></div><div>Fred</div><div><div><div><div><br></div><blockquote type="cite"><span><div><div><div dir="ltr"><div><div><div><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr">
<div>I don't know what's going on here. Have I been staring at this
for too long (yes)?</div>
<div><br></div>
<div>
<div>Our other servers are chugging along happily now, using a
connection pool of 20, no errors, no timeouts (different db
though).</div>
</div>
<div><br></div>
<div>Has anyone got any suggestions / seen this? Is there something
fundamentally wrong with the way we're establishing a connection to
the external dbs? Surely this is MySQL related</div>
<div><br></div>
<div>Thanks for listening, </div>
<div><br></div>
<div>S</div>
<div><br></div>
<div><br></div>
<div>
<div>Simon Morley<br>
<span style="font-family:arial;font-size:small"><br></span></div>
<div>
<div>
<div>Got an unlicensed Meraki? Set it free with Cucumber</div>
</div>
</div>
</div>
<div><a href="http://cucumberwifi.io/meraki" target="_blank">cucumberwifi.io/meraki</a></div>
<div><br></div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div>On 15 January 2016 at 13:58, Gerhard Lazu
<span dir="ltr"><<a href="mailto:gerhard@lazu.co.uk" target="_blank">gerhard@lazu.co.uk</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
The understanding of difficult problems/bugs and the learning that
comes with it cannot be rushed. Each and every one of us has his /
her own pace, and all "speeds" are perfectly fine. The only
question that really matters is whether it's worth it (a.k.a. the
cost of lost opportunity). If the answer is yes, plough on. If not,
look for alternatives.</div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
Not everyone likes or wants to run their own infrastructure. The
monthly savings on the PaaS, IaaS advertised costs are undisputed,
but few like to think - never mind talk - about how many hours /
days / weeks have been spent debugging obscure problems which
"solve themselves" on a managed environment. Don't get me started
on those that are building their own Docker-based PaaS-es without
even realising it...</div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
As a side-note, I've been dealing with a similar TCP-related
problem for a while now, so I could empathise with your struggles
the second I've seen your post. One of us is bound to solve it
first, and I hope it will be you ; )</div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
Have a good one, Gerhard.</div>
<div>
<div>
<div><div><br></div>
<div>On Fri, Jan 15, 2016 at 10:01 AM, Simon
Morley <span dir="ltr"><<a href="mailto:simon@polkaspots.com" target="_blank">simon@polkaspots.com</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">You must be more patient that I am. It's been a long
month - having said that, I'm excited to find the cause.
<div><br></div>
<div>I misunderstood you re. file descriptors. We checked the
kernel limits / files open on the systems before and during and
there's nothing untoward. </div>
<div><br></div>
<div>Since writing in, it's not happened as before - no doubt it'll
take place during our forthcoming office move today.</div>
<div><br></div>
<div>I ran a strace (thanks for that suggestion John) on a couple
of processes yesterday and saw redis blocking. Restarted a few
redis servers to see if that helped. Can't be certain yet.</div>
<div><br></div>
<div>As soon as it's on, I'll run a tcpdump. How I'd not thought
about that I don't know...</div>
<div><br></div>
<div>Actually, this is one thing I dislike about Rails - it's so
nice and easy to do everything, one forgets we're dealing with the
real servers / components / connections. It's too abstract in ways,
but that's a whole other debate :)</div>
<div><span><br></span></div>
<div><span>S</span></div>
</div>
<div><span><br></span>
<div>
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div><span><br></span></div>
<div><span><br></span></div>
<div>
<div><span>Simon Morley<br>
<span style="font-family:arial;font-size:small"><div><br></div>
</span></span></div>
<div><span style="font-family:arial;font-size:small">Big
Chief</span> | PolkaSpots Supafly Wi-Fi
<div>Bigger Chief | Cucumber Tony</div>
<div><div><br></div>
<div><span style="font-size:small;font-family:arial"><a href="mailto:simon@polkaspots.com" target="_blank">simon@PolkaSpots.com</a></span><br style="font-size:small;font-family:arial">
<div style="font-size:small;font-family:arial">Linkedin: I'm on it
again and it still sucks</div>
<div style="font-size:small;font-family:arial"><a value="+442071831471">020
7183 1471</a></div>
</div>
</div>
</div>
</div>
<div style="font-size:small;font-family:arial"><br></div>
<div>🚀💥<br></div>
</div>
</div>
</div>
</div>
</div>
<br>
<div>
<div>
<div>On 15 January 2016 at 06:53, Gerhard Lazu
<span dir="ltr"><<a href="mailto:gerhard@lazu.co.uk" target="_blank">gerhard@lazu.co.uk</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
File descriptors, for traditional reasons, include TCP
connections.<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
Are you logging all requests to a central location? When the
problem occurs, it might help taking a closer look at the type of
requests you're receiving.</div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
Depending on how long the mischief lasts, a tcpdump to pcap, then
wireshark might help. Same for an strace on the Puma processes,
similar to what John suggested . Those are low level tools though,
verbose, complex and complete, it's easy to get lost unless you
know what you're looking for.</div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
In summary, CLOSE_WAITs piling up from haproxy (client role) to
Puma (server role) indicates the app not closing connections in
time (or maybe ever) - why? It's a fun one to troubleshoot ;
)</div>
</div>
<div>
<div>
<div><div><br></div>
<div>On Thu, Jan 14, 2016 at 11:35 PM, Simon
Morley <span dir="ltr"><<a href="mailto:simon@polkaspots.com" target="_blank">simon@polkaspots.com</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">Right now, none of the servers have any issues. No
close_waits.
<div><div><br></div>
All is well. Seemingly.</div>
<div><br></div>
<div>When it occurs ALL the servers end up going. Sometimes real
fast. That's why I thought we had a db bottleneck. It happens
pretty quickly, randomly, no particular times.</div>
<div><br></div>
<div>We don't ever really get spikes of traffic, there's an even
load inbound throughout.</div>
<div><br></div>
<div>I thought we had someone running a slow loris style attack on
us. So I added some rules to HA Proxy and Cloudflare ain't seen
nofin honest guv.</div>
<div><br></div>
<div>Will find a way to chart it and send a link over.</div>
<div><br></div>
<div>Will see if we're not closing any files - not much of that
going on. There's some manual gzipping happening - we've had that
in place for over a year though - not sure why it'd start playing
up now. Memory usage is high but consistent and doesn't
increase.</div>
<div><span><br></span></div>
<div><span>S</span></div>
</div>
<div><span><br></span>
<div>
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div><span><br></span></div>
<div><span><br></span></div>
<div>
<div><span>Simon Morley<br>
<span style="font-family:arial;font-size:small"><div><br></div>
</span></span></div>
<div><span style="font-family:arial;font-size:small">Big
Chief</span> | PolkaSpots Supafly Wi-Fi
<div>Bigger Chief | Cucumber Tony</div>
<div><div><br></div>
<div><span style="font-size:small;font-family:arial"><a href="mailto:simon@polkaspots.com" target="_blank">simon@PolkaSpots.com</a></span><br style="font-size:small;font-family:arial">
<div style="font-size:small;font-family:arial">Linkedin: I'm on it
again and it still sucks</div>
<div style="font-size:small;font-family:arial"><a value="+442071831471">020
7183 1471</a></div>
</div>
</div>
</div>
</div>
<div style="font-size:small;font-family:arial"><br></div>
<div>🚀💥<br></div>
</div>
</div>
</div>
</div>
</div>
<br>
<div>
<div>
<div>On 14 January 2016 at 22:14, Gerhard Lazu
<span dir="ltr"><<a href="mailto:gerhard@lazu.co.uk" target="_blank">gerhard@lazu.co.uk</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
That sounds like a file descriptor leak. Are the CLOSE_WAITs
growing over time?</div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
You're right, New Relic is too high level, this is a layer 4-5
issue.</div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
The simplest thing that can plot some graphs will work. Throw the
dirtiest script together that curls the data out if it comes easy,
it doesn't matter how you get those metrics as long as you have
them.</div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
This is a great blog post opportunity ; )</div>
</div>
<div>
<div>
<div><div><br></div>
<div>On Thu, Jan 14, 2016 at 8:40 PM, Simon
Morley <span dir="ltr"><<a href="mailto:simon@polkaspots.com" target="_blank">simon@polkaspots.com</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="auto">
<div>I would ordinarily agree with you about the connection however
they hang around for hours sometimes. </div>
<div><br></div>
<div>The 500 in the hyproxy config was actually left over from a
previous experiment. Realistically I know they won't cope with
that.</div>
<div><br></div>
<div>Using another server was to find any issues with puma. I'm
still going to try unicorn just in case. <br>
<br>
Will up the numbers too - thanks for that suggestion. </div>
<div><br></div>
<div>I'll look at a better monitoring tool too. So far new relic
hasn't helped much. </div>
<div><br></div>
<div>Thanks</div>
<div><br></div>
<div>S</div>
<div><span><br></span>
<div style="font-family:Helvetica;font-size:medium"><span>Simon
Morley</span></div>
<div style="font-family:Helvetica;font-size:medium"><span>Big Chief
| PolkaSpots Supafly Wi-Fi</span></div>
<div style="font-family:Helvetica;font-size:medium">
<span><br></span></div>
<div style="font-family:Helvetica;font-size:medium">
<span><br></span></div>
<div style="font-family:Helvetica;font-size:medium">
<span><br></span></div>
<div style="font-family:Helvetica;font-size:medium">I'm doing it
with Cucumber Tony. Are you?</div>
</div>
<div>
<div>
<div><div><br></div>
On 14 Jan 2016, at 20:30, Gerhard Lazu <<a href="mailto:gerhard@lazu.co.uk" target="_blank">gerhard@lazu.co.uk</a>> wrote:<br>
<br></div>
<blockquote type="cite">
<div>
<div dir="ltr">
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
Hi Simon,</div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
CLOSE_WAIT suggests that Puma is not closing connections fast
enough. The client has asked for the connection to be closed,
but Puma is busy.</div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
Quickest win would be to increase your Puma instances. Unicorn
won't help - or any other Rack web server for the matter.</div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
Based on your numbers, start with 10 Puma instances. Anything more
than 100 connections for a Rails instance is not realistic. I would
personally go with 50, just to be safe. I think I saw 500
conns in your haproxy config, which is way
too optimistic.</div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
You want metrics
for detailed CPU usage by process, connections open with state
by process, and memory usage, by process. Without these, you're
flying blind. Any suggestions anyone makes without real
metrics - including myself - are just guesses. You'll get
there, but you're making it far too difficult for
yourself.</div>
<div dir="ltr"><br></div>
<div dir="ltr">Let me know how it goes, Gerhard.</div>
<div><div><br></div>
<div>On Thu, Jan 14, 2016 at 3:16 PM, Simon
Morley <span dir="ltr"><<a>simon@polkaspots.com</a>></span>
wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">Hello All
<div><br></div>
<div>We've been battling with Puma for a long while now, I'm
looking for some help / love / attention / advice / anything to
prevent further hair loss.</div>
<div><br></div>
<div>We're using it in a reasonably typical Rails 4 application
behind Nginx. <br>
<div>
<div>
<div dir="ltr">
<div dir="ltr">
<div><br></div>
<div>Over the last 3 months, our requests have gone from 500 rpm to
a little over 1000 depending on the hour. Over this period, we've
been seeing weird CLOSE_WAIT conns appearing in netstat, which
eventually kill the servers.</div>
<div><br></div>
<div>We have 3 Rails servers behind Haproxy running things. Load is
generally even.</div>
<div><br></div>
<div>Running netstat on the servers shows a pile of connections in
the CLOSE_WAIT state with varying recv-q values as so:<br></div>
<div><br></div>
<div>
<div>tcp 2784 0 localhost:58786
localhost:5100
CLOSE_WAIT</div>
<div>tcp 717 0
localhost:35794 localhost:5100
CLOSE_WAIT</div>
<div>tcp 784 0
localhost:55712 localhost:5100
CLOSE_WAIT<br></div>
<div>tcp 0 0
localhost:38639 localhost:5100
CLOSE_WAIT</div>
</div>
<div><br></div>
<div>
<div>That's just a snippet. A wc reveals over 400 of these on each
server.</div>
</div>
<div><br></div>
<div>Puma is running on port 5100 btw. We've tried puma with
multiple threads and a single one - same result. Latest version as
of today.</div>
<div><br></div>
<div>I've checked haproxy and don't see much lingering
around.</div>
<div><br></div>
<div>Only a kill -9 can stop Puma - otherwise, it says something
like 'waiting for requests to finish'</div>
<div><br></div>
<div>I ran GDB to see if I could debug the process however I can't
claim I knew what I was looking at. The processes that seemed
apparent were event machine and mongo.</div>
<div><br></div>
<div>We then ditched EM (we were using the AMQP gem) in favour of
Bunny. That made zero difference.</div>
<div><br></div>
<div>So we upgraded Mongo and Mongoid to the latest versions,
neither of which helped.</div>
<div><br></div>
<div>I thought we might have a bottleneck somewhere - Mongo, ES or
MySQL. But, none of those services seem to have any issues /
latencies.</div>
<div><br></div>
<div>It's also 100% random. Might happen 10 times in an hour, then
not at all for a week.</div>
<div><br></div>
<div>The puma issues on github don't shed much light.</div>
<div><br></div>
<div>I don't really know where to turn at the moment or what to do
next? I was going to resort back to Unicorn but I don't think the
issue is that side and I wanted to fix the problem, not just patch
it up.</div>
<div><br></div>
<div>It's starting to look like a nasty in my code somewhere but I
don't want to go down that route just yet...</div>
<div><br></div>
<div>Sorry for the long email, thanks in advance. Stuff.</div>
<div><br></div>
<div>I hope someone can help!</div>
<div><br></div>
<div>S</div>
<div><br></div>
<div>
<div>Simon Morley<br>
<span style="font-family:arial;font-size:small"><br></span></div>
<div><span style="font-family:arial;font-size:small">Big
Chief</span> | PolkaSpots Supafly Wi-Fi
<div>Bigger Chief | Cucumber Tony</div>
<div><div><br></div>
<div><span style="font-size:small;font-family:arial"><a href="mailto:simon@polkaspots.com" target="_blank">simon@PolkaSpots.com</a></span><br style="font-size:small;font-family:arial">
<div style="font-size:small;font-family:arial">Linkedin: I'm on it
again and it still sucks</div>
<div style="font-size:small;font-family:arial"><a value="+442071831471">020
7183 1471</a></div>
</div>
</div>
</div>
</div>
<div style="font-size:small;font-family:arial"><br></div>
<div>🚀💥<br></div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
_______________________________________________<br>
Chat mailing list<br>
<a>Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote>
</div>
<br></div>
</div>
</blockquote>
<blockquote type="cite">
<div>
<span>_______________________________________________</span><br>
<span>Chat mailing list</span><br>
<span><a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a></span><br>
<span>Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a></span><br>
<span>Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a></span><br>
<span>List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a></span><br>
</div>
</blockquote>
</div>
</div>
</div>
<br>
_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote>
</div>
<br></div>
</div>
</div>
<br>
_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote>
</div>
<br></div>
</div>
</div>
<br>
_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote>
</div>
<br></div>
</div>
</div>
<br>
_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote>
</div>
<br></div>
</div>
</div>
<br>
_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote>
</div>
<br></div>
</div>
</div>
</div>
<br>
_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote>
</div>
<br></div>
</div>
</div>
_______________________________________________
<br>Chat mailing list
<br><a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a>
<br>Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a>
<br>Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a>
<br>List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a>
<br></div></div></span></blockquote></div></div></div></div></div></blockquote></div></div></div><br></div>
<br>_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote></div><br><br><div><br></div></div></div><span>-- <br><div><div dir="ltr">Riccardo Tacconi<br><br><a href="http://github.com/rtacconi" target="_blank">http://github.com/rtacconi</a><br><a href="http://twitter.com/rtacconi" target="_blank">http://twitter.com/rtacconi</a></div></div>
</span></div>
<br>_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote></div><br></div></div></div>
<br>_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote></div><br><br><div><br></div>-- <br><div><div dir="ltr">Riccardo Tacconi<br><br><a href="http://github.com/rtacconi" target="_blank">http://github.com/rtacconi</a><br><a href="http://twitter.com/rtacconi" target="_blank">http://twitter.com/rtacconi</a></div></div>
</div></div></div></div></div>
<br>_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote></div><br><br><div><br></div>-- <br><div><div dir="ltr">Riccardo Tacconi<br><br><a href="http://github.com/rtacconi" target="_blank">http://github.com/rtacconi</a><br><a href="http://twitter.com/rtacconi" target="_blank">http://twitter.com/rtacconi</a></div></div>
</div>
</div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>Chat mailing list</span><br><span><a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a></span><br><span>Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a></span><br><span>Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a></span><br><span>List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a></span><br></div></blockquote></div>_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
</blockquote></div>
</blockquote></div></div><br>_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote></div><br></div></div></div></div><div class="gmail_extra"><div><div><br clear="all"><div><div><div dir="ltr"><div><div dir="ltr"><div><br></div><div><br></div><div><div>Simon Morley<br><span style="font-family:arial;font-size:small"><br></span></div><div><span style="font-family:arial;font-size:small">Big Chief</span> | PolkaSpots Supafly Wi-Fi<div>Bigger Chief | Cucumber Tony</div><div><br><div><span style="font-size:small;font-family:arial"><a href="mailto:simon@polkaspots.com">simon@PolkaSpots.com</a></span><br style="font-size:small;font-family:arial"><div style="font-size:small;font-family:arial">Linkedin: I'm on it again and it still sucks</div><div style="font-size:small;font-family:arial"><a href="tel:020%207183%201471" value="+442071831471" target="_blank">020 7183 1471</a></div></div></div></div></div><div style="font-size:small;font-family:arial"><br></div><div><font face="arial" size="2">🚀💥</font><br></div></div></div></div></div></div>
<br></div></div><div><div><div class="gmail_quote">On 19 February 2016 at 10:21, Ben Lovell <span dir="ltr"><<a href="mailto:benjamin.lovell@gmail.com" target="_blank">benjamin.lovell@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Feb 19 2016, at 9:20 am, Jon Wood <<a href="mailto:jon@ninjagiraffes.co.uk" target="_blank">jon@ninjagiraffes.co.uk</a>> wrote:
<br>
<p dir="ltr">Is there some documentation on using rb_backtrace and gdb for this? We occasionally see similar issues and it would be great to know how to properly debug them rather than the gut feel approach we've been using so far.</p></blockquote><div><br></div></span><div>Some quick googling threw up both [0] and [1] which at first glance seem like good introductory texts. Perhaps I should give a talk on these things...</div><div><br></div><div>[0] <a href="https://blog.newrelic.com/2013/04/29/debugging-stuck-ruby-processes-what-to-do-before-you-kill-9/" target="_blank">https://blog.newrelic.com/2013/04/29/debugging-stuck-ruby-processes-what-to-do-before-you-kill-9/</a></div><div>[1] <a href="https://robots.thoughtbot.com/using-gdb-to-inspect-a-running-ruby-process" target="_blank">https://robots.thoughtbot.com/using-gdb-to-inspect-a-running-ruby-process</a></div><div><div><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br><div><div dir="ltr">On Fri, 19 Feb 2016 08:52 Ben Lovell <<a href="mailto:benjamin.lovell@gmail.com" target="_blank">benjamin.lovell@gmail.com</a>> wrote:<br></div><blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div><br></div><div><div><br></div><div><div><br></div><br>Sent from my iPhone</div></div></div><div dir="auto"><div>On 19 Feb 2016, at 08:33, Riccardo Tacconi <<a href="mailto:rtacconi@gmail.com" target="_blank">rtacconi@gmail.com</a>> wrote:<br><br></div></div><div dir="auto"><blockquote type="cite"><div><div dir="ltr">Yes there are, you move from blocking to non-blocking, but with JRuby you have parallel processing, so why sticking with MRI?</div><div><br></div></div></blockquote><div><br></div></div><div dir="auto"><div>I'm one of JRuby's greatest fans and would always recommend it, but you're saying this like it's a trivial change. It isn't. This is also probably *the* worst time to be chucking semi-random tech into a stack for some kind of trial-and-error approach. </div><div><br></div><div>I'd recommend you attach a gdb to one of the stuck processes and rb_backtrace() your way out. Once attached you can (depending on the state of the stuck process) also execute arbitrary Ruby code to help with you investigations by using ruby_eval(...).</div><div><br></div><div>There should be plenty written about this, I'd recommend some googling. Otherwise get in touch, I'll happily help you out (for a fee)</div><div><br></div><div>Good luck,</div><div>Ben</div></div><div dir="auto"><div><br></div><br><blockquote type="cite"><div><div><div>On 18 February 2016 at 23:36, Glenn @ Ruby Pond Ltd <span dir="ltr"><<a href="mailto:glenn@rubypond.com" target="_blank">glenn@rubypond.com</a>></span> wrote:<br><blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">There's still a lot of benefit to using Puma, even if you're on MRI. Admittedly not as much benefit as using it with JRuby.<div><br></div><div>This comment from when Heroku recommended customer switch to Puma goes through a basic example: <a href="https://www.reddit.com/r/ruby/comments/2vjoxe/puma_is_now_the_recommended_ruby_webserver_on/coiypgp" target="_blank">https://www.reddit.com/r/ruby/comments/2vjoxe/puma_is_now_the_recommended_ruby_webserver_on/coiypgp</a></div></div><div><div><div><div><br></div><div>On 18 February 2016 at 23:43, Riccardo Tacconi <span dir="ltr"><<a href="mailto:rtacconi@gmail.com" target="_blank">rtacconi@gmail.com</a>></span> wrote:<br><blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Ruby MRI? If yes what's the point of using Puma? With MRI you have one worker and one thread, which is very inefficient. Would be possible to split HTTP requests handling from querying the DB? From Puma you could send requests to a topic (MOM), and multiple workers could process requests and each worker will have a DB connection. This could work with MRI, although you will need more RAM. However I would try rubinius of jruby first.<div><br></div><div>Sorry if I misunderstood, I did not follow the whole thread.<div><div><div><br></div><div><div><br></div><div>On 18 February 2016 at 12:28, Simon Morley <span dir="ltr"><<a href="mailto:simon@polkaspots.com" target="_blank">simon@polkaspots.com</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Ruby 2.2.2 <div>Rails 4.2.5.1</div><div>mysql2 0.4.2 (tried a few)</div><div>Puma 2.16.0</div></div><div><span><div><br></div><div><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><br></div><div><br></div><div><div>Simon Morley<br><span style="font-family:arial;font-size:small"><br></span></div><div><span style="font-family:arial;font-size:small">Big Chief</span> | PolkaSpots Supafly Wi-Fi<div>Bigger Chief | Cucumber Tony</div><div><div><br></div><div>Got an unlicensed Meraki? Set it free with Cucumber</div></div></div></div><div><a href="http://cucumberwifi.io/meraki" target="_blank">cucumberwifi.io/meraki</a></div><div><br></div></div></div></div></div></div></div></div>
<br></span><div><div><div>On 18 February 2016 at 12:24, Riccardo Tacconi <span dir="ltr"><<a href="mailto:rtacconi@gmail.com" target="_blank">rtacconi@gmail.com</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Which version of Ruby are you using?</div><div><div><div><div><br></div><div>On 18 February 2016 at 12:17, Simon Morley <span dir="ltr"><<a href="mailto:simon@polkaspots.com" target="_blank">simon@polkaspots.com</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Actually puma docs suggest doing that when using preload_app and ActiveRecord...<div><br></div><div><a href="https://github.com/puma/puma#clustered-mode" target="_blank">https://github.com/puma/puma#clustered-mode</a><br></div></div><div><div><br></div><div><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><br></div><div><br></div><div><div>Simon Morley<br><span style="font-family:arial;font-size:small"><br></span></div><div><span><span style="font-family:arial;font-size:small">Big Chief</span> | PolkaSpots Supafly Wi-Fi<div>Bigger Chief | Cucumber Tony</div></span><div><div><br></div><span><div>Got an unlicensed Meraki? Set it free with Cucumber</div></span></div></div></div><div><a href="http://cucumberwifi.io/meraki" target="_blank">cucumberwifi.io/meraki</a></div><div><br></div></div></div></div></div></div></div></div>
<br><div><span>On 18 February 2016 at 12:05, Frederick Cheung <span dir="ltr"><<a href="mailto:frederick.cheung@gmail.com" target="_blank">frederick.cheung@gmail.com</a>></span> wrote:<br></span><div><div><blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><span><div style="font-family:Helvetica,Arial;font-size:13px;color:rgb(0,0,0);margin:0px"><div><br></div><br></div> <br><p>On 18 February 2016 at 11:17:34, Simon Morley (<a href="mailto:simon@polkaspots.com" target="_blank">simon@polkaspots.com</a>) wrote:</p> <blockquote type="cite"><span><div><div></div><div>
<div dir="ltr">
<div><br></div><div><div><div><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr">
<div>class RadiusDatabase </div>
<div>
<div> self.abstract_class = true</div>
<div> establish_connection "radius_#{Rails.env}".to_sym</div>
</div>
<div>end</div>
<div><br></div>
<div>class Radacct < RadiusDatabase</div>
<div>end</div>
<div><br></div>
<div>Then I decreased our database pool from 20 to 5 and added a
wait_timeout of 5 (since there seems to be some discrepancies with
this). Things got much better (but weren't fixed).</div>
<div><br></div>
<div>I tried querying differently, including
using <span style="color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;font-size:12px;line-height:22.4px;white-space:pre-wrap">connection_pool.with_connection.</span>
I've tried closing the connections manually and also used
ActiveRecord::Base.clear_active_connections! periodically. No
joy.</div>
<div><br></div>
<div>By this point, we were running 2-4 instances - handling around
very little traffic in total (about 50rpm). Every few hours, they'd
block, all of them. At the same time, we'd see a load of rack
timeouts - same DB. I've checked the connections - they were each
opening only a few to MySQL and MySQL was looking good.</div>
<div><br></div>
<div>One day, by chance, I reduced the 4 instances to 1. <b>And the
problem is solved!!! WHAT</b>? Obviously the problem isn't solved,
we can only use a single server. </div>
<div><br></div></div></div></div></div></div></div></div></div></div></div></div></span></blockquote><div><br></div></span>Are you using puma in the mode where it forks workers? if so, then you want to reconnect post fork or multiple processes will share the same file descriptor and really weird shit will happen.<div><br></div><div>The puma readme advises to do this:</div><div><br></div><div><div>before_fork do</div><div> ActiveRecord::Base.connection_pool.disconnect!</div><div>end</div><div><br></div><div>I don't know off the top of my head whether that will do the job for classes that have established a connection to a different db - presumably they have a separate connection pool</div><div><br></div><div>Fred</div><div><div><div><div><br></div><blockquote type="cite"><span><div><div><div dir="ltr"><div><div><div><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr">
<div>I don't know what's going on here. Have I been staring at this
for too long (yes)?</div>
<div><br></div>
<div>
<div>Our other servers are chugging along happily now, using a
connection pool of 20, no errors, no timeouts (different db
though).</div>
</div>
<div><br></div>
<div>Has anyone got any suggestions / seen this? Is there something
fundamentally wrong with the way we're establishing a connection to
the external dbs? Surely this is MySQL related</div>
<div><br></div>
<div>Thanks for listening, </div>
<div><br></div>
<div>S</div>
<div><br></div>
<div><br></div>
<div>
<div>Simon Morley<br>
<span style="font-family:arial;font-size:small"><br></span></div>
<div>
<div>
<div>Got an unlicensed Meraki? Set it free with Cucumber</div>
</div>
</div>
</div>
<div><a href="http://cucumberwifi.io/meraki" target="_blank">cucumberwifi.io/meraki</a></div>
<div><br></div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div>On 15 January 2016 at 13:58, Gerhard Lazu
<span dir="ltr"><<a href="mailto:gerhard@lazu.co.uk" target="_blank">gerhard@lazu.co.uk</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
The understanding of difficult problems/bugs and the learning that
comes with it cannot be rushed. Each and every one of us has his /
her own pace, and all "speeds" are perfectly fine. The only
question that really matters is whether it's worth it (a.k.a. the
cost of lost opportunity). If the answer is yes, plough on. If not,
look for alternatives.</div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
Not everyone likes or wants to run their own infrastructure. The
monthly savings on the PaaS, IaaS advertised costs are undisputed,
but few like to think - never mind talk - about how many hours /
days / weeks have been spent debugging obscure problems which
"solve themselves" on a managed environment. Don't get me started
on those that are building their own Docker-based PaaS-es without
even realising it...</div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
As a side-note, I've been dealing with a similar TCP-related
problem for a while now, so I could empathise with your struggles
the second I've seen your post. One of us is bound to solve it
first, and I hope it will be you ; )</div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
Have a good one, Gerhard.</div>
<div>
<div>
<div><div><br></div>
<div>On Fri, Jan 15, 2016 at 10:01 AM, Simon
Morley <span dir="ltr"><<a href="mailto:simon@polkaspots.com" target="_blank">simon@polkaspots.com</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">You must be more patient that I am. It's been a long
month - having said that, I'm excited to find the cause.
<div><br></div>
<div>I misunderstood you re. file descriptors. We checked the
kernel limits / files open on the systems before and during and
there's nothing untoward. </div>
<div><br></div>
<div>Since writing in, it's not happened as before - no doubt it'll
take place during our forthcoming office move today.</div>
<div><br></div>
<div>I ran a strace (thanks for that suggestion John) on a couple
of processes yesterday and saw redis blocking. Restarted a few
redis servers to see if that helped. Can't be certain yet.</div>
<div><br></div>
<div>As soon as it's on, I'll run a tcpdump. How I'd not thought
about that I don't know...</div>
<div><br></div>
<div>Actually, this is one thing I dislike about Rails - it's so
nice and easy to do everything, one forgets we're dealing with the
real servers / components / connections. It's too abstract in ways,
but that's a whole other debate :)</div>
<div><span><br></span></div>
<div><span>S</span></div>
</div>
<div><span><br></span>
<div>
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div><span><br></span></div>
<div><span><br></span></div>
<div>
<div><span>Simon Morley<br>
<span style="font-family:arial;font-size:small"><div><br></div>
</span></span></div>
<div><span style="font-family:arial;font-size:small">Big
Chief</span> | PolkaSpots Supafly Wi-Fi
<div>Bigger Chief | Cucumber Tony</div>
<div><div><br></div>
<div><span style="font-size:small;font-family:arial"><a href="mailto:simon@polkaspots.com" target="_blank">simon@PolkaSpots.com</a></span><br style="font-size:small;font-family:arial">
<div style="font-size:small;font-family:arial">Linkedin: I'm on it
again and it still sucks</div>
<div style="font-size:small;font-family:arial"><a value="+442071831471">020
7183 1471</a></div>
</div>
</div>
</div>
</div>
<div style="font-size:small;font-family:arial"><br></div>
<div>🚀💥<br></div>
</div>
</div>
</div>
</div>
</div>
<br>
<div>
<div>
<div>On 15 January 2016 at 06:53, Gerhard Lazu
<span dir="ltr"><<a href="mailto:gerhard@lazu.co.uk" target="_blank">gerhard@lazu.co.uk</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
File descriptors, for traditional reasons, include TCP
connections.<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
Are you logging all requests to a central location? When the
problem occurs, it might help taking a closer look at the type of
requests you're receiving.</div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
Depending on how long the mischief lasts, a tcpdump to pcap, then
wireshark might help. Same for an strace on the Puma processes,
similar to what John suggested . Those are low level tools though,
verbose, complex and complete, it's easy to get lost unless you
know what you're looking for.</div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
In summary, CLOSE_WAITs piling up from haproxy (client role) to
Puma (server role) indicates the app not closing connections in
time (or maybe ever) - why? It's a fun one to troubleshoot ;
)</div>
</div>
<div>
<div>
<div><div><br></div>
<div>On Thu, Jan 14, 2016 at 11:35 PM, Simon
Morley <span dir="ltr"><<a href="mailto:simon@polkaspots.com" target="_blank">simon@polkaspots.com</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">Right now, none of the servers have any issues. No
close_waits.
<div><div><br></div>
All is well. Seemingly.</div>
<div><br></div>
<div>When it occurs ALL the servers end up going. Sometimes real
fast. That's why I thought we had a db bottleneck. It happens
pretty quickly, randomly, no particular times.</div>
<div><br></div>
<div>We don't ever really get spikes of traffic, there's an even
load inbound throughout.</div>
<div><br></div>
<div>I thought we had someone running a slow loris style attack on
us. So I added some rules to HA Proxy and Cloudflare ain't seen
nofin honest guv.</div>
<div><br></div>
<div>Will find a way to chart it and send a link over.</div>
<div><br></div>
<div>Will see if we're not closing any files - not much of that
going on. There's some manual gzipping happening - we've had that
in place for over a year though - not sure why it'd start playing
up now. Memory usage is high but consistent and doesn't
increase.</div>
<div><span><br></span></div>
<div><span>S</span></div>
</div>
<div><span><br></span>
<div>
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div><span><br></span></div>
<div><span><br></span></div>
<div>
<div><span>Simon Morley<br>
<span style="font-family:arial;font-size:small"><div><br></div>
</span></span></div>
<div><span style="font-family:arial;font-size:small">Big
Chief</span> | PolkaSpots Supafly Wi-Fi
<div>Bigger Chief | Cucumber Tony</div>
<div><div><br></div>
<div><span style="font-size:small;font-family:arial"><a href="mailto:simon@polkaspots.com" target="_blank">simon@PolkaSpots.com</a></span><br style="font-size:small;font-family:arial">
<div style="font-size:small;font-family:arial">Linkedin: I'm on it
again and it still sucks</div>
<div style="font-size:small;font-family:arial"><a value="+442071831471">020
7183 1471</a></div>
</div>
</div>
</div>
</div>
<div style="font-size:small;font-family:arial"><br></div>
<div>🚀💥<br></div>
</div>
</div>
</div>
</div>
</div>
<br>
<div>
<div>
<div>On 14 January 2016 at 22:14, Gerhard Lazu
<span dir="ltr"><<a href="mailto:gerhard@lazu.co.uk" target="_blank">gerhard@lazu.co.uk</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
That sounds like a file descriptor leak. Are the CLOSE_WAITs
growing over time?</div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
You're right, New Relic is too high level, this is a layer 4-5
issue.</div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
The simplest thing that can plot some graphs will work. Throw the
dirtiest script together that curls the data out if it comes easy,
it doesn't matter how you get those metrics as long as you have
them.</div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
This is a great blog post opportunity ; )</div>
</div>
<div>
<div>
<div><div><br></div>
<div>On Thu, Jan 14, 2016 at 8:40 PM, Simon
Morley <span dir="ltr"><<a href="mailto:simon@polkaspots.com" target="_blank">simon@polkaspots.com</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="auto">
<div>I would ordinarily agree with you about the connection however
they hang around for hours sometimes. </div>
<div><br></div>
<div>The 500 in the hyproxy config was actually left over from a
previous experiment. Realistically I know they won't cope with
that.</div>
<div><br></div>
<div>Using another server was to find any issues with puma. I'm
still going to try unicorn just in case. <br>
<br>
Will up the numbers too - thanks for that suggestion. </div>
<div><br></div>
<div>I'll look at a better monitoring tool too. So far new relic
hasn't helped much. </div>
<div><br></div>
<div>Thanks</div>
<div><br></div>
<div>S</div>
<div><span><br></span>
<div style="font-family:Helvetica;font-size:medium"><span>Simon
Morley</span></div>
<div style="font-family:Helvetica;font-size:medium"><span>Big Chief
| PolkaSpots Supafly Wi-Fi</span></div>
<div style="font-family:Helvetica;font-size:medium">
<span><br></span></div>
<div style="font-family:Helvetica;font-size:medium">
<span><br></span></div>
<div style="font-family:Helvetica;font-size:medium">
<span><br></span></div>
<div style="font-family:Helvetica;font-size:medium">I'm doing it
with Cucumber Tony. Are you?</div>
</div>
<div>
<div>
<div><div><br></div>
On 14 Jan 2016, at 20:30, Gerhard Lazu <<a href="mailto:gerhard@lazu.co.uk" target="_blank">gerhard@lazu.co.uk</a>> wrote:<br>
<br></div>
<blockquote type="cite">
<div>
<div dir="ltr">
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
Hi Simon,</div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
CLOSE_WAIT suggests that Puma is not closing connections fast
enough. The client has asked for the connection to be closed,
but Puma is busy.</div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
Quickest win would be to increase your Puma instances. Unicorn
won't help - or any other Rack web server for the matter.</div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
Based on your numbers, start with 10 Puma instances. Anything more
than 100 connections for a Rails instance is not realistic. I would
personally go with 50, just to be safe. I think I saw 500
conns in your haproxy config, which is way
too optimistic.</div>
<div style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<br></div>
You want metrics
for detailed CPU usage by process, connections open with state
by process, and memory usage, by process. Without these, you're
flying blind. Any suggestions anyone makes without real
metrics - including myself - are just guesses. You'll get
there, but you're making it far too difficult for
yourself.</div>
<div dir="ltr"><br></div>
<div dir="ltr">Let me know how it goes, Gerhard.</div>
<div><div><br></div>
<div>On Thu, Jan 14, 2016 at 3:16 PM, Simon
Morley <span dir="ltr"><<a>simon@polkaspots.com</a>></span>
wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">Hello All
<div><br></div>
<div>We've been battling with Puma for a long while now, I'm
looking for some help / love / attention / advice / anything to
prevent further hair loss.</div>
<div><br></div>
<div>We're using it in a reasonably typical Rails 4 application
behind Nginx. <br>
<div>
<div>
<div dir="ltr">
<div dir="ltr">
<div><br></div>
<div>Over the last 3 months, our requests have gone from 500 rpm to
a little over 1000 depending on the hour. Over this period, we've
been seeing weird CLOSE_WAIT conns appearing in netstat, which
eventually kill the servers.</div>
<div><br></div>
<div>We have 3 Rails servers behind Haproxy running things. Load is
generally even.</div>
<div><br></div>
<div>Running netstat on the servers shows a pile of connections in
the CLOSE_WAIT state with varying recv-q values as so:<br></div>
<div><br></div>
<div>
<div>tcp 2784 0 localhost:58786
localhost:5100
CLOSE_WAIT</div>
<div>tcp 717 0
localhost:35794 localhost:5100
CLOSE_WAIT</div>
<div>tcp 784 0
localhost:55712 localhost:5100
CLOSE_WAIT<br></div>
<div>tcp 0 0
localhost:38639 localhost:5100
CLOSE_WAIT</div>
</div>
<div><br></div>
<div>
<div>That's just a snippet. A wc reveals over 400 of these on each
server.</div>
</div>
<div><br></div>
<div>Puma is running on port 5100 btw. We've tried puma with
multiple threads and a single one - same result. Latest version as
of today.</div>
<div><br></div>
<div>I've checked haproxy and don't see much lingering
around.</div>
<div><br></div>
<div>Only a kill -9 can stop Puma - otherwise, it says something
like 'waiting for requests to finish'</div>
<div><br></div>
<div>I ran GDB to see if I could debug the process however I can't
claim I knew what I was looking at. The processes that seemed
apparent were event machine and mongo.</div>
<div><br></div>
<div>We then ditched EM (we were using the AMQP gem) in favour of
Bunny. That made zero difference.</div>
<div><br></div>
<div>So we upgraded Mongo and Mongoid to the latest versions,
neither of which helped.</div>
<div><br></div>
<div>I thought we might have a bottleneck somewhere - Mongo, ES or
MySQL. But, none of those services seem to have any issues /
latencies.</div>
<div><br></div>
<div>It's also 100% random. Might happen 10 times in an hour, then
not at all for a week.</div>
<div><br></div>
<div>The puma issues on github don't shed much light.</div>
<div><br></div>
<div>I don't really know where to turn at the moment or what to do
next? I was going to resort back to Unicorn but I don't think the
issue is that side and I wanted to fix the problem, not just patch
it up.</div>
<div><br></div>
<div>It's starting to look like a nasty in my code somewhere but I
don't want to go down that route just yet...</div>
<div><br></div>
<div>Sorry for the long email, thanks in advance. Stuff.</div>
<div><br></div>
<div>I hope someone can help!</div>
<div><br></div>
<div>S</div>
<div><br></div>
<div>
<div>Simon Morley<br>
<span style="font-family:arial;font-size:small"><br></span></div>
<div><span style="font-family:arial;font-size:small">Big
Chief</span> | PolkaSpots Supafly Wi-Fi
<div>Bigger Chief | Cucumber Tony</div>
<div><div><br></div>
<div><span style="font-size:small;font-family:arial"><a href="mailto:simon@polkaspots.com" target="_blank">simon@PolkaSpots.com</a></span><br style="font-size:small;font-family:arial">
<div style="font-size:small;font-family:arial">Linkedin: I'm on it
again and it still sucks</div>
<div style="font-size:small;font-family:arial"><a value="+442071831471">020
7183 1471</a></div>
</div>
</div>
</div>
</div>
<div style="font-size:small;font-family:arial"><br></div>
<div>🚀💥<br></div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
_______________________________________________<br>
Chat mailing list<br>
<a>Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote>
</div>
<br></div>
</div>
</blockquote>
<blockquote type="cite">
<div>
<span>_______________________________________________</span><br>
<span>Chat mailing list</span><br>
<span><a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a></span><br>
<span>Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a></span><br>
<span>Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a></span><br>
<span>List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a></span><br>
</div>
</blockquote>
</div>
</div>
</div>
<br>
_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote>
</div>
<br></div>
</div>
</div>
<br>
_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote>
</div>
<br></div>
</div>
</div>
<br>
_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote>
</div>
<br></div>
</div>
</div>
<br>
_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote>
</div>
<br></div>
</div>
</div>
<br>
_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote>
</div>
<br></div>
</div>
</div>
</div>
<br>
_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote>
</div>
<br></div>
</div>
</div>
_______________________________________________
<br>Chat mailing list
<br><a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a>
<br>Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a>
<br>Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a>
<br>List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a>
<br></div></div></span></blockquote></div></div></div></div></div></blockquote></div></div></div><br></div>
<br>_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote></div><br><br><div><br></div></div></div><span>-- <br><div><div dir="ltr">Riccardo Tacconi<br><br><a href="http://github.com/rtacconi" target="_blank">http://github.com/rtacconi</a><br><a href="http://twitter.com/rtacconi" target="_blank">http://twitter.com/rtacconi</a></div></div>
</span></div>
<br>_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote></div><br></div></div></div>
<br>_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote></div><br><br><div><br></div>-- <br><div><div dir="ltr">Riccardo Tacconi<br><br><a href="http://github.com/rtacconi" target="_blank">http://github.com/rtacconi</a><br><a href="http://twitter.com/rtacconi" target="_blank">http://twitter.com/rtacconi</a></div></div>
</div></div></div></div></div>
<br>_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote></div><br><br><div><br></div>-- <br><div><div dir="ltr">Riccardo Tacconi<br><br><a href="http://github.com/rtacconi" target="_blank">http://github.com/rtacconi</a><br><a href="http://twitter.com/rtacconi" target="_blank">http://twitter.com/rtacconi</a></div></div>
</div>
</div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>Chat mailing list</span><br><span><a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a></span><br><span>Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a></span><br><span>Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a></span><br><span>List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a></span><br></div></blockquote></div>_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
</blockquote></div>
</blockquote></div></div><br>_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote></div><br></div></div></div>
<br>_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" rel="noreferrer" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr">Riccardo Tacconi<br><br><a href="http://github.com/rtacconi" target="_blank">http://github.com/rtacconi</a><br><a href="http://twitter.com/rtacconi" target="_blank">http://twitter.com/rtacconi</a></div></div>
</div>
</div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>Chat mailing list</span><br><span><a href="mailto:Chat@lists.lrug.org">Chat@lists.lrug.org</a></span><br><span>Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org">http://lists.lrug.org/pipermail/chat-lrug.org</a></span><br><span>Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org">http://lists.lrug.org/options.cgi/chat-lrug.org</a></span><br><span>List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a></span><br></div></blockquote></body></html>