<div dir="ltr">Ok. Back to the drawing board. I'll try that.<div><br>I'm assuming therefore there's no biggy running a job within a job, so to speak..</div></div><div class="gmail_extra"><br clear="all"><div><div dir="ltr">
Simon Morley<br><span style="font-family:arial;font-size:small">Big Chief</span> | PolkaSpots Supafly Wi-Fi<div><br><div>simon@PolkaSpots.com<br><div>020 7183 1471<br><br>Follow our updates on twitter: <a href="http://twitter.com/polkaspots" target="_blank">twitter.com/polkaspots</a><br>
Try Wi-Fi As A Service (WaaS) here: <a href="http://bit.ly/15UymJ8" style="padding:0px;margin:0px;text-decoration:none;color:rgb(97,179,222);display:inline-block;max-width:150px;text-overflow:ellipsis;overflow:hidden;white-space:nowrap;vertical-align:top;height:17px;line-height:17px;font-family:adelle,Georgia,Helvetica,Arial,sans-serif;font-style:italic;font-weight:bold" target="_blank"><span style="padding:0px;margin:0px;width:1px;height:1px;line-height:1px;display:block;overflow:hidden">http://</span>bit.ly/15UymJ8</a><br>
<br><img src="http://ps-ec2-web-1.polkaspots.com/global/images/polkaspots.png" width="200" height="104"><br><br><br><br><div><br></div></div></div></div></div></div>
<br><br><div class="gmail_quote">On 21 March 2014 13:11, Jim Myhrberg <span dir="ltr"><<a href="mailto:contact@jimeh.me" target="_blank">contact@jimeh.me</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">In that case I'd take your Ruby worker out of the equation, and stress test your notification queue and Node.js worker by having a custom script or something flood the notification queue with messages, upping the message rate until you're starting to see the Node.js worker misbehaving. In theory you should be able to do this even on your local dev machine, unless production and dev run different OSs and the issue stems from something OS specific that RabbitMQ or Node.js relies upon.</div>
<div class="gmail_extra"><br clear="all"><div><div dir="ltr"><div><br></div><div><br></div>--jimeh</div></div><div><div class="h5">
<br><br><div class="gmail_quote">On Fri, Mar 21, 2014 at 12:47 PM, Simon Morley <span dir="ltr"><<a href="mailto:simon@polkaspots.com" target="_blank">simon@polkaspots.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi Jim<div><br></div><div>Thanks for the email. I have two coffees on my desk right now, I can send one over..</div><div><br></div><div>We were actually using bunny until last night but swapped over to AMQP gem as a test. There's a bunny issue that's just been closed that sort of addressed our problem (<a href="https://groups.google.com/forum/#!topic/ruby-amqp/_ZShRP6FRXI" target="_blank">https://groups.google.com/forum/#!topic/ruby-amqp/_ZShRP6FRXI</a>). Wanted to eliminate that part despite running a later version.</div>
<div><br></div><div>It's actually the latter we seem to be struggling with. They seem to disappear between Rabbit and Node. Thanks for the info though, good to learn new stuff.</div><span><font color="#888888"><div>
<br></div><div>S</div><div><br>
</div><div><br></div></font></span></div><div class="gmail_extra"><div><br clear="all"><div><div dir="ltr">Simon Morley<br><span style="font-family:arial;font-size:small">Big Chief</span> | PolkaSpots Supafly Wi-Fi<div>
<br><div>simon@PolkaSpots.com<br>
<div><a href="tel:020%207183%201471" value="+442071831471" target="_blank">020 7183 1471</a><br><br>Follow our updates on twitter: <a href="http://twitter.com/polkaspots" target="_blank">twitter.com/polkaspots</a><br>Try Wi-Fi As A Service (WaaS) here: <a href="http://bit.ly/15UymJ8" style="padding:0px;margin:0px;text-decoration:none;color:rgb(97,179,222);display:inline-block;max-width:150px;text-overflow:ellipsis;overflow:hidden;white-space:nowrap;vertical-align:top;min-height:17px;line-height:17px;font-family:adelle,Georgia,Helvetica,Arial,sans-serif;font-style:italic;font-weight:bold" target="_blank"><span style="padding:0px;margin:0px;width:1px;min-height:1px;line-height:1px;display:block;overflow:hidden">http://</span>bit.ly/15UymJ8</a><br>
<br><img src="http://ps-ec2-web-1.polkaspots.com/global/images/polkaspots.png" width="200" height="104"><br><br><br><br><div><br></div></div></div></div></div></div>
<br><br></div><div><div><div class="gmail_quote">On 21 March 2014 12:30, Jim Myhrberg <span dir="ltr"><<a href="mailto:contact@jimeh.me" target="_blank">contact@jimeh.me</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hey Simon,<div><br></div><div>My AMQP skills have been gathering dust for a couple of years at this point, but I'll see if I can help.</div><div><br></div><div>First, maybe I haven't had enough caffeine today, so would mind clarifying if the issue is that the notification messages don't get pushed from the Ruby worker, or they do show up fine in RabbitMQ and then disappear between RabbitMQ and the Node.js worker?</div>
<div><br></div><div>If it's the former, one thing to keep in mind when using the amqp gem, is that calling #push doesn't mean the message has been pushed, it means it will be pushed at some point in the future when the event loop gets to it. Typically if I recall, anything/everything pushed within processing one AMQP message, get's all gets pushed at once after the ACK for the initial message being processed is sent back to RabbitMQ. Another thing to keep in mind is that killing the connection/process/whatever even after finishing processing the initial message might cut off your messages being pushed.</div>
<div><br></div><div>The async nature of the amqp gem can cause some weird timing issues. Hence when I last used AMQP we ended up replacing the amqp gem where async wasn't specifically required with the synchronous bunny gem instead to make things simpler.</div>
<div><br></div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div dir="ltr"><div><br></div><div><br></div>--jimeh</div></div><div><div>
<br><br><div class="gmail_quote">On Fri, Mar 21, 2014 at 12:10 PM, Simon Morley <span dir="ltr"><<a href="mailto:simon@polkaspots.com" target="_blank">simon@polkaspots.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi Tom, Steve<div><br></div><div>Thanks for your emails.<br><div><br></div><div>We have ack on for everything right now. We can see the 'parent' job being queued, the follow up jobs don't seem to be queued - I'll confirm this shortly.</div>
<div><br></div><div>I have no unacked messages either right now.</div><div><br></div><div>I'd thought the consumer might be dying which is why we ran the child job in a loop. No problems encountered.</div><div><br></div>
<div>Just can't see why they work independently, just not as one.</div><div><br></div><div>S</div></div></div><div class="gmail_extra"><br clear="all"><div><div dir="ltr"><div>Simon Morley<br><span style="font-family:arial;font-size:small">Big Chief</span> | PolkaSpots Supafly Wi-Fi</div>
<div>
<br><div>simon@PolkaSpots.com<br><div><a href="tel:020%207183%201471" value="+442071831471" target="_blank">020 7183 1471</a><br><br>Follow our updates on twitter: <a href="http://twitter.com/polkaspots" target="_blank">twitter.com/polkaspots</a><br>
Try Wi-Fi As A Service (WaaS) here: <a href="http://bit.ly/15UymJ8" style="padding:0px;margin:0px;text-decoration:none;color:rgb(97,179,222);display:inline-block;max-width:150px;text-overflow:ellipsis;overflow:hidden;white-space:nowrap;vertical-align:top;min-height:17px;line-height:17px;font-family:adelle,Georgia,Helvetica,Arial,sans-serif;font-style:italic;font-weight:bold" target="_blank"><span style="padding:0px;margin:0px;width:1px;min-height:1px;line-height:1px;display:block;overflow:hidden">http://</span>bit.ly/15UymJ8</a><br>
<br><img src="http://ps-ec2-web-1.polkaspots.com/global/images/polkaspots.png" width="200" height="104"><br><br><br><br><div><br></div></div></div></div></div></div><div><div>
<br><br><div class="gmail_quote">On 21 March 2014 11:59, Tom - Droplet <span dir="ltr"><<a href="mailto:tom@dropletpay.com" target="_blank">tom@dropletpay.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word"><div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">Hi Simon,</div><div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">
<br></div><div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">What are your publisher confirm settings and consumer ack settings? Could it be that the consumer dies during processing the job but you have subscribed to the queue without requiring acks, so the message is essentially lost?</div>
<div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto"><br></div><div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">
The RabbitMQ management page can be pretty useful for debugging this kind of stuff sometimes, you can check that messages are getting put in the queue as you expect and pull them out to see the message.</div><div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">
<br></div> <div><div style="font-family:helvetica,arial;font-size:13px">-- <br>Tom Livesey<br></div><div style="font-family:helvetica,arial;font-size:13px">Lead engineer - Droplet</div><div style="font-family:helvetica,arial;font-size:13px">
<a href="http://dropletpay.com" target="_blank">dropletpay.com</a></div><div style="font-family:helvetica,arial;font-size:13px"><br></div></div><div><div> <br><p>On 21 March 2014 at 11:48:05, Simon Morley (<a href="mailto:simon@polkaspots.com" target="_blank">simon@polkaspots.com</a>) wrote:</p>
</div></div><blockquote type="cite"><span><div><div></div><div><div><div>
<div dir="ltr">Morning All
<div><br></div>
<div>Having asked SO, I remembered we were supposed to be
encouraging more questions here! I hope you can help (sorry this is
a long message).</div>
<div><br></div>
<div>Relatively new to the world of RabbitMQ, I'm getting stuck
debugging the problem of missing messages. All was / is fine in my
dev. environment, the problems arise when we start chucking
thousand of messages at things. </div>
<div><br></div>
<div>I've narrowed the problem down to messages that are sent from
within another job. For example, we have a job that moves a
customer's subscription when they sign up. We publish this to our
rabbit servers using the AMQP gem and EM using something like
this:</div>
<div><br></div>
<div><a href="https://gist.github.com/simonmorley/9684010" target="_blank">https://gist.github.com/simonmorley/9684010</a><br>
</div>
<div><br></div>
<div>def queue_process_subscription </div>
<div> user. process_subscription</div>
<div>end</div>
<div><br></div>
<div>def process_subscription</div>
<div> hammer_time</div>
<div> MESSAGE_QUEUE.push(json_message,
'ps.routing.key') </div>
<div>end</div>
<div><br></div>
<div>After the job's been completed, we use MESSAGE_QUEUE to push a
message to our notification servers (which run NodeJS, node-amqp
and SocketIO).</div>
<div><br></div>
<div>On my lonely laptop, the jobs process perfectly, the
notification are sent and my SocketIO client receives them. Happy
days.</div>
<div><br></div>
<div>On my production servers, the job is queued in RabbitMq,
processed perfectly and the notification is published apparently.
The problem is, it doesn't appear to be received by the node amqp
consumer 100% of the time. Maybe 1 in 10 get delivered (and
sometimes they don't look like the full message). The rest just
vanish.. It gets worse as we load the servers up.</div>
<div><br></div>
<div>I thought it was the AMQP on the Node servers but I created a
loop to run MESSAGE_QUEUE 10,000 times and they were all
consumed..</div>
<div><br></div>
<div>So, I'm assuming the problem is that we're sending a message
from within another thread. At least, that's what I think it is. My
knowledge of such things has now come to an end...</div>
<div><br></div>
<div>We've now replaced the last notification with Redis publish
which works a treat. But that's another cog to think about.</div>
<div><br></div>
<div>Like I said, I'm new to this messaging / threading stuff,
and would appreciate it if someone can point me in the right
direction, tell me what to read or suggestion something else to
try. Or tell me to give up and go home.</div>
<div><br></div>
<div>Cheers</div>
<div><br></div>
<div>Simon</div>
<div><br></div>
<div><br></div>
<div><br></div>
<div>
<div>
<div dir="ltr">Simon Morley<br>
<span style="font-family:arial;font-size:small">Big
Chief</span> | PolkaSpots Supafly Wi-Fi
<div><br>
<div>simon@PolkaSpots.com<br>
<div><a href="tel:020%207183%201471" value="+442071831471" target="_blank">020 7183 1471</a><br>
@polkaspots</div>
<div><br>
<img src="http://ps-ec2-web-1.polkaspots.com/global/images/polkaspots.png" width="200" height="104"><br>
<br>
<br>
<br>
<div><br></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div></div><pre>---------------------------------------------------------------------------------------
The information contained in this email and its attachments is confidential and may be the subject of legal, professional or other privilege. It is intended only for the named addressees and may not be disclosed to anyone else without consent from PolkaSpots Limited. If you are not the named addressee you must not use, disclose, distribute, copy, print or rely on the contents of this email and should destroy it immediately. Whilst PolkaSpots Limited takes care to protect its systems from electronic virus attack or other harmful event, the firm gives no warranty that this email message (including any attachments to it) is free of any virus or other harmful matter and accepts no responsibility for any loss or damage resulting from the recipient receiving, opening or using it. If you need any further information, please contact the originator of this message on <a href="tel:%2B44%20%280%29%2020%207183%201471" value="+442071831471" target="_blank">+44 (0) 20 7183 1471</a>.
PolkaSpots Limited is registered in the UK, number 05508105. Registered Office: 62 Britton Street (behind the bike sheds), Clerkenwell, London EC1M 5UY
Website: <a href="http://polkaspots.com" target="_blank">polkaspots.com</a>
WiFi Shop: <a href="http://shop.polkaspots.com" target="_blank">shop.polkaspots.com</a>
Blog: <a href="http://polkaspots.com/blog" target="_blank">polkaspots.com/blog</a>
---------------------------------------------------------------------------------------
</pre>
_______________________________________________
<br>Chat mailing list
<br><a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a>
<br><a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a>
<br></div></div></span></blockquote></div></blockquote></div><br></div></div></div><div><div>
<pre>---------------------------------------------------------------------------------------
The information contained in this email and its attachments is confidential and may be the subject of legal, professional or other privilege. It is intended only for the named addressees and may not be disclosed to anyone else without consent from PolkaSpots Limited. If you are not the named addressee you must not use, disclose, distribute, copy, print or rely on the contents of this email and should destroy it immediately. Whilst PolkaSpots Limited takes care to protect its systems from electronic virus attack or other harmful event, the firm gives no warranty that this email message (including any attachments to it) is free of any virus or other harmful matter and accepts no responsibility for any loss or damage resulting from the recipient receiving, opening or using it. If you need any further information, please contact the originator of this message on <a href="tel:%2B44%20%280%29%2020%207183%201471" value="+442071831471" target="_blank">+44 (0) 20 7183 1471</a>.
PolkaSpots Limited is registered in the UK, number 05508105. Registered Office: 62 Britton Street (behind the bike sheds), Clerkenwell, London EC1M 5UY
Website: <a href="http://polkaspots.com" target="_blank">polkaspots.com</a>
WiFi Shop: <a href="http://shop.polkaspots.com" target="_blank">shop.polkaspots.com</a>
Blog: <a href="http://polkaspots.com/blog" target="_blank">polkaspots.com/blog</a>
---------------------------------------------------------------------------------------
</pre></div></div><br>_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
<a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote></div><br></div></div></div>
</blockquote></div><br></div></div></div><div><div>
<pre>---------------------------------------------------------------------------------------
The information contained in this email and its attachments is confidential and may be the subject of legal, professional or other privilege. It is intended only for the named addressees and may not be disclosed to anyone else without consent from PolkaSpots Limited. If you are not the named addressee you must not use, disclose, distribute, copy, print or rely on the contents of this email and should destroy it immediately. Whilst PolkaSpots Limited takes care to protect its systems from electronic virus attack or other harmful event, the firm gives no warranty that this email message (including any attachments to it) is free of any virus or other harmful matter and accepts no responsibility for any loss or damage resulting from the recipient receiving, opening or using it. If you need any further information, please contact the originator of this message on <a href="tel:%2B44%20%280%29%2020%207183%201471" value="+442071831471" target="_blank">+44 (0) 20 7183 1471</a>.
PolkaSpots Limited is registered in the UK, number 05508105. Registered Office: 62 Britton Street (behind the bike sheds), Clerkenwell, London EC1M 5UY
Website: <a href="http://polkaspots.com" target="_blank">polkaspots.com</a>
WiFi Shop: <a href="http://shop.polkaspots.com" target="_blank">shop.polkaspots.com</a>
Blog: <a href="http://polkaspots.com/blog" target="_blank">polkaspots.com/blog</a>
---------------------------------------------------------------------------------------
</pre></div></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div>
<pre>
---------------------------------------------------------------------------------------
The information contained in this email and its attachments is confidential and may be the subject of legal, professional or other privilege. It is intended only for the named addressees and may not be disclosed to anyone else without consent from PolkaSpots Limited. If you are not the named addressee you must not use, disclose, distribute, copy, print or rely on the contents of this email and should destroy it immediately. Whilst PolkaSpots Limited takes care to protect its systems from electronic virus attack or other harmful event, the firm gives no warranty that this email message (including any attachments to it) is free of any virus or other harmful matter and accepts no responsibility for any loss or damage resulting from the recipient receiving, opening or using it. If you need any further information, please contact the originator of this message on +44 (0) 20 7183 1471.
PolkaSpots Limited is registered in the UK, number 05508105. Registered Office: 62 Britton Street (behind the bike sheds), Clerkenwell, London EC1M 5UY
Website: polkaspots.com
WiFi Shop: shop.polkaspots.com
Blog: polkaspots.com/blog
---------------------------------------------------------------------------------------