<html><head><style>body{font-family:Helvetica,Arial;font-size:13px}</style></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><div id="bloop_customfont" style="line-height: 19.5px; margin: 0px;">Hello there from the trenches,</div><div id="bloop_customfont" style="line-height: 19.5px; margin: 0px;"><br></div><div id="bloop_customfont" style="line-height: 19.5px; margin: 0px;">One other thing to note is that SQS may ramp up a significant bill if you have busy queues. The cost increases significantly if your workers live outside the Amazon cloud, as then they charge for data transfer as well. Conversely, if you deploy inside the cloud and have computationally-intensive jobs, it may prove to be a false economy as you will now pay the AWS CPU-clock tax. </div><div id="bloop_customfont" style="line-height: 19.5px; margin: 0px;"><br></div><div id="bloop_customfont" style="line-height: 19.5px; margin: 0px;">If we’re not talking millions of messages, then ignore the above.</div><div id="bloop_customfont" style="line-height: 19.5px; margin: 0px;"><br></div><div id="bloop_customfont" style="line-height: 19.5px; margin: 0px;">Also, I wouldn’t worry so much about whether Shoryuken implements long polling, which is simply a glorified name for a loop that hits SQS repeatedly. You can achieve the same thing by reducing the poll interval of Shoryuken workers to something acceptable if the default of 15s is not:</div><div id="bloop_customfont" style="line-height: 19.5px; margin: 0px;"><br></div><div id="bloop_customfont" style="line-height: 19.5px; margin: 0px;"><a href="https://github.com/mperham/sidekiq/wiki/Scheduled-Jobs">https://github.com/mperham/sidekiq/wiki/Scheduled-Jobs</a></div><div id="bloop_customfont" style="line-height: 19.5px; margin: 0px;"><br></div><div id="bloop_customfont" style="line-height: 19.5px; margin: 0px;">My quite unscientific rule-of-thumb is that if you hit SQS a few times and it doesn’t return anything, you can be pretty sure there’s nothing in the queue. It’s sort of like quantum physics, where you’re never completely sure about anything and resign to reasonable compromises.</div><div id="bloop_customfont" style="line-height: 19.5px; margin: 0px;"><br></div><div id="bloop_customfont" style="line-height: 19.5px; margin: 0px;">Also, a barely-related aside: If you happen to go metal and bump many things from queue, rather one per job, you’re better off in terms of performance to delete the messages in batches as well. The last time I checked, the AWS gem deleted them one by one, which slows things down significantly.</div><div id="bloop_customfont" style="line-height: 19.5px; margin: 0px;"><br></div></div><p class="airmail_on" style="color:#000;">On 29 May 2015 at 11:57:17, Garry Shutler (<a href="mailto:garry@robustsoftware.co.uk">garry@robustsoftware.co.uk</a>) wrote:</p> <blockquote type="cite" class="clean_bq"><span><div><div></div><div>
<title></title>
<div dir="ltr">Nice tip Fred, thanks.
<div><br></div>
<div>I had a look and Shoryuken doesn't support it at the gem
level, there's some issues going into why (<a href="https://github.com/phstc/shoryuken/issues/30">#30</a>, <a href="https://github.com/phstc/shoryuken/issues/92">#92</a>).</div>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div class="gmail_signature">
<div dir="ltr"><b>Garry Shutler</b>
<div>
<div><a href="http://twitter.com/gshutler" target="_blank">@gshutler</a></div>
<a href="http://gshutler.com/" target="_blank">gshutler.com</a></div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On 29 May 2015 at 11:40, Frederick Cheung
<span dir="ltr"><<a href="mailto:frederick.cheung@gmail.com" target="_blank">frederick.cheung@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">
SQS largely takes care of itself. One issue I’ve run into with SQS
is that in its default mode reading messages from a nearly empty
queue may return no messages (the reason being that the queue is
distributed across many machines and reads by default will hit a
single partition in the queue, if that one has no messages then the
api call will return no message).</div>
<div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">
<br></div>
<div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">
You can address this by using the long polling mode - calls to read
a message will block up to a configurable limit (upto 20s) before
either returning no message or a message that has arrived in the
wait period and this mode doesn’t suffer from the limitation I
describe. This can either be configured as a setting on the queue
or passed as a parameter to individual api calls. If Shoryuken
doesn’t support the latter then you might want to set this as a
global queue property.</div>
<br>
<div>Fred</div>
<div>
<div class="h5"><br>
<p style="color:#000">On 29 May 2015 at 11:31:30, Garry Shutler
(<a href="mailto:garry@robustsoftware.co.uk" target="_blank">garry@robustsoftware.co.uk</a>) wrote:</p>
</div>
</div>
<blockquote type="cite">
<div>
<div>
<div>
<div class="h5">
<div dir="ltr"><span>Hi all,</span>
<div><span><br></span></div>
<div><span>I'm looking into moving our messaging from Resque to
<a href="http://aws.amazon.com/sqs/" target="_blank">SQS</a>.</span></div>
<div><span><br></span></div>
<div><span>I'm looking at using <a href="https://github.com/phstc/shoryuken" target="_blank">Shoryuken</a>
as it seems to be the most mature gem, follows the patterns of
Sidekiq, and supports ActiveJob which means we'll be able to switch
in Resque/Sidekiq for development as there's no way to run SQS
locally.</span></div>
<div><span><br></span></div>
<div><span>Do any of you have any "from the trenches" experience
you can share around SQS and/or Shoryuken? In particular I've not
found much useful stuff on monitoring and administering the queues
themselves.<br></span></div>
<div><span><br></span></div>
<div><span>Thanks in advance,</span></div>
<div><span>Garry</span></div>
<div><span><br clear="all"></span>
<div>
<div>
<div dir="ltr"><span><b>Garry Shutler</b></span>
<div>
<div><span><a href="http://twitter.com/gshutler" target="_blank">@gshutler</a></span></div>
<span><a href="http://gshutler.com/" target="_blank">gshutler.com</a></span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<span>_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br></span></div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br></div>
_______________________________________________
<br>Chat mailing list
<br>Chat@lists.lrug.org
<br>Archives: http://lists.lrug.org/pipermail/chat-lrug.org
<br>Manage your subscription: http://lists.lrug.org/options.cgi/chat-lrug.org
<br>List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
<br></div></div></span></blockquote></body></html>