[LRUG] Amazon Simple Queue Service (SQS)

[Hakan Ensari]

Hello there from the trenches,

One other thing to note is that SQS may ramp up a significant bill if you have busy queues. The cost increases significantly if your workers live outside the Amazon cloud, as then they charge for data transfer as well. Conversely, if you deploy inside the cloud and have computationally-intensive jobs, it may prove to be a false economy as you will now pay the AWS CPU-clock tax. 

If we’re not talking millions of messages, then ignore the above.

Also, I wouldn’t worry so much about whether Shoryuken implements long polling, which is simply a glorified name for a loop that hits SQS repeatedly. You can achieve the same thing by reducing the poll interval of Shoryuken workers to something acceptable if the default of 15s is not:

https://github.com/mperham/sidekiq/wiki/Scheduled-Jobs

My quite unscientific rule-of-thumb is that if you hit SQS a few times and it doesn’t return anything, you can be pretty sure there’s nothing in the queue. It’s sort of like quantum physics, where you’re never completely sure about anything and resign to reasonable compromises.

Also, a barely-related aside: If you happen to go metal and bump many things from queue, rather one per job, you’re better off in terms of performance to delete the messages in batches as well. The last time I checked, the AWS gem deleted them one by one, which slows things down significantly.

On 29 May 2015 at 11:57:17, Garry Shutler (garry at robustsoftware.co.uk) wrote:

Nice tip Fred, thanks.

I had a look and Shoryuken doesn't support it at the gem level, there's some issues going into why (#30, #92).

Garry Shutler
@gshutler
gshutler.com

On 29 May 2015 at 11:40, Frederick Cheung <frederick.cheung at gmail.com> wrote:
SQS largely takes care of itself. One issue I’ve run into with SQS is that in its default mode reading messages from a nearly empty queue may return no messages (the reason being that the queue is distributed across many machines and reads by default will hit a single partition in the queue, if that one has no messages then the api call will return no message).

You can address this by using the long polling mode - calls to read a message will block up to a configurable limit (upto 20s) before either returning no message or a message that has arrived in the wait period and this mode doesn’t suffer from the limitation I describe. This can either be configured as a setting on the queue or passed as a parameter to individual api calls. If Shoryuken doesn’t support the latter then you might want to set this as a global queue property.

Fred

On 29 May 2015 at 11:31:30, Garry Shutler (garry at robustsoftware.co.uk) wrote:

Hi all,

I'm looking into moving our messaging from Resque to SQS.

I'm looking at using Shoryuken as it seems to be the most mature gem, follows the patterns of Sidekiq, and supports ActiveJob which means we'll be able to switch in Resque/Sidekiq for development as there's no way to run SQS locally.

Do any of you have any "from the trenches" experience you can share around SQS and/or Shoryuken? In particular I've not found much useful stuff on monitoring and administering the queues themselves.

Thanks in advance,
Garry

Garry Shutler
@gshutler
gshutler.com
_______________________________________________
Chat mailing list
Chat at lists.lrug.org
Archives: http://lists.lrug.org/pipermail/chat-lrug.org
Manage your subscription: http://lists.lrug.org/options.cgi/chat-lrug.org
List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org

_______________________________________________  
Chat mailing list  
Chat at lists.lrug.org  
Archives: http://lists.lrug.org/pipermail/chat-lrug.org  
Manage your subscription: http://lists.lrug.org/options.cgi/chat-lrug.org  
List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20150529/8424428d/attachment.html>