[LRUG] ActiveMQ advice?

[David Salgado]

Hi James

> I think we [1] found that it's better to concentrate on processing
> messages as quickly as possible to avoid too much of a backlog
> building up. That way having persistent messages is not so important

Good advice. In my particular application, each message can be
processed reasonably quickly, most of the time, but I'm still going to
need concurrency to achieve the traffic and latency targets.
So, the queue is just a way to delegate concurrency issues, so that my
message processor daemons don't need to worry if one of their clones
is also trying to process their current message, at the same time.

> and in any case is usually easy to retro-fit. If you start from the
> idea that your queue is going to be persistent, it's easy to be lazy
> at making message processing efficient. Then you start having a big
> backlog and you rely heavily on the queue persistence mechanism. I
> think this is where we felt the pain with ActiveMQ.

Yes, I can see how that would be an easy trap to fall into.

I think my best approach is to run with a lightweight, in-memory
queue, for speed, and have some kind of single-threaded process that
will  (re-)queue any messages that got left behind after a queue
outage.

AFAICT from a quick look at the beanstalk docs/google group, you can
ask a beanstalkd if a particular jobid already exists in a queue,
which should make the queue recovery process quite straightforward.

Thanks again for all the good advice.

David