[LRUG] Queuing systems

Graham Ashton graham at effectif.com
Wed Sep 7 02:29:15 PDT 2011


On 7 Sep 2011, at 10:18, Neil Middleton wrote:

> Hundreds, possibly thousands of new jobs per second.

I was actually wondering in terms of bytes per second, but that's still a useful answer.

I know I'm still avoiding the question here, but I'm now wondering how you're thinking of going about processing them. That would probably have an impact on how I'd approach queueing.

i.e. If your job processing component went off line for six hours for some reason, would it (when it came back up) be more important to process the earliest queued data, or the most recent data?

If your job processing stuff got a long way behind would it make sense to drop really old jobs and just start on recent stuff? I'm wondering if this data needs persisting to disk, or whether you can get away with stashing it in RAM (possibly in a persistence backed key value store like Redis).


More information about the Chat mailing list