[rabbitmq-discuss] Millions of Queues
Ryan Williams (Which)
rdw at lindenlab.com
Tue Feb 17 22:28:08 GMT 2009
Hey all, we have a different use case for rabbit than usual. We'd like
to create a rabbit cluster that handles at least 250,000 queues, with at
leat 1 million total bindings; ideally we would have 20+ million and 80+
million respectively. Basically we want infinite scalability along the
# of queues axis. We've been testing this configuration by using a
modified ScalabilityTest that simply created oodles of queues with 4
bindings each. We observed that we were able to get up around 200,000
queues before the test would halt. At this time the server was
consuming a large amount of physical memory and was entering swap.
We repeated the experiment with a cluster of two machines and achieved
basically the same result -- except the "primary" machine (the one
ScalabilityTest was interacting with) had most of its memory consumed,
and the "secondary" machine had 40% of its memory consumed by beam
processes. Would we have achieved better results by manually targeting
ScalabilityTest at the secondary machine as well?
Are we doing something wrong in our setup here? What's the maximum
number of queues that has been achieved by anyone on this list, and how
did you get there?
I see that in this email:
, Ben Hood mentions that routing complexity is O(n), where n is the
number of bindings, which doesn't bode well for our particular
application. Assuming that I'm interpreting that correctly, is there
anything we can do to tackle that problem to enable huge numbers of queues?
More information about the rabbitmq-discuss