I was a bit disappointed by the result. It's known that Pulsar shards it's topics so that one topic is handled by one broker - for subscribers and publishers. However, at least at that time, it appeared as the broker looped trough all it's subscribers in a single thread, effectively enforcing a linear limit on the number of subscribers it could handle on a topic.
I'm just mentioning it, because when I tested, I could definitely not handle 100k shared consumers.
(Eventually, I ended up writing a consumer-proxy that could scale the number of subscribers indefinitely, using just one or a few subscriber-connections to the broker. Unfortunately, it's not open source.)
Interesting. That's the same path the article goes down. That is:
"Pulsar uses two individual tasks to complete the message-sending process - one for sending the message and the other for the ACK. For each message sent by the consumer, Pulsar generates these two tasks and pushes them to OrderedExecutor. To guarantee the order of messages, Pulsar always adds them to the same thread...As shown in Figure 12, Pulsar generates 200K tasks, all of which are inserted into a single thread."
It does, at the beginning, talk about other ways to address this, like "shadow topics". But it goes on to solve it instead with a combination of virtual cursors, and this:
"Therefore, to solve this problem, we used a random thread for ACK tasks instead of the same thread."
This looks like a blog post, but it's actually a writeup by a user/customer (aka case study or similar). Would recommend putting that into the post header somewhere, so as not to confuse readers who didn't navigate there by hand.
https://github.com/jgaa/pulsar-test-consumer-scale
I was a bit disappointed by the result. It's known that Pulsar shards it's topics so that one topic is handled by one broker - for subscribers and publishers. However, at least at that time, it appeared as the broker looped trough all it's subscribers in a single thread, effectively enforcing a linear limit on the number of subscribers it could handle on a topic.
I'm just mentioning it, because when I tested, I could definitely not handle 100k shared consumers.
(Eventually, I ended up writing a consumer-proxy that could scale the number of subscribers indefinitely, using just one or a few subscriber-connections to the broker. Unfortunately, it's not open source.)