gRPC connection pool for subscriptions

Clients
1

Hello there

I find that the go client blocks the current routine when calling SubscribeToStream more than 100 times without canceling the supplied context (keeping the subscriptions open). If the contexts are closed on each iteration, the call will not block.
In wireshark i see that there is created a single TCP connection for each go client.

I suspect the blocking is due to 100 being the default limit of gRPC connections in a single HTTP2 connection?
I haven’t found any hints of such a connection limit in the docs.

Can you verify that this is intended behavior (also across the other clients?), and if so suggest a strategy on how to use the client in an application with an amount of subscription in the 10^4 range?

As a consumer of the client it would be very nice, if i could avoid having to handle this limit and the client itself opened a new HTTP2 connection when needed.

I also suspect, that we have made som bad choices in creating that many subscriptions. That is something we will probably rework over time. If that is indeed an anti-pattern, maybe that should be mentioned in the docs too?

I have tested this on both:
Eventstore v24.2.0 and v22.10.2
And:
EventStore-Client-Go/v4 and EventStore-Client-Go/v3

Thanks in advance
Bjørn Wiegell

1 Like
2

Hey @bj.wiegell

When you say the client blocks the current routine, you mean the SubscribeToStream call is actually blocking? Looking at the code, SubscribeToStream returns only if it received a subscription confirmation from the server or an error happened in the meantime.

I’m suspecting the blocking part is due to not receiving that confirmation. If you see this behavior happening like clockwork after 100 calls on SubscribeToStream and your server is still available for other instances of your application (those are still able to create subscriptions) then there is substance to your claim.

To be honest, I’m not aware of such limitation. I don’t think a new connection is created when creating a subscription as the gRPC protocol should be able to multiplex multiple operations at the same time on the same connection.

Could you share a snippet on how that subscription creation and consumption is happening?

3

Yes SubscribeToStream is blocking
It’s like clockwork at 100 connections yes. I have made an example here:

Where you can see that each client can only support 100 active subscriptions. Multiple clients can go to 100 each.

4

Seems you were on the money. I did find a default limit on max in-flight streams limit:

That constant is used when creating an HTTP/2 client. Unfortunately, I didn’t find a way for us (grpc-go users) to override that settings.

5

The topic has been discussed here, where it is related to how .net has solved the problem:

I suppose you could allow a larger amount of grpc connections by emitting the setting from the esdb server. However i think the better solution for now is what the final post suggests:

create multiple grpc.ClientConn s and manually spread your RPCs across them

I would love if this was done internally in your client. That would save some overhead and complexity from having to create multiple esdb clients in our consumer code.

6

What prevents you from creating as many esdb clients as you need when your application starts? It shouldn’t change your code that much.

7

We a subscribing to ESDB dynamically when we receive incoming GraphQL subscriptions, so we don’t know the amount of subscriptions in advance.

8

Frankly, that’s a tall order you are asking here if the solution is to implement that logic internally. That ‘load-balancing’ is not that straightforward to implement especially if we are talking about connecting to a cluster of nodes and new election cycles happen.

I need to investigate how a server-side solution would look like.