.Net grpc client connection pool

lekhasy · June 30, 2023, 10:52pm

Hi, I’m trying to build an application using Microsoft Orleans and EventStoreDb.
In the document for grpc client, I don’t see any place saying that we have a connection pool build into the client itself, so I assume I have to close the connection after I use it.
I also search the source code of the grpc client for ‘pool’ and can’t find anything.

If that was true, then the recommended approach would be open and close the connection right after I’m done using it?
If so, should we update the document to make it clear?

yves.lorphelin · July 3, 2023, 7:48am

We don’t have pooling ; it’s http/2 under the hood.

Ususally there is 1 connection per process and there is no need to close it.

In the case of an actor model you could have 1 actor holding the connection and responding to read / append requests from other actors ( assuming you have an actor instance per entity and read model in your system)

alexey.zimarev · July 6, 2023, 4:02pm

We know about Orleans-based systems where each actor reads and appends events using an injected EventStoreClient singleton. I would still recommend doing it similarly to what Yves wrote, but maybe not with one, but several stateless worker grains that would serve read calls. Appends are fine (I think) but the main issue happens when a silo starts up and starts to spawn a lot of grains at the same, triggering lots of concurrent reads. It can lead to the database being overloaded with read requests. The database doesn’t have any connection limits, so it will be like self-inflicted DDoS attack.

lekhasy · July 20, 2023, 5:39am

Hi @alexey.zimarev , @yves.lorphelin ,

That is great suggestions, I have never thought about that before.
The thing is after weeks trying to get things to work with confident, I decided to use a library instead, and Eventuous is the one I choose.
My strategy is to use AggregateStore inside a grain to load the aggregate. The grain now is just a container for synchronize commands/events to the aggregate.
The reason I need actor model is it help eliminate the need to reason about optimistic concurrency exception.

Now the new concern of self DDOS make the strategy go bankrupt.

But luckily I found this github issue from 2018:

Which have Orleans developer confirm that Orleans will not migrate existing grain when a new node join the cluster. Auto migration only happen on Akka at that time.

And in 2022, Orleans implement auto grain migration, which migrate grains automatically to rebalance the cluster:

github.com/dotnet/orleans

[Proposal] Grain migration

opened 10:20PM - 16 Apr 22 UTC

closed 07:12PM - 06 Jun 23 UTC

ReubenBond

Status: Fixed area-grains area-persistence epic

This is a proposal to implement live workload migration in Orleans. Live grain m…igration gives grains the capability to move from one host to another without dropping requests or losing in-memory state. The proposal is for a live migration *mechanism*. As for the *policy*, which determines when, where to, and which grains to migrate, that is a separate concern. There may be multiple policies which perform migration: * During graceful shutdown/upgrade * For load leveling * Programmatically, from application or library code * As a part of self-optimization ## Motivation ### Capacity utilization improvements for rolling upgrades & scale-out When a silo is added to a cluster, active grains are not moved to the new silo. When Orleans was originally released, it was common to use blue/green deployments (on Azure Cloud Services Worker Roles), so the cluster was much less dynamic than it is today where rolling deployments are fairly commonplace. Depending on the workload (number of grains, calling patterns), rolling deployments can result in grains distribution becoming heavily skewed towards the silos which were upgraded first. Therefore, some hosts may become heavily loaded while others are under-utilized and adding additional resources will not promptly reduce utilization of the existing resources. There are two problems here: * Rolling upgrades can result in a load skew * Scale-out does not promptly reduce the load on existing hosts If we have a mechanism for live migration of grains, we could move a portion of the load on heavily loaded servers to under-utilized servers. #### Why deactivation is not enough Deactivating some portion of grains on the heavily loaded servers is not enough to implement load leveling. We will explore why that is in this section. Grains are responsible for registering themselves in the grain directory as a part of the activation process. When a grain deactivates, it is also responsible for removing its own registration. Thus, if a grain calls `DeactivateOnIdle()` today, there will be a window of time where the grain is not registered anywhere, and any silo is free to try to place it somewhere. In most cases, silos will choose different targets and there will be a race for new activations on each silo to register themselves in the grain directory. Only one activation can win this race and the others will need to forward pending messages to the winning activation and update their caches. However, silos cache directory lookups and those caches aren't immediately invalidated when a grain is deactivated. We no longer require that messages carry an `ActivationId` which must match an existing activation or set the `NewActivation` flag on the message (which has also been removed). Hence, messages sent to the original silo will reactivate the grain without causing a message rejection and cache invalidation. This means that once a frequently messaged grain is activated on a silo, voluntarily deactivating the grain will not likely cause that grain to be reactivated on a different silo because caches will point to the original silo, which will happily reactivate the grain upon receiving a message. The rationale for the changes which lead to this behavior are (I believe) sound: it reduces the flurry of rerouting and multiple directory lookups caused by activating a new grain and it simplifies the process, however it has this arguable downside of making grain placement stickier than desirable in all circumstances. ### Soft-state preservation during upgrades Some customers have told us that they would like to preserve running grain state during migrations. This state tends to be *soft state*, state which can be reconstructed (perhaps loaded from a database) at a cost. If a live grain migration mechanism includes a facility to preserve soft state across a migration, then that could be utilized by these customers to preserve soft state. Additionally, it can be utilized by grain state providers to avoid needing to reload state from the database during the migration, lessening database load. ### Self-optimizing placement [Previous research (referred to as ActOp)](https://www.microsoft.com/en-us/research/publication/optimizing-distributed-actor-systems-for-dynamic-interactive-services/) into optimizing actor-based systems (in particular, Orleans) indicates that substantial benefits can be obtained by co-locating grains which frequently communicate with each other. The ActOp research involved implementing an efficient distributed graph-partitioning algorithm to incrementally optimize inter-host RPC calls by collocating the groups of grains which communicate with each other the most without violating load-levelling constraints. In the Halo Presence scenario which the paper targeted, experiments showed a decrease from around 90% of calls being remote (using random placement) to around 12% of calls being remote. As a result, 99th percentile latency decreased by more than 3x and throughput increased by 2x. The improvements would vary substantially based upon cluster size and the particulars of the workload, but they demonstrate the potential gains. ## Summary of the functionality 1. Provides a mechanism to migrate running grains from one host to another 1. Components of the grain can preserve their in-memory state using a provided mechanism. i.e, the runtime will not automatically preserve in-memory state without a component's explicit cooperation. 1. Migration preserves any pending requests and messages sent to the grain's previous address are forwarded to the grain's new address * Consider what happens when a message is forwarded to a new location which then crashes: that *should* be made to result in a prompt `SiloUnavailableException` on the call chain. This issue exists today with activation races and with clients calling gateways, but currently they are left to timeout. 1. Migration can be triggered programmatically from application or library code running in the context of the grain * This could allow specifying a migration target so that developers can implement their own placement optimizations 1. Migration can be triggered for a grain externally, from code which is not running in the context of the grain * This is how load leveling and self-optimization can be implemented. 1. Grain *activations* can indicate their migration cost when queried * This is useful for more advanced self-optimization 1. Grain *types* can opt out of automated migration * Some grain types, such as system targets, have fixed placement and are not eligible to be moved 1. Migration atomically updates the grain's address in the grain directory * This could be implemented using the state preservation mechanism. 1. Concurrently executing requests are allowed to finish before migration is initiated 1. Grains are deactivated using the regular grain deactivation process (`OnDeactivateAsync`) ## Design Lifecycle of the grain will be modified with two new hooks, for initiating and completing migration. Any process can crash at any point. That fact is omitted below for simplicity. ``` mermaid stateDiagram-v2 direction LR [*] --> Constructor Constructor --> OnActivateAsync Constructor --> OnRehydrate OnActivateAsync --> Running OnRehydrate --> OnActivateAsync OnDeactivateAsync --> OnDehydrate Running --> OnDeactivateAsync OnDeactivateAsync --> DisposeAsync OnDehydrate --> DisposeAsync DisposeAsync --> [*] ``` The grain class and components injected into the grain can enlist in a per-activation migration lifecycle hook: ``` C# // Created per-activation, on-demand public interface IGrainLifecycle { void Add(IGrainMigrationParticipant participant); void Remove(IGrainMigrationParticipant participant); } // Interface for components of a grain (or the grain class itself) to implement in order to participate in the // migration process (adding or consuming migration context) public interface IGrainMigrationParticipant { // Called on the original activation when migration is initiated, before OnDeactivateAsync. // The participant can access and update the migration context dictionary. // Methods are void, since we do not want components to perform any IO before // the grain is activated. void OnDehydrate(IDictionary<string, object> migrationContext); // Called on the new activation after a migration, before OnActivateAsync. // The participant can restore state from the migration context dictionary. // Method is void since we only want to set in-memory context here and do void OnRehydrate(IReadOnlyDictionary<string, object> migrationContext); // Future work: // Perhaps this should return an `int` or a `float` to express the migration // cost. The interface can be updated later to add this, with a default return // value indicating no/minor cost. MigrationCost GetMigrationCost(); } ``` When `OnDeactivateAsync` is called, the reason will be set to 'Dehydrating' so that app code can decide whether to persist state or not. Any developer performing a state write in `OnDeactivateAsync` either doesn't require that the state is written or doesn't realize that a process can crash at any moment and therefore they cannot rely on `OnDeactivateAsync` being called to preserve state. **Discussion:** What terminology should be used here? This migration functionality could be used to put a grain into storage (eg, cached on disk or in a database), so if it was repurposed for that then maybe a more generic terminology should be used. Eg, *dehydrating* vs *rehydrating*, *hibernating* vs *waking* (*estivate* is the reverse of hibernate, but that's a very esoteric word), *pause* vs *resume*, or *freeze* vs *thaw*. Python uses the terms *pickle* & *unpickle*, but in the context of serialization which this involves but which isn't the purpose. Personally, I favor *dehydrate* vs *rehydrate* since we are using the analogue of *grains*. A little research shows that preserving grains for long periods largely involves removing oxygen, reducing moisture, and reducing temperature. ## Enhancements / future work * [ ] Track/propagate forwarding targets in order to promptly fail requests forwarded to hosts which crash shortly after. * [ ] Prevent deactivation of grains due to loss of a directory partition by pausing request processing and salvaging the entries. See: https://github.com/dotnet/orleans/issues/2656 * It is straightforward to pause processing of new requests while the directory registration is in doubt. Pausing the processing of background tasks is not straightforward: they can capture the activation's `TaskScheduler` and schedule any work they like on it. Perhaps we deem this acceptable, but a better solution would be to have a more resilient in-memory directory. * [ ] In-memory directory replication to improve resiliency to shutdown/failure and prevent duplicate activations * [ ] Add a CheckAndSet operation to IGrainDirectory so that we can atomically update grain directory entries and avoid races Related: https://github.com/dotnet/orleans/issues/4694

github.com/dotnet/orleans

Live grain migration (#7692)

dotnet:main ← ReubenBond:feature/grain-migration/1

opened 05:47PM - 25 May 23 UTC

ReubenBond

+2141 -552

This PR implements live grain migration and resolves #7692 It contains only t…he mechanism for migration for now, no policies for automatic grain migration. The intention is to introduce those in separate PRs. The mechanism is accessible in two ways: 1. Within a grain, call `this.MigrateOnIdle()` 1. With a grain reference, call `grain.Cast<IGrainManagementExtension>().MigrateOnIdle()` Migration supports preserving in-memory state without needing to refresh from storage. To access this functionality, grains *can* implement `IGrainMigrationParticipant` and components can inject `IGrainContext`, access `ObservableLifecycle`, and use `AddGrainMigrationParticipant`. I do not intend for grains to implement `IGrainMigrationParticipant` and the relevant interfaces are optimized for performance over usability. Instead, components of the grain which know how to preserve their state should implement the interface. The relevant interfaces here are: ``` csharp public interface IGrainMigrationParticipant { /// <summary> /// Called on the original activation when migration is initiated, before <see cref="IGrainBase.OnDeactivateAsync(DeactivationReason, CancellationToken)"/>. /// The participant can access and update the dehydration context. /// </summary> /// <param name="dehydrationContext"></param> void OnDehydrate(IDehydrationContext dehydrationContext); /// <summary> /// Called on the new activation after a migration, before <see cref="IGrainBase.OnActivateAsync(CancellationToken)"/>. /// The participant can restore state from the migration context. /// </summary> /// <param name="rehydrationContext">The rehydration context.</param> void OnRehydrate(IRehydrationContext rehydrationContext); } /// <summary> /// Records the state of a grain activation for migration. /// </summary> public interface IDehydrationContext { /// <summary> /// Gets the keys in the context. /// </summary> IEnumerable<string> Keys { get; } /// <summary> /// Adds a value to the dehydration context, associated with the provided key. /// </summary> /// <param name="key">The key.</param> /// <param name="value">The value.</param> void Add(string key, ReadOnlySpan<byte> value); /// <summary> /// Adds a value to the dehydration context, associated with the provided key. /// </summary> /// <param name="key">The key.</param> /// <param name="valueWriter">A delegate used to write the provided value to the context.</param> /// <param name="value">The value to provide to <paramref name="valueWriter"/>.</param> void Add(string key, Action<object, IBufferWriter<byte>> valueWriter, object value); } /// <summary> /// Contains the state of a grain activation for migration. /// </summary> public interface IRehydrationContext { /// <summary> /// Gets the keys in the context. /// </summary> IEnumerable<string> Keys { get; } /// <summary> /// Tries to get a value from the rehydration context, associated with the provided key. /// </summary> /// <param name="key">The key.</param> /// <param name="value">The value, if present.</param> /// <returns><see langword="true"/> if the key exists in the context, otherwise <see langword="false"/>.</returns> bool TryGetValue(string key, out ReadOnlySequence<byte> value); } ``` The lifecycle of a grain therefore becomes: ``` mermaid stateDiagram-v2 direction LR [*] --> Constructor Constructor --> OnActivateAsync Constructor --> OnRehydrate OnActivateAsync --> Running OnRehydrate --> OnActivateAsync OnDeactivateAsync --> OnDehydrate Running --> OnDeactivateAsync OnDeactivateAsync --> DisposeAsync OnDehydrate --> DisposeAsync DisposeAsync --> [*] ``` Note that rehydration occurs *before* `OnActivateAsync` is called and dehydration occurs *after* `OnDeactivateAsync` is called. This PR also involves adding an additional parameter to IGrainDirectory.Register so that registrations can be updated (ideally) atomically. TODO before merge: * [x] Split PooledBuffer into a separate PR (#8453) * [x] Add additional test coverage * [x] Add default implementation for new IGrainDirectory.Register overload (make it a new overload) which calls unregister+register in sequence * [x] Add specific IGrainDirectory.Register tests * [x] Implement `IGrainMigrationParticipant` for storage providers to preserve state ###### Microsoft Reviewers: [Open in CodeFlow](https://portal.fabricbot.ms/api/codeflow?pullrequest=https://github.com/dotnet/orleans/pull/8452)

And again, luckily it preserve soft state:

Soft-state preservation during upgrades

Some customers have told us that they would like to preserve running grain state during migrations.
This state tends to be soft state, state which can be reconstructed (perhaps loaded from a database) at a cost.
If a live grain migration mechanism includes a facility to preserve soft state across a migration, then that could be utilized by these customers to preserve soft state.
Additionally, it can be utilized by grain state providers to avoid needing to reload state from the database during the migration, lessening database load.

Which mean the aggregate that I already load into memory still there after the migration.
So, what i need to do is to check for the in-memory aggregate inside the OnActivateAsync method of the grain, if the aggregate is there then I don’t need to reload it.

Personally I feel like actor model and event sourcing is born for the other, it is a beautiful combination.
It would be nice if Eventuous have an extension to support this architecture (I would if I have a bit more experience on Orleans, maybe 6 months from now).

alexey.zimarev · July 20, 2023, 6:36am

Eventuous has a cached state feature planned, and when combined with session affinity, it solves some of the problems that Actor Model promises to solve. Of course, Actor Model shines on higher load, like you said, you would never get optimistic concurrency exceptions, and all the processing will become linear.

About your impression that Actor Model and Event Sourcing play nicely together - I agree, I have the same impression, and I participated in implementing several actor-based systems using Porto.Actor and EventStoreDB. By the way, Porto.Actor persistence is event-sourced from the start, so it was quite easy to use. Unlike the Event Sourcing idea in Orleans, which is done quite badly. It was quite shocking when I found out how it works in Orleans as they persist the whole event stream as one object, and instead of granular appends you quite fast get a gargantuan blob being pushed to persistence.

Btw, I am keen to include support for Actor Model frameworks such as Orleans and Porto.Actor to Eventuous, so I am open for contributions. It’s actually quite easy to implement, I think. I like your thought that the actor is not an aggregate, I see it way too often. It’s indeed much closer to the command service concept, taking care of persistence and calling aggregate functions.