Eventstore Replicator stopping

thomas.mantegazzi1 · January 19, 2022, 11:07am

Hi,

I am testing evenstore replicator docker compose locally.

I have setup two docker containers for source (port 2114. Version 21.10.1) and sink (2113. Version 21.10.1). With grpc clients.

The replicator is configured like this:
replicator:
reader:
connectionString: “esdb://host.docker.internal:2114?tls=false”
protocol: grpc
sink:
connectionString: “esdb://host.docker.internal:2113?tls=false”
protocol: grpc
partitionCount: 1
bufferSize: 1000
scavenge: true
# transform:
# type: js
# config: ./config/transform.js
# bufferSize: 1000
filters: []
checkpoint:
path: “./checkpoint”

for some reason the replicator stops at around 20% of the progress, then it stops, looking at the various logs there are no particular errors.

Some additional information:

In the source database I have restored a backup from a v5 into a v21.10
Running both source and sink into Single node (Cluster size = 1) and insecure
I have EVENSTORE_RUN_PROJECTIONS=All on both source and sink
I have some persistent subscriptions on the source
There are no clients attached to either source or sink, i am just making a simple test, as I am planning a migration to eventstore cloud, so I want to make sure i am able to migrate my database.

I can not share the database, as it contains client data. But if there are any other info I can provide let me know.

Any Idea what could be the issue?

Thank you!

alexey.zimarev · January 19, 2022, 11:21am

With the information provided, it’s nearly impossible to diagnose the issue. During the development I made tens of replication cycles with millions of events, using both local instances in Docker, and local -> ES Cloud, it never failed. We also have customer reports of successful replication.

I remember an issue with a non-admin user, but that was it.

thomas.mantegazzi1 · January 19, 2022, 1:07pm

Thank you for the reply, yeah, I understand, not much to go on.
I tried adding verbose log level but nothing. although,

source-eventstore-eventstore-1  | [    1,37,12:58:23.085,VRB] Subscription 9d8624b7-ba66-4613-b5a4-bf1c4e409533 to $all seen event "C:70739772889/P:70739772889".
source-eventstore-eventstore-1  | [    1,47,12:58:23.090,INF] Live subscription 9d8624b7-ba66-4613-b5a4-bf1c4e409533 to $all running from "C:70739773079/P:0"...
source-eventstore-eventstore-1  | [    1, 9,12:58:23.098,VRB] Live subscription 9d8624b7-ba66-4613-b5a4-bf1c4e409533 to $all confirmed at "C:70739772889/P:70739772889".
source-eventstore-eventstore-1  | [    1, 9,12:58:23.099,VRB] Live subscription 9d8624b7-ba66-4613-b5a4-bf1c4e409533 to $all loading any missed events starting from "C:70739773079/P:0".
source-eventstore-eventstore-1  | [    1, 9,12:58:23.099,VRB] Subscription 9d8624b7-ba66-4613-b5a4-bf1c4e409533 to $all reading next page starting from "C:70739773079/P:0".
source-eventstore-eventstore-1  | [    1,23,12:58:23.104,VRB] Live subscription 9d8624b7-ba66-4613-b5a4-bf1c4e409533 to $all caught up at "C:70739773079/P:0" because the end of stream was reached.

Seems to say it’s done with reading. Could it be possible that the replicator shows 20% progress, but it’s actually done? Maybe because of the System projection events being skipped? or some deleted streams?

I noticed that there are quite some “IDEMPOTENT WRITE TO STREAM” in my sink log. Which to my understanding it means that system is trying to write the same event twice. Could this have have any impact?

Would you have some suggestions on where to start for finding the issue?

Are there some limitations on events?
What about System projections? Is it ok if I have them turned on on the sink? or should I keep it off?

I am trying to setup a user to try with an authenticated user. I am still trying to figure out how to do that, because I think that since I am running it with insecure=true it doesn’t have any users at all, and doesn’t require authentication.

thomas.mantegazzi1 · January 19, 2022, 1:20pm

Actually nevermind, i just tried to run my application i can clearly see that the migration was not completed. So it is definetly showing the right percentage…

I am afraid it could be that I have something messed up in the database itself…

alexey.zimarev · January 19, 2022, 1:23pm

The subscription to $all is only used to ensure that the replicator receives live updates of stream metadata, so it doesn’t replicate streams that were deleted after it has read that stream (that’s the moment when it gets and caches the stream meta). The live subscription always starts from the end, so it’s always catching up, but it doesn’t tell you anything about the current replication status.

What do you see in the UI? Are all the buffers full?

thomas.mantegazzi1 · January 19, 2022, 3:44pm

Thx for the clarification.

About the UI, and the buffers I am not entirely sure what you mean, so here is the screenshot of the replicator ui, source and sink:

Not sure if this is what you were asking for, sorry.

alexey.zimarev · January 19, 2022, 6:38pm

Maybe you can post the docker compose file here, so I can run it.

thomas.mantegazzi1 · January 19, 2022, 6:53pm

Thank you.

The replicator docker compose is:

version: '3.7'

services:

  replicator:
    container_name: repl-replicator
    image: eu.gcr.io/esc-platform-advocacy/eventstore/replicator:latest
    ports:
      - "5000:5000"
    volumes:
      - ./replicator.yml:/app/config/appsettings.yaml
#      - ./transform.js:/app/config/transform.js
#    environment:
#      REPLICATOR_DEBUG: 1
      
  prometheus:
    container_name: repl-prometheus
    image: prom/prometheus:v2.17.1
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

  grafana:
    container_name: repl-grafana
    image: grafana/grafana:6.7.2
    ports:
      - "3000:3000"
    volumes:
      - ./grafana/dashboards.yml:/etc/grafana/provisioning/dashboards/rabbitmq.yaml
      - ./grafana/datasources.yml:/etc/grafana/provisioning/datasources/prometheus.yaml
      - ./grafana/dashboards:/dashboards

While the docker compose for the eventstores is

version: '3.7'

services:

  eventstore:
    container_name: repl-test-eventstore
    image: eventstore/eventstore:21.10.1-buster-slim
    ports:
      - '2113:2113'
      - '1113:1113'
    environment:
      EVENTSTORE_INSECURE: 'true'
      EVENTSTORE_CLUSTER_SIZE: 1
      EVENTSTORE_EXT_TCP_PORT: 1113
      EVENTSTORE_HTTP_PORT: 2113
      EVENTSTORE_ENABLE_EXTERNAL_TCP: 'true'
      EVENTSTORE_RUN_PROJECTIONS: all
      EVENTSTORE_ENABLE_ATOM_PUB_OVER_HTTP: "true"
    volumes:
      - './data:/var/lib/eventstore'

  esdb:
    container_name: repl-test-esdb
    image: eventstore/eventstore:21.10.1-buster-slim
    ports:
      - '2114:2114'
      - '1114:1114'
    environment:
      EVENTSTORE_INSECURE: 'true'
      EVENTSTORE_CLUSTER_SIZE: 1
      EVENTSTORE_EXT_TCP_PORT: 1114
      EVENTSTORE_HTTP_PORT: 2114
      EVENTSTORE_ENABLE_EXTERNAL_TCP: 'true'
      EVENTSTORE_RUN_PROJECTIONS: all
      EVENTSTORE_ENABLE_ATOM_PUB_OVER_HTTP: "true"

networks:
  default:
    name: repl-test-network

In the ./data folder I have a backup of my staging environment. I think the original backup was done in eventstore V5 .

And the replicator file:

replicator:
  reader:
    connectionString: "esdb://host.docker.internal:2113?tls=false"
    protocol: grpc
  sink:
    connectionString: "esdb://host.docker.internal:2114?tls=false"
    protocol: grpc
    partitionCount: 1
    bufferSize: 100
  scavenge: true
  filters: []
  checkpoint:
    path: "./checkpoint"

I just found out that i was missing an error in the replicator logs which might actually be the cause:

{"@t":“2022-01-19T18:24:13.3430836Z”,
“@m”:“Unable to read metadata for stream “volumetemplate-b7baf02f-c5bd-4ad0-86eb-a96b1586d2b1"”,”@i":“b7df7f78”,"@l":“Warning”,"@x":“System.InvalidOperationException: Nullable object must have a value.\n at EventStore.Replicator.Esdb.Grpc.ConnectionExtensions.GetStreamMeta(EventStoreClient client, String stream) in /app/src/EventStore.Replicator.Esdb.Grpc/ConnectionExtensions.cs:line 32\n at EventStore.Replicator.Shared.Extensions.DictionaryExtensions.GetOrAddAsync[T](ConcurrentDictionary2 dict, String key, Func1 get) in /app/src/EventStore.Replicator.Shared/Extensions/DictionaryExtensions.cs:line 15\n at EventStore.Replicator.Esdb.Grpc.StreamMetaCache.GetOrAddStreamMeta(String stream, Func`2 getMeta) in /app/src/EventStore.Replicator.Esdb.Grpc/StreamMetaCache.cs:line 28”,“Stream”:“volumetemplate-b7baf02f-c5bd-4ad0-86eb-a96b1586d2b1”,“SourceContext”:“EventStore.Replicator.Esdb.Grpc.StreamMetaCache”}

It seems like the replicator is not able to read metadatas of streams. I am currently investigating this error.

As it might also be related to why I have “IDEMPOTENT WRITE TO STREAM” on the target eventstore.

alexey.zimarev · January 19, 2022, 7:05pm

IDEMPOTENT WRITE TO STREAM message indicates that the same event(s) is (are) being written again as a result of a retry on the append operation, or replicator restart with the checkpoint deviating from the latest one.

Maybe you can check what that particular stream has in its meta, and I will check the code from the stacktrace.

thomas.mantegazzi1 · January 19, 2022, 7:36pm

So I have checked the metadata of the stream (from the AdminUI clicked on metadata after selecting the stream) and I have nothing:

The only metadata that we have added to the stream events is the EventId. And in the events our metadata looks like this:

also after turning of the Replicator debug logs, i found this error message at the end, sorry i didn’t find this first.

{"@t":“2022-01-19T19:46:44.7272290Z”,"@m":“Error occured in the “ReaderContext” pipe: “Status(StatusCode=\“DeadlineExceeded\”, Detail=\”\”)"","@i":“c9acd0a5”,"@l":“Error”,"@x":“Grpc.Core.RpcException: Status(StatusCode=“DeadlineExceeded”, Detail=”")\n at EventStore.Client.Interceptors.TypedExceptionInterceptor.AsyncStreamReader1.MoveNext(CancellationToken cancellationToken)\n at Grpc.Core.AsyncStreamReaderExtensions.ReadAllAsyncCore[T](IAsyncStreamReader1 streamReader, CancellationToken cancellationToken)+MoveNext()\n at System.Linq.AsyncEnumerable.SelectEnumerableAsyncIterator2.MoveNextCore() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Select.cs:line 210\n at System.Linq.AsyncIteratorBase1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 77\n at System.Linq.AsyncIteratorBase1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 77\n at EventStore.Client.EventStoreClient.ReadInternal(ReadReq request, EventStoreClientOperationOptions operationOptions, UserCredentials userCredentials, CancellationToken cancellationToken)+MoveNext()\n at EventStore.Client.EventStoreClient.ReadInternal(ReadReq request, EventStoreClientOperationOptions operationOptions, UserCredentials userCredentials, CancellationToken cancellationToken)+MoveNext()\n at EventStore.Client.EventStoreClient.ReadAllAsync(Direction direction, Position position, Int64 maxCount, EventStoreClientOperationOptions operationOptions, Boolean resolveLinkTos, UserCredentials userCredentials, CancellationToken cancellationToken)+MoveNext()\n at EventStore.Client.EventStoreClient.ReadAllAsync(Direction direction, Position position, Int64 maxCount, EventStoreClientOperationOptions operationOptions, Boolean resolveLinkTos, UserCredentials userCredentials, CancellationToken cancellationToken)+MoveNext()\n at Ubiquitous.Metrics.Metrics.Measure[T](Func1 action, IHistogramMetric metric, ICountMetric errorCount, String[] labels, Int32 count)\n at EventStore.Replicator.Esdb.Grpc.GrpcEventReader.ReadEvents(Position fromPosition, Func2 next, CancellationToken cancellationToken) in /app/src/EventStore.Replicator.Esdb.Grpc/GrpcEventReader.cs:line 111\n at EventStore.Replicator.Read.ReaderPipe.<>c__DisplayClass1_0.<<-ctor>g__Reader|1>d.MoveNext() in /app/src/EventStore.Replicator/Read/ReaderPipe.cs:line 61\n--- End of stack trace from previous location ---\n at GreenPipes.Filters.AsyncDelegateFilter1.<>c__DisplayClass3_0.<g__SendAsync|0>d.MoveNext()\n— End of stack trace from previous location —\n at EventStore.Replicator.LoggingFilter1.Send(T context, IPipe1 next) in /app/src/EventStore.Replicator/Logging.cs:line 19",“Type”:“ReaderContext”,“Message”:“Status(StatusCode=“DeadlineExceeded”, Detail=”")",“SourceContext”:“EventStore.Replicator.LoggingFilter`1[T]”}

thomas.mantegazzi1 · January 19, 2022, 8:23pm

I finally managed to make it work, apparently it has something to do with the source client being grpc.
I found another topic on this forum saying that there were som issues with the grpc client and that switching to tcp client would fix it. So that is what I did and it worked!

So just changing the replication configuration to this:
replicator:
reader:
connectionString: “ConnectTo=tcp://admin:[email protected]:1113; HeartBeatTimeout=500; UseSslConnection=false;”
protocol: tcp
sink:
connectionString: “esdb://host.docker.internal:2114?tls=false”
protocol: grpc
partitionCount: 1
bufferSize: 1000
scavenge: true
filters: []
checkpoint:
path: “./checkpoint”

Resulted in the replication to work! I will run some more tests, but it seems to be fine now.

Thank you for the help!

alexey.zimarev · January 20, 2022, 11:55am

I think it might be related to the concurrent calls issue when using gRPC client. As I use it to read events, and read the last event from $all for metrics, the latter can fail (I’ve seen it in Eventuous), and if it’s not handled, the metrics collection breaks. I am not sure if the whole thing goes down, it might be that only metrics stop reporting and you see no progress. But, I will try to fix it. Thanks for reporting it and for debugging

thomas.mantegazzi1 · January 21, 2022, 7:22am

In my case it was completeli stopping, I could not see any new log in the replicator logs, or anything new seemed to be processing on either the eventstore instances.
Of course I might have missed something, but from what I recall nothing was happening anymore, so it was not only the monitoring in my case. Hope this can help.

Thanks again for your help!

alexey.zimarev · February 21, 2022, 3:34pm

Sorry for delayed response, but what are the last messages in the log?

thomas.mantegazzi1 · February 23, 2022, 8:44am

Hi, no worries

I managed to recreate the issue, here are the logs of the two eventstoredbs:

repl-test-esdb | [ 1,13,08:37:56.001,DBG] IDEMPOTENT WRITE TO STREAM ClientCorrelationID 27ec0ae8-3487-43d4-a762-0778a2291e83, “EventStreamId: master-e01c085f-3750-405f-b2cb-1e27deecc705, CorrelationId: 27ec0ae8-3487-43d4-a762-0778a2291e83, FirstEventNumber: 24, LastEventNumber: 24”.
repl-test-esdb | [ 1,13,08:37:56.018,DBG] IDEMPOTENT WRITE TO STREAM ClientCorrelationID 3c6dbd4e-1433-4ff2-aec2-cfe7bd490e17, “EventStreamId: master-e01c085f-3750-405f-b2cb-1e27deecc705, CorrelationId: 3c6dbd4e-1433-4ff2-aec2-cfe7bd490e17, FirstEventNumber: 25, LastEventNumber: 25”.
repl-test-esdb | [ 1,13,08:38:00.645,DBG] IDEMPOTENT WRITE TO STREAM ClientCorrelationID 490141ed-0d68-40a5-bde9-2e7039ce2a7b, “EventStreamId: master-22621603-98cc-4f93-8943-633aefaaa7a1, CorrelationId: 490141ed-0d68-40a5-bde9-2e7039ce2a7b, FirstEventNumber: 21, LastEventNumber: 21”.
repl-test-esdb | [ 1,35,08:38:18.184,DBG] “streamExistenceFilter” is flushing at 65,211,979. Diff 8,912,935 …
repl-test-esdb | [ 1,35,08:38:18.208,DBG] “streamExistenceFilter” has flushed at 65,211,979. Diff 8,912,935. Took 00:00:00.0241070
repl-test-esdb | [ 1,34,08:38:23.211,DBG] “streamExistenceFilter” took checkpoint at position: 65,211,979.
repl-test-eventstore | [ 1,49,08:38:32.737,DBG] Read All Stream Events Forward operation has expired for C:15111909651/P:0. Operation Expired at 02/23/2022 08:38:32
repl-test-esdb | [ 1,37,08:38:53.177,DBG] “streamExistenceFilter” is flushing at 71,134,387. Diff 5,922,408 …
repl-test-esdb | [ 1,37,08:38:53.181,DBG] “streamExistenceFilter” has flushed at 71,134,387. Diff 5,922,408. Took 00:00:00.0043580
repl-test-esdb | [ 1,36,08:38:58.184,DBG] “streamExistenceFilter” took checkpoint at position: 71,134,387.
repl-test-esdb | [ 1,37,08:39:28.150,DBG] “streamExistenceFilter” is flushing at 73,099,922. Diff 1,965,535 …
repl-test-esdb | [ 1,37,08:39:28.154,DBG] “streamExistenceFilter” has flushed at 73,099,922. Diff 1,965,535. Took 00:00:00.0032885

And here are the last messages of the replicator

repl-replicator | {"@t":“2022-02-23T08:39:05.5812462Z”,"@m":“Reader stopped”,"@i":“49a9146f”,“SourceContext”:“EventStore.Replicator.Read.ReaderPipe”}
repl-replicator | {"@t":“2022-02-23T08:39:05.5818926Z”,"@m":“Error occured in the “ReaderContext” pipe: “Status(StatusCode=\“DeadlineExceeded\”, Detail=\”\”)"","@i":“c9acd0a5”,"@l":“Error”,"@x":“Grpc.Core.RpcException: Status(StatusCode=“DeadlineExceeded”, Detail=”")\n at EventStore.Client.Interceptors.TypedExceptionInterceptor.AsyncStreamReader1.MoveNext(CancellationToken cancellationToken)\n at Grpc.Core.AsyncStreamReaderExtensions.ReadAllAsyncCore[T](IAsyncStreamReader1 streamReader, CancellationToken cancellationToken)+MoveNext()\n at System.Linq.AsyncEnumerable.SelectEnumerableAsyncIterator2.MoveNextCore() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Select.cs:line 210\n at System.Linq.AsyncIteratorBase1.MoveNextAsync() in //Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 77\n at System.Linq.AsyncIteratorBase1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 77\n at EventStore.Client.EventStoreClient.ReadInternal(ReadReq request, EventStoreClientOperationOptions operationOptions, UserCredentials userCredentials, CancellationToken cancellationToken)+MoveNext()\n at EventStore.Client.EventStoreClient.ReadInternal(ReadReq request, EventStoreClientOperationOptions operationOptions, UserCredentials userCredentials, CancellationToken cancellationToken)+MoveNext()\n at EventStore.Client.EventStoreClient.ReadAllAsync(Direction direction, Position position, Int64 maxCount, EventStoreClientOperationOptions operationOptions, Boolean resolveLinkTos, UserCredentials userCredentials, CancellationToken cancellationToken)+MoveNext()\n at EventStore.Client.EventStoreClient.ReadAllAsync(Direction direction, Position position, Int64 maxCount, EventStoreClientOperationOptions operationOptions, Boolean resolveLinkTos, UserCredentials userCredentials, CancellationToken cancellationToken)+MoveNext()\n at Ubiquitous.Metrics.Metrics.Measure[T](Func1 action, IHistogramMetric metric, ICountMetric errorCount, String[] labels, Int32 count)\n at EventStore.Replicator.Esdb.Grpc.GrpcEventReader.ReadEvents(Position fromPosition, Func2 next, CancellationToken cancellationToken) in /app/src/EventStore.Replicator.Esdb.Grpc/GrpcEventReader.cs:line 111\n at EventStore.Replicator.Read.ReaderPipe.<>c__DisplayClass1_0.<<-ctor>g__Reader|1>d.MoveNext() in /app/src/EventStore.Replicator/Read/ReaderPipe.cs:line 61\n--- End of stack trace from previous location ---\n at GreenPipes.Filters.AsyncDelegateFilter1.<>c__DisplayClass3_0.<g__SendAsync|0>d.MoveNext()\n— End of stack trace from previous location —\n at EventStore.Replicator.LoggingFilter1.Send(T context, IPipe1 next) in /app/src/EventStore.Replicator/Logging.cs:line 19",“Type”:“ReaderContext”,“Message”:“Status(StatusCode=“DeadlineExceeded”, Detail=”")",“SourceContext”:"EventStore.Replicator.LoggingFilter1[T]"} repl-replicator | {"@t":"2022-02-23T08:39:05.5843898Z","@m":"Error: \"Status(StatusCode=\\\"DeadlineExceeded\\\", Detail=\\\"\\\")\", will fail","@i":"02639a3a","@l":"Error","@x":"Grpc.Core.RpcException: Status(StatusCode=\"DeadlineExceeded\", Detail=\"\")\n at EventStore.Client.Interceptors.TypedExceptionInterceptor.AsyncStreamReader1.MoveNext(CancellationToken cancellationToken)\n at Grpc.Core.AsyncStreamReaderExtensions.ReadAllAsyncCore[T](IAsyncStreamReader1 streamReader, CancellationToken cancellationToken)+MoveNext()\n at System.Linq.AsyncEnumerable.SelectEnumerableAsyncIterator2.MoveNextCore() in //Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Select.cs:line 210\n at System.Linq.AsyncIteratorBase1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 77\n at System.Linq.AsyncIteratorBase1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 77\n at EventStore.Client.EventStoreClient.ReadInternal(ReadReq request, EventStoreClientOperationOptions operationOptions, UserCredentials userCredentials, CancellationToken cancellationToken)+MoveNext()\n at EventStore.Client.EventStoreClient.ReadInternal(ReadReq request, EventStoreClientOperationOptions operationOptions, UserCredentials userCredentials, CancellationToken cancellationToken)+MoveNext()\n at EventStore.Client.EventStoreClient.ReadAllAsync(Direction direction, Position position, Int64 maxCount, EventStoreClientOperationOptions operationOptions, Boolean resolveLinkTos, UserCredentials userCredentials, CancellationToken cancellationToken)+MoveNext()\n at EventStore.Client.EventStoreClient.ReadAllAsync(Direction direction, Position position, Int64 maxCount, EventStoreClientOperationOptions operationOptions, Boolean resolveLinkTos, UserCredentials userCredentials, CancellationToken cancellationToken)+MoveNext()\n at EventStore.Client.EventStoreClient.ReadAllAsync(Direction direction, Position position, Int64 maxCount, EventStoreClientOperationOptions operationOptions, Boolean resolveLinkTos, UserCredentials userCredentials, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()\n at Ubiquitous.Metrics.Metrics.Measure[T](Func1 action, IHistogramMetric metric, ICountMetric errorCount, String[] labels, Int32 count)\n at EventStore.Replicator.Esdb.Grpc.GrpcEventReader.ReadEvents(Position fromPosition, Func2 next, CancellationToken cancellationToken) in /app/src/EventStore.Replicator.Esdb.Grpc/GrpcEventReader.cs:line 111\n at EventStore.Replicator.Read.ReaderPipe.<>c__DisplayClass1_0.<<-ctor>g__Reader|1>d.MoveNext() in /app/src/EventStore.Replicator/Read/ReaderPipe.cs:line 45\n— End of stack trace from previous location —\n at GreenPipes.Filters.AsyncDelegateFilter1.<>c__DisplayClass3_0.<<Send>g__SendAsync|0>d.MoveNext()\n--- End of stack trace from previous location ---\n at EventStore.Replicator.LoggingFilter1.Send(T context, IPipe1 next) in /app/src/EventStore.Replicator/Logging.cs:line 19\n at GreenPipes.Filters.RetryFilter1.GreenPipes.IFilter.Send(TContext context, IPipe1 next)","Error":"Status(StatusCode=\"DeadlineExceeded\", Detail=\"\")","SourceContext":"EventStore.Replicator.Observers.LoggingRetryObserver"} repl-replicator | {"@t":"2022-02-23T08:39:05.5873102Z","@m":"Unable to read metadata for stream \"master-787b206f-4460-411b-8ab7-076ad1aaf7ca\"","@i":"b7df7f78","@l":"Warning","@x":"System.InvalidOperationException: Nullable object must have a value.\n at EventStore.Replicator.Esdb.Grpc.ConnectionExtensions.GetStreamMeta(EventStoreClient client, String stream) in /app/src/EventStore.Replicator.Esdb.Grpc/ConnectionExtensions.cs:line 32\n at EventStore.Replicator.Shared.Extensions.DictionaryExtensions.GetOrAddAsync[T](ConcurrentDictionary2 dict, String key, Func1 get) in /app/src/EventStore.Replicator.Shared/Extensions/DictionaryExtensions.cs:line 15\n at EventStore.Replicator.Esdb.Grpc.StreamMetaCache.GetOrAddStreamMeta(String stream, Func2 getMeta) in /app/src/EventStore.Replicator.Esdb.Grpc/StreamMetaCache.cs:line 28",“Stream”:“master-787b206f-4460-411b-8ab7-076ad1aaf7ca”,“SourceContext”:“EventStore.Replicator.Esdb.Grpc.StreamMetaCache”}

hope this helps!