Hi,
I’m running the following user defined projection:
Which has the following configuration:
The streams that this projection creates are used to replay the tenant readsides in our multi tenant instance. We have 1 eventstore per instance. Each tenant has it’s own Catchup subscription that uses a singleton connection. We replay from scratch. So during a 26 pods (k8s) deployment there are potentially 26 catchup subscriptions reading from [tenantCode].TenantEvents stream.
During such a replay it seems like events can’t be written by the projection to a [tenantCode].TenantEvents stream. Why is that? Is there some kind of ‘read-lock’? It retries 5 times and then the projection is pushed into a faulty state.
[00001,29,07:49:24.978] Checkpoint has been written for projection "TenantEvents" at sequence number 700 (current)
[00001,29,19:04:52.268] Failed to write events to stream "savills.TenantEvents". Error: "CommitTimeout"
[00001,29,19:04:55.292] Failed to write events to stream "savills.TenantEvents". Error: "CommitTimeout"
[00001,29,19:04:59.326] Failed to write events to stream "savills.TenantEvents". Error: "CommitTimeout"
[00001,29,19:05:02.345] Failed to write events to stream "esdemo.TenantEvents". Error: "CommitTimeout"
[00001,29,19:05:04.351] Failed to write events to stream "savills.TenantEvents". Error: "CommitTimeout"
[00001,29,19:05:05.356] Failed to write events to stream "esdemo.TenantEvents". Error: "CommitTimeout"
[00001,29,19:05:10.397] Failed to write events to stream "savills.TenantEvents". Error: "CommitTimeout"
[00001,29,19:05:17.437] Failed to write events to stream "savills.TenantEvents". Error: "CommitTimeout"
[00001,29,19:05:17.444] PROJECTIONS: Scheduling the writing of "$faulted" to "$projections-$master". Current status of Writer: Busy: false
[00001,29,19:05:18.996] PROJECTIONS: Finished writing events to "$projections-$master": "$faulted"
[00001,29,07:36:49.896] PROJECTIONS: Command received: 97@"$dispose"
[00001,29,07:36:49.901] PROJECTIONS: Command received: 98@"$create-and-prepare"
[00001,29,07:36:49.909] PROJECTIONS: Scheduling the writing of "$prepared" to "$projections-$master". Current status of Writer: Busy: true
[00001,29,07:36:49.918] PROJECTIONS: Finished writing events to "$projections-$master": "$prepared"
[00001,29,07:36:49.930] PROJECTIONS: Command received: 99@"$start"
[00001,29,07:36:49.936] PROJECTIONS: Scheduling the writing of "$started" to "$projections-$master". Current status of Writer: Busy: true
[00001,29,07:36:49.946] PROJECTIONS: Finished writing events to "$projections-$master": "$started"
Looking forward to hear from you.
Kind regards,