Hi,
I have an EventStore cluster setup across 3 VM’s hosted in Azure.
The system runs along nicely, but every-so-often the projections, “$by_category”, “$by_event_type”, “$stream_by_category” and “$streams” appear to reset to 0% DONE and then slowly climb back up to 100%. As this is a test system, it can often sit idol for a few hours, or sometime a couple of days.
What would be the cause of a reset like this? Oftentimes, it can take a while to reach 100% again.
Thank you for your advice.
Are you by chance resetting the projections?
Hi Greg,
No we’re not resetting them. Our applications fail, so we login to the gES portal and see that projections are back to single figures and slowly climbing back up. What scenarios would trigger this? Our gES cluster is deployed into Azure VM’s - anything in that - that might be causing the issue?
Thanks
HI,
No we’re not resetting the projections. Looking at the logs, they might be suggesting IO issues or perhaps network… (there’s 3 Windows VM’s hosted in Azure)…
[PID:04448:033 2018.09.07 09:40:50.050 TRACE QueuedHandlerThreadP] SLOW QUEUE MSG [StorageReaderQueue #4]: ReadAllEventsForward - 1328ms. Q: 0/2.
[PID:04448:011 2018.09.07 09:40:51.285 TRACE GossipServiceBase ] Looks like node [10.23.64.16:2113] is DEAD (Gossip send failed).
[PID:04448:011 2018.09.07 09:40:51.285 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip send failed to [10.23.64.16:2113])
[PID:04448:011 2018.09.07 09:40:51.285 TRACE GossipServiceBase ] Old:
[PID:04448:011 2018.09.07 09:40:51.285 TRACE GossipServiceBase ] VND {d6ab9373-7f0a-4e8a-8e1b-bd9b86654228} [Initializing, 10.23.64.16:1113, n/a, 10.23.64.16:1112, n/a, 10.23.64.16:2113, 10.23.64.16:2114] 23959898675/28077299741/28077299741/E7960@28072929564:{6c31c15e-4b10-4138-ba48-8d387b4ae298} | 2018-09-07 09:40:50.060
[PID:04448:011 2018.09.07 09:40:51.285 TRACE GossipServiceBase ] VND {863c9768-140d-4546-9aa8-9bc1038597b5} [Master, 10.23.64.12:1113, 10.23.64.12:0, 10.23.64.12:1112, 10.23.64.12:0, 10.23.64.12:2113, 10.23.64.12:2114] 28078217448/28078235202/28078235202/E7965@28077427524:{921b6866-4004-4e87-b40c-bd6ef8380549} | 2018-09-07 09:40:51.270
[PID:04448:011 2018.09.07 09:40:51.285 TRACE GossipServiceBase ] VND {6212f2f3-0854-4dc1-b2ac-f962a0f44379} [Slave, 10.23.64.11:1113, n/a, 10.23.64.11:1112, n/a, 10.23.64.11:2113, 10.23.64.11:2114] 28078217448/28078235202/28078235202/E7965@28077427524:{921b6866-4004-4e87-b40c-bd6ef8380549} | 2018-09-07 09:40:50.534
[PID:04448:011 2018.09.07 09:40:51.285 TRACE GossipServiceBase ] New:
[PID:04448:011 2018.09.07 09:40:51.285 TRACE GossipServiceBase ] VND {d6ab9373-7f0a-4e8a-8e1b-bd9b86654228} [Initializing, 10.23.64.16:1113, n/a, 10.23.64.16:1112, n/a, 10.23.64.16:2113, 10.23.64.16:2114] 23959898675/28077299741/28077299741/E7960@28072929564:{6c31c15e-4b10-4138-ba48-8d387b4ae298} | 2018-09-07 09:40:51.285
[PID:04448:011 2018.09.07 09:40:51.285 TRACE GossipServiceBase ] VND {863c9768-140d-4546-9aa8-9bc1038597b5} [Master, 10.23.64.12:1113, 10.23.64.12:0, 10.23.64.12:1112, 10.23.64.12:0, 10.23.64.12:2113, 10.23.64.12:2114] 28078217448/28078235202/28078235202/E7965@28077427524:{921b6866-4004-4e87-b40c-bd6ef8380549} | 2018-09-07 09:40:51.270
[PID:04448:011 2018.09.07 09:40:51.285 TRACE GossipServiceBase ] VND {6212f2f3-0854-4dc1-b2ac-f962a0f44379} [Slave, 10.23.64.11:1113, n/a, 10.23.64.11:1112, n/a, 10.23.64.11:2113, 10.23.64.11:2114] 28078217448/28078235202/28078235202/E7965@28077427524:{921b6866-4004-4e87-b40c-bd6ef8380549} | 2018-09-07 09:40:50.534
[PID:04448:011 2018.09.07 09:40:51.285 TRACE GossipServiceBase ] --------------------------------------------------------------------------------
[PID:04448:021 2018.09.07 09:40:51.301 TRACE QueuedHandlerThreadP] SLOW QUEUE MSG [StorageReaderQueue #1]: ReadAllEventsForward - 1281ms. Q: 0/1.
[PID:04448:007 2018.09.07 09:40:51.363 TRACE QueuedHandlerThreadP] SLOW QUEUE MSG [StorageReaderQueue #3]: ReadAllEventsForward - 1297ms. Q: 0/2.
[PID:04448:033 2018.09.07 09:40:51.379 TRACE QueuedHandlerThreadP] SLOW QUEUE MSG [StorageReaderQueue #4]: ReadAllEventsForward - 1312ms. Q: 0/2.
[PID:04448:011 2018.09.07 09:40:51.770 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [10.23.64.16:2113])
[PID:04448:011 2018.09.07 09:40:51.770 TRACE GossipServiceBase ] Old:
[PID:04448:011 2018.09.07 09:40:51.770 TRACE GossipServiceBase ] VND {d6ab9373-7f0a-4e8a-8e1b-bd9b86654228} [Initializing, 10.23.64.16:1113, n/a, 10.23.64.16:1112, n/a, 10.23.64.16:2113, 10.23.64.16:2114] 23959898675/28077299741/28077299741/E7960@28072929564:{6c31c15e-4b10-4138-ba48-8d387b4ae298} | 2018-09-07 09:40:51.285
[PID:04448:011 2018.09.07 09:40:51.770 TRACE GossipServiceBase ] VND {863c9768-140d-4546-9aa8-9bc1038597b5} [Master, 10.23.64.12:1113, 10.23.64.12:0, 10.23.64.12:1112, 10.23.64.12:0, 10.23.64.12:2113, 10.23.64.12:2114] 28078217448/28078235202/28078235202/E7965@28077427524:{921b6866-4004-4e87-b40c-bd6ef8380549} | 2018-09-07 09:40:51.770
[PID:04448:011 2018.09.07 09:40:51.770 TRACE GossipServiceBase ] VND {6212f2f3-0854-4dc1-b2ac-f962a0f44379} [Slave, 10.23.64.11:1113, n/a, 10.23.64.11:1112, n/a, 10.23.64.11:2113, 10.23.64.11:2114] 28078217448/28078235202/28078235202/E7965@28077427524:{921b6866-4004-4e87-b40c-bd6ef8380549} | 2018-09-07 09:40:50.534
[PID:04448:011 2018.09.07 09:40:51.770 TRACE GossipServiceBase ] New:
[PID:04448:011 2018.09.07 09:40:51.770 TRACE GossipServiceBase ] VND {d6ab9373-7f0a-4e8a-8e1b-bd9b86654228} [Initializing, 10.23.64.16:1113, n/a, 10.23.64.16:1112, n/a, 10.23.64.16:2113, 10.23.64.16:2114] 23964009476/28077299741/28077299741/E7960@28072929564:{6c31c15e-4b10-4138-ba48-8d387b4ae298} | 2018-09-07 09:40:51.763
[PID:04448:011 2018.09.07 09:40:51.770 TRACE GossipServiceBase ] VND {863c9768-140d-4546-9aa8-9bc1038597b5} [Master, 10.23.64.12:1113, 10.23.64.12:0, 10.23.64.12:1112, 10.23.64.12:0, 10.23.64.12:2113, 10.23.64.12:2114] 28078217448/28078235202/28078235202/E7965@28077427524:{921b6866-4004-4e87-b40c-bd6ef8380549} | 2018-09-07 09:40:51.770
[PID:04448:011 2018.09.07 09:40:51.770 TRACE GossipServiceBase ] VND {6212f2f3-0854-4dc1-b2ac-f962a0f44379} [Slave, 10.23.64.11:1113, n/a, 10.23.64.11:1112, n/a, 10.23.64.11:2113, 10.23.64.11:2114] 28078217448/28078235202/28078235202/E7965@28077427524:{921b6866-4004-4e87-b40c-bd6ef8380549} | 2018-09-07 09:40:51.545
[PID:04448:011 2018.09.07 09:40:51.770 TRACE GossipServiceBase ] --------------------------------------------------------------------------------
[PID:04448:036 2018.09.07 09:40:52.707 TRACE QueuedHandlerThreadP] SLOW QUEUE MSG [StorageReaderQueue #2]: ReadAllEventsForward - 1343ms. Q: 0/1.
I don’t understand why the MASTER node change would trigger a projection reset. Any help much appreciated.