intermittent cluster node crash

Ryan_Anthony · January 21, 2019, 3:43pm

I am seeing one of my nodes crash with the following log (added a few log message prior for context).
Seems to happen every few days. This is happening on a development cluster. Haven’t seen it on any other clusters (test and prod).

At this point the dev cluster is getting hit harder than the other two (we are in early stage of production).

[00001,345,22:23:47.713] SLOW BUS MSG [Worker #4 Bus]: SendOverHttp - 308ms. Handler: HttpSendService.

[00001,345,22:23:47.833] SLOW QUEUE MSG [Worker #4]: SendOverHttp - 428ms. Q: 28/31.

[00001,15,22:36:23.731] SLOW BUS MSG [MainBus]: GossipReceived - 50ms. Handler: NodeGossipService.

[00001,15,22:36:23.731] SLOW QUEUE MSG [MainQueue]: GossipReceived - 51ms. Q: 0/1.

[00001,15,22:40:03.211] SLOW BUS MSG [MainBus]: SendViewChangeProof - 154ms. Handler: ElectionsService.

[00001,15,22:40:03.211] SLOW QUEUE MSG [MainQueue]: SendViewChangeProof - 154ms. Q: 49/55.

[00001,133,22:40:04.046] SLOW BUS MSG [Worker #3 Bus]: SendOverHttp - 586ms. Handler: HttpSendService.

[00001,133,22:40:04.046] SLOW QUEUE MSG [Worker #3]: SendOverHttp - 586ms. Q: 13/56.

[00001,268,22:40:04.222] SLOW BUS MSG [Worker #4 Bus]: SendOverHttp - 756ms. Handler: HttpSendService.

[00001,268,22:40:04.222] SLOW QUEUE MSG [Worker #4]: SendOverHttp - 757ms. Q: 12/55.

[00001,200,22:40:04.231] SLOW BUS MSG [Worker #2 Bus]: SendOverHttp - 319ms. Handler: HttpSendService.

[00001,200,22:40:04.231] SLOW QUEUE MSG [Worker #2]: SendOverHttp - 319ms. Q: 37/38.

[00001,154,22:40:04.234] SLOW BUS MSG [Worker #1 Bus]: SendOverHttp - 803ms. Handler: HttpSendService.

[00001,154,22:40:04.234] SLOW QUEUE MSG [Worker #1]: SendOverHttp - 803ms. Q: 8/51.

[00001,200,22:40:04.452] SLOW BUS MSG [Worker #2 Bus]: SendOverHttp - 213ms. Handler: HttpSendService.

[00001,200,22:40:04.472] SLOW QUEUE MSG [Worker #2]: SendOverHttp - 232ms. Q: 6/7.

[00001,133,22:40:04.452] SLOW BUS MSG [Worker #3 Bus]: SendOverHttp - 241ms. Handler: HttpSendService.

[00001,133,22:40:04.472] SLOW QUEUE MSG [Worker #3]: SendOverHttp - 261ms. Q: 28/29.

[00001,82,22:40:04.572] SLOW BUS MSG [Worker #5 Bus]: SendOverHttp - 303ms. Handler: HttpSendService.

[00001,82,22:40:04.572] SLOW QUEUE MSG [Worker #5]: SendOverHttp - 304ms. Q: 4/6.

[00001,15,22:40:05.484] Looks like node [10.244.6.215:2112] is DEAD (Gossip send failed).

[00001,15,22:40:05.484] CLUSTER HAS CHANGED (gossip send failed to [10.244.6.215:2112])

[00001,15,22:40:05.484] Old:

[00001,15,22:40:05.484] VND {aaf84877-fbbe-4720-8454-bc01a0905e27} [Slave, 10.244.6.215:1112, n/a, 10.244.6.215:1113, n/a, 10.244.6.215:2112, 10.244.6.215:2113] 692567660/692585752/692585752/E4980@692549374:{e0ff3759-933a-4c63-906f-fb0ed6ce0fec} | 2019-01-19 22:40:05.297

[00001,15,22:40:05.484] VND {ae7ea0fc-f9b4-44d2-9fd3-93121b0c198b} [Master, 10.244.4.209:1112, n/a, 10.244.4.209:1113, n/a, 10.244.4.209:2112, 10.244.4.209:2113] 692567660/692585752/692585752/E4980@692549374:{e0ff3759-933a-4c63-906f-fb0ed6ce0fec} | 2019-01-19 22:40:05.477

[00001,15,22:40:05.484] VND {fcd88c89-c673-4984-8c88-37d06bd9f509} [Slave, 10.244.3.224:1112, 10.244.3.224:0, 10.244.3.224:1113, 10.244.3.224:0, 10.244.3.224:2112, 10.244.3.224:2113] 692567660/692585752/692585752/E4980@692549374:{e0ff3759-933a-4c63-906f-fb0ed6ce0fec} | 2019-01-19 22:40:05.477

[00001,15,22:40:05.484] New:

[00001,15,22:40:05.484] VND {aaf84877-fbbe-4720-8454-bc01a0905e27} [Slave, 10.244.6.215:1112, n/a, 10.244.6.215:1113, n/a, 10.244.6.215:2112, 10.244.6.215:2113] 692567660/692585752/692585752/E4980@692549374:{e0ff3759-933a-4c63-906f-fb0ed6ce0fec} | 2019-01-19 22:40:05.484

[00001,15,22:40:05.484] VND {ae7ea0fc-f9b4-44d2-9fd3-93121b0c198b} [Master, 10.244.4.209:1112, n/a, 10.244.4.209:1113, n/a, 10.244.4.209:2112, 10.244.4.209:2113] 692567660/692585752/692585752/E4980@692549374:{e0ff3759-933a-4c63-906f-fb0ed6ce0fec} | 2019-01-19 22:40:05.477

[00001,15,22:40:05.484] VND {fcd88c89-c673-4984-8c88-37d06bd9f509} [Slave, 10.244.3.224:1112, 10.244.3.224:0, 10.244.3.224:1113, 10.244.3.224:0, 10.244.3.224:2112, 10.244.3.224:2113] 692567660/692585752/692585752/E4980@692549374:{e0ff3759-933a-4c63-906f-fb0ed6ce0fec} | 2019-01-19 22:40:05.477

[00001,15,22:40:05.484] --------------------------------------------------------------------------------

[00001,15,22:40:05.892] CLUSTER HAS CHANGED (gossip received from [10.244.6.215:2112])

[00001,15,22:40:05.892] Old:

[00001,15,22:40:05.892] VND {aaf84877-fbbe-4720-8454-bc01a0905e27} [Slave, 10.244.6.215:1112, n/a, 10.244.6.215:1113, n/a, 10.244.6.215:2112, 10.244.6.215:2113] 692567660/692585752/692585752/E4980@692549374:{e0ff3759-933a-4c63-906f-fb0ed6ce0fec} | 2019-01-19 22:40:05.484

[00001,15,22:40:05.892] VND {ae7ea0fc-f9b4-44d2-9fd3-93121b0c198b} [Master, 10.244.4.209:1112, n/a, 10.244.4.209:1113, n/a, 10.244.4.209:2112, 10.244.4.209:2113] 692567660/692585752/692585752/E4980@692549374:{e0ff3759-933a-4c63-906f-fb0ed6ce0fec} | 2019-01-19 22:40:05.713

[00001,15,22:40:05.892] VND {fcd88c89-c673-4984-8c88-37d06bd9f509} [Slave, 10.244.3.224:1112, 10.244.3.224:0, 10.244.3.224:1113, 10.244.3.224:0, 10.244.3.224:2112, 10.244.3.224:2113] 692567660/692585752/692585752/E4980@692549374:{e0ff3759-933a-4c63-906f-fb0ed6ce0fec} | 2019-01-19 22:40:05.884

[00001,15,22:40:05.892] New:

[00001,15,22:40:05.892] VND {aaf84877-fbbe-4720-8454-bc01a0905e27} [Slave, 10.244.6.215:1112, n/a, 10.244.6.215:1113, n/a, 10.244.6.215:2112, 10.244.6.215:2113] 692567660/692585752/692585752/E4980@692549374:{e0ff3759-933a-4c63-906f-fb0ed6ce0fec} | 2019-01-19 22:40:05.891

[00001,15,22:40:05.892] VND {ae7ea0fc-f9b4-44d2-9fd3-93121b0c198b} [Master, 10.244.4.209:1112, n/a, 10.244.4.209:1113, n/a, 10.244.4.209:2112, 10.244.4.209:2113] 692567660/692585752/692585752/E4980@692549374:{e0ff3759-933a-4c63-906f-fb0ed6ce0fec} | 2019-01-19 22:40:05.892

[00001,15,22:40:05.892] VND {fcd88c89-c673-4984-8c88-37d06bd9f509} [Slave, 10.244.3.224:1112, 10.244.3.224:0, 10.244.3.224:1113, 10.244.3.224:0, 10.244.3.224:2112, 10.244.3.224:2113] 692567660/692585752/692585752/E4980@692549374:{e0ff3759-933a-4c63-906f-fb0ed6ce0fec} | 2019-01-19 22:40:05.892

[00001,15,22:40:05.892] --------------------------------------------------------------------------------

[00001,15,22:40:56.779] SLOW BUS MSG [MainBus]: SendOverHttp - 55ms. Handler: WideningHandler`2.

[00001,15,22:40:56.779] SLOW QUEUE MSG [MainQueue]: SendOverHttp - 56ms. Q: 8/10.

[00001,354,22:46:09.685] SLOW BUS MSG [Worker #4 Bus]: SendOverHttp - 308ms. Handler: HttpSendService.

[00001,354,22:46:09.695] SLOW QUEUE MSG [Worker #4]: SendOverHttp - 318ms. Q: 46/53.

[00001,116,22:46:09.685] SLOW BUS MSG [Worker #2 Bus]: SendOverHttp - 307ms. Handler: HttpSendService.

[00001,116,22:46:09.700] SLOW QUEUE MSG [Worker #2]: SendOverHttp - 322ms. Q: 19/26.

[00001,133,22:46:09.685] SLOW BUS MSG [Worker #3 Bus]: SendOverHttp - 306ms. Handler: HttpSendService.

[00001,133,22:46:09.700] SLOW QUEUE MSG [Worker #3]: SendOverHttp - 321ms. Q: 53/60.

[00001,294,22:46:09.687] SLOW BUS MSG [Worker #1 Bus]: SendOverHttp - 301ms. Handler: HttpSendService.

[00001,294,22:46:09.709] SLOW QUEUE MSG [Worker #1]: SendOverHttp - 322ms. Q: 37/44.

STATE CUE CARD: (? means a positive number, usually 1 or 2, * means any number)

0x0 - starting (GOOD, unless the thread is running managed code)

0x1 - running (BAD, unless it’s the gc thread)

0x2 - detached (GOOD, unless the thread is running managed code)

0x?03 - async suspended (GOOD)

0x?04 - self suspended (GOOD)

0x?05 - async suspend requested (BAD)

0x?06 - self suspend requested (BAD)

0x*07 - blocking (GOOD)

0x?08 - blocking with pending suspend (GOOD)

–thread 0x7f3d5001c050 id 0x7f3cfe6f8700 [(nil)] state 1

–thread 0x7f3d4406b0b0 id 0x7f3cfe8f9700 [(nil)] state 1

–thread 0x7f3d48036ac0 id 0x7f3cfeafa700 [(nil)] state 1

–thread 0x7f3d3c024b40 id 0x7f3cfecfb700 [(nil)] state 1

–thread 0x7f3d400140d0 id 0x7f3cfeefc700 [(nil)] state 1

–thread 0x7f3d3402e8b0 id 0x7f3cff0fd700 [(nil)] state 1

–thread 0x7f3d38018910 id 0x7f3cff2fe700 [(nil)] state 1

–thread 0x7f3d2c010ac0 id 0x7f3cff4ff700 [(nil)] state 1

–thread 0x7f3d3000ef40 id 0x7f3cff8ff700 [(nil)] state 1

–thread 0x7f3d54049c80 id 0x7f3cffefe700 [(nil)] state 1

–thread 0x7f3dc00274f0 id 0x7f3d000ff700 [(nil)] state 1

–thread 0x4011e40 id 0x7f3d004ff700 [(nil)] state 1

–thread 0x7f3db802f950 id 0x7f3d008f7700 [(nil)] state 1

–thread 0x7f3db401d4b0 id 0x7f3d00af8700 [(nil)] state 1

–thread 0x7f3dac016d20 id 0x7f3d00cf9700 [(nil)] state 1

–thread 0x7f3db008b2b0 id 0x7f3d00efa700 [(nil)] state 1

–thread 0x7f3da409df60 id 0x7f3d010fb700 [(nil)] state 1

–thread 0x7f3da8063110 id 0x7f3d012fc700 [(nil)] state 1

–thread 0x7f3d9c185440 id 0x7f3d014fd700 [(nil)] state 1

–thread 0x7f3da0038fc0 id 0x7f3d016fe700 [(nil)] state 1

–thread 0x7f3d8407c8e0 id 0x7f3d018ff700 [(nil)] state 1

–thread 0x7f3d7c052d10 id 0x7f3d01ce7700 [(nil)] state 1

–thread 0x7f3d80016550 id 0x7f3d01ee8700 [(nil)] state 1

–thread 0x7f3d740dd3f0 id 0x7f3d020e9700 [(nil)] state 1

–thread 0x7f3d7807fcf0 id 0x7f3d022ea700 [(nil)] state 1

–thread 0x7f3d6c00f9b0 id 0x7f3d024eb700 [(nil)] state 1

–thread 0x7f3d64132cd0 id 0x7f3d026ec700 [(nil)] state 1

–thread 0x7f3d680b1d50 id 0x7f3d028ed700 [(nil)] state 1

–thread 0x7f3d600a59d0 id 0x7f3d02aee700 [(nil)] state 1

–thread 0x7f3d54048f20 id 0x7f3d02cef700 [(nil)] state 1

–thread 0x7f3d4c0a4630 id 0x7f3d02ef0700 [(nil)] state 1

–thread 0x7f3d5001b0d0 id 0x7f3d030f1700 [(nil)] state 1

–thread 0x7f3d4406a550 id 0x7f3d032f2700 [(nil)] state 1

–thread 0x7f3d48035860 id 0x7f3d034f3700 [(nil)] state 1

–thread 0x7f3d3c021950 id 0x7f3d036f4700 [(nil)] state 1

–thread 0x7f3d40013190 id 0x7f3d038f5700 [(nil)] state 1

–thread 0x7f3d34028380 id 0x7f3d03af6700 [(nil)] state 1

–thread 0x7f3d380173d0 id 0x7f3d03cf7700 [(nil)] state 1

–thread 0x7f3d2c00f920 id 0x7f3d03ef8700 [(nil)] state 1

–thread 0x7f3d3000db20 id 0x7f3d040f9700 [(nil)] state 1

–thread 0x3b72da0 id 0x7f3d042fa700 [(nil)] state 1

–thread 0x7f3dc00269c0 id 0x7f3d044fb700 [(nil)] state 1

–thread 0x7f3db8059cb0 id 0x7f3d046fc700 [(nil)] state 1

–thread 0x7f3db401bf90 id 0x7f3d048fd700 [(nil)] state 1

–thread 0x7f3dac015ac0 id 0x7f3d04afe700 [(nil)] state 1

–thread 0x7f3db0086b70 id 0x7f3d04cff700 [(nil)] state 1

–thread 0x7f3da4094cf0 id 0x7f3d050ec700 [(nil)] state 1 GC INITIATOR

–thread 0x7f3da8090c30 id 0x7f3d052ed700 [(nil)] state 1

–thread 0x7f3d9c1843c0 id 0x7f3d054ee700 [(nil)] state 1

–thread 0x7f3da0038080 id 0x7f3d056ef700 [(nil)] state 1

–thread 0x7f3d8407b9a0 id 0x7f3d058f0700 [(nil)] state 1

–thread 0x7f3d7c051fb0 id 0x7f3d05af1700 [(nil)] state 1

–thread 0x7f3d80015070 id 0x7f3d05cf2700 [(nil)] state 1

–thread 0x7f3d740de400 id 0x7f3d05ef3700 [(nil)] state 1

–thread 0x7f3d7807e9c0 id 0x7f3d060f4700 [(nil)] state 1

–thread 0x7f3d6c00d580 id 0x7f3d062f5700 [(nil)] state 1

–thread 0x7f3d7000be90 id 0x7f3d064f6700 [(nil)] state 1

–thread 0x7f3d64122fa0 id 0x7f3d066f7700 [(nil)] state 1

–thread 0x7f3d680ddd30 id 0x7f3d068f8700 [(nil)] state 1

–thread 0x7f3d5c0d6d40 id 0x7f3d06af9700 [(nil)] state 1

–thread 0x7f3d6009e540 id 0x7f3d06cfa700 [(nil)] state 1

–thread 0x7f3d5404b700 id 0x7f3d06efb700 [(nil)] state 1

–thread 0x7f3d4c0a3260 id 0x7f3d070fc700 [(nil)] state 1

–thread 0x7f3d5001cb90 id 0x7f3d072fd700 [(nil)] state 1

–thread 0x7f3d44069a20 id 0x7f3d074fe700 [(nil)] state 1

–thread 0x7f3d48034920 id 0x7f3d076ff700 [(nil)] state 1

–thread 0x7f3d700430e0 id 0x7f3d07aee700 [(nil)] state 1

–thread 0x7f3d5c0e83c0 id 0x7f3d07cef700 [(nil)] state 1

–thread 0x7f3d3c020ce0 id 0x7f3d07ef0700 [(nil)] state 1

–thread 0x7f3d4000e480 id 0x7f3d080f1700 [(nil)] state 1

–thread 0x7f3d3402b560 id 0x7f3d082f2700 [(nil)] state 1

–thread 0x7f3d380162f0 id 0x7f3d084f3700 [(nil)] state 1

–thread 0x7f3d2c00e450 id 0x7f3d086f4700 [(nil)] state 1

–thread 0x7f3d30001b70 id 0x7f3d088f5700 [(nil)] state 1

–thread 0x49f4820 id 0x7f3d08af6700 [(nil)] state 1

–thread 0x7f3dc0006a60 id 0x7f3d08cf7700 [(nil)] state 1

–thread 0x7f3db401b050 id 0x7f3d08ef8700 [(nil)] state 1

–thread 0x7f3dac014b80 id 0x7f3d090f9700 [(nil)] state 1

–thread 0x7f3db0084e70 id 0x7f3d092fa700 [(nil)] state 1

–thread 0x7f3da409a970 id 0x7f3d094fb700 [(nil)] state 1

–thread 0x7f3da8062440 id 0x7f3d096fc700 [(nil)] state 1

–thread 0x7f3d9c169350 id 0x7f3d098fd700 [(nil)] state 1

–thread 0x7f3da0036b10 id 0x7f3d09afe700 [(nil)] state 1

–thread 0x7f3d8407aa60 id 0x7f3d09cff700 [(nil)] state 1

–thread 0x7f3d7c051480 id 0x7f3d0a0ff700 [(nil)] state 1

–thread 0x7f3d80013840 id 0x7f3d0a6ff700 [(nil)] state 1

–thread 0x7f3db802aa40 id 0x7f3d0aa96700 [(nil)] state 1

–thread 0x7f3d640b6480 id 0x7f3d0e1fb700 [(nil)] state 1

–thread 0x7f3d6800eae0 id 0x7f3d0e3fc700 [(nil)] state 1

–thread 0x7f3d5c0449f0 id 0x7f3d0e5fd700 [(nil)] state 1

–thread 0x7f3d54035cc0 id 0x7f3d0e7fe700 [(nil)] state 1

–thread 0x7f3d4c073490 id 0x7f3d0e9ff700 [(nil)] state 1

–thread 0x7f3d50016040 id 0x7f3d0edff700 [(nil)] state 1

–thread 0x7f3d6002b8d0 id 0x7f3d0f3ff700 [(nil)] state 1

–thread 0x7f3d44043450 id 0x7f3d0fdfe700 [(nil)] state 1

–thread 0x7f3db40090f0 id 0x7f3d0ffff700 [(nil)] state 1

–thread 0x7f3db0084340 id 0x7f3d103fe700 [(nil)] state 1

–thread 0x7f3da40999f0 id 0x7f3d105ff700 [(nil)] state 1

–thread 0x7f3da8059790 id 0x7f3d109fd700 [(nil)] state 1

–thread 0x7f3d9c168290 id 0x7f3d10bfe700 [(nil)] state 1

–thread 0x7f3da0039ac0 id 0x7f3d10dff700 [(nil)] state 1

–thread 0x7f3d34007d60 id 0x7f3d111ed700 [(nil)] state 1

–thread 0x7f3d7c008310 id 0x7f3d113ee700 [(nil)] state 1

–thread 0x7f3d8000f960 id 0x7f3d115ef700 [(nil)] state 1

–thread 0x7f3d740df520 id 0x7f3d117f0700 [(nil)] state 1

–thread 0x7f3d7807a2a0 id 0x7f3d119f1700 [(nil)] state 1

–thread 0x7f3d6c00b020 id 0x7f3d11bf2700 [(nil)] state 1

–thread 0x7f3d7000ac10 id 0x7f3d11df3700 [(nil)] state 1

–thread 0x7f3d640b5540 id 0x7f3d11ff4700 [(nil)] state 1

–thread 0x7f3d6800d620 id 0x7f3d121f5700 [(nil)] state 1

–thread 0x7f3d5c043ab0 id 0x7f3d123f6700 [(nil)] state 1

–thread 0x7f3d48032770 id 0x7f3d125f7700 [(nil)] state 1

–thread 0x7f3d3c01d6e0 id 0x7f3d127f8700 [(nil)] state 1

–thread 0x7f3d40011d80 id 0x7f3d129f9700 [(nil)] state 1

–thread 0x7f3d340086a0 id 0x7f3d12bfa700 [(nil)] state 1

–thread 0x7f3d38013d30 id 0x7f3d12dfb700 [(nil)] state 1

–thread 0x7f3d2c00c020 id 0x7f3d12ffc700 [(nil)] state 1

–thread 0x7f3d3000ac10 id 0x7f3d131fd700 [(nil)] state 1

–thread 0x3b71c80 id 0x7f3d133fe700 [(nil)] state 1

–thread 0x7f3dc00051c0 id 0x7f3d135ff700 [(nil)] state 1

–thread 0x7f3db8016550 id 0x7f3d13bff700 [(nil)] state 1

–thread 0x7f3dac0120b0 id 0x7f3d13ffe700 [(nil)] state 1

–thread 0x7f3d540382f0 id 0x7f3d141ff700 [(nil)] state 1

–thread 0x7f3d4c076140 id 0x7f3d145fe700 [(nil)] state 1

–thread 0x7f3d50015100 id 0x7f3d147ff700 [(nil)] state 1

–thread 0x7f3d44042510 id 0x7f3d14bfe700 [(nil)] state 1

–thread 0x7f3db4008210 id 0x7f3d14dff700 [(nil)] state 1

–thread 0x7f3db008dd60 id 0x7f3d151fc700 [(nil)] state 1

–thread 0x7f3da4084e70 id 0x7f3d153fd700 [(nil)] state 1

–thread 0x7f3da80652d0 id 0x7f3d155fe700 [(nil)] state 1

–thread 0x7f3d9c17c640 id 0x7f3d157ff700 [(nil)] state 1

–thread 0x7f3dac012a00 id 0x7f3d15bf9700 [(nil)] state 1

–thread 0x7f3d780790a0 id 0x7f3d15dfa700 [(nil)] state 1

–thread 0x7f3d6c009df0 id 0x7f3d15ffb700 [(nil)] state 1

–thread 0x7f3d70009a70 id 0x7f3d161fc700 [(nil)] state 1

–thread 0x7f3d640bbec0 id 0x7f3d163fd700 [(nil)] state 1

–thread 0x7f3d680105f0 id 0x7f3d165fe700 [(nil)] state 1

–thread 0x7f3d5c042550 id 0x7f3d167ff700 [(nil)] state 1

–thread 0x7f3d60040a10 id 0x7f3d16bfc700 [(nil)] state 1

–thread 0x7f3d54037020 id 0x7f3d16dfd700 [(nil)] state 1

–thread 0x7f3d4c074ae0 id 0x7f3d16ffe700 [(nil)] state 1

–thread 0x7f3d500141c0 id 0x7f3d171ff700 [(nil)] state 1

–thread 0x7f3da0047100 id 0x7f3d175fb700 [(nil)] state 1

–thread 0x7f3d44041ef0 id 0x7f3d177fc700 [(nil)] state 1

–thread 0x7f3d48031210 id 0x7f3d179fd700 [(nil)] state 1

–thread 0x7f3d3c01fb30 id 0x7f3d17bfe700 [(nil)] state 1

–thread 0x7f3d40002e00 id 0x7f3d17dff700 [(nil)] state 1

–thread 0x7f3d30009ce0 id 0x7f3d181fd700 [(nil)] state 1

–thread 0x3759390 id 0x7f3d183fe700 [(nil)] state 1

–thread 0x7f3dc000cba0 id 0x7f3d185ff700 [(nil)] state 1

–thread 0x7f3d340061c0 id 0x7f3d189fa700 [(nil)] state 1

–thread 0x7f3db8015250 id 0x7f3d18bfb700 [(nil)] state 1

–thread 0x7f3db4003920 id 0x7f3d18dfc700 [(nil)] state 1

–thread 0x7f3dac005480 id 0x7f3d18ffd700 [(nil)] state 1

–thread 0x7f3db008c830 id 0x7f3d191fe700 [(nil)] state 1

–thread 0x7f3da40836a0 id 0x7f3d193ff700 [(nil)] state 1

–thread 0x7f3da8075830 id 0x7f3d197ff700 [(nil)] state 1

–thread 0x7f3d84072e00 id 0x7f3d19bf8700 [(nil)] state 1

–thread 0x7f3d9c1892d0 id 0x7f3d19df9700 [(nil)] state 1

–thread 0x7f3da003f0b0 id 0x7f3d19ffa700 [(nil)] state 1

–thread 0x7f3d84071820 id 0x7f3d1a1fb700 [(nil)] state 1

–thread 0x7f3d7c007cc0 id 0x7f3d1a3fc700 [(nil)] state 1

–thread 0x7f3d8000bb00 id 0x7f3d1a5fd700 [(nil)] state 1

–thread 0x7f3d740dc450 id 0x7f3d1a7fe700 [(nil)] state 1

–thread 0x7f3d78077fb0 id 0x7f3d1a9ff700 [(nil)] state 1

–thread 0x7f3d38012bc0 id 0x7f3d1adff700 [(nil)] state 1

–thread 0x7f3d4000eea0 id 0x7f3d1b1ff700 [(nil)] state 1

–thread 0x7f3d2c00aa70 id 0x7f3d1b5fa700 [(nil)] state 1

–thread 0x7f3d7000ec80 id 0x7f3d1b7fb700 [(nil)] state 1

–thread 0x7f3d6800f6b0 id 0x7f3d1b9fc700 [(nil)] state 1

–thread 0x7f3d5c0414e0 id 0x7f3d1bbfd700 [(nil)] state 1

–thread 0x7f3d6002d290 id 0x7f3d1bdfe700 [(nil)] state 1

–thread 0x7f3d5401d9b0 id 0x7f3d1bfff700 [(nil)] state 1

–thread 0x7f3d4c085900 id 0x7f3d1c3fa700 [(nil)] state 1

–thread 0x7f3d500122f0 id 0x7f3d1c5fb700 [(nil)] state 1

–thread 0x7f3d44001410 id 0x7f3d1c7fc700 [(nil)] state 1

–thread 0x7f3d480301a0 id 0x7f3d1c9fd700 [(nil)] state 1

–thread 0x7f3d3000be10 id 0x7f3d1cbfe700 [(nil)] state 1

–thread 0x7f3d40001df0 id 0x7f3d1cdff700 [(nil)] state 1

–thread 0x7f3d34005080 id 0x7f3d1d1fa700 [(nil)] state 1

–thread 0x7f3d38010f00 id 0x7f3d1d3fb700 [(nil)] state 1

–thread 0x7f3d2c0071d0 id 0x7f3d1d5fc700 [(nil)] state 1

–thread 0x7f3d30007e20 id 0x7f3d1d7fd700 [(nil)] state 1

–thread 0x4eb85e0 id 0x7f3d1d9fe700 [(nil)] state 1

–thread 0x7f3dc000c070 id 0x7f3d1dbff700 [(nil)] state 1

–thread 0x7f3d640baf80 id 0x7f3d1dff9700 [(nil)] state 1

–thread 0x7f3db8018860 id 0x7f3d1e1fa700 [(nil)] state 1

–thread 0x7f3d8408d5e0 id 0x7f3d1e3fb700 [(nil)] state 1

–thread 0x7f3d7c0070b0 id 0x7f3d1e5fc700 [(nil)] state 1

–thread 0x7f3d8000cc60 id 0x7f3d1e7fd700 [(nil)] state 1

–thread 0x7f3d740e1650 id 0x7f3d1e9fe700 [(nil)] state 1

–thread 0x7f3d7807d3e0 id 0x7f3d1ebff700 [(nil)] state 1

–thread 0x7f3db4002970 id 0x7f3d1effc700 [(nil)] state 1

–thread 0x7f3dac004540 id 0x7f3d1f1fd700 [(nil)] state 1

–thread 0x7f3d7c014c50 id 0x7f3d1f3fe700 [(nil)] state 1

–thread 0x7f3d6c006df0 id 0x7f3d1f5ff700 [(nil)] state 1

–thread 0x7f3da409c050 id 0x7f3d1f9f8700 [(nil)] state 1

–thread 0x7f3da80748f0 id 0x7f3d1fbf9700 [(nil)] state 1

–thread 0x7f3d9c0bad30 id 0x7f3d1fdfa700 [(nil)] state 1

–thread 0x7f3da003dac0 id 0x7f3d1fffb700 [(nil)] state 1

–thread 0x7f3d7000da40 id 0x7f3d201fc700 [(nil)] state 1

–thread 0x7f3d64074340 id 0x7f3d203fd700 [(nil)] state 1

–thread 0x7f3d6800c490 id 0x7f3d205fe700 [(nil)] state 1

–thread 0x7f3d5c03a230 id 0x7f3d207ff700 [(nil)] state 1

–thread 0x7f3d6002c010 id 0x7f3d20bff700 [(nil)] state 1

–thread 0x7f3d54036330 id 0x7f3d20fff700 [(nil)] state 1

–thread 0x7f3d4c073a80 id 0x7f3d213f7700 [(nil)] state 1

–thread 0x7f3d50011140 id 0x7f3d215f8700 [(nil)] state 1

–thread 0x7f3d380105b0 id 0x7f3d217f9700 [(nil)] state 1

–thread 0x7f3d2c0066a0 id 0x7f3d219fa700 [(nil)] state 1

–thread 0x7f3d30006b80 id 0x7f3d21bfb700 [(nil)] state 1

–thread 0x4eb7ab0 id 0x7f3d21dfc700 [(nil)] state 1

–thread 0x7f3d3c01e620 id 0x7f3d21ffd700 [(nil)] state 1

–thread 0x7f3db80175f0 id 0x7f3d221fe700 [(nil)] state 1

–thread 0x7f3db4006cb0 id 0x7f3d223ff700 [(nil)] state 1

–thread 0x7f3dac000b50 id 0x7f3d227fe700 [(nil)] state 1

–thread 0x7f3db008a370 id 0x7f3d229ff700 [(nil)] state 1

–thread 0x7f3d44000ed0 id 0x7f3d22dfd700 [(nil)] state 1

–thread 0x7f3da004b620 id 0x7f3d22ffe700 [(nil)] state 1

–thread 0x7f3d8408c6a0 id 0x7f3d231ff700 [(nil)] state 1

–thread 0x7f3d7c00f310 id 0x7f3d235ff700 [(nil)] state 1

–thread 0x7f3d740c2280 id 0x7f3d239f9700 [(nil)] state 1

–thread 0x7f3d7807c2a0 id 0x7f3d23bfa700 [(nil)] state 1

–thread 0x7f3d6c0058f0 id 0x7f3d23dfb700 [(nil)] state 1

–thread 0x7f3d70037790 id 0x7f3d23ffc700 [(nil)] state 1

–thread 0x7f3d640731d0 id 0x7f3d241fd700 [(nil)] state 1

–thread 0x7f3d6800b550 id 0x7f3d243fe700 [(nil)] state 1

–thread 0x7f3d5c038e60 id 0x7f3d245ff700 [(nil)] state 1

–thread 0x7f3dc000ba80 id 0x7f3d249ff700 [(nil)] state 1

–thread 0x7f3d4802f260 id 0x7f3d24ffe700 [(nil)] state 1

–thread 0x7f3d2c00d4b0 id 0x7f3d251ff700 [(nil)] state 1

–thread 0x7f3d480337e0 id 0x7f3d255ff700 [(nil)] state 1

–thread 0x7f3d4000a1f0 id 0x7f3d259f6700 [(nil)] state 1

–thread 0x7f3d3401abc0 id 0x7f3d25bf7700 [(nil)] state 1

–thread 0x7f3d80003340 id 0x7f3d25df8700 [(nil)] state 1

–thread 0x7f3d50010200 id 0x7f3d25ff9700 [(nil)] state 1

–thread 0x7f3d44044f60 id 0x7f3d261fa700 [(nil)] state 1

–thread 0x7f3d4801fad0 id 0x7f3d263fb700 [(nil)] state 1

–thread 0x7f3d3800f150 id 0x7f3d265fc700 [(nil)] state 1

–thread 0x7f3d2c004450 id 0x7f3d267fd700 [(nil)] state 1

–thread 0x7f3d30004810 id 0x7f3d269fe700 [(nil)] state 1

–thread 0x3ee7540 id 0x7f3d26bff700 [(nil)] state 1

–thread 0x7f3dc0004920 id 0x7f3d26fff700 [(nil)] state 1

–thread 0x7f3d3c01c7a0 id 0x7f3d273f6700 [(nil)] state 1

–thread 0x7f3db8006620 id 0x7f3d275f7700 [(nil)] state 1

–thread 0x7f3db4006360 id 0x7f3d277f8700 [(nil)] state 1

–thread 0x7f3dac003180 id 0x7f3d279f9700 [(nil)] state 1

–thread 0x7f3db007ed10 id 0x7f3d27bfa700 [(nil)] state 1

–thread 0x7f3da409fd90 id 0x7f3d27dfb700 [(nil)] state 1

–thread 0x7f3da805b7e0 id 0x7f3d27ffc700 [(nil)] state 1

–thread 0x7f3d9c179db0 id 0x7f3d281fd700 [(nil)] state 1

–thread 0x7f3da004a430 id 0x7f3d283fe700 [(nil)] state 1

–thread 0x7f3d70026a20 id 0x7f3d285ff700 [(nil)] state 1

–thread 0x7f3d84080920 id 0x7f3d289fb700 [(nil)] state 1

–thread 0x7f3d7c00e3d0 id 0x7f3d28bfc700 [(nil)] state 1

–thread 0x7f3d80001980 id 0x7f3d28dfd700 [(nil)] state 1

–thread 0x7f3d740c2b70 id 0x7f3d28ffe700 [(nil)] state 1

–thread 0x7f3d640ca420 id 0x7f3d291ff700 [(nil)] state 1

–thread 0x7f3d780829c0 id 0x7f3d295f8700 [(nil)] state 1

–thread 0x7f3d6c003bc0 id 0x7f3d297f9700 [(nil)] state 1

–thread 0x7f3d4c076b80 id 0x7f3d299fa700 [(nil)] state 1

–thread 0x7f3d50001fa0 id 0x7f3d29bfb700 [(nil)] state 1

–thread 0x7f3d4403d4c0 id 0x7f3d29dfc700 [(nil)] state 1

–thread 0x7f3d4801e8d0 id 0x7f3d29ffd700 [(nil)] state 1

–thread 0x7f3d3c01b860 id 0x7f3d2a1fe700 [(nil)] state 1

–thread 0x7f3d40004100 id 0x7f3d2a3ff700 [(nil)] state 1

–thread 0x7f3d400092b0 id 0x7f3d2a7f4700 [(nil)] state 1

–thread 0x7f3d68002f10 id 0x7f3d2a9f5700 [(nil)] state 1

–thread 0x7f3d5c037800 id 0x7f3d2abf6700 [(nil)] state 1

–thread 0x7f3d340188f0 id 0x7f3d2adf7700 [(nil)] state 1

–thread 0x7f3d38002ee0 id 0x7f3d2aff8700 [(nil)] state 1

–thread 0x7f3d2c002f40 id 0x7f3d2b1f9700 [(nil)] state 1

–thread 0x7f3dc000df20 id 0x7f3d2b3fa700 [(nil)] state 1

–thread 0x7f3db80056e0 id 0x7f3d2b5fb700 [(nil)] state 1

–thread 0x7f3db4004fb0 id 0x7f3d2b7fc700 [(nil)] state 1

–thread 0x7f3dac0019f0 id 0x7f3d2b9fd700 [(nil)] state 1

–thread 0x7f3db0082470 id 0x7f3d2bbfe700 [(nil)] state 1

–thread 0x7f3da40a3ab0 id 0x7f3d2bdff700 [(nil)] state 1

–thread 0x7f3da8060230 id 0x7f3d583f7700 [(nil)] state 1

–thread 0x7f3d80005e70 id 0x7f3d585f8700 [(nil)] state 1

–thread 0x7f3da004bd40 id 0x7f3d587f9700 [(nil)] state 1

–thread 0x7f3d7c004f10 id 0x7f3d589fa700 [(nil)] state 1

–thread 0x7f3d78091180 id 0x7f3d58bfb700 [(nil)] state 1

–thread 0x7f3da40a0d60 id 0x7f3d58dfc700 [(nil)] state 1

–thread 0x7f3d70011090 id 0x7f3d58ffd700 [(nil)] state 1

–thread 0x7f3d64077710 id 0x7f3d591fe700 [(nil)] state 1

–thread 0x7f3d680041f0 id 0x7f3d593ff700 [(nil)] state 1

–thread 0x7f3d30003250 id 0x7f3d597f5700 [(nil)] state 1

–thread 0x7f3d38014fe0 id 0x7f3d599f6700 [(nil)] state 1

–thread 0x7f3d84075be0 id 0x7f3d59bf7700 [(nil)] state 1

–thread 0x7f3d80007500 id 0x7f3d59df8700 [(nil)] state 1

–thread 0x7f3d54024730 id 0x7f3d59ff9700 [(nil)] state 1

–thread 0x7f3d4c09f100 id 0x7f3d5a1fa700 [(nil)] state 1

–thread 0x7f3d5000a810 id 0x7f3d5a3fb700 [(nil)] state 1

–thread 0x7f3d4403ddc0 id 0x7f3d5a5fc700 [(nil)] state 1

–thread 0x7f3d4802c900 id 0x7f3d5a7fd700 [(nil)] state 1

–thread 0x7f3d3c017840 id 0x7f3d5a9fe700 [(nil)] state 1

–thread 0x7f3d40005250 id 0x7f3d5abff700 [(nil)] state 1

–thread 0x7f3d340092c0 id 0x7f3d5affd700 [(nil)] state 1

–thread 0x7f3d380019f0 id 0x7f3d5b1fe700 [(nil)] state 1

–thread 0x7f3d2c001cf0 id 0x7f3d5b3ff700 [(nil)] state 1

–thread 0x4ea0380 id 0x7f3d5b7fe700 [(nil)] state 1

–thread 0x7f3d300008e0 id 0x7f3d5b9ff700 [(nil)] state 1

–thread 0x7f3d740c6180 id 0x7f3d5bdff700 [(nil)] state 1

–thread 0x7f3d2c0008e0 id 0x7f3d883fe700 [(nil)] state 1

–thread 0x7f3d380008e0 id 0x7f3d885ff700 [(nil)] state 1

–thread 0x7f3d5c03c050 id 0x7f3d889fc700 [(nil)] state 1

–thread 0x7f3d6001fc10 id 0x7f3d88bfd700 [(nil)] state 1

–thread 0x7f3db801a710 id 0x7f3d88dfe700 [(nil)] state 1

–thread 0x7f3d600008e0 id 0x7f3d88fff700 [(nil)] state 1

–thread 0x7f3d5c0008e0 id 0x7f3d893ff700 [(nil)] state 1

–thread 0x7f3d600378a0 id 0x7f3d8977b700 [(nil)] state 1

–thread 0x7f3d680008e0 id 0x7f3d899fe700 [(nil)] state 1

–thread 0x7f3d640008e0 id 0x7f3d89bff700 [(nil)] state 1

–thread 0x7f3d700008e0 id 0x7f3d8a5ff700 [(nil)] state 1

–thread 0x7f3d3c0008e0 id 0x7f3dbc5fe700 [(nil)] state 1

–thread 0x7f3d480008e0 id 0x7f3dbc7ff700 [(nil)] state 1

–thread 0x7f3d340008e0 id 0x7f3dbcdfd700 [(nil)] state 1

–thread 0x7f3d400008e0 id 0x7f3dbcffe700 [(nil)] state 1

–thread 0x7f3d4c0008e0 id 0x7f3dbd1ff700 [(nil)] state 1

–thread 0x7f3d540008e0 id 0x7f3dbd5ff700 [(nil)] state 1

–thread 0x7f3d4c0a0750 id 0x7f3dbd942700 [(nil)] state 1

–thread 0x7f3d500008e0 id 0x7f3dbdb43700 [(nil)] state 1

–thread 0x7f3d780008e0 id 0x7f3dbe4ff700 [(nil)] state 1

–thread 0x7f3d740008e0 id 0x7f3dbe883700 [(nil)] state 1

–thread 0x7f3d800008e0 id 0x7f3dbea84700 [(nil)] state 1

–thread 0x7f3d7c0008e0 id 0x7f3dbec85700 [(nil)] state 1

–thread 0x7f3d840008e0 id 0x7f3dbee86700 [(nil)] state 1

–thread 0x7f3d74138370 id 0x7f3dbf15b700 [(nil)] state 1

–thread 0x4ca6310 id 0x7f3dbf5fa700 [(nil)] state 1

–thread 0x7f3d9c0008e0 id 0x7f3dbf7fb700 [(nil)] state 1

–thread 0x7f3da80008e0 id 0x7f3dbf9fc700 [(nil)] state 1

–thread 0x7f3da40008e0 id 0x7f3dbfbfd700 [(nil)] state 1

–thread 0x7f3db00008e0 id 0x7f3dbfdfe700 [(nil)] state 1

–thread 0x7f3db40008e0 id 0x7f3dbffff700 [(nil)] state 1

–thread 0x7f3db80008e0 id 0x7f3dc43ff700 [(nil)] state 1

–thread 0x7f3d6c0008e0 id 0x7f3dc457b700 [(nil)] state 1

–thread 0x7f3dc00008e0 id 0x7f3dc47fe700 [(nil)] state 1

Stacktrace:

at <0xffffffff>

at (wrapper managed-to-native) System.Net.Sockets.Socket.cancel_blocking_socket_operation (System.Threading.Thread) <0x0005a>

at System.Net.Sockets.SafeSocketHandle.ReleaseHandle () <0x0027b>

–thread 0x337a1d0 id 0x7f3dc7dbb740 [(nil)] state 1

WAITING for 1 threads, got 0 suspended

_wapi_connect: error looking up socket handle 0x2a

suspend_thread suspend took 556 ms, which is more than the allowed 200 ms

[00001,94,22:46:09.685] SLOW BUS MSG [Worker #5 Bus]: SendOverHttp - 328ms. Handler: HttpSendService.

[00001,94,22:46:11.259] SLOW QUEUE MSG [Worker #5]: SendOverHttp - 1901ms. Q: 18/60.

[00001,94,22:46:11.259] Dropping HTTP send message due to TTL being over. ViewChangeProof To : 10.244.4.209:2112

[00001,94,22:46:11.259] Dropping HTTP send message due to TTL being over. ViewChangeProof To : 10.244.6.215:2112

[00001,294,22:46:09.930] SLOW BUS MSG [Worker #1 Bus]: SendOverHttp - 208ms. Handler: HttpSendService.

[00001,294,22:46:11.260] SLOW QUEUE MSG [Worker #1]: SendOverHttp - 1538ms. Q: 10/13.

[00001,294,22:46:11.260] Dropping HTTP send message due to TTL being over. ViewChangeProof To : 10.244.4.209:2112

[00001,294,22:46:11.260] Dropping HTTP send message due to TTL being over. ViewChangeProof To : 10.244.6.215:2112

[00001,294,22:46:11.260] Dropping HTTP send message due to TTL being over. ViewChangeProof To : 10.244.4.209:2112

[00001,294,22:46:11.260] Dropping HTTP send message due to TTL being over. ViewChangeProof To : 10.244.6.215:2112

[00001,294,22:46:11.260] Dropping HTTP send message due to TTL being over. ViewChangeProof To : 10.244.4.209:2112

[00001,294,22:46:11.260] Dropping HTTP send message due to TTL being over. ViewChangeProof To : 10.244.6.215:2112

[00001,294,22:46:11.260] Dropping HTTP send message due to TTL being over. ViewChangeProof To : 10.244.4.209:2112

at System.Runtime.InteropServices.SafeHandle.DangerousReleaseInternal (bool) <0x00194>

at System.Runtime.InteropServices.SafeHandle.InternalDispose () <0x00027>

at System.Runtime.InteropServices.SafeHandle.Dispose (bool) <0x00023>

at System.Runtime.InteropServices.SafeHandle.Dispose () <0x00015>

at System.Net.Sockets.Socket.Dispose (bool) <0x00073>

at System.Net.Sockets.Socket.Dispose () <0x00015>

at System.Net.Sockets.Socket.Close () <0x0001b>

at System.Net.WebConnection.Close (bool) <0x00173>

at System.Net.WebConnection.ReadDone (System.IAsyncResult) <0x00137>

at System.Net.Sockets.SocketAsyncResult/c__AnonStorey0.<>m__0 (object) <0x0001d>

at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () <0x0002f>

at System.Threading.ThreadPoolWorkQueue.Dispatch () <0x001f0>

at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () <0x0000b>

at (wrapper runtime-invoke) .runtime_invoke_bool (object,intptr,intptr,intptr) <0x0005a>

Native stacktrace:

eventstored() [0x4432dd]

/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7f3dc727e390]

/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38) [0x7f3dc6cc2428]

/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a) [0x7f3dc6cc402a]

eventstored() [0x5a0739]

eventstored() [0x5a0947]

eventstored() [0x5a09f2]

eventstored() [0x596d0c]

eventstored() [0x597c5d]

[0x4077cf4b]

Debug info from gdb:

jen20 · January 21, 2019, 4:35pm

Hi Ryan,

This doesn’t look good - can you confirm a few details so we can investigate further:

which version of Event Store are you using, and what is the source of the binary (download vs build etc)
which distribution/version of Linux are you using, and which kernel version

Thanks,

James

Ryan_Anthony · January 21, 2019, 4:43pm

Source is docker image:
eventstore/eventstore:release-4.1.1-hotfix1

Running in k8s.

Ryan

Ryan_Anthony · January 22, 2019, 4:38pm

James,

Do I have any options here?

Ryan

jen20 · January 22, 2019, 4:53pm

Hi,

There’s still no kernel or distribution information - being in Docker or Kubernetes does not abstract these details, and we’ll need them to make any meaningful progress on this.

Thanks,

James

Ryan_Anthony · January 22, 2019, 6:27pm

uname -a

Linux aks-ssh-66cf68f4c7-4d45k 4.15.0-1030-azure #31~16.04.1-Ubuntu SMP Tue Oct 30 19:40:01 UTC 2018 x86_64 GNU/Linux

Ryan_Anthony · January 22, 2019, 10:04pm

Attached is the k8s statefulSet configuration.

Of note are the configurations:
EVENTSTORE_MAX_MEM_TABLE_SIZE = 250000

EVENTSTORE_STATS_PERIOD_SEC = 3600
EVENTSTORE_SKIP_DB_VERIFY = true

EVENTSTORE_GOSSIP_TIMEOUT_MS = 750

I changed EVENTSTORE_GOSSIP_TIMEOUT_MS in an attempt to help the problem, as the issue seemed to have something to do with timeouts & gossip.

es_statefulset.yml (2.85 KB)

Ryan_Anthony · January 24, 2019, 3:53pm

Node crashed again. Different cluster (production this time) Adding more lines prior to the failure.

[00001,13,05:19:24.963] ELECTIONS: STARTING ELECTIONS.

[00001,13,05:19:24.963] ELECTIONS: (V=658) SHIFT TO LEADER ELECTION.

[00001,13,05:19:24.963] ELECTIONS: (V=658) VIEWCHANGE FROM [10.244.5.194:2112, {9164fbfb-dd74-43b6-8136-7e83d73dd99b}].

[00001,13,05:19:25.245] ELECTIONS: (V=658) VIEWCHANGE FROM [10.244.4.218:2112, {1e4cc1a3-88eb-4254-bd49-24caf642ca39}].

[00001,13,05:19:25.245] ELECTIONS: (V=658) MAJORITY OF VIEWCHANGE.

[00001,13,05:19:25.245] ELECTIONS: (V=658) SHIFT TO PREPARE PHASE.

[00001,13,05:19:25.245] ELECTIONS: (V=658) PREPARE_OK FROM 10.244.5.194:2112,{9164fbfb-dd74-43b6-8136-7e83d73dd99b}.

[00001,13,05:19:25.278] ELECTIONS: (V=658) PREPARE_OK FROM 10.244.4.218:2112,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}.

[00001,13,05:19:25.278] ELECTIONS: (V=658) SHIFT TO REG_LEADER.

[00001,13,05:19:25.278] ELECTIONS: (V=658) SENDING PROPOSAL CANDIDATE: 10.244.4.218:2112,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}, ME: 10.244.5.194:2112,{9164fbfb-dd74-43b6-8136-7e83d73dd99b}.

[00001,13,05:19:25.278] ELECTIONS: (V=658) ACCEPT FROM [10.244.5.194:2112,{9164fbfb-dd74-43b6-8136-7e83d73dd99b}] M=[10.244.4.218:2112,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}]).

[00001,13,05:19:25.283] ELECTIONS: (V=658) ACCEPT FROM [10.244.4.218:2112,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}] M=[10.244.4.218:2112,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}]).

[00001,13,05:19:25.283] ELECTIONS: (V=658) DONE. ELECTED MASTER = 10.244.4.218:2112,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}. ME=10.244.5.194:2112,{9164fbfb-dd74-43b6-8136-7e83d73dd99b}.

[00001,13,05:19:26.774] ========== [10.244.5.194:2112] PRE-REPLICA STATE, WAITING FOR CHASER TO CATCH UP… MASTER IS [10.244.4.218:2112,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}]

[00001,25,05:19:26.778] Subscriptions received state change to PreReplica stopping listening.

[00001,13,05:19:26.778] Closing connection ‘master-normal’ [10.244.4.218:1112, L10.244.5.194:37076, {1eed9fc5-c9f7-47e6-b2b3-3979c3a157ac}] cleanly. Reason: Reconnecting from old master [10.244.4.218:1112] to new master: [10.244.4.218:1112].

[00001,08,05:19:26.779] Connection ‘master-normal’ ({019c1b95-bfd7-4a15-9d61-26a0dc9975af}) to [10.244.4.218:1112] established.

[00001,13,05:19:26.779] CLUSTER HAS CHANGED (gossip received from [10.244.4.218:2112])

[00001,13,05:19:26.779] Old:

[00001,13,05:19:26.779] VND {36f99263-a4be-44cc-9333-6be98f4a00e9} [Slave, 10.244.6.223:1112, n/a, 10.244.6.223:1113, n/a, 10.244.6.223:2112, 10.244.6.223:2113] 782786516/782801319/782801319/E5350@782750240:{c690d0a9-7231-4eee-a1cd-29fadb99b13e} | 2019-01-23 05:19:25.257

[00001,13,05:19:26.779] VND {9164fbfb-dd74-43b6-8136-7e83d73dd99b} [PreReplica, 10.244.5.194:1112, 10.244.5.194:0, 10.244.5.194:1113, 10.244.5.194:0, 10.244.5.194:2112, 10.244.5.194:2113] 782786516/782801319/782801319/E5350@782750240:{c690d0a9-7231-4eee-a1cd-29fadb99b13e} | 2019-01-23 05:19:26.779

[00001,13,05:19:26.779] VND {1e4cc1a3-88eb-4254-bd49-24caf642ca39} [Master, 10.244.4.218:1112, n/a, 10.244.4.218:1113, n/a, 10.244.4.218:2112, 10.244.4.218:2113] 782786516/782801319/782801319/E5350@782750240:{c690d0a9-7231-4eee-a1cd-29fadb99b13e} | 2019-01-23 05:19:25.259

[00001,13,05:19:26.780] New:

[00001,13,05:19:26.780] VND {36f99263-a4be-44cc-9333-6be98f4a00e9} [Slave, 10.244.6.223:1112, n/a, 10.244.6.223:1113, n/a, 10.244.6.223:2112, 10.244.6.223:2113] 782786516/782801547/782801547/E5351@782801319:{9e6d759b-65dd-47b2-a70f-3cfabb1dbc7d} | 2019-01-23 05:19:25.376

[00001,13,05:19:26.780] VND {9164fbfb-dd74-43b6-8136-7e83d73dd99b} [PreReplica, 10.244.5.194:1112, 10.244.5.194:0, 10.244.5.194:1113, 10.244.5.194:0, 10.244.5.194:2112, 10.244.5.194:2113] 782786516/782801319/782801319/E5350@782750240:{c690d0a9-7231-4eee-a1cd-29fadb99b13e} | 2019-01-23 05:19:26.779

[00001,13,05:19:26.780] VND {1e4cc1a3-88eb-4254-bd49-24caf642ca39} [Master, 10.244.4.218:1112, n/a, 10.244.4.218:1113, n/a, 10.244.4.218:2112, 10.244.4.218:2113] 782786516/782801547/782801547/E5351@782801319:{9e6d759b-65dd-47b2-a70f-3cfabb1dbc7d} | 2019-01-23 05:19:25.754

[00001,13,05:19:27.091] --------------------------------------------------------------------------------

[00001,13,05:19:27.091] SLOW BUS MSG [MainBus]: GossipReceived - 312ms. Handler: NodeGossipService.

[00001,13,05:19:27.091] SLOW QUEUE MSG [MainQueue]: GossipReceived - 312ms. Q: 12/21.

[00001,13,05:19:27.092] Subscribing at LogPosition: 782801319 (0x2EA899A7) to MASTER [10.244.4.218:1112, {1e4cc1a3-88eb-4254-bd49-24caf642ca39}] as replica with SubscriptionId: {179bbc5c-86b9-44a0-b444-f693b73af6be}, ConnectionId: {019c1b95-bfd7-4a15-9d61-26a0dc9975af}, LocalEndPoint: [10.244.5.194:49260], Epochs:

E5350@782750240:{c690d0a9-7231-4eee-a1cd-29fadb99b13e}

E5349@782596837:{c5a9bdd3-3c7d-46d0-b388-f8fadb904a04}

E5348@782545638:{33982e78-311c-4f6d-9a27-4c681c168472}

E5347@782545410:{755e5c90-dcbe-47c9-a72c-3c1736bb3793}

E5346@782443075:{492631ab-c3ac-46ff-8564-74f2ba99d48f}

E5345@782442847:{718f5839-6c43-4d93-8055-451d95873e61}

E5344@782442619:{bba3b76c-d157-40de-a9b9-5290d0d1761a}

E5343@782427518:{8e563a79-8334-4be5-8d16-6e32ddec1e63}

E5342@782427290:{98b04c50-29c7-4d40-bd0c-c6417f50bf97}

E5341@782427062:{604ec455-0491-44b2-96d4-ee408f21fa3d}

E5340@782426834:{cddf469a-760c-45ba-b978-1f691a0f078e}

E5339@782426606:{bf17640c-77db-4a33-aca9-532e909d1c6e}

E5338@782426378:{4ad0b740-a7b2-4e2c-83a3-8ba376a44aa1}

E5337@782426150:{4a28a6db-9607-4fb8-8eb6-2fefd5a6a3ac}

E5336@782425922:{c849f7eb-58b8-4a4f-be2e-80e582c07f2f}

E5335@782407581:{f1c5eee2-7ff1-4349-af80-cef82b0e7b15}

E5334@782407353:{b23874fe-80ba-47df-93ad-4de62296a8f9}

E5333@782407125:{8a3f21cc-99a2-442c-a63e-504913707bb6}

E5332@782406897:{7f4ae8e7-8d43-47e8-8b6c-ac672e285598}

E5331@782406669:{664f15bc-ae6a-47e8-abfa-523e0f011438}…

.

[00001,10,05:19:27.115] === SUBSCRIBED to [10.244.4.218:1112,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}] at 782801319 (0x2EA899A7). SubscriptionId: {179bbc5c-86b9-44a0-b444-f693b73af6be}.

[00001,13,05:19:27.119] ========== [10.244.5.194:2112] IS CATCHING UP… MASTER IS [10.244.4.218:2112,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}]

[00001,57,05:19:27.122] Subscriptions received state change to CatchingUp stopping listening.

[00001,13,05:19:27.381] ========== [10.244.5.194:2112] CLONE ASSIGNMENT RECEIVED FROM [10.244.4.218:1112,n/a,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}].

[00001,13,05:19:27.381] ========== [10.244.5.194:2112] IS CLONE… MASTER IS [10.244.4.218:2112,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}]

[00001,13,05:19:27.381] ========== [10.244.5.194:2112] SLAVE ASSIGNMENT RECEIVED FROM [10.244.4.218:1112,n/a,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}].

[00001,13,05:19:27.381] ========== [10.244.5.194:2112] IS SLAVE… MASTER IS [10.244.4.218:2112,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}]

[00001,13,05:19:27.484] SLOW QUEUE MSG [MainQueue]: SlaveAssignment - 102ms. Q: 1/3.

[00001,44,05:19:27.484] Subscriptions received state change to Clone stopping listening.

[00001,44,05:19:27.484] Subscriptions received state change to Slave stopping listening.

[00001,10,05:19:28.082] SLOW QUEUE MSG [StorageWriterQueue]: DataChunkBulk - 680ms. Q: 1/2.

[00001,14,05:19:28.086] === Update Last Epoch E5351@782801319:{9e6d759b-65dd-47b2-a70f-3cfabb1dbc7d} (previous epoch at 782750240).

[00001,07,05:19:29.323] SLOW BUS MSG [Worker #2 Bus]: SendOverHttp - 205ms. Handler: HttpSendService.

[00001,07,05:19:29.323] SLOW QUEUE MSG [Worker #2]: SendOverHttp - 206ms. Q: 0/0.

[00001,75,05:19:30.078] SLOW BUS MSG [Worker #2 Bus]: SendOverHttp - 613ms. Handler: HttpSendService.

[00001,75,05:19:30.765] SLOW QUEUE MSG [Worker #2]: SendOverHttp - 1300ms. Q: 4/11.

[00001,75,05:19:30.765] Dropping HTTP send message due to TTL being over. ViewChangeProof To : 10.244.6.223:2112

[00001,75,05:19:30.765] Dropping HTTP send message due to TTL being over. ViewChangeProof To : 10.244.4.218:2112

[00001,75,05:19:30.765] Dropping HTTP send message due to TTL being over. ViewChangeProof To : 10.244.6.223:2112

[00001,75,05:19:30.765] Dropping HTTP send message due to TTL being over. ViewChangeProof To : 10.244.4.218:2112

[00001,23,05:19:30.777] ES TcpConnection closed [05:19:30.777: N10.244.4.218:1112, L10.244.5.194:49260, {019c1b95-bfd7-4a15-9d61-26a0dc9975af}]:Received bytes: 523, Sent bytes: 717

[00001,23,05:19:30.777] ES TcpConnection closed [05:19:30.777: N10.244.4.218:1112, L10.244.5.194:49260, {019c1b95-bfd7-4a15-9d61-26a0dc9975af}]:Send calls: 3, callbacks: 3

[00001,23,05:19:30.777] ES TcpConnection closed [05:19:30.778: N10.244.4.218:1112, L10.244.5.194:49260, {019c1b95-bfd7-4a15-9d61-26a0dc9975af}]:Receive calls: 5, callbacks: 5

[00001,13,05:19:30.786] Looks like master [10.244.4.218:2112, {1e4cc1a3-88eb-4254-bd49-24caf642ca39}] is DEAD (Gossip send failed), though we wait for TCP to decide.

[00001,13,05:19:30.844] SLOW BUS MSG [MainBus]: GossipSendFailed - 58ms. Handler: NodeGossipService.

[00001,13,05:19:30.844] SLOW QUEUE MSG [MainQueue]: GossipSendFailed - 58ms. Q: 3/3.

[00001,23,05:19:30.844] ES TcpConnection closed [05:19:30.845: N10.244.4.218:1112, L10.244.5.194:49260, {019c1b95-bfd7-4a15-9d61-26a0dc9975af}]:Close reason: [Success] Socket closed

[00001,23,05:19:31.082] Connection ‘master-normal’ [10.244.4.218:1112, {019c1b95-bfd7-4a15-9d61-26a0dc9975af}] closed: Success.

[00001,13,05:19:31.082] Looks like node [10.244.4.218:1112] is DEAD (TCP connection lost).

[00001,13,05:19:31.082] CLUSTER HAS CHANGED (TCP connection lost to [10.244.4.218:1112])

[00001,13,05:19:31.082] Old:

[00001,13,05:19:31.082] VND {36f99263-a4be-44cc-9333-6be98f4a00e9} [Slave, 10.244.6.223:1112, n/a, 10.244.6.223:1113, n/a, 10.244.6.223:2112, 10.244.6.223:2113] 782786516/782801547/782801547/E5351@782801319:{9e6d759b-65dd-47b2-a70f-3cfabb1dbc7d} | 2019-01-23 05:19:30.406

[00001,13,05:19:31.082] VND {9164fbfb-dd74-43b6-8136-7e83d73dd99b} [Slave, 10.244.5.194:1112, 10.244.5.194:0, 10.244.5.194:1113, 10.244.5.194:0, 10.244.5.194:2112, 10.244.5.194:2113] 782786516/782801547/782801547/E5351@782801319:{9e6d759b-65dd-47b2-a70f-3cfabb1dbc7d} | 2019-01-23 05:19:30.844

[00001,13,05:19:31.082] VND {1e4cc1a3-88eb-4254-bd49-24caf642ca39} [Master, 10.244.4.218:1112, n/a, 10.244.4.218:1113, n/a, 10.244.4.218:2112, 10.244.4.218:2113] 782786516/782801547/782801547/E5351@782801319:{9e6d759b-65dd-47b2-a70f-3cfabb1dbc7d} | 2019-01-23 05:19:30.844

[00001,13,05:19:31.346] New:

[00001,13,05:19:31.348] VND {36f99263-a4be-44cc-9333-6be98f4a00e9} [Slave, 10.244.6.223:1112, n/a, 10.244.6.223:1113, n/a, 10.244.6.223:2112, 10.244.6.223:2113] 782786516/782801547/782801547/E5351@782801319:{9e6d759b-65dd-47b2-a70f-3cfabb1dbc7d} | 2019-01-23 05:19:30.406

[00001,13,05:19:31.348] VND {9164fbfb-dd74-43b6-8136-7e83d73dd99b} [Slave, 10.244.5.194:1112, 10.244.5.194:0, 10.244.5.194:1113, 10.244.5.194:0, 10.244.5.194:2112, 10.244.5.194:2113] 782786516/782801547/782801547/E5351@782801319:{9e6d759b-65dd-47b2-a70f-3cfabb1dbc7d} | 2019-01-23 05:19:30.844

[00001,13,05:19:31.348] VND {1e4cc1a3-88eb-4254-bd49-24caf642ca39} [Master, 10.244.4.218:1112, n/a, 10.244.4.218:1113, n/a, 10.244.4.218:2112, 10.244.4.218:2113] 782786516/782801547/782801547/E5351@782801319:{9e6d759b-65dd-47b2-a70f-3cfabb1dbc7d} | 2019-01-23 05:19:31.082

[00001,13,05:19:31.348] --------------------------------------------------------------------------------

[00001,13,05:19:31.348] SLOW BUS MSG [MainBus]: VNodeConnectionLost - 266ms. Handler: NodeGossipService.

[00001,13,05:19:31.348] SLOW QUEUE MSG [MainQueue]: VNodeConnectionLost - 266ms. Q: 0/7.

[00001,13,05:19:31.350] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {1e4cc1a3-88eb-4254-bd49-24caf642ca39}, InternalTcp: 10.244.4.218:1112, InternalSecureTcp: , ExternalTcp: 10.244.4.218:1113, ExternalSecureTcp: , InternalHttp: 10.244.4.218:2112, ExternalHttp: 10.244.4.218:2113].

[00001,13,05:19:31.350] ELECTIONS: STARTING ELECTIONS.

[00001,13,05:19:31.350] ELECTIONS: (V=659) SHIFT TO LEADER ELECTION.

[00001,13,05:19:31.350] ELECTIONS: (V=659) VIEWCHANGE FROM [10.244.5.194:2112, {9164fbfb-dd74-43b6-8136-7e83d73dd99b}].

[00001,13,05:19:31.375] CLUSTER HAS CHANGED (gossip received from [10.244.4.218:2112])

[00001,13,05:19:31.391] Old:

[00001,13,05:19:31.391] VND {36f99263-a4be-44cc-9333-6be98f4a00e9} [Slave, 10.244.6.223:1112, n/a, 10.244.6.223:1113, n/a, 10.244.6.223:2112, 10.244.6.223:2113] 782786516/782801547/782801547/E5351@782801319:{9e6d759b-65dd-47b2-a70f-3cfabb1dbc7d} | 2019-01-23 05:19:30.406

[00001,13,05:19:31.391] VND {9164fbfb-dd74-43b6-8136-7e83d73dd99b} [Slave, 10.244.5.194:1112, 10.244.5.194:0, 10.244.5.194:1113, 10.244.5.194:0, 10.244.5.194:2112, 10.244.5.194:2113] 782786516/782801547/782801547/E5351@782801319:{9e6d759b-65dd-47b2-a70f-3cfabb1dbc7d} | 2019-01-23 05:19:31.350

[00001,13,05:19:31.391] VND {1e4cc1a3-88eb-4254-bd49-24caf642ca39} [Master, 10.244.4.218:1112, n/a, 10.244.4.218:1113, n/a, 10.244.4.218:2112, 10.244.4.218:2113] 782786516/782801547/782801547/E5351@782801319:{9e6d759b-65dd-47b2-a70f-3cfabb1dbc7d} | 2019-01-23 05:19:31.082

[00001,13,05:19:31.391] New:

[00001,13,05:19:31.391] VND {36f99263-a4be-44cc-9333-6be98f4a00e9} [Slave, 10.244.6.223:1112, n/a, 10.244.6.223:1113, n/a, 10.244.6.223:2112, 10.244.6.223:2113] 782786516/782801547/782801547/E5351@782801319:{9e6d759b-65dd-47b2-a70f-3cfabb1dbc7d} | 2019-01-23 05:19:30.788

[00001,13,05:19:31.391] VND {9164fbfb-dd74-43b6-8136-7e83d73dd99b} [Slave, 10.244.5.194:1112, 10.244.5.194:0, 10.244.5.194:1113, 10.244.5.194:0, 10.244.5.194:2112, 10.244.5.194:2113] 782786516/782801547/782801547/E5351@782801319:{9e6d759b-65dd-47b2-a70f-3cfabb1dbc7d} | 2019-01-23 05:19:31.375

[00001,13,05:19:31.392] VND {1e4cc1a3-88eb-4254-bd49-24caf642ca39} [Master, 10.244.4.218:1112, n/a, 10.244.4.218:1113, n/a, 10.244.4.218:2112, 10.244.4.218:2113] 782786516/782801547/782801547/E5351@782801319:{9e6d759b-65dd-47b2-a70f-3cfabb1dbc7d} | 2019-01-23 05:19:31.370

[00001,13,05:19:31.392] --------------------------------------------------------------------------------

[00001,13,05:19:31.734] ELECTIONS: (IV=659) VIEWCHANGEPROOF FROM [10.244.6.223:2112, {36f99263-a4be-44cc-9333-6be98f4a00e9}]. JUMPING TO NON-LEADER STATE.

[00001,13,05:19:31.734] ELECTIONS: (V=659) SHIFT TO REG_NONLEADER.

[00001,13,05:19:31.859] ========== [10.244.5.194:2112] PRE-REPLICA STATE, WAITING FOR CHASER TO CATCH UP… MASTER IS [10.244.4.218:2112,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}]

[00001,13,05:19:31.859] Closing connection ‘master-normal’ [10.244.4.218:1112, L10.244.5.194:49260, {019c1b95-bfd7-4a15-9d61-26a0dc9975af}] cleanly. Reason: Reconnecting from old master [10.244.4.218:1112] to new master: [10.244.4.218:1112].

[00001,54,05:19:31.859] Subscriptions received state change to PreReplica stopping listening.

[00001,13,05:19:31.861] Subscribing at LogPosition: 782801547 (0x2EA89A8B) to MASTER [10.244.4.218:1112, {1e4cc1a3-88eb-4254-bd49-24caf642ca39}] as replica with SubscriptionId: {b0d39851-06f8-47f4-a0d4-1d77b6413246}, ConnectionId: {dd7d4b83-d9ad-49f0-bd9f-f7c147418419}, LocalEndPoint: [], Epochs:

E5351@782801319:{9e6d759b-65dd-47b2-a70f-3cfabb1dbc7d}

E5350@782750240:{c690d0a9-7231-4eee-a1cd-29fadb99b13e}

E5349@782596837:{c5a9bdd3-3c7d-46d0-b388-f8fadb904a04}

E5348@782545638:{33982e78-311c-4f6d-9a27-4c681c168472}

E5347@782545410:{755e5c90-dcbe-47c9-a72c-3c1736bb3793}

E5346@782443075:{492631ab-c3ac-46ff-8564-74f2ba99d48f}

E5345@782442847:{718f5839-6c43-4d93-8055-451d95873e61}

E5344@782442619:{bba3b76c-d157-40de-a9b9-5290d0d1761a}

E5343@782427518:{8e563a79-8334-4be5-8d16-6e32ddec1e63}

E5342@782427290:{98b04c50-29c7-4d40-bd0c-c6417f50bf97}

E5341@782427062:{604ec455-0491-44b2-96d4-ee408f21fa3d}

E5340@782426834:{cddf469a-760c-45ba-b978-1f691a0f078e}

E5339@782426606:{bf17640c-77db-4a33-aca9-532e909d1c6e}

E5338@782426378:{4ad0b740-a7b2-4e2c-83a3-8ba376a44aa1}

E5337@782426150:{4a28a6db-9607-4fb8-8eb6-2fefd5a6a3ac}

E5336@782425922:{c849f7eb-58b8-4a4f-be2e-80e582c07f2f}

E5335@782407581:{f1c5eee2-7ff1-4349-af80-cef82b0e7b15}

E5334@782407353:{b23874fe-80ba-47df-93ad-4de62296a8f9}

E5333@782407125:{8a3f21cc-99a2-442c-a63e-504913707bb6}

E5332@782406897:{7f4ae8e7-8d43-47e8-8b6c-ac672e285598}…

.

[00001,90,05:19:31.868] Connection ‘master-normal’ ({dd7d4b83-d9ad-49f0-bd9f-f7c147418419}) to [10.244.4.218:1112] established.

[00001,13,05:19:31.885] ========== [10.244.5.194:2112] IS CATCHING UP… MASTER IS [10.244.4.218:2112,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}]

[00001,10,05:19:31.885] === SUBSCRIBED to [10.244.4.218:1112,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}] at 782801547 (0x2EA89A8B). SubscriptionId: {b0d39851-06f8-47f4-a0d4-1d77b6413246}.

[00001,39,05:19:31.885] Subscriptions received state change to CatchingUp stopping listening.

[00001,13,05:19:31.900] ========== [10.244.5.194:2112] CLONE ASSIGNMENT RECEIVED FROM [10.244.4.218:1112,n/a,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}].

[00001,13,05:19:31.900] ========== [10.244.5.194:2112] IS CLONE… MASTER IS [10.244.4.218:2112,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}]

[00001,13,05:19:31.900] ========== [10.244.5.194:2112] SLAVE ASSIGNMENT RECEIVED FROM [10.244.4.218:1112,n/a,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}].

[00001,13,05:19:31.900] ========== [10.244.5.194:2112] IS SLAVE… MASTER IS [10.244.4.218:2112,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}]

[00001,24,05:19:31.901] Subscriptions received state change to Clone stopping listening.

[00001,24,05:19:31.901] Subscriptions received state change to Slave stopping listening.

[00001,14,05:19:31.925] === Update Last Epoch E5352@782801547:{5bd8dbab-1167-49b1-9871-944ddc82a71b} (previous epoch at 782801319).

[00001,13,05:19:32.353] ELECTIONS: (V=659) TIMED OUT! (S=NonLeader, M=).

[00001,13,05:19:32.353] ELECTIONS: (V=660) SHIFT TO LEADER ELECTION.

[00001,13,05:19:32.353] ELECTIONS: (V=660) VIEWCHANGE FROM [10.244.5.194:2112, {9164fbfb-dd74-43b6-8136-7e83d73dd99b}].

[00001,13,05:19:32.442] ELECTIONS: (V=660) VIEWCHANGE FROM [10.244.4.218:2112, {1e4cc1a3-88eb-4254-bd49-24caf642ca39}].

[00001,13,05:19:32.442] ELECTIONS: (V=660) MAJORITY OF VIEWCHANGE.

[00001,13,05:19:32.442] ELECTIONS: (V=660) VIEWCHANGE FROM [10.244.6.223:2112, {36f99263-a4be-44cc-9333-6be98f4a00e9}].

[00001,13,05:19:32.442] ELECTIONS: (V=660) PREPARE FROM [10.244.6.223:2112, {36f99263-a4be-44cc-9333-6be98f4a00e9}].

[00001,13,05:19:32.442] ELECTIONS: (V=660) SHIFT TO REG_NONLEADER.

[00001,13,05:19:32.442] CLUSTER HAS CHANGED (gossip received from [10.244.4.218:2112])

[00001,13,05:19:32.442] Old:

[00001,13,05:19:32.442] VND {36f99263-a4be-44cc-9333-6be98f4a00e9} [Slave, 10.244.6.223:1112, n/a, 10.244.6.223:1113, n/a, 10.244.6.223:2112, 10.244.6.223:2113] 782786516/782801775/782801547/E5351@782801319:{9e6d759b-65dd-47b2-a70f-3cfabb1dbc7d} | 2019-01-23 05:19:31.410

[00001,13,05:19:32.443] VND {9164fbfb-dd74-43b6-8136-7e83d73dd99b} [Slave, 10.244.5.194:1112, 10.244.5.194:0, 10.244.5.194:1113, 10.244.5.194:0, 10.244.5.194:2112, 10.244.5.194:2113] 782786516/782801775/782801775/E5352@782801547:{5bd8dbab-1167-49b1-9871-944ddc82a71b} | 2019-01-23 05:19:32.353

[00001,13,05:19:32.443] VND {1e4cc1a3-88eb-4254-bd49-24caf642ca39} [Master, 10.244.4.218:1112, n/a, 10.244.4.218:1113, n/a, 10.244.4.218:2112, 10.244.4.218:2113] 782786516/782801547/782801547/E5351@782801319:{9e6d759b-65dd-47b2-a70f-3cfabb1dbc7d} | 2019-01-23 05:19:31.868

[00001,13,05:19:32.443] New:

[00001,13,05:19:32.443] VND {36f99263-a4be-44cc-9333-6be98f4a00e9} [Slave, 10.244.6.223:1112, n/a, 10.244.6.223:1113, n/a, 10.244.6.223:2112, 10.244.6.223:2113] 782786516/782801775/782801775/E5352@782801547:{5bd8dbab-1167-49b1-9871-944ddc82a71b} | 2019-01-23 05:19:31.798

[00001,13,05:19:32.443] VND {9164fbfb-dd74-43b6-8136-7e83d73dd99b} [Slave, 10.244.5.194:1112, 10.244.5.194:0, 10.244.5.194:1113, 10.244.5.194:0, 10.244.5.194:2112, 10.244.5.194:2113] 782786516/782801775/782801775/E5352@782801547:{5bd8dbab-1167-49b1-9871-944ddc82a71b} | 2019-01-23 05:19:32.443

[00001,13,05:19:32.443] VND {1e4cc1a3-88eb-4254-bd49-24caf642ca39} [Master, 10.244.4.218:1112, n/a, 10.244.4.218:1113, n/a, 10.244.4.218:2112, 10.244.4.218:2113] 782786516/782801775/782801775/E5352@782801547:{5bd8dbab-1167-49b1-9871-944ddc82a71b} | 2019-01-23 05:19:32.390

[00001,13,05:19:32.443] --------------------------------------------------------------------------------

[00001,13,05:19:32.443] ELECTIONS: (V=660) PROPOSAL FROM [10.244.6.223:2112,{36f99263-a4be-44cc-9333-6be98f4a00e9}] M=10.244.4.218:2112,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}. ME=10.244.5.194:2112,{9164fbfb-dd74-43b6-8136-7e83d73dd99b}.

[00001,13,05:19:32.443] ELECTIONS: (V=660) ACCEPT FROM [10.244.6.223:2112,{36f99263-a4be-44cc-9333-6be98f4a00e9}] M=[10.244.4.218:2112,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}]).

[00001,13,05:19:32.443] ELECTIONS: (V=660) ACCEPT FROM [10.244.5.194:2112,{9164fbfb-dd74-43b6-8136-7e83d73dd99b}] M=[10.244.4.218:2112,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}]).

[00001,13,05:19:32.443] ELECTIONS: (V=660) DONE. ELECTED MASTER = 10.244.4.218:2112,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}. ME=10.244.5.194:2112,{9164fbfb-dd74-43b6-8136-7e83d73dd99b}.

[00001,13,05:19:32.443] ELECTIONS: (V=660) ACCEPT FROM [10.244.4.218:2112,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}] M=[10.244.4.218:2112,{1e4cc1a3-88eb-4254-bd49-24caf642ca39}]).

[00001,14,05:19:32.481] === Update Last Epoch E5353@782801775:{38f4f59a-e9c3-4b11-b637-18f6fa7485fa} (previous epoch at 782801547).

[00001,13,05:19:32.854] CLUSTER HAS CHANGED (gossip received from [10.244.4.218:2112])

[00001,13,05:19:32.854] Old:

[00001,13,05:19:32.854] VND {36f99263-a4be-44cc-9333-6be98f4a00e9} [Slave, 10.244.6.223:1112, n/a, 10.244.6.223:1113, n/a, 10.244.6.223:2112, 10.244.6.223:2113] 782786516/782801775/782801775/E5352@782801547:{5bd8dbab-1167-49b1-9871-944ddc82a71b} | 2019-01-23 05:19:32.411

[00001,13,05:19:32.854] VND {9164fbfb-dd74-43b6-8136-7e83d73dd99b} [Slave, 10.244.5.194:1112, 10.244.5.194:0, 10.244.5.194:1113, 10.244.5.194:0, 10.244.5.194:2112, 10.244.5.194:2113] 782786516/782801775/782801775/E5352@782801547:{5bd8dbab-1167-49b1-9871-944ddc82a71b} | 2019-01-23 05:19:32.444

[00001,13,05:19:32.855] VND {1e4cc1a3-88eb-4254-bd49-24caf642ca39} [Master, 10.244.4.218:1112, n/a, 10.244.4.218:1113, n/a, 10.244.4.218:2112, 10.244.4.218:2113] 782786516/782801775/782801775/E5352@782801547:{5bd8dbab-1167-49b1-9871-944ddc82a71b} | 2019-01-23 05:19:32.390

[00001,13,05:19:32.855] New:

[00001,13,05:19:32.855] VND {36f99263-a4be-44cc-9333-6be98f4a00e9} [Slave, 10.244.6.223:1112, n/a, 10.244.6.223:1113, n/a, 10.244.6.223:2112, 10.244.6.223:2113] 782786516/782801775/782801775/E5352@782801547:{5bd8dbab-1167-49b1-9871-944ddc82a71b} | 2019-01-23 05:19:32.411

[00001,13,05:19:32.855] VND {9164fbfb-dd74-43b6-8136-7e83d73dd99b} [Slave, 10.244.5.194:1112, 10.244.5.194:0, 10.244.5.194:1113, 10.244.5.194:0, 10.244.5.194:2112, 10.244.5.194:2113] 782786516/782802003/782802003/E5353@782801775:{38f4f59a-e9c3-4b11-b637-18f6fa7485fa} | 2019-01-23 05:19:32.854

[00001,13,05:19:32.855] VND {1e4cc1a3-88eb-4254-bd49-24caf642ca39} [Master, 10.244.4.218:1112, n/a, 10.244.4.218:1113, n/a, 10.244.4.218:2112, 10.244.4.218:2113] 782786516/782802003/782802003/E5353@782801775:{38f4f59a-e9c3-4b11-b637-18f6fa7485fa} | 2019-01-23 05:19:32.806

[00001,13,05:19:32.855] --------------------------------------------------------------------------------

[00001,13,05:19:33.570] CLUSTER HAS CHANGED (gossip received from [10.244.6.223:2112])

[00001,13,05:19:33.570] Old:

[00001,13,05:19:33.570] VND {36f99263-a4be-44cc-9333-6be98f4a00e9} [Slave, 10.244.6.223:1112, n/a, 10.244.6.223:1113, n/a, 10.244.6.223:2112, 10.244.6.223:2113] 782786516/782801775/782801775/E5352@782801547:{5bd8dbab-1167-49b1-9871-944ddc82a71b} | 2019-01-23 05:19:32.411

[00001,13,05:19:33.570] VND {9164fbfb-dd74-43b6-8136-7e83d73dd99b} [Slave, 10.244.5.194:1112, 10.244.5.194:0, 10.244.5.194:1113, 10.244.5.194:0, 10.244.5.194:2112, 10.244.5.194:2113] 782786516/782802003/782802003/E5353@782801775:{38f4f59a-e9c3-4b11-b637-18f6fa7485fa} | 2019-01-23 05:19:33.426

[00001,13,05:19:33.570] VND {1e4cc1a3-88eb-4254-bd49-24caf642ca39} [Master, 10.244.4.218:1112, n/a, 10.244.4.218:1113, n/a, 10.244.4.218:2112, 10.244.4.218:2113] 782786516/782802003/782802003/E5353@782801775:{38f4f59a-e9c3-4b11-b637-18f6fa7485fa} | 2019-01-23 05:19:32.806

[00001,13,05:19:33.570] New:

[00001,13,05:19:33.578] VND {36f99263-a4be-44cc-9333-6be98f4a00e9} [Slave, 10.244.6.223:1112, n/a, 10.244.6.223:1113, n/a, 10.244.6.223:2112, 10.244.6.223:2113] 782786516/782802003/782802003/E5353@782801775:{38f4f59a-e9c3-4b11-b637-18f6fa7485fa} | 2019-01-23 05:19:33.447

[00001,13,05:19:33.578] VND {9164fbfb-dd74-43b6-8136-7e83d73dd99b} [Slave, 10.244.5.194:1112, 10.244.5.194:0, 10.244.5.194:1113, 10.244.5.194:0, 10.244.5.194:2112, 10.244.5.194:2113] 782786516/782802003/782802003/E5353@782801775:{38f4f59a-e9c3-4b11-b637-18f6fa7485fa} | 2019-01-23 05:19:33.570

[00001,13,05:19:33.578] VND {1e4cc1a3-88eb-4254-bd49-24caf642ca39} [Master, 10.244.4.218:1112, n/a, 10.244.4.218:1113, n/a, 10.244.4.218:2112, 10.244.4.218:2113] 782786516/782802003/782802003/E5353@782801775:{38f4f59a-e9c3-4b11-b637-18f6fa7485fa} | 2019-01-23 05:19:33.570

[00001,13,05:19:33.578] --------------------------------------------------------------------------------

[00001,45,05:20:04.173] SLOW BUS MSG [Worker #3 Bus]: AuthenticatedHttpRequestMessage - 383ms. Handler: AuthenticatedHttpRequestProcessor.

[00001,45,05:20:04.173] SLOW QUEUE MSG [Worker #3]: AuthenticatedHttpRequestMessage - 383ms. Q: 0/6.

[00001,23,05:20:08.823] SLOW BUS MSG [Worker #1 Bus]: AuthenticatedHttpRequestMessage - 268ms. Handler: AuthenticatedHttpRequestProcessor.

[00001,23,05:20:08.823] SLOW QUEUE MSG [Worker #1]: AuthenticatedHttpRequestMessage - 268ms. Q: 0/2.

[00001,88,05:20:10.704] SLOW BUS MSG [Worker #2 Bus]: IODispatcherDelayedMessage - 589ms. Handler: IODispatcher.

[00001,88,05:20:10.704] SLOW QUEUE MSG [Worker #2]: IODispatcherDelayedMessage - 589ms. Q: 2/2.

[00001,63,05:20:10.704] SLOW BUS MSG [SubscriptionsBus]: CheckPollTimeout - 496ms. Handler: SubscriptionsService.

[00001,37,05:20:11.635] ES TcpConnection closed [05:20:11.635: N10.244.4.218:1112, L10.244.5.194:49358, {dd7d4b83-d9ad-49f0-bd9f-f7c147418419}]:Received bytes: 1372, Sent bytes: 1245

[00001,37,05:20:11.635] ES TcpConnection closed [05:20:11.635: N10.244.4.218:1112, L10.244.5.194:49358, {dd7d4b83-d9ad-49f0-bd9f-f7c147418419}]:Send calls: 28, callbacks: 27

[00001,37,05:20:11.635] ES TcpConnection closed [05:20:11.635: N10.244.4.218:1112, L10.244.5.194:49358, {dd7d4b83-d9ad-49f0-bd9f-f7c147418419}]:Receive calls: 31, callbacks: 31

[00001,37,05:20:11.635] ES TcpConnection closed [05:20:11.635: N10.244.4.218:1112, L10.244.5.194:49358, {dd7d4b83-d9ad-49f0-bd9f-f7c147418419}]:Close reason: [Success] Socket closed

[00001,11,05:20:15.892] Closing connection ‘master-normal’ [10.244.4.218:1112, L10.244.5.194:49358, {dd7d4b83-d9ad-49f0-bd9f-f7c147418419}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 30

[00001,42,05:20:15.960] Dropping HTTP send message due to TTL being over. ViewChangeProof To : 10.244.4.218:2112

[00001,42,05:20:16.484] SLOW BUS MSG [Worker #1 Bus]: SendOverHttp - 523ms. Handler: HttpSendService.

[00001,22,05:20:15.960] Dropping HTTP send message due to TTL being over. SendGossip To : 10.244.4.218:2112

[00001,22,05:20:16.486] SLOW BUS MSG [Worker #2 Bus]: SendOverHttp - 526ms. Handler: HttpSendService.

[00001,22,05:20:16.486] SLOW QUEUE MSG [Worker #2]: SendOverHttp - 526ms. Q: 14/16.

[00001,22,05:20:16.486] Dropping HTTP send message due to TTL being over. ViewChangeProof To : 10.244.6.223:2112

[00001,22,05:20:16.486] Dropping HTTP send message due to TTL being over. ViewChangeProof To : 10.244.4.218:2112

[00001,22,05:20:16.486] Dropping HTTP send message due to TTL being over. ViewChangeProof To : 10.244.6.223:2112

[00001,18,05:20:15.960] Dropping HTTP send message due to TTL being over. ViewChangeProof To : 10.244.6.223:2112

[00001,66,05:20:15.960] Dropping HTTP send message due to TTL being over. ViewChangeProof To : 10.244.6.223:2112

[00001,66,05:20:19.745] SLOW BUS MSG [Worker #5 Bus]: SendOverHttp - 3785ms. Handler: HttpSendService.

[00001,66,05:20:19.745] SLOW QUEUE MSG [Worker #5]: SendOverHttp - 3785ms. Q: 14/16.