Problem with subscriptions

Ivan_Rodchenkov · April 26, 2017, 1:53pm

Hi!

We run Event Store on cluster of 3 nodes.

Recently we faced really nasty problem: all checkpoints were reset to 0 when cluster recovered itself after master node’s crushing. So when the client reconnected, all subscriptions started pushing events from the beginnig of the stream.

How can we avoid such problem in future?

Thanks.

Here are logs:

Old master node:

[PID:23590:038 2017.04.01 23:10:16.146 TRACE InMemoryBus ] SLOW BUS MSG [MainBus]: GossipReceived - 68ms. Handler: NodeGossipService.

[PID:23590:038 2017.04.01 23:10:16.146 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [MainQueue]: GossipReceived - 68ms. Q: 0/2.

[PID:23590:038 2017.04.01 23:20:28.653 TRACE InMemoryBus ] SLOW BUS MSG [MainBus]: GossipReceived - 64ms. Handler: NodeGossipService.

[PID:23590:038 2017.04.01 23:20:28.653 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [MainQueue]: GossipReceived - 65ms. Q: 0/6.

[PID:23590:038 2017.04.01 23:23:04.189 TRACE InMemoryBus ] SLOW BUS MSG [MainBus]: GossipReceived - 52ms. Handler: NodeGossipService.

[PID:23590:038 2017.04.01 23:23:04.190 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [MainQueue]: GossipReceived - 53ms. Q: 0/6.

[PID:23590:132 2017.04.01 23:23:55.063 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-es-stream-v2_01::system-blacklist-checkpoint: 4392241

[PID:23590:132 2017.04.01 23:23:55.063 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-es-stream-v2_01::web-adapter-out-checkpoint: 4392241

[PID:23590:132 2017.04.01 23:23:55.063 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-es-stream-v2_01::system-notifications-checkpoint: 4392241

[PID:23590:132 2017.04.01 23:23:55.063 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-es-stream-v2_01::statistics-checkpoint: 4392241

[PID:23590:132 2017.04.01 23:23:55.082 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-es-stream-v2_01::system-web-hire-checkpoint: 4392241

[PID:23590:132 2017.04.01 23:23:55.082 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-es-stream-v2_01::system-persons-web-checkpoint: 4392241

[PID:23590:132 2017.04.01 23:23:55.082 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-es-stream-v2_01::system-persons-checkpoint: 4392241

[PID:23590:132 2017.04.01 23:23:55.082 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-es-stream-v2_01::system-event-processing-checkpoint: 4392241

[PID:23590:038 2017.04.01 23:23:55.088 DEBUG PersistentSubscripti] Checkpoint write successful for $persistentsubscription-es-stream-v2_01::system-blacklist-checkpoint

[PID:23590:038 2017.04.01 23:23:55.100 DEBUG PersistentSubscripti] Checkpoint write successful for $persistentsubscription-es-stream-v2_01::web-adapter-out-checkpoint

[PID:23590:038 2017.04.01 23:23:55.100 DEBUG PersistentSubscripti] Checkpoint write successful for $persistentsubscription-es-stream-v2_01::system-notifications-checkpoint

[PID:23590:038 2017.04.01 23:23:55.110 DEBUG PersistentSubscripti] Checkpoint write successful for $persistentsubscription-es-stream-v2_01::statistics-checkpoint

[PID:23590:038 2017.04.01 23:23:55.112 DEBUG PersistentSubscripti] Checkpoint write successful for $persistentsubscription-es-stream-v2_01::system-persons-web-checkpoint

[PID:23590:038 2017.04.01 23:23:55.112 DEBUG PersistentSubscripti] Checkpoint write successful for $persistentsubscription-es-stream-v2_01::system-persons-checkpoint

[PID:23590:038 2017.04.01 23:23:55.112 DEBUG PersistentSubscripti] Checkpoint write successful for $persistentsubscription-es-stream-v2_01::system-event-processing-checkpoint

[PID:23590:038 2017.04.01 23:23:55.113 DEBUG PersistentSubscripti] Checkpoint write successful for $persistentsubscription-es-stream-v2_01::system-web-hire-checkpoint

[PID:23590:038 2017.04.01 23:23:56.818 TRACE InMemoryBus ] SLOW BUS MSG [MainBus]: GossipReceived - 69ms. Handler: NodeGossipService.

[PID:23590:038 2017.04.01 23:23:56.818 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [MainQueue]: GossipReceived - 70ms. Q: 0/2.

[PID:23590:038 2017.04.01 23:28:10.026 TRACE InMemoryBus ] SLOW BUS MSG [MainBus]: GossipReceived - 64ms. Handler: NodeGossipService.

[PID:23590:038 2017.04.01 23:28:10.027 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [MainQueue]: GossipReceived - 65ms. Q: 0/2.

[PID:23590:038 2017.04.01 23:30:04.691 TRACE GossipServiceBase ] Looks like node [10.80.42.217:2112] is DEAD (Gossip send failed).

[PID:23590:038 2017.04.01 23:30:04.691 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip send failed to [10.80.42.217:2112])

[PID:23590:038 2017.04.01 23:30:04.691 TRACE GossipServiceBase ] Old:

[PID:23590:038 2017.04.01 23:30:04.692 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [Slave, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243526992/23243541020/23243541020/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:03.181

[PID:23590:038 2017.04.01 23:30:04.692 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Master, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243526992/23243541126/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:04.174

[PID:23590:038 2017.04.01 23:30:04.692 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [Slave, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243541020/23243541020/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:03.058

[PID:23590:038 2017.04.01 23:30:04.692 TRACE GossipServiceBase ] New:

[PID:23590:038 2017.04.01 23:30:04.693 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [Slave, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243526992/23243541020/23243541020/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:03.181

[PID:23590:038 2017.04.01 23:30:04.693 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Master, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243526992/23243541126/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:04.174

[PID:23590:038 2017.04.01 23:30:04.693 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [Slave, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243541020/23243541020/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:04.691

[PID:23590:038 2017.04.01 23:30:04.693 TRACE GossipServiceBase ] --------------------------------------------------------------------------------

[PID:23590:036 2017.04.01 23:30:05.149 TRACE TcpConnectionManager] Closing connection ‘internal-normal’ [10.80.42.219:51983, L10.80.42.218:1112, {a22e1a59-342d-4f12-8074-aca1ad4b0e4e}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 4164

[PID:23590:036 2017.04.01 23:30:05.150 INFO TcpConnection ] ES TcpConnection closed [23:30:05.150: N10.80.42.219:51983, L10.80.42.218:1112, {a22e1a59-342d-4f12-8074-aca1ad4b0e4e}]:Received bytes: 2228240, Sent bytes: 7691503

[PID:23590:036 2017.04.01 23:30:05.150 INFO TcpConnection ] ES TcpConnection closed [23:30:05.150: N10.80.42.219:51983, L10.80.42.218:1112, {a22e1a59-342d-4f12-8074-aca1ad4b0e4e}]:Send calls: 4452, callbacks: 4452

[PID:23590:036 2017.04.01 23:30:05.150 INFO TcpConnection ] ES TcpConnection closed [23:30:05.151: N10.80.42.219:51983, L10.80.42.218:1112, {a22e1a59-342d-4f12-8074-aca1ad4b0e4e}]:Receive calls: 4166, callbacks: 4165

[PID:23590:036 2017.04.01 23:30:05.151 INFO TcpConnection ] ES TcpConnection closed [23:30:05.151: N10.80.42.219:51983, L10.80.42.218:1112, {a22e1a59-342d-4f12-8074-aca1ad4b0e4e}]:Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 4164

[PID:23590:036 2017.04.01 23:30:05.152 INFO TcpConnectionManager] Connection ‘internal-normal’ [10.80.42.219:51983, {a22e1a59-342d-4f12-8074-aca1ad4b0e4e}] closed: Success.

[PID:23590:134 2017.04.01 23:30:05.152 DEBUG PersistentSubscripti] Lost connection from 10.80.42.219:51983

[PID:23590:038 2017.04.01 23:30:05.418 TRACE GossipServiceBase ] Looks like node [10.80.42.219:1112] is DEAD (TCP connection lost).

[PID:23590:038 2017.04.01 23:30:05.418 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection lost to [10.80.42.219:1112])

[PID:23590:038 2017.04.01 23:30:05.418 TRACE GossipServiceBase ] Old:

[PID:23590:038 2017.04.01 23:30:05.419 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [Slave, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243526992/23243541020/23243541020/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:03.181

[PID:23590:038 2017.04.01 23:30:05.419 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Master, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243526992/23243541126/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:05.175

[PID:23590:038 2017.04.01 23:30:05.419 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [Slave, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243541020/23243541020/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:04.691

[PID:23590:038 2017.04.01 23:30:05.420 TRACE GossipServiceBase ] New:

[PID:23590:038 2017.04.01 23:30:05.420 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [Slave, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243526992/23243541020/23243541020/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:05.418

[PID:23590:038 2017.04.01 23:30:05.420 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Master, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243526992/23243541126/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:05.175

[PID:23590:038 2017.04.01 23:30:05.421 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [Slave, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243541020/23243541020/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:04.691

[PID:23590:038 2017.04.01 23:30:05.421 TRACE GossipServiceBase ] --------------------------------------------------------------------------------

[PID:23590:036 2017.04.01 23:30:05.650 TRACE TcpConnectionManager] Closing connection ‘internal-normal’ [10.80.42.217:48477, L10.80.42.218:1112, {cc93676f-cf08-43a1-99f2-4852fb08526d}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 4331

[PID:23590:036 2017.04.01 23:30:05.650 INFO TcpConnection ] ES TcpConnection closed [23:30:05.650: N10.80.42.217:48477, L10.80.42.218:1112, {cc93676f-cf08-43a1-99f2-4852fb08526d}]:Received bytes: 2724922, Sent bytes: 7691771

[PID:23590:036 2017.04.01 23:30:05.650 INFO TcpConnection ] ES TcpConnection closed [23:30:05.650: N10.80.42.217:48477, L10.80.42.218:1112, {cc93676f-cf08-43a1-99f2-4852fb08526d}]:Send calls: 4461, callbacks: 4461

[PID:23590:036 2017.04.01 23:30:05.650 INFO TcpConnection ] ES TcpConnection closed [23:30:05.650: N10.80.42.217:48477, L10.80.42.218:1112, {cc93676f-cf08-43a1-99f2-4852fb08526d}]:Receive calls: 4334, callbacks: 4333

[PID:23590:036 2017.04.01 23:30:05.650 INFO TcpConnection ] ES TcpConnection closed [23:30:05.650: N10.80.42.217:48477, L10.80.42.218:1112, {cc93676f-cf08-43a1-99f2-4852fb08526d}]:Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 4331

[PID:23590:036 2017.04.01 23:30:05.650 INFO TcpConnectionManager] Connection ‘internal-normal’ [10.80.42.217:48477, {cc93676f-cf08-43a1-99f2-4852fb08526d}] closed: Success.

[PID:23590:122 2017.04.01 23:30:05.650 DEBUG PersistentSubscripti] Lost connection from 10.80.42.217:48477

[PID:23590:122 2017.04.01 23:30:07.295 DEBUG ElectController ] Error while reading request: EndRead failure.

[PID:23590:135 2017.04.01 23:30:07.295 DEBUG ElectController ] Error while reading request: EndRead failure.

[PID:23590:038 2017.04.01 23:30:08.998 INFO ClusterVNodeControll] === NO QUORUM EMERGED WITHIN TIMEOUT… RETIRING…

[PID:23590:038 2017.04.01 23:30:09.000 INFO ClusterVNodeControll] ========== [10.80.42.218:2112] IS UNKNOWN…

[PID:23590:038 2017.04.01 23:30:09.000 DEBUG ElectionsService ] ELECTIONS: STARTING ELECTIONS.

[PID:23590:103 2017.04.01 23:30:09.000 DEBUG PersistentSubscripti] Subscriptions received state change to Unknown stopping listening.

[PID:23590:060 2017.04.01 23:30:09.000 DEBUG ProjectionCoreCoordi] PROJECTIONS: Stopping Projections Core Coordinator. (Node State : Unknown)

[PID:23590:038 2017.04.01 23:30:09.000 DEBUG ElectionsService ] ELECTIONS: (V=6660) SHIFT TO LEADER ELECTION.

[PID:23590:038 2017.04.01 23:30:09.000 DEBUG ElectionsService ] ELECTIONS: (V=6660) VIEWCHANGE FROM [10.80.42.218:2112, {32945cdd-2c8b-4079-86d1-22b58ec2392b}].

[PID:23590:060 2017.04.01 23:30:09.002 DEBUG ProjectionManager ] PROJECTIONS: Stopping Projections Manager. (Node State : Unknown)

[PID:23590:061 2017.04.01 23:30:09.033 INFO ProjectionCoreServic] _projections is not empty after all the projections have been killed

[PID:23590:062 2017.04.01 23:30:09.033 INFO ProjectionCoreServic] _projections is not empty after all the projections have been killed

[PID:23590:063 2017.04.01 23:30:09.033 INFO ProjectionCoreServic] _projections is not empty after all the projections have been killed

[PID:23590:061 2017.04.01 23:30:09.033 DEBUG ProjectionCoreServic] PROJECTIONS: Stopping Projection Core Reader (c7250f6f159c4cb2864a07af5c40d728)

[PID:23590:062 2017.04.01 23:30:09.033 DEBUG ProjectionCoreServic] PROJECTIONS: Stopping Projection Core Reader (03d3b27c331a401787f533553bbc2ddd)

[PID:23590:063 2017.04.01 23:30:09.033 DEBUG ProjectionCoreServic] PROJECTIONS: Stopping Projection Core Reader (53b82ad67dc3408cb95d1e0ecd63f805)

[PID:23590:061 2017.04.01 23:30:09.034 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed

[PID:23590:063 2017.04.01 23:30:09.034 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed

[PID:23590:062 2017.04.01 23:30:09.034 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed

[PID:23590:103 2017.04.01 23:30:09.322 DEBUG ElectController ] Error while reading request: EndRead failure.

[PID:23590:038 2017.04.01 23:30:09.375 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [10.80.42.219:2112])

[PID:23590:038 2017.04.01 23:30:09.375 TRACE GossipServiceBase ] Old:

[PID:23590:038 2017.04.01 23:30:09.375 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [Slave, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243526992/23243541020/23243541020/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:05.418

[PID:23590:038 2017.04.01 23:30:09.375 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Unknown, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243526992/23243541232/23243541232/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:09.179

[PID:23590:038 2017.04.01 23:30:09.376 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [Slave, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243541020/23243541020/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:04.691

[PID:23590:038 2017.04.01 23:30:09.376 TRACE GossipServiceBase ] New:

[PID:23590:038 2017.04.01 23:30:09.376 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [Slave, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243526992/23243541126/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:09.330

[PID:23590:038 2017.04.01 23:30:09.376 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Unknown, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243526992/23243541232/23243541232/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:09.375

[PID:23590:038 2017.04.01 23:30:09.377 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [Slave, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243541020/23243541020/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:04.691

[PID:23590:038 2017.04.01 23:30:09.377 TRACE GossipServiceBase ] --------------------------------------------------------------------------------

[PID:23590:038 2017.04.01 23:30:09.388 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [10.80.42.219:2112])

[PID:23590:038 2017.04.01 23:30:09.388 TRACE GossipServiceBase ] Old:

[PID:23590:038 2017.04.01 23:30:09.388 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [Slave, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243526992/23243541126/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:09.330

[PID:23590:038 2017.04.01 23:30:09.388 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Unknown, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243526992/23243541232/23243541232/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:09.375

[PID:23590:038 2017.04.01 23:30:09.389 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [Slave, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243541020/23243541020/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:04.691

[PID:23590:038 2017.04.01 23:30:09.389 TRACE GossipServiceBase ] New:

[PID:23590:038 2017.04.01 23:30:09.389 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [Slave, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243526992/23243541126/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:09.385

[PID:23590:038 2017.04.01 23:30:09.389 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Unknown, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243526992/23243541232/23243541232/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:09.388

[PID:23590:038 2017.04.01 23:30:09.389 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [Slave, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243541126/23243541020/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:08.923

[PID:23590:038 2017.04.01 23:30:09.389 TRACE GossipServiceBase ] --------------------------------------------------------------------------------

[PID:23590:133 2017.04.01 23:30:09.833 INFO TcpService ] Internal TCP connection accepted: [Normal, 10.80.42.219:54178, L10.80.42.218:1112, {be986dba-eec6-4041-9896-d30d75d97fd2}].

[PID:23590:038 2017.04.01 23:30:10.001 DEBUG ElectionsService ] ELECTIONS: (V=6660) TIMED OUT! (S=ElectingLeader, M=).

[PID:23590:038 2017.04.01 23:30:10.001 DEBUG ElectionsService ] ELECTIONS: (V=6661) SHIFT TO LEADER ELECTION.

[PID:23590:038 2017.04.01 23:30:10.001 DEBUG ElectionsService ] ELECTIONS: (V=6661) VIEWCHANGE FROM [10.80.42.218:2112, {32945cdd-2c8b-4079-86d1-22b58ec2392b}].

[PID:23590:038 2017.04.01 23:30:10.007 DEBUG ElectionsService ] ELECTIONS: (V=6661) VIEWCHANGE FROM [10.80.42.219:2112, {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72}].

[PID:23590:038 2017.04.01 23:30:10.007 DEBUG ElectionsService ] ELECTIONS: (V=6661) MAJORITY OF VIEWCHANGE.

[PID:23590:038 2017.04.01 23:30:10.008 DEBUG ElectionsService ] ELECTIONS: (V=6661) SHIFT TO PREPARE PHASE.

[PID:23590:038 2017.04.01 23:30:10.008 DEBUG ElectionsService ] ELECTIONS: (V=6661) PREPARE_OK FROM 10.80.42.218:2112,{32945cdd-2c8b-4079-86d1-22b58ec2392b}.

[PID:23590:038 2017.04.01 23:30:10.013 DEBUG ElectionsService ] ELECTIONS: (V=6661) PREPARE_OK FROM 10.80.42.219:2112,{1221bd3c-d4b0-41d2-9b5e-2c764b14ac72}.

[PID:23590:038 2017.04.01 23:30:10.013 DEBUG ElectionsService ] ELECTIONS: (V=6661) SHIFT TO REG_LEADER.

[PID:23590:038 2017.04.01 23:30:10.013 DEBUG ElectionsService ] ELECTIONS: (V=6661) SENDING PROPOSAL CANDIDATE: 10.80.42.218:2112,{32945cdd-2c8b-4079-86d1-22b58ec2392b}, ME: 10.80.42.218:2112,{32945cdd-2c8b-4079-86d1-22b58ec2392b}.

[PID:23590:038 2017.04.01 23:30:10.013 DEBUG ElectionsService ] ELECTIONS: (V=6661) ACCEPT FROM [10.80.42.218:2112,{32945cdd-2c8b-4079-86d1-22b58ec2392b}] M=[10.80.42.218:2112,{32945cdd-2c8b-4079-86d1-22b58ec2392b}]).

[PID:23590:038 2017.04.01 23:30:10.018 DEBUG ElectionsService ] ELECTIONS: (V=6661) ACCEPT FROM [10.80.42.219:2112,{1221bd3c-d4b0-41d2-9b5e-2c764b14ac72}] M=[10.80.42.218:2112,{32945cdd-2c8b-4079-86d1-22b58ec2392b}]).

[PID:23590:038 2017.04.01 23:30:10.018 INFO ElectionsService ] ELECTIONS: (V=6661) DONE. ELECTED MASTER = 10.80.42.218:2112,{32945cdd-2c8b-4079-86d1-22b58ec2392b}. ME=10.80.42.218:2112,{32945cdd-2c8b-4079-86d1-22b58ec2392b}.

[PID:23590:038 2017.04.01 23:30:10.018 INFO ClusterVNodeControll] ========== [10.80.42.218:2112] PRE-MASTER STATE, WAITING FOR CHASER TO CATCH UP…

[PID:23590:132 2017.04.01 23:30:10.026 DEBUG PersistentSubscripti] Subscriptions received state change to PreMaster stopping listening.

[PID:23590:038 2017.04.01 23:30:10.026 INFO ClusterVNodeControll] ========== [10.80.42.218:2112] IS MASTER… SPARTA!

[PID:23590:060 2017.04.01 23:30:10.027 DEBUG ProjectionCoreCoordi] PROJECTIONS: Starting Projections Core Coordinator. (Node State : Master)

[PID:23590:060 2017.04.01 23:30:10.027 DEBUG ProjectionManager ] PROJECTIONS: Starting Projections Manager. (Node State : Master)

[PID:23590:062 2017.04.01 23:30:10.027 DEBUG ProjectionCoreServic] PROJECTIONS: Starting Projection Core Reader (reads from $projections-$03d3b27c331a401787f533553bbc2ddd)

[PID:23590:060 2017.04.01 23:30:10.027 DEBUG ProjectionManagerRes] PROJECTIONS: There was an active cancellation scope, cancelling now

[PID:23590:063 2017.04.01 23:30:10.027 DEBUG ProjectionCoreServic] PROJECTIONS: Starting Projection Core Reader (reads from $projections-$53b82ad67dc3408cb95d1e0ecd63f805)

[PID:23590:062 2017.04.01 23:30:10.029 DEBUG ResponseWriter ] PROJECTIONS: Resetting Master Writer

[PID:23590:061 2017.04.01 23:30:10.029 DEBUG ProjectionCoreServic] PROJECTIONS: Starting Projection Core Reader (reads from $projections-$c7250f6f159c4cb2864a07af5c40d728)

[PID:23590:132 2017.04.01 23:30:10.029 DEBUG PersistentSubscripti] Subscriptions Became Master so now handling subscriptions

[PID:23590:060 2017.04.01 23:30:10.029 DEBUG ProjectionManagerRes] PROJECTIONS: Starting Projection Manager Response Reader (reads from $projections-$master)

[PID:23590:063 2017.04.01 23:30:10.029 DEBUG ResponseWriter ] PROJECTIONS: Resetting Master Writer

[PID:23590:035 2017.04.01 23:30:10.029 DEBUG EpochManager ] === Writing E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91} (previous epoch at 23236027113).

[PID:23590:063 2017.04.01 23:30:10.030 DEBUG ProjectionCoreServic] PROJECTIONS: Starting read control from 80

[PID:23590:060 2017.04.01 23:30:10.030 DEBUG MultiStreamMessageWr] PROJECTIONS: Resetting Worker Writer

[PID:23590:062 2017.04.01 23:30:10.030 DEBUG ProjectionCoreServic] PROJECTIONS: Starting read control from 80

[PID:23590:061 2017.04.01 23:30:10.030 DEBUG ResponseWriter ] PROJECTIONS: Resetting Master Writer

[PID:23590:060 2017.04.01 23:30:10.032 DEBUG ProjectionManager ] PROJECTIONS: Starting Projections Manager. (Node State : Master)

[PID:23590:061 2017.04.01 23:30:10.032 DEBUG ProjectionCoreServic] PROJECTIONS: Starting read control from 80

[PID:23590:060 2017.04.01 23:30:10.032 DEBUG ProjectionManagerRes] PROJECTIONS: There was an active cancellation scope, cancelling now

[PID:23590:038 2017.04.01 23:30:10.033 DEBUG PersistentSubscripti] Subscription es-stream-v2_01::system-blacklist: read checksum 4392241

[PID:23590:060 2017.04.01 23:30:10.034 DEBUG ProjectionManagerRes] PROJECTIONS: Starting Projection Manager Response Reader (reads from $projections-$master)

[PID:23590:038 2017.04.01 23:30:10.034 DEBUG PersistentSubscripti] Subscription es-stream-v2_01::web-adapter-out: read checksum 4392241

[PID:23590:060 2017.04.01 23:30:10.034 DEBUG MultiStreamMessageWr] PROJECTIONS: Resetting Worker Writer

[PID:23590:038 2017.04.01 23:30:10.034 DEBUG PersistentSubscripti] Subscription es-stream-v2_01::system-web-hire: read checksum 4392241

[PID:23590:038 2017.04.01 23:30:10.035 DEBUG PersistentSubscripti] Subscription es-stream-v2_01::system-persons-web: read checksum 4392241

[PID:23590:038 2017.04.01 23:30:10.035 DEBUG PersistentSubscripti] Subscription es-stream-v2_01::system-notifications: read checksum 4392241

[PID:23590:038 2017.04.01 23:30:10.035 DEBUG PersistentSubscripti] Subscription es-stream-v2_01::statistics: read checksum 4392241

[PID:23590:038 2017.04.01 23:30:10.036 DEBUG PersistentSubscripti] Subscription es-stream-v2_01::system-event-processing: read checksum 4392241

[PID:23590:038 2017.04.01 23:30:10.036 DEBUG PersistentSubscripti] Subscription es-stream-v2_01::system-persons: read checksum 4392241

[PID:23590:035 2017.04.01 23:30:11.373 DEBUG EpochManager ] === Update Last Epoch E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91} (previous epoch at 23236027113).

[PID:23590:035 2017.04.01 23:30:11.373 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [StorageWriterQueue]: BecomeMaster - 1346ms. Q: 0/11.

[PID:23590:038 2017.04.01 23:30:11.373 TRACE InMemoryBus ] SLOW BUS MSG [MainBus]: Gossip - 1192ms. Handler: NodeGossipService.

[PID:23590:038 2017.04.01 23:30:11.373 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [MainQueue]: Gossip - 1192ms. Q: 0/23.

[PID:23590:038 2017.04.01 23:30:11.373 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [10.80.42.219:2112])

[PID:23590:038 2017.04.01 23:30:11.373 TRACE GossipServiceBase ] Old:

[PID:23590:038 2017.04.01 23:30:11.373 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [Slave, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243526992/23243541126/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:09.385

[PID:23590:038 2017.04.01 23:30:11.374 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Master, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243526992/23243541464/23243541232/E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91} | 2017-04-01 23:30:11.373

[PID:23590:038 2017.04.01 23:30:11.374 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [Slave, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243541126/23243541020/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:08.923

[PID:23590:038 2017.04.01 23:30:11.374 TRACE GossipServiceBase ] New:

[PID:23590:038 2017.04.01 23:30:11.374 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [PreReplica, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243526992/23243541126/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:10.367

[PID:23590:038 2017.04.01 23:30:11.374 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Master, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243526992/23243541623/23243541464/E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91} | 2017-04-01 23:30:11.373

[PID:23590:038 2017.04.01 23:30:11.374 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [Slave, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243541126/23243541020/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:08.923

[PID:23590:038 2017.04.01 23:30:11.375 TRACE GossipServiceBase ] --------------------------------------------------------------------------------

[PID:23590:135 2017.04.01 23:30:11.375 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-es-stream-v2_01::system-blacklist-checkpoint: 0

[PID:23590:038 2017.04.01 23:30:11.375 INFO MasterReplicationSer] SUBSCRIBE REQUEST from [10.80.42.219:1112,C:{be986dba-eec6-4041-9896-d30d75d97fd2},S:{275d0208-e134-4cdb-876c-5b04c9f8672f},23243541126(0x5696C4A86),E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396}, E3058@23236026882:{19249e5d-6b27-4d08-8d19-2b55bd32b84a}, E3057@23235720749:{73fa7146-b343-4f70-9e49-54ba9e466264}, E3056@23235706408:{a21d5f90-07a6-499b-baa7-75515fee141a}, E3055@23230975805:{e820e911-590e-4f99-b6f7-452a5c479c9a}, E3054@23230975573:{ac0c638d-987a-42f9-8089-4554fe310522}, E3053@23230975341:{ad13323b-53fb-4291-b9b7-c00817bbf280}, E3052@23230387757:{043aa4e3-9dd9-4724-b0ee-b9551f9cfae8}, E3051@23230155892:{20e19804-1b44-46cf-ab5f-cf1cf980f340}, E3050@23176837486:{2f5c1336-5fec-4079-8548-be05447da8bf}, E3049@23176837254:{841d7292-ad06-4b5e-af7b-13d508e12b2f}, E3048@23092852866:{ecc191fc-ff2f-4d48-b369-290035db53f5}, E3047@23092525361:{d9465e8b-00b3-4ce3-8c19-4b912aa23fe9}, E3046@23092524133:{82d88259-7b38-4ea5-87eb-86c694562adb}, E3045@23092523901:{1a8729c5-ccbc-4a09-8b9f-2ae1cf7749f9}, E3044@23070038265:{86f6baaa-570e-4c7d-acad-fa1e4a7d5331}, E3043@23070037037:{09ba0853-7718-456a-bdf2-d628de343288}, E3042@23070035810:{d097b597-ec42-4b10-ad14-a7e458cbf943}, E3041@23069607034:{bb8588db-6bec-4d7f-8890-8732315e39c0}, E3040@23069605806:{5133a54a-262b-46d5-9a37-446b4c4b4fe8}]…

[PID:23590:135 2017.04.01 23:30:11.375 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-es-stream-v2_01::web-adapter-out-checkpoint: 0

[PID:23590:135 2017.04.01 23:30:11.375 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-es-stream-v2_01::system-notifications-checkpoint: 0

[PID:23590:038 2017.04.01 23:30:11.375 INFO MasterReplicationSer] Subscribed replica [10.80.42.219:1112,S:275d0208-e134-4cdb-876c-5b04c9f8672f] for data send at 23243541126 (0x5696C4A86).

[PID:23590:135 2017.04.01 23:30:11.376 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-es-stream-v2_01::statistics-checkpoint: 0

[PID:23590:135 2017.04.01 23:30:11.376 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-es-stream-v2_01::system-web-hire-checkpoint: 0

[PID:23590:135 2017.04.01 23:30:11.376 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-es-stream-v2_01::system-persons-web-checkpoint: 0

[PID:23590:135 2017.04.01 23:30:11.376 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-es-stream-v2_01::system-persons-checkpoint: 0

[PID:23590:135 2017.04.01 23:30:11.376 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-es-stream-v2_01::system-event-processing-checkpoint: 0

[PID:23590:038 2017.04.01 23:30:11.892 TRACE GossipServiceBase ] Looks like node [10.80.42.217:2112] is DEAD (Gossip send failed).

[PID:23590:038 2017.04.01 23:30:11.892 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip send failed to [10.80.42.217:2112])

[PID:23590:038 2017.04.01 23:30:11.893 TRACE GossipServiceBase ] Old:

[PID:23590:038 2017.04.01 23:30:11.893 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [PreReplica, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243526992/23243541126/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:11.377

[PID:23590:038 2017.04.01 23:30:11.893 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Master, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243526992/23243541623/23243541464/E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91} | 2017-04-01 23:30:11.376

[PID:23590:038 2017.04.01 23:30:11.894 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [Slave, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243541126/23243541020/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:08.923

[PID:23590:038 2017.04.01 23:30:11.894 TRACE GossipServiceBase ] New:

[PID:23590:038 2017.04.01 23:30:11.894 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [PreReplica, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243526992/23243541126/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:11.377

[PID:23590:038 2017.04.01 23:30:11.895 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Master, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243526992/23243541623/23243541464/E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91} | 2017-04-01 23:30:11.376

[PID:23590:038 2017.04.01 23:30:11.895 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [Slave, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243541126/23243541020/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:11.893

[PID:23590:038 2017.04.01 23:30:11.895 TRACE GossipServiceBase ] --------------------------------------------------------------------------------

[PID:23590:038 2017.04.01 23:30:12.421 ERROR StorageScavenger ] Failed to write the $maxAge of 30 days metadata for the $scavenges stream. Reason: CommitTimeout

[PID:23590:038 2017.04.01 23:30:12.480 TRACE TwoPhaseRequestManag] IDEMPOTENT WRITE TO STREAM ClientCorrelationID a0b03893-7cc6-4623-978e-544af602654f, EventStreamId: $projections-$master, CorrelationId: a0b03893-7cc6-4623-978e-544af602654f, FirstEventNumber: 4298, LastEventNumber: 4298.

[PID:23590:038 2017.04.01 23:30:12.480 TRACE TwoPhaseRequestManag] IDEMPOTENT WRITE TO STREAM ClientCorrelationID 4cff8ca4-f761-4cb0-bba2-86ac38d3a87f, EventStreamId: $projections-$master, CorrelationId: 4cff8ca4-f761-4cb0-bba2-86ac38d3a87f, FirstEventNumber: 4296, LastEventNumber: 4296.

[PID:23590:038 2017.04.01 23:30:12.481 TRACE TwoPhaseRequestManag] IDEMPOTENT WRITE TO STREAM ClientCorrelationID ff8a8b37-aca3-4b3f-9f57-a41f753df0f1, EventStreamId: $projections-$master, CorrelationId: ff8a8b37-aca3-4b3f-9f57-a41f753df0f1, FirstEventNumber: 4297, LastEventNumber: 4297.

[PID:23590:038 2017.04.01 23:30:12.892 TRACE GossipServiceBase ] Looks like node [10.80.42.219:2112] is DEAD (Gossip send failed).

[PID:23590:038 2017.04.01 23:30:12.893 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip send failed to [10.80.42.219:2112])

[PID:23590:038 2017.04.01 23:30:12.893 TRACE GossipServiceBase ] Old:

[PID:23590:038 2017.04.01 23:30:12.894 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [PreReplica, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243526992/23243541126/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:11.377

[PID:23590:038 2017.04.01 23:30:12.894 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Master, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243544247/23243544437/23243544437/E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91} | 2017-04-01 23:30:12.377

[PID:23590:038 2017.04.01 23:30:12.896 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [Slave, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243541126/23243541020/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:11.893

[PID:23590:038 2017.04.01 23:30:12.896 TRACE GossipServiceBase ] New:

[PID:23590:038 2017.04.01 23:30:12.896 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [PreReplica, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243526992/23243541126/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:12.893

[PID:23590:038 2017.04.01 23:30:12.897 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Master, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243544247/23243544437/23243544437/E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91} | 2017-04-01 23:30:12.377

[PID:23590:038 2017.04.01 23:30:12.897 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [Slave, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243541126/23243541020/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:11.893

[PID:23590:038 2017.04.01 23:30:12.897 TRACE GossipServiceBase ] --------------------------------------------------------------------------------

[PID:23590:038 2017.04.01 23:30:13.423 DEBUG PersistentSubscripti] Error writing checkpoint for $persistentsubscription-es-stream-v2_01::system-blacklist-checkpoint: CommitTimeout

[PID:23590:038 2017.04.01 23:30:13.423 DEBUG PersistentSubscripti] Error writing checkpoint for $persistentsubscription-es-stream-v2_01::system-persons-web-checkpoint: CommitTimeout

[PID:23590:038 2017.04.01 23:30:13.423 DEBUG PersistentSubscripti] Error writing checkpoint for $persistentsubscription-es-stream-v2_01::system-persons-checkpoint: CommitTimeout

[PID:23590:038 2017.04.01 23:30:13.423 DEBUG PersistentSubscripti] Error writing checkpoint for $persistentsubscription-es-stream-v2_01::system-notifications-checkpoint: CommitTimeout

[PID:23590:038 2017.04.01 23:30:13.423 DEBUG PersistentSubscripti] Error writing checkpoint for $persistentsubscription-es-stream-v2_01::web-adapter-out-checkpoint: CommitTimeout

[PID:23590:038 2017.04.01 23:30:13.423 DEBUG PersistentSubscripti] Error writing checkpoint for $persistentsubscription-es-stream-v2_01::system-web-hire-checkpoint: CommitTimeout

[PID:23590:038 2017.04.01 23:30:13.423 DEBUG PersistentSubscripti] Error writing checkpoint for $persistentsubscription-es-stream-v2_01::system-event-processing-checkpoint: CommitTimeout

[PID:23590:038 2017.04.01 23:30:13.424 DEBUG PersistentSubscripti] Error writing checkpoint for $persistentsubscription-es-stream-v2_01::statistics-checkpoint: CommitTimeout

[PID:23590:036 2017.04.01 23:30:13.477 TRACE TcpConnectionManager] Closing connection ‘internal-normal’ [10.80.42.219:54178, L10.80.42.218:1112, {be986dba-eec6-4041-9896-d30d75d97fd2}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 2

[PID:23590:036 2017.04.01 23:30:13.477 INFO TcpConnection ] ES TcpConnection closed [23:30:13.477: N10.80.42.219:54178, L10.80.42.218:1112, {be986dba-eec6-4041-9896-d30d75d97fd2}]:Received bytes: 1346, Sent bytes: 4584

[PID:23590:036 2017.04.01 23:30:13.477 INFO TcpConnection ] ES TcpConnection closed [23:30:13.478: N10.80.42.219:54178, L10.80.42.218:1112, {be986dba-eec6-4041-9896-d30d75d97fd2}]:Send calls: 8, callbacks: 8

[PID:23590:036 2017.04.01 23:30:13.478 INFO TcpConnection ] ES TcpConnection closed [23:30:13.478: N10.80.42.219:54178, L10.80.42.218:1112, {be986dba-eec6-4041-9896-d30d75d97fd2}]:Receive calls: 3, callbacks: 2

[PID:23590:036 2017.04.01 23:30:13.478 INFO TcpConnection ] ES TcpConnection closed [23:30:13.478: N10.80.42.219:54178, L10.80.42.218:1112, {be986dba-eec6-4041-9896-d30d75d97fd2}]:Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 2

[PID:23590:036 2017.04.01 23:30:13.478 INFO TcpConnectionManager] Connection ‘internal-normal’ [10.80.42.219:54178, {be986dba-eec6-4041-9896-d30d75d97fd2}] closed: Success.

[PID:23590:130 2017.04.01 23:30:13.479 DEBUG PersistentSubscripti] Lost connection from 10.80.42.219:54178

[PID:23590:038 2017.04.01 23:30:16.653 INFO ClusterVNodeControll] === NO QUORUM EMERGED WITHIN TIMEOUT… RETIRING…

[PID:23590:038 2017.04.01 23:30:16.653 INFO ClusterVNodeControll] ========== [10.80.42.218:2112] IS UNKNOWN…

[PID:23590:132 2017.04.01 23:30:16.653 DEBUG PersistentSubscripti] Subscriptions received state change to Unknown stopping listening.

[PID:23590:038 2017.04.01 23:30:16.653 DEBUG ElectionsService ] ELECTIONS: STARTING ELECTIONS.

[PID:23590:060 2017.04.01 23:30:16.653 DEBUG ProjectionCoreCoordi] PROJECTIONS: Stopping Projections Core Coordinator. (Node State : Unknown)

[PID:23590:038 2017.04.01 23:30:16.654 DEBUG ElectionsService ] ELECTIONS: (V=6662) SHIFT TO LEADER ELECTION.

[PID:23590:062 2017.04.01 23:30:16.654 DEBUG ProjectionCoreServic] PROJECTIONS: Stopping Projection Core Reader (03d3b27c331a401787f533553bbc2ddd)

[PID:23590:063 2017.04.01 23:30:16.654 DEBUG ProjectionCoreServic] PROJECTIONS: Stopping Projection Core Reader (53b82ad67dc3408cb95d1e0ecd63f805)

[PID:23590:061 2017.04.01 23:30:16.654 DEBUG ProjectionCoreServic] PROJECTIONS: Stopping Projection Core Reader (c7250f6f159c4cb2864a07af5c40d728)

[PID:23590:062 2017.04.01 23:30:16.655 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed

[PID:23590:063 2017.04.01 23:30:16.656 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed

[PID:23590:038 2017.04.01 23:30:16.655 DEBUG ElectionsService ] ELECTIONS: (V=6662) VIEWCHANGE FROM [10.80.42.218:2112, {32945cdd-2c8b-4079-86d1-22b58ec2392b}].

[PID:23590:061 2017.04.01 23:30:16.657 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed

[PID:23590:038 2017.04.01 23:30:17.216 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [10.80.42.219:2112])

[PID:23590:038 2017.04.01 23:30:17.216 TRACE GossipServiceBase ] Old:

[PID:23590:038 2017.04.01 23:30:17.216 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [PreReplica, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243526992/23243541126/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:12.893

[PID:23590:038 2017.04.01 23:30:17.216 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Unknown, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243545606/23243545871/23243545871/E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91} | 2017-04-01 23:30:16.653

[PID:23590:038 2017.04.01 23:30:17.217 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [Slave, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243541126/23243541020/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:11.893

[PID:23590:038 2017.04.01 23:30:17.217 TRACE GossipServiceBase ] New:

[PID:23590:038 2017.04.01 23:30:17.217 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [Slave, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243526992/23243541623/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:17.204

[PID:23590:038 2017.04.01 23:30:17.218 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Unknown, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243545606/23243545871/23243545871/E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91} | 2017-04-01 23:30:17.215

[PID:23590:038 2017.04.01 23:30:17.218 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [Slave, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243541126/23243541020/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:11.893

[PID:23590:038 2017.04.01 23:30:17.218 TRACE GossipServiceBase ] --------------------------------------------------------------------------------

[PID:23590:038 2017.04.01 23:30:17.262 DEBUG ElectionsService ] ELECTIONS: (V=6662) VIEWCHANGE FROM [10.80.42.219:2112, {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72}].

[PID:23590:038 2017.04.01 23:30:17.263 DEBUG ElectionsService ] ELECTIONS: (V=6662) MAJORITY OF VIEWCHANGE.

[PID:23590:130 2017.04.01 23:30:17.306 INFO TcpService ] Internal TCP connection accepted: [Normal, 10.80.42.217:49909, L10.80.42.218:1112, {4be22212-d7cc-4436-af83-9890eba17e8c}].

[PID:23590:038 2017.04.01 23:30:17.331 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [10.80.42.217:2112])

[PID:23590:038 2017.04.01 23:30:17.331 TRACE GossipServiceBase ] Old:

[PID:23590:038 2017.04.01 23:30:17.331 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [Slave, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243526992/23243541623/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:17.204

[PID:23590:038 2017.04.01 23:30:17.332 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Unknown, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243545606/23243545871/23243545871/E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91} | 2017-04-01 23:30:17.215

[PID:23590:038 2017.04.01 23:30:17.332 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [Slave, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243541126/23243541020/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:11.893

[PID:23590:038 2017.04.01 23:30:17.332 TRACE GossipServiceBase ] New:

[PID:23590:038 2017.04.01 23:30:17.333 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [Slave, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243526992/23243541623/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:17.204

[PID:23590:038 2017.04.01 23:30:17.333 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Unknown, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243545606/23243545871/23243545871/E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91} | 2017-04-01 23:30:17.330

[PID:23590:038 2017.04.01 23:30:17.333 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [PreReplica, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243541126/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:17.300

[PID:23590:038 2017.04.01 23:30:17.333 TRACE GossipServiceBase ] --------------------------------------------------------------------------------

[PID:23590:038 2017.04.01 23:30:17.388 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [10.80.42.219:2112])

[PID:23590:038 2017.04.01 23:30:17.388 TRACE GossipServiceBase ] Old:

[PID:23590:038 2017.04.01 23:30:17.388 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [Slave, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243526992/23243541623/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:17.204

[PID:23590:038 2017.04.01 23:30:17.388 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Unknown, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243545606/23243545871/23243545871/E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91} | 2017-04-01 23:30:17.381

[PID:23590:038 2017.04.01 23:30:17.388 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [PreReplica, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243541126/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:17.300

[PID:23590:038 2017.04.01 23:30:17.389 TRACE GossipServiceBase ] New:

[PID:23590:038 2017.04.01 23:30:17.389 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [Slave, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243544889/23243545154/23243545154/E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91} | 2017-04-01 23:30:17.384

[PID:23590:038 2017.04.01 23:30:17.389 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Unknown, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243545606/23243545871/23243545871/E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91} | 2017-04-01 23:30:17.388

[PID:23590:038 2017.04.01 23:30:17.390 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [PreReplica, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243541126/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:17.300

[PID:23590:038 2017.04.01 23:30:17.390 TRACE GossipServiceBase ] --------------------------------------------------------------------------------

[PID:23590:038 2017.04.01 23:30:17.657 DEBUG ElectionsService ] ELECTIONS: (V=6662) TIMED OUT! (S=ElectingLeader, M=).

[PID:23590:038 2017.04.01 23:30:17.657 DEBUG ElectionsService ] ELECTIONS: (V=6663) SHIFT TO LEADER ELECTION.

[PID:23590:038 2017.04.01 23:30:17.658 DEBUG ElectionsService ] ELECTIONS: (V=6663) VIEWCHANGE FROM [10.80.42.218:2112, {32945cdd-2c8b-4079-86d1-22b58ec2392b}].

[PID:23590:038 2017.04.01 23:30:17.664 DEBUG ElectionsService ] ELECTIONS: (V=6663) VIEWCHANGE FROM [10.80.42.219:2112, {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72}].

[PID:23590:038 2017.04.01 23:30:17.665 DEBUG ElectionsService ] ELECTIONS: (V=6663) MAJORITY OF VIEWCHANGE.

[PID:23590:038 2017.04.01 23:30:17.665 DEBUG ElectionsService ] ELECTIONS: (V=6663) VIEWCHANGE FROM [10.80.42.217:2112, {705359a8-be32-4a13-aafb-32c100f6dc6c}].

[PID:23590:038 2017.04.01 23:30:17.668 DEBUG ElectionsService ] ELECTIONS: (V=6663) PREPARE FROM [10.80.42.219:2112, {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72}].

[PID:23590:038 2017.04.01 23:30:17.668 DEBUG ElectionsService ] ELECTIONS: (V=6663) SHIFT TO REG_NONLEADER.

[PID:23590:128 2017.04.01 23:30:17.669 INFO TcpConnection ] ES TcpConnection closed [23:30:17.669: N10.80.42.217:49909, L10.80.42.218:1112, {4be22212-d7cc-4436-af83-9890eba17e8c}]:Received bytes: 18028, Sent bytes: 24

[PID:23590:128 2017.04.01 23:30:17.669 INFO TcpConnection ] ES TcpConnection closed [23:30:17.670: N10.80.42.217:49909, L10.80.42.218:1112, {4be22212-d7cc-4436-af83-9890eba17e8c}]:Send calls: 1, callbacks: 1

[PID:23590:128 2017.04.01 23:30:17.670 INFO TcpConnection ] ES TcpConnection closed [23:30:17.670: N10.80.42.217:49909, L10.80.42.218:1112, {4be22212-d7cc-4436-af83-9890eba17e8c}]:Receive calls: 4, callbacks: 4

[PID:23590:128 2017.04.01 23:30:17.670 INFO TcpConnection ] ES TcpConnection closed [23:30:17.670: N10.80.42.217:49909, L10.80.42.218:1112, {4be22212-d7cc-4436-af83-9890eba17e8c}]:Close reason: [Success] Socket closed

[PID:23590:128 2017.04.01 23:30:17.670 INFO TcpConnectionManager] Connection ‘internal-normal’ [10.80.42.217:49909, {4be22212-d7cc-4436-af83-9890eba17e8c}] closed: Success.

[PID:23590:130 2017.04.01 23:30:17.671 DEBUG PersistentSubscripti] Lost connection from 10.80.42.217:49909

[PID:23590:038 2017.04.01 23:30:18.388 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [10.80.42.217:2112])

[PID:23590:038 2017.04.01 23:30:18.388 TRACE GossipServiceBase ] Old:

[PID:23590:038 2017.04.01 23:30:18.388 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [Slave, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243544889/23243545154/23243545154/E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91} | 2017-04-01 23:30:17.384

[PID:23590:038 2017.04.01 23:30:18.388 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Unknown, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243545606/23243545871/23243545871/E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91} | 2017-04-01 23:30:18.382

[PID:23590:038 2017.04.01 23:30:18.389 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [PreReplica, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243541126/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:17.300

[PID:23590:038 2017.04.01 23:30:18.389 TRACE GossipServiceBase ] New:

[PID:23590:038 2017.04.01 23:30:18.389 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [Slave, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243544889/23243545154/23243545154/E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91} | 2017-04-01 23:30:18.387

[PID:23590:038 2017.04.01 23:30:18.389 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Unknown, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243545606/23243545871/23243545871/E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91} | 2017-04-01 23:30:18.388

[PID:23590:038 2017.04.01 23:30:18.389 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [Slave, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243545386/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:18.387

[PID:23590:038 2017.04.01 23:30:18.389 TRACE GossipServiceBase ] --------------------------------------------------------------------------------

[PID:23590:038 2017.04.01 23:30:18.457 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [10.80.42.219:2112])

[PID:23590:038 2017.04.01 23:30:18.457 TRACE GossipServiceBase ] Old:

[PID:23590:038 2017.04.01 23:30:18.457 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [Slave, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243544889/23243545154/23243545154/E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91} | 2017-04-01 23:30:18.387

[PID:23590:038 2017.04.01 23:30:18.457 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Unknown, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243545606/23243545871/23243545871/E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91} | 2017-04-01 23:30:18.388

[PID:23590:038 2017.04.01 23:30:18.457 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [Slave, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243545386/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:18.387

[PID:23590:038 2017.04.01 23:30:18.458 TRACE GossipServiceBase ] New:

[PID:23590:038 2017.04.01 23:30:18.458 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [Master, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243544889/23243545386/23243545386/E3061@23243545154:{ec5dc7e7-ecf9-4ab3-bf72-39d4e4608b2e} | 2017-04-01 23:30:18.449

[PID:23590:038 2017.04.01 23:30:18.458 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Unknown, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243545606/23243545871/23243545871/E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91} | 2017-04-01 23:30:18.457

[PID:23590:038 2017.04.01 23:30:18.458 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [Slave, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243545386/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:18.387

[PID:23590:038 2017.04.01 23:30:18.458 TRACE GossipServiceBase ] --------------------------------------------------------------------------------

[PID:23590:038 2017.04.01 23:30:18.659 DEBUG ElectionsService ] ELECTIONS: (V=6663) TIMED OUT! (S=NonLeader, M=).

[PID:23590:038 2017.04.01 23:30:18.660 DEBUG ElectionsService ] ELECTIONS: (V=6664) SHIFT TO LEADER ELECTION.

[PID:23590:038 2017.04.01 23:30:18.660 DEBUG ElectionsService ] ELECTIONS: (V=6664) VIEWCHANGE FROM [10.80.42.218:2112, {32945cdd-2c8b-4079-86d1-22b58ec2392b}].

[PID:23590:038 2017.04.01 23:30:18.672 DEBUG ElectionsService ] ELECTIONS: (V=6664) VIEWCHANGE FROM [10.80.42.219:2112, {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72}].

[PID:23590:038 2017.04.01 23:30:18.672 DEBUG ElectionsService ] ELECTIONS: (V=6664) MAJORITY OF VIEWCHANGE.

[PID:23590:038 2017.04.01 23:30:18.672 DEBUG ElectionsService ] ELECTIONS: (V=6664) SHIFT TO PREPARE PHASE.

[PID:23590:038 2017.04.01 23:30:18.673 DEBUG ElectionsService ] ELECTIONS: (V=6664) PREPARE_OK FROM 10.80.42.218:2112,{32945cdd-2c8b-4079-86d1-22b58ec2392b}.

[PID:23590:038 2017.04.01 23:30:18.678 DEBUG ElectionsService ] ELECTIONS: (V=6664) PREPARE_OK FROM 10.80.42.219:2112,{1221bd3c-d4b0-41d2-9b5e-2c764b14ac72}.

[PID:23590:038 2017.04.01 23:30:18.678 DEBUG ElectionsService ] ELECTIONS: (V=6664) SHIFT TO REG_LEADER.

[PID:23590:038 2017.04.01 23:30:18.679 DEBUG ElectionsService ] ELECTIONS: (V=6664) SENDING PROPOSAL CANDIDATE: 10.80.42.218:2112,{32945cdd-2c8b-4079-86d1-22b58ec2392b}, ME: 10.80.42.218:2112,{32945cdd-2c8b-4079-86d1-22b58ec2392b}.

[PID:23590:038 2017.04.01 23:30:18.679 DEBUG ElectionsService ] ELECTIONS: (V=6664) ACCEPT FROM [10.80.42.218:2112,{32945cdd-2c8b-4079-86d1-22b58ec2392b}] M=[10.80.42.218:2112,{32945cdd-2c8b-4079-86d1-22b58ec2392b}]).

[PID:23590:038 2017.04.01 23:30:19.333 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [10.80.42.217:2112])

[PID:23590:038 2017.04.01 23:30:19.334 TRACE GossipServiceBase ] Old:

[PID:23590:038 2017.04.01 23:30:19.334 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [Master, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243544889/23243545386/23243545386/E3061@23243545154:{ec5dc7e7-ecf9-4ab3-bf72-39d4e4608b2e} | 2017-04-01 23:30:18.449

[PID:23590:038 2017.04.01 23:30:19.334 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Unknown, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243545606/23243545871/23243545871/E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91} | 2017-04-01 23:30:18.457

[PID:23590:038 2017.04.01 23:30:19.334 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [Slave, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243526992/23243545386/23243541126/E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396} | 2017-04-01 23:30:18.387

[PID:23590:038 2017.04.01 23:30:19.335 TRACE GossipServiceBase ] New:

[PID:23590:038 2017.04.01 23:30:19.335 TRACE GossipServiceBase ] VND {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72} [Master, 10.80.42.219:1112, n/a, 10.80.42.219:1113, n/a, 10.80.42.219:2112, 10.80.42.219:2113] 23243544889/23243545386/23243545386/E3061@23243545154:{ec5dc7e7-ecf9-4ab3-bf72-39d4e4608b2e} | 2017-04-01 23:30:18.450

[PID:23590:038 2017.04.01 23:30:19.335 TRACE GossipServiceBase ] VND {32945cdd-2c8b-4079-86d1-22b58ec2392b} [Unknown, 10.80.42.218:1112, 10.80.42.218:0, 10.80.42.218:1113, 10.80.42.218:0, 10.80.42.218:2112, 10.80.42.218:2113] 23243545606/23243545871/23243545871/E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91} | 2017-04-01 23:30:19.332

[PID:23590:038 2017.04.01 23:30:19.336 TRACE GossipServiceBase ] VND {705359a8-be32-4a13-aafb-32c100f6dc6c} [Slave, 10.80.42.217:1112, n/a, 10.80.42.217:1113, n/a, 10.80.42.217:2112, 10.80.42.217:2113] 23243549115/23243549248/23243549248/E3061@23243545154:{ec5dc7e7-ecf9-4ab3-bf72-39d4e4608b2e} | 2017-04-01 23:30:19.328

[PID:23590:038 2017.04.01 23:30:19.336 TRACE GossipServiceBase ] --------------------------------------------------------------------------------

[PID:23590:038 2017.04.01 23:30:19.661 DEBUG ElectionsService ] ELECTIONS: (V=6664) TIMED OUT! (S=Leader, M=).

[PID:23590:038 2017.04.01 23:30:19.661 DEBUG ElectionsService ] ELECTIONS: (V=6665) SHIFT TO LEADER ELECTION.

[PID:23590:038 2017.04.01 23:30:19.661 DEBUG ElectionsService ] ELECTIONS: (V=6665) VIEWCHANGE FROM [10.80.42.218:2112, {32945cdd-2c8b-4079-86d1-22b58ec2392b}].

[PID:23590:038 2017.04.01 23:30:19.666 DEBUG ElectionsService ] ELECTIONS: (V=6665) VIEWCHANGE FROM [10.80.42.219:2112, {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72}].

[PID:23590:038 2017.04.01 23:30:19.666 DEBUG ElectionsService ] ELECTIONS: (V=6665) MAJORITY OF VIEWCHANGE.

[PID:23590:038 2017.04.01 23:30:19.666 DEBUG ElectionsService ] ELECTIONS: (V=6665) VIEWCHANGE FROM [10.80.42.217:2112, {705359a8-be32-4a13-aafb-32c100f6dc6c}].

[PID:23590:038 2017.04.01 23:30:19.667 DEBUG ElectionsService ] ELECTIONS: (V=6665) PREPARE FROM [10.80.42.217:2112, {705359a8-be32-4a13-aafb-32c100f6dc6c}].

[PID:23590:038 2017.04.01 23:30:19.667 DEBUG ElectionsService ] ELECTIONS: (V=6665) SHIFT TO REG_NONLEADER.

[PID:23590:038 2017.04.01 23:30:19.671 DEBUG ElectionsService ] ELECTIONS: (V=6665) PROPOSAL FROM [10.80.42.217:2112,{705359a8-be32-4a13-aafb-32c100f6dc6c}] M=10.80.42.219:2112,{1221bd3c-d4b0-41d2-9b5e-2c764b14ac72}. ME=10.80.42.218:2112,{32945cdd-2c8b-4079-86d1-22b58ec2392b}.

[PID:23590:038 2017.04.01 23:30:19.671 DEBUG ElectionsService ] ELECTIONS: (V=6665) ACCEPT FROM [10.80.42.217:2112,{705359a8-be32-4a13-aafb-32c100f6dc6c}] M=[10.80.42.219:2112,{1221bd3c-d4b0-41d2-9b5e-2c764b14ac72}]).

[PID:23590:038 2017.04.01 23:30:19.672 DEBUG ElectionsService ] ELECTIONS: (V=6665) ACCEPT FROM [10.80.42.218:2112,{32945cdd-2c8b-4079-86d1-22b58ec2392b}] M=[10.80.42.219:2112,{1221bd3c-d4b0-41d2-9b5e-2c764b14ac72}]).

[PID:23590:038 2017.04.01 23:30:19.672 INFO ElectionsService ] ELECTIONS: (V=6665) DONE. ELECTED MASTER = 10.80.42.219:2112,{1221bd3c-d4b0-41d2-9b5e-2c764b14ac72}. ME=10.80.42.218:2112,{32945cdd-2c8b-4079-86d1-22b58ec2392b}.

[PID:23590:038 2017.04.01 23:30:19.672 INFO ClusterVNodeControll] ========== [10.80.42.218:2112] PRE-REPLICA STATE, WAITING FOR CHASER TO CATCH UP… MASTER IS [10.80.42.219:2112,{1221bd3c-d4b0-41d2-9b5e-2c764b14ac72}]

[PID:23590:123 2017.04.01 23:30:19.672 DEBUG PersistentSubscripti] Subscriptions received state change to PreReplica stopping listening.

[PID:23590:038 2017.04.01 23:30:19.673 DEBUG ElectionsService ] ELECTIONS: (V=6665) ACCEPT FROM [10.80.42.219:2112,{1221bd3c-d4b0-41d2-9b5e-2c764b14ac72}] M=[10.80.42.219:2112,{1221bd3c-d4b0-41d2-9b5e-2c764b14ac72}]).

[PID:23590:130 2017.04.01 23:30:19.673 INFO TcpConnectionManager] Connection ‘master-normal’ ({37f7bfde-1480-417c-9a40-0bc15c07f20b}) to [10.80.42.219:1112] established.

[PID:23590:038 2017.04.01 23:30:19.673 INFO ReplicaService ] Subscribing at LogPosition: 23243545871 (0x5696C5D0F) to MASTER [10.80.42.219:1112, {1221bd3c-d4b0-41d2-9b5e-2c764b14ac72}] as replica with SubscriptionId: {7e76881c-9a26-41d7-9e81-2eb448b737f1}, ConnectionId: {37f7bfde-1480-417c-9a40-0bc15c07f20b}, LocalEndPoint: [10.80.42.218:33459], Epochs:

E3060@23243541232:{58bf9227-ac0c-4edb-a373-ee67a8b84e91}

E3059@23236027113:{5ec2f9f7-77e7-435b-900a-872a78cdf396}

E3058@23236026882:{19249e5d-6b27-4d08-8d19-2b55bd32b84a}

E3057@23235720749:{73fa7146-b343-4f70-9e49-54ba9e466264}

E3056@23235706408:{a21d5f90-07a6-499b-baa7-75515fee141a}

E3055@23230975805:{e820e911-590e-4f99-b6f7-452a5c479c9a}

E3054@23230975573:{ac0c638d-987a-42f9-8089-4554fe310522}

E3053@23230975341:{ad13323b-53fb-4291-b9b7-c00817bbf280}

E3052@23230387757:{043aa4e3-9dd9-4724-b0ee-b9551f9cfae8}

E3051@23230155892:{20e19804-1b44-46cf-ab5f-cf1cf980f340}

E3050@23176837486:{2f5c1336-5fec-4079-8548-be05447da8bf}

E3049@23176837254:{841d7292-ad06-4b5e-af7b-13d508e12b2f}

E3048@23092852866:{ecc191fc-ff2f-4d48-b369-290035db53f5}

E3047@23092525361:{d9465e8b-00b3-4ce3-8c19-4b912aa23fe9}

E3046@23092524133:{82d88259-7b38-4ea5-87eb-86c694562adb}

E3045@23092523901:{1a8729c5-ccbc-4a09-8b9f-2ae1cf7749f9}

E3044@23070038265:{86f6baaa-570e-4c7d-acad-fa1e4a7d5331}

E3043@23070037037:{09ba0853-7718-456a-bdf2-d628de343288}

E3042@23070035810:{d097b597-ec42-4b10-ad14-a7e458cbf943}

E3041@23069607034:{bb8588db-6bec-4d7f-8890-8732315e39c0}…

.

[PID:23590:038 2017.04.01 23:30:19.703 INFO ClusterVNodeControll] ========== [10.80.42.218:2112] IS CATCHING UP… MASTER IS [10.80.42.219:2112,{1221bd3c-d4b0-41d2-9b5e-2c764b14ac72}]

[PID:23590:035 2017.04.01 23:30:19.703 INFO ClusterStorageWriter] === SUBSCRIBED to [10.80.42.219:1112,{1221bd3c-d4b0-41d2-9b5e-2c764b14ac72}] at 23243545154 (0x5696C5A42). SubscriptionId: {7e76881c-9a26-41d7-9e81-2eb448b737f1}.

[PID:23590:128 2017.04.01 23:30:19.703 DEBUG PersistentSubscripti] Subscriptions received state change to CatchingUp stopping listening.

[PID:23590:035 2017.04.01 23:30:19.703 INFO ClusterStorageWriter] Master [10.80.42.219:1112,{1221bd3c-d4b0-41d2-9b5e-2c764b14ac72}] subscribed us at 23243545154 (0x5696C5A42), which is less than our writer checkpoint 23243545871 (0x5696C5D0F). TRUNCATION IS NEEDED.

[PID:23590:035 2017.04.01 23:30:19.704 INFO ClusterStorageWriter] OFFLINE TRUNCATION IS NEEDED (SubscribedAt 23243545154 (0x5696C5A42) <= LastCommitPosition 23243545606 (0x5696C5C06)). SHUTTING DOWN NODE.

[PID:23590:038 2017.04.01 23:30:19.717 INFO ClusterVNodeControll] ========== [10.80.42.218:2112] CLONE ASSIGNMENT RECEIVED FROM [10.80.42.219:1112,n/a,{1221bd3c-d4b0-41d2-9b5e-2c764b14ac72}].

[PID:23590:038 2017.04.01 23:30:19.717 INFO ClusterVNodeControll] ========== [10.80.42.218:2112] IS CLONE… MASTER IS [10.80.42.219:2112,{1221bd3c-d4b0-41d2-9b5e-2c764b14ac72}]

[PID:23590:103 2017.04.01 23:30:19.717 DEBUG PersistentSubscripti] Subscriptions received state change to Clone stopping listening.

[PID:23590:038 2017.04.01 23:30:19.717 INFO ClusterVNodeControll] ========== [10.80.42.218:2112] SLAVE ASSIGNMENT RECEIVED FROM [10.80.42.219:1112,n/a,{1221bd3c-d4b0-41d2-9b5e-2c764b14ac72}].

[PID:23590:038 2017.04.01 23:30:19.718 INFO ClusterVNodeControll] ========== [10.80.42.218:2112] IS SLAVE… MASTER IS [10.80.42.219:2112,{1221bd3c-d4b0-41d2-9b5e-2c764b14ac72}]

[PID:23590:134 2017.04.01 23:30:19.718 DEBUG PersistentSubscripti] Subscriptions received state change to Slave stopping listening.

[PID:23590:060 2017.04.01 23:30:19.718 DEBUG ProjectionCoreCoordi] PROJECTIONS: Starting Projections Core Coordinator. (Node State : Slave)

[PID:23590:063 2017.04.01 23:30:19.719 DEBUG ProjectionCoreServic] PROJECTIONS: Starting Projection Core Reader (reads from $projections-$53b82ad67dc3408cb95d1e0ecd63f805)

[PID:23590:061 2017.04.01 23:30:19.719 DEBUG ProjectionCoreServic] PROJECTIONS: Starting Projection Core Reader (reads from $projections-$c7250f6f159c4cb2864a07af5c40d728)

[PID:23590:062 2017.04.01 23:30:19.719 DEBUG ProjectionCoreServic] PROJECTIONS: Starting Projection Core Reader (reads from $projections-$03d3b27c331a401787f533553bbc2ddd)

[PID:23590:061 2017.04.01 23:30:19.720 DEBUG ResponseWriter ] PROJECTIONS: Resetting Master Writer

[PID:23590:062 2017.04.01 23:30:19.720 DEBUG ResponseWriter ] PROJECTIONS: Resetting Master Writer

[PID:23590:063 2017.04.01 23:30:19.719 DEBUG ResponseWriter ] PROJECTIONS: Resetting Master Writer

[PID:23590:061 2017.04.01 23:30:19.720 DEBUG ProjectionCoreServic] PROJECTIONS: Starting read control from 80

[PID:23590:063 2017.04.01 23:30:19.720 DEBUG ProjectionCoreServic] PROJECTIONS: Starting read control from 80

[PID:23590:062 2017.04.01 23:30:19.720 DEBUG ProjectionCoreServic] PROJECTIONS: Starting read control from 80

[PID:23590:035 2017.04.01 23:30:19.724 TRACE StorageWriterService] Blocking message DataChunkBulk in StorageWriterService. Message:

DataChunkBulk message: MasterId: 1221bd3c-d4b0-41d2-9b5e-2c764b14ac72, SubscriptionId: 7e76881c-9a26-41d7-9e81-2eb448b737f1, ChunkStartNumber: 86, ChunkEndNumber: 86, SubscriptionPosition: 23243545154, DataBytes length: 4592, CompleteChunk: False

[PID:23590:038 2017.04.01 23:30:19.726 INFO ClusterVNodeControll] ========== [10.80.42.218:2112] IS SHUTTING DOWN…

[PID:23590:038 2017.04.01 23:30:19.828 TRACE InMemoryBus ] SLOW BUS MSG [MainBus]: BecomeShuttingDown - 100ms. Handler: WideningHandler`2.

[PID:23590:128 2017.04.01 23:30:19.838 FATAL GLOBAL-LOGGER ] Global Unhandled Exception occurred.

System.ObjectDisposedException: The object was used after being disposed.

at System.Net.Sockets.Socket.AcceptAsync (System.Net.Sockets.SocketAsyncEventArgs e) [0x00000] in :0

at System.Net.EndPointListener.Accept (System.Net.Sockets.Socket socket, System.Net.Sockets.SocketAsyncEventArgs e) [0x00000] in :0

at System.Net.EndPointListener.ProcessAccept (System.Net.Sockets.SocketAsyncEventArgs args) [0x00000] in :0

at System.Net.EndPointListener.OnAccept (System.Object sender, System.Net.Sockets.SocketAsyncEventArgs e) [0x00000] in :0

at System.Net.Sockets.SocketAsyncEventArgs.OnCompleted (System.Net.Sockets.SocketAsyncEventArgs e) [0x00000] in :0

at System.Net.Sockets.SocketAsyncEventArgs.AcceptCallback (IAsyncResult ares) [0x00000] in :0

at System.Net.Sockets.SocketAsyncEventArgs.DispatcherCB (IAsyncResult ares) [0x00000] in :0

[PID:23590:038 2017.04.01 23:30:19.840 ERROR HttpAsyncServer ] Error while shutting down http server

System.IO.IOException: Write failure —> System.Net.Sockets.SocketException: Connection reset by peer

at System.Net.Sockets.Socket.Send (System.Byte[] buf, Int32 offset, Int32 size, SocketFlags flags) [0x00000] in :0

at System.Net.Sockets.NetworkStream.Write (System.Byte[] buffer, Int32 offset, Int32 size) [0x00000] in :0

— End of inner exception stack trace —

at System.Net.Sockets.NetworkStream.Write (System.Byte[] buffer, Int32 offset, Int32 size) [0x00000] in :0

at System.Net.ResponseStream.InternalWrite (System.Byte[] buffer, Int32 offset, Int32 count) [0x00000] in :0

at System.Net.ResponseStream.Close () [0x00000] in :0

at System.Net.HttpConnection.Close (Boolean force_close) [0x00000] in :0

at System.Net.HttpListener.Cleanup (Boolean close_existing) [0x00000] in :0

at System.Net.HttpListener.Close (Boolean force) [0x00000] in :0

at System.Net.HttpListener.Abort () [0x00000] in :0

at EventStore.Transport.Http.Server.HttpAsyncServer.Shutdown () [0x00000] in :0

System.Net.Sockets.SocketException: Connection reset by peer

at System.Net.Sockets.Socket.Send (System.Byte[] buf, Int32 offset, Int32 size, SocketFlags flags) [0x00000] in :0

at System.Net.Sockets.NetworkStream.Write (System.Byte[] buffer, Int32 offset, Int32 size) [0x00000] in :0

[PID:23590:132 2017.04.01 23:30:19.841 FATAL GLOBAL-LOGGER ] Global Unhandled Exception occurred.