[PID:03068:012 2015.07.04 02:14:23.740 TRACE InMemoryBus ] SLOW BUS MSG [bus]: UnwrapEnvelopeMessage - 99ms. Handler: UnwrapEnvelopeHandler. [PID:03068:012 2015.07.04 02:14:23.740 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [Projection Core #0]: UnwrapEnvelopeMessage - 117ms. Q: 0/2. [PID:03068:010 2015.07.04 04:25:37.103 TRACE GossipServiceBase ] Looks like master [192.168.33.12:2113, {5181f33b-6a45-4afc-a3a1-bc797944b79d}] is DEAD (Gossip send failed), though we wait for TCP to decide. [PID:03068:010 2015.07.04 07:05:39.159 TRACE GossipServiceBase ] Looks like master [192.168.33.12:2113, {5181f33b-6a45-4afc-a3a1-bc797944b79d}] is DEAD (Gossip send failed), though we wait for TCP to decide. [PID:03068:150 2015.07.04 08:37:20.783 TRACE InMemoryBus ] SLOW BUS MSG [SubscriptionsBus]: CheckPollTimeout - 99ms. Handler: SubscriptionsService. [PID:03068:010 2015.07.04 09:42:51.003 TRACE GossipServiceBase ] Looks like master [192.168.33.12:2113, {5181f33b-6a45-4afc-a3a1-bc797944b79d}] is DEAD (Gossip send failed), though we wait for TCP to decide. [PID:03068:010 2015.07.04 12:01:02.660 TRACE GossipServiceBase ] Looks like master [192.168.33.12:2113, {5181f33b-6a45-4afc-a3a1-bc797944b79d}] is DEAD (Gossip send failed), though we wait for TCP to decide. [PID:03068:010 2015.07.04 13:17:50.616 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.13:2113]) Old: VND {5181f33b-6a45-4afc-a3a1-bc797944b79d} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36593099/36610231/36610231/E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} | 2015-07-04 13:17:50.012 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36593099/36610231/36610231/E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} | 2015-07-04 13:17:50.003 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Unknown, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 482703/499692/499692/E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} | 2015-07-04 13:17:50.532 VND {5181f33b-6a45-4afc-a3a1-bc797944b79d} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36593099/36610231/36610231/E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} | 2015-07-04 13:17:50.012 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36593099/36610231/36610231/E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} | 2015-07-04 13:17:50.616 -------------------------------------------------------------------------------- [PID:03068:175 2015.07.04 13:17:51.769 INFO TcpConnection ] ES TcpConnection closed [13:17:51.769: N192.168.33.12:1111, L192.168.33.11:51455, {ce79e6d3-8e42-4c37-92fa-7ff0a2fa7772}]: Received bytes: 39284342, Sent bytes: 891583 Send calls: 38856, callbacks: 38856 Receive calls: 46443, callbacks: 46443 Close reason: [Success] Socket closed [PID:03068:175 2015.07.04 13:17:51.769 INFO TcpConnectionManager] Connection 'master-normal' [192.168.33.12:1111, {ce79e6d3-8e42-4c37-92fa-7ff0a2fa7772}] closed: Success. [PID:03068:010 2015.07.04 13:17:51.769 TRACE GossipServiceBase ] Looks like node [192.168.33.12:1111] is DEAD (TCP connection lost). [PID:03068:010 2015.07.04 13:17:51.769 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection lost to [192.168.33.12:1111]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Unknown, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 482703/499692/499692/E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} | 2015-07-04 13:17:51.239 VND {5181f33b-6a45-4afc-a3a1-bc797944b79d} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36593099/36610231/36610231/E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} | 2015-07-04 13:17:50.873 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36593099/36610231/36610231/E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} | 2015-07-04 13:17:51.276 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Unknown, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 482703/499692/499692/E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} | 2015-07-04 13:17:51.239 VND {5181f33b-6a45-4afc-a3a1-bc797944b79d} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36593099/36610231/36610231/E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} | 2015-07-04 13:17:51.770 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36593099/36610231/36610231/E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} | 2015-07-04 13:17:51.276 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:17:51.769 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {5181f33b-6a45-4afc-a3a1-bc797944b79d}, InternalTcp: 192.168.33.12:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.12:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.12:2113, ExternalHttp: 192.168.33.12:2114]. [PID:03068:010 2015.07.04 13:17:51.769 DEBUG ElectionsService ] ELECTIONS: STARTING ELECTIONS. [PID:03068:010 2015.07.04 13:17:51.769 DEBUG ElectionsService ] ELECTIONS: (V=40) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:17:51.769 DEBUG ElectionsService ] ELECTIONS: (V=40) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:17:52.271 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] PRE-REPLICA STATE, WAITING FOR CHASER TO CATCH UP... MASTER IS [192.168.33.12:2113,{5181f33b-6a45-4afc-a3a1-bc797944b79d}] [PID:03068:010 2015.07.04 13:17:52.271 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L192.168.33.11:51455, {ce79e6d3-8e42-4c37-92fa-7ff0a2fa7772}] cleanly. Reason: Reconnecting from old master [192.168.33.12:1111] to new master: [192.168.33.12:1111]. [PID:03068:010 2015.07.04 13:17:52.272 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {5181f33b-6a45-4afc-a3a1-bc797944b79d}, InternalTcp: 192.168.33.12:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.12:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.12:2113, ExternalHttp: 192.168.33.12:2114]. [PID:03068:013 2015.07.04 13:17:52.272 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:03068:012 2015.07.04 13:17:52.272 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:03068:014 2015.07.04 13:17:52.272 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:03068:010 2015.07.04 13:17:52.272 INFO ReplicaService ] Subscribing at LogPosition: 36610231 (0x22EA0B7) to MASTER [192.168.33.12:1111, {5181f33b-6a45-4afc-a3a1-bc797944b79d}] as replica with SubscriptionId: {f82ebebf-c28f-4a17-b52e-f39877da0913}, ConnectionId: {19d9bdd4-6a7d-4da6-b568-62aaa841e37e}, LocalEndPoint: [], Epochs: E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740} E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65} E30@310956:{26cd0216-4e18-43aa-a899-11057cb44b2f} E29@293661:{e83f9ab8-c0e1-4b7f-ad41-504a907b90d1} E28@286573:{219b52c3-b7dd-4a47-834a-730aa0e69066} E27@247371:{d635cf1b-4fac-4c60-b507-8ba566601c8f} E26@247151:{44eaa2de-fcaa-41db-b5b7-c0fb3add327d} E25@246931:{fcf0df26-2741-43a6-a8d6-3cdb1b0d9206} E24@229645:{1d2c878f-7071-4e70-9e12-bc951df9797f} E23@195248:{26766374-f4bf-4628-afd9-d948e7e4d2ea} E22@195029:{1ca233f2-7ead-48f8-8a13-3f392cf6dfa5}... . [PID:03068:175 2015.07.04 13:17:52.274 INFO TcpConnectionManager] Connection 'master-normal' ({19d9bdd4-6a7d-4da6-b568-62aaa841e37e}) to [192.168.33.12:1111] failed: ConnectionRefused. [PID:03068:010 2015.07.04 13:17:52.771 DEBUG ElectionsService ] ELECTIONS: (V=40) TIMED OUT! (S=ElectingLeader, M=). [PID:03068:010 2015.07.04 13:17:52.771 DEBUG ElectionsService ] ELECTIONS: (V=41) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:17:52.771 DEBUG ElectionsService ] ELECTIONS: (V=41) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:17:52.774 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L, {19d9bdd4-6a7d-4da6-b568-62aaa841e37e}] cleanly. Reason: Reconnecting from old master [192.168.33.12:1111] to new master: [192.168.33.12:1111]. [PID:03068:010 2015.07.04 13:17:52.774 INFO TcpConnection ] ES TcpConnection closed [13:17:52.775: N192.168.33.12:1111, L, {19d9bdd4-6a7d-4da6-b568-62aaa841e37e}]: Received bytes: 0, Sent bytes: 0 Send calls: 0, callbacks: 0 Receive calls: 0, callbacks: 0 Close reason: [Success] Reconnecting from old master [192.168.33.12:1111] to new master: [192.168.33.12:1111]. [PID:03068:175 2015.07.04 13:17:52.776 INFO TcpConnectionManager] Connection 'master-normal' ({14bff519-c100-4ed7-8510-be799b8db478}) to [192.168.33.12:1111] failed: ConnectionRefused. [PID:03068:010 2015.07.04 13:17:53.274 INFO ReplicaService ] Subscribing at LogPosition: 36610231 (0x22EA0B7) to MASTER [192.168.33.12:1111, {5181f33b-6a45-4afc-a3a1-bc797944b79d}] as replica with SubscriptionId: {de3a4996-b38f-45ad-b6b4-99dcd7b7c9d9}, ConnectionId: {14bff519-c100-4ed7-8510-be799b8db478}, LocalEndPoint: [], Epochs: E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740} E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65} E30@310956:{26cd0216-4e18-43aa-a899-11057cb44b2f} E29@293661:{e83f9ab8-c0e1-4b7f-ad41-504a907b90d1} E28@286573:{219b52c3-b7dd-4a47-834a-730aa0e69066} E27@247371:{d635cf1b-4fac-4c60-b507-8ba566601c8f} E26@247151:{44eaa2de-fcaa-41db-b5b7-c0fb3add327d} E25@246931:{fcf0df26-2741-43a6-a8d6-3cdb1b0d9206} E24@229645:{1d2c878f-7071-4e70-9e12-bc951df9797f} E23@195248:{26766374-f4bf-4628-afd9-d948e7e4d2ea} E22@195029:{1ca233f2-7ead-48f8-8a13-3f392cf6dfa5}... . [PID:03068:010 2015.07.04 13:17:53.276 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L, {14bff519-c100-4ed7-8510-be799b8db478}] cleanly. Reason: Reconnecting from old master [192.168.33.12:1111] to new master: [192.168.33.12:1111]. [PID:03068:010 2015.07.04 13:17:53.276 INFO TcpConnection ] ES TcpConnection closed [13:17:53.276: N192.168.33.12:1111, L, {14bff519-c100-4ed7-8510-be799b8db478}]: Received bytes: 0, Sent bytes: 0 Send calls: 0, callbacks: 0 Receive calls: 0, callbacks: 0 Close reason: [Success] Reconnecting from old master [192.168.33.12:1111] to new master: [192.168.33.12:1111]. [PID:03068:177 2015.07.04 13:17:53.276 INFO TcpConnectionManager] Connection 'master-normal' ({52b20836-a351-4658-8142-fba6a135fd05}) to [192.168.33.12:1111] failed: ConnectionRefused. [PID:03068:010 2015.07.04 13:17:53.772 DEBUG ElectionsService ] ELECTIONS: (V=41) TIMED OUT! (S=ElectingLeader, M=). [PID:03068:010 2015.07.04 13:17:53.772 DEBUG ElectionsService ] ELECTIONS: (V=42) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:17:53.772 DEBUG ElectionsService ] ELECTIONS: (V=42) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:17:53.777 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L, {52b20836-a351-4658-8142-fba6a135fd05}] cleanly. Reason: Reconnecting from old master [192.168.33.12:1111] to new master: [192.168.33.12:1111]. [PID:03068:010 2015.07.04 13:17:53.777 INFO TcpConnection ] ES TcpConnection closed [13:17:53.777: N192.168.33.12:1111, L, {52b20836-a351-4658-8142-fba6a135fd05}]: Received bytes: 0, Sent bytes: 0 Send calls: 0, callbacks: 0 Receive calls: 0, callbacks: 0 Close reason: [Success] Reconnecting from old master [192.168.33.12:1111] to new master: [192.168.33.12:1111]. [PID:03068:175 2015.07.04 13:17:53.777 INFO TcpConnectionManager] Connection 'master-normal' ({9ba23915-203d-4be8-b10a-aec2a4be79be}) to [192.168.33.12:1111] failed: ConnectionRefused. [PID:03068:010 2015.07.04 13:17:54.045 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {5181f33b-6a45-4afc-a3a1-bc797944b79d}, InternalTcp: 192.168.33.12:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.12:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.12:2113, ExternalHttp: 192.168.33.12:2114]. [PID:03068:010 2015.07.04 13:17:54.051 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {5181f33b-6a45-4afc-a3a1-bc797944b79d}, InternalTcp: 192.168.33.12:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.12:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.12:2113, ExternalHttp: 192.168.33.12:2114]. [PID:03068:010 2015.07.04 13:17:54.052 DEBUG ElectionsService ] ELECTIONS: (V=41) VIEWCHANGE FROM [192.168.33.13:2113, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:17:54.052 DEBUG ElectionsService ] ELECTIONS: (V=41) MAJORITY OF VIEWCHANGE. [PID:03068:010 2015.07.04 13:17:54.052 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {5181f33b-6a45-4afc-a3a1-bc797944b79d}, InternalTcp: 192.168.33.12:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.12:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.12:2113, ExternalHttp: 192.168.33.12:2114]. [PID:03068:010 2015.07.04 13:17:54.053 DEBUG ElectionsService ] ELECTIONS: (V=42) VIEWCHANGE FROM [192.168.33.13:2113, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:17:54.053 DEBUG ElectionsService ] ELECTIONS: (V=42) PREPARE FROM [192.168.33.13:2113, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:17:54.053 DEBUG ElectionsService ] ELECTIONS: (V=42) SHIFT TO REG_NONLEADER. [PID:03068:010 2015.07.04 13:17:54.067 DEBUG ElectionsService ] ELECTIONS: (V=42) PROPOSAL FROM [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] M=[192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}](L=36593099,W=36610231,C=36610231,E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b}). ME=[192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}](L=36593099,W=36610231,C=36610231,E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b}). [PID:03068:010 2015.07.04 13:17:54.067 DEBUG ElectionsService ] ELECTIONS: (V=42) ACCEPT FROM [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] M=[192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]). [PID:03068:010 2015.07.04 13:17:54.067 DEBUG ElectionsService ] ELECTIONS: (V=42) ACCEPT FROM [192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}] M=[192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]). [PID:03068:010 2015.07.04 13:17:54.067 INFO ElectionsService ] ELECTIONS: (V=42) DONE. ELECTED MASTER = [192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}](L=36593099,W=36610231,C=36610231,E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b}). ME=[192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}](L=36593099,W=36610231,C=36610231,E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b}). [PID:03068:010 2015.07.04 13:17:54.067 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] PRE-MASTER STATE, WAITING FOR CHASER TO CATCH UP... [PID:03068:010 2015.07.04 13:17:54.067 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L, {9ba23915-203d-4be8-b10a-aec2a4be79be}] cleanly. Reason: Node state changed to PreMaster. Closing replication connection. [PID:03068:010 2015.07.04 13:17:54.067 INFO TcpConnection ] ES TcpConnection closed [13:17:54.067: N192.168.33.12:1111, L, {9ba23915-203d-4be8-b10a-aec2a4be79be}]: Received bytes: 0, Sent bytes: 0 Send calls: 0, callbacks: 0 Receive calls: 0, callbacks: 0 Close reason: [Success] Node state changed to PreMaster. Closing replication connection. [PID:03068:010 2015.07.04 13:17:54.069 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS MASTER!!! SPARTA!!! [PID:03068:007 2015.07.04 13:17:54.071 DEBUG EpochManager ] === Writing E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} (previous epoch at 499692). [PID:03068:007 2015.07.04 13:17:54.073 DEBUG EpochManager ] === Update Last Epoch E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} (previous epoch at 499692). [PID:03068:177 2015.07.04 13:17:54.088 INFO TcpService ] Internal TCP connection accepted: [Normal, 192.168.33.13:32966, L192.168.33.11:1111, {a579d8ed-4512-4c80-a2b7-775aab971341}]. [PID:03068:175 2015.07.04 13:17:54.098 TRACE MonitoringService ] Created stats stream '$stats-192.168.33.11:2114', code = WrongExpectedVersion [PID:03068:010 2015.07.04 13:17:54.151 INFO MasterReplicationSer] SUBSCRIBE REQUEST from [192.168.33.13:1111,C:{a579d8ed-4512-4c80-a2b7-775aab971341},S:{77e047f5-0755-4a60-9606-2873104d3a7a},499692(0x79FEC),E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76}, E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812}, E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9}, E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845}, E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864}, E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed}, E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0}, E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff}, E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740}, E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65}, E30@310956:{26cd0216-4e18-43aa-a899-11057cb44b2f}, E29@293661:{e83f9ab8-c0e1-4b7f-ad41-504a907b90d1}, E28@286573:{219b52c3-b7dd-4a47-834a-730aa0e69066}, E27@247371:{d635cf1b-4fac-4c60-b507-8ba566601c8f}, E26@247151:{44eaa2de-fcaa-41db-b5b7-c0fb3add327d}, E25@246931:{fcf0df26-2741-43a6-a8d6-3cdb1b0d9206}, E24@229645:{1d2c878f-7071-4e70-9e12-bc951df9797f}, E23@195248:{26766374-f4bf-4628-afd9-d948e7e4d2ea}, E22@195029:{1ca233f2-7ead-48f8-8a13-3f392cf6dfa5}, E21@194809:{85807fde-e0c2-48de-9e1d-a296dfdea4fe}]... [PID:03068:010 2015.07.04 13:17:54.152 INFO MasterReplicationSer] Subscribed replica [192.168.33.13:1111,S:77e047f5-0755-4a60-9606-2873104d3a7a] for data send at 499692 (0x79FEC). [PID:03068:010 2015.07.04 13:17:54.447 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Unknown, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 482703/499692/499692/E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} | 2015-07-04 13:17:54.153 VND {5181f33b-6a45-4afc-a3a1-bc797944b79d} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36593099/36610231/36610231/E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} | 2015-07-04 13:17:54.045 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36593099/36610231/36610231/E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} | 2015-07-04 13:17:54.069 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Unknown, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 482703/499692/499692/E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} | 2015-07-04 13:17:54.153 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Unknown, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36593099/36610231/36610231/E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} | 2015-07-04 13:17:54.119 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36611409/36611582/36611582/E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} | 2015-07-04 13:17:54.447 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:17:55.199 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.13:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Unknown, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 482703/499692/499692/E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} | 2015-07-04 13:17:54.153 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Unknown, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36593099/36610231/36610231/E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} | 2015-07-04 13:17:54.119 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36611409/36611582/36611582/E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} | 2015-07-04 13:17:54.997 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [CatchingUp, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 5489801/5797840/5506905/E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} | 2015-07-04 13:17:54.579 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Unknown, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36593099/36610231/36610231/E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} | 2015-07-04 13:17:54.119 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36611409/36611582/36611582/E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} | 2015-07-04 13:17:55.199 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:17:56.494 TRACE TwoPhaseRequestManag] IDEMPOTENT WRITE TO STREAM ClientCorrelationID d4be1867-1237-4a6b-ae2b-b829338213bf, EventStreamId: $projections-$master, CorrelationId: d4be1867-1237-4a6b-ae2b-b829338213bf, FirstEventNumber: 325, LastEventNumber: 325. [PID:03068:010 2015.07.04 13:17:56.494 TRACE TwoPhaseRequestManag] IDEMPOTENT WRITE TO STREAM ClientCorrelationID 451ed08f-db2a-42e3-8be1-b83eba4334bd, EventStreamId: $projections-$master, CorrelationId: 451ed08f-db2a-42e3-8be1-b83eba4334bd, FirstEventNumber: 326, LastEventNumber: 326. [PID:03068:010 2015.07.04 13:17:56.494 TRACE TwoPhaseRequestManag] IDEMPOTENT WRITE TO STREAM ClientCorrelationID 1574db80-8a02-49af-a4cd-b43db8aedf38, EventStreamId: $projections-$master, CorrelationId: 1574db80-8a02-49af-a4cd-b43db8aedf38, FirstEventNumber: 324, LastEventNumber: 324. [PID:03068:175 2015.07.04 13:17:56.969 DEBUG GossipController ] Error while reading request (gossip): EndRead failure [PID:03068:010 2015.07.04 13:17:57.709 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.13:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [CatchingUp, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 29605047/29622159/29622159/E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} | 2015-07-04 13:17:56.091 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Unknown, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36593099/36610231/36610231/E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} | 2015-07-04 13:17:54.119 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36612034/36612193/36612193/E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} | 2015-07-04 13:17:56.998 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36612034/36612193/36612193/E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} | 2015-07-04 13:17:57.669 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Unknown, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36593099/36610231/36610231/E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} | 2015-07-04 13:17:54.119 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36612034/36612193/36612193/E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} | 2015-07-04 13:17:57.708 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:17:57.974 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36612034/36612193/36612193/E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} | 2015-07-04 13:17:57.669 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Unknown, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36593099/36610231/36610231/E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} | 2015-07-04 13:17:56.089 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36612034/36612193/36612193/E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} | 2015-07-04 13:17:57.964 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Clone, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36612034/36612193/36612193/E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} | 2015-07-04 13:17:57.967 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Unknown, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36593099/36610231/36610231/E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} | 2015-07-04 13:17:57.975 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36612034/36612193/36612193/E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} | 2015-07-04 13:17:57.974 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:17:58.450 DEBUG ElectionsService ] ELECTIONS: (V=43) VIEWCHANGE FROM [192.168.33.12:2113, {6648e0c4-835f-4575-a12e-0f82442ed57e}]. [PID:03068:010 2015.07.04 13:17:58.450 DEBUG ElectionsService ] ELECTIONS: (V=43) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:17:58.450 DEBUG ElectionsService ] ELECTIONS: (V=43) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:17:58.450 DEBUG ElectionsService ] ELECTIONS: (V=43) MAJORITY OF VIEWCHANGE. [PID:03068:010 2015.07.04 13:17:58.450 DEBUG ElectionsService ] ELECTIONS: (V=43) SHIFT TO PREPARE PHASE. [PID:03068:010 2015.07.04 13:17:58.450 DEBUG ElectionsService ] ELECTIONS: (V=43) PREPARE_OK FROM [192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}](L=36612034,W=36612193,C=36612193,E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5}). [PID:03068:010 2015.07.04 13:17:58.879 DEBUG ElectionsService ] ELECTIONS: (V=43) PREPARE_OK FROM [192.168.33.12:2113,{6648e0c4-835f-4575-a12e-0f82442ed57e}](L=36593099,W=36610231,C=36610231,E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b}). [PID:03068:010 2015.07.04 13:17:58.879 DEBUG ElectionsService ] ELECTIONS: (V=43) SHIFT TO REG_LEADER. [PID:03068:010 2015.07.04 13:17:58.879 DEBUG ElectionsService ] ELECTIONS: (V=43) SENDING PROPOSAL CANDIDATE: [192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}](L=36612034,W=36612193,C=36612193,E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5}), ME: [192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}](L=36612034,W=36612193,C=36612193,E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5}). [PID:03068:010 2015.07.04 13:17:58.879 DEBUG ElectionsService ] ELECTIONS: (V=43) ACCEPT FROM [192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}] M=[192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]). [PID:03068:177 2015.07.04 13:17:58.900 INFO TcpService ] Internal TCP connection accepted: [Normal, 192.168.33.12:38609, L192.168.33.11:1111, {f6710670-1730-49c6-b3ba-9fd60b480cfb}]. [PID:03068:010 2015.07.04 13:17:58.948 DEBUG ElectionsService ] ELECTIONS: (V=43) ACCEPT FROM [192.168.33.12:2113,{6648e0c4-835f-4575-a12e-0f82442ed57e}] M=[192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]). [PID:03068:010 2015.07.04 13:17:58.948 INFO ElectionsService ] ELECTIONS: (V=43) DONE. ELECTED MASTER = [192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}](L=36612034,W=36612193,C=36612193,E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5}). ME=[192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}](L=36612034,W=36612193,C=36612193,E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5}). [PID:03068:007 2015.07.04 13:17:58.949 DEBUG EpochManager ] === Writing E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} (previous epoch at 36610231). [PID:03068:007 2015.07.04 13:17:58.954 DEBUG EpochManager ] === Update Last Epoch E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} (previous epoch at 36610231). [PID:03068:010 2015.07.04 13:17:58.991 INFO MasterReplicationSer] SUBSCRIBE REQUEST from [192.168.33.12:1111,C:{f6710670-1730-49c6-b3ba-9fd60b480cfb},S:{71b0b116-4f48-46fc-a949-9efea4e11a18},36610231(0x22EA0B7),E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b}, E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76}, E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812}, E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9}, E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845}, E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864}, E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed}, E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0}, E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff}, E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740}, E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65}, E30@310956:{26cd0216-4e18-43aa-a899-11057cb44b2f}, E29@293661:{e83f9ab8-c0e1-4b7f-ad41-504a907b90d1}, E28@286573:{219b52c3-b7dd-4a47-834a-730aa0e69066}, E27@247371:{d635cf1b-4fac-4c60-b507-8ba566601c8f}, E26@247151:{44eaa2de-fcaa-41db-b5b7-c0fb3add327d}, E25@246931:{fcf0df26-2741-43a6-a8d6-3cdb1b0d9206}, E24@229645:{1d2c878f-7071-4e70-9e12-bc951df9797f}, E23@195248:{26766374-f4bf-4628-afd9-d948e7e4d2ea}, E22@195029:{1ca233f2-7ead-48f8-8a13-3f392cf6dfa5}]... [PID:03068:010 2015.07.04 13:17:58.991 INFO MasterReplicationSer] Subscribed replica [192.168.33.12:1111,S:71b0b116-4f48-46fc-a949-9efea4e11a18] for data send at 36610231 (0x22EA0B7). [PID:03068:010 2015.07.04 13:17:59.002 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.13:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Clone, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36612034/36612193/36612193/E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} | 2015-07-04 13:17:57.967 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Unknown, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36593099/36610231/36610231/E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} | 2015-07-04 13:17:58.991 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36612034/36612417/36612417/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:17:58.999 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36612034/36612417/36612417/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:17:58.966 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Unknown, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36593099/36610231/36610231/E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} | 2015-07-04 13:17:58.991 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36612034/36612417/36612417/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:17:59.002 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:17:59.709 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.13:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36612034/36612417/36612417/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:17:58.966 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Unknown, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36593099/36610231/36610231/E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} | 2015-07-04 13:17:58.991 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36612034/36612417/36612417/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:17:59.002 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36623545/36624215/36623736/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:17:59.671 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Clone, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36612034/36612417/36612417/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:17:59.092 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36624215/36624682/36624682/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:17:59.708 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:18:00.014 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36623545/36624215/36623736/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:17:59.671 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Clone, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36612034/36612417/36612417/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:17:59.092 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36631972/36632510/36632510/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:17:59.999 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36623545/36624215/36623736/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:17:59.671 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36631972/36632510/36632510/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:18:00.023 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36631972/36632510/36632510/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:18:00.014 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:19:10.070 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36650241/36667084/36667084/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:09.791 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36650241/36667084/36667084/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:08.191 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36650241/36667084/36667084/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:10.064 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36650241/36667084/36667084/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:10.009 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36650241/36667084/36667084/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:10.077 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36650241/36667084/36667084/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:10.070 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:19:10.831 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.13:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36650241/36667084/36667084/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:10.009 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36650241/36667084/36667084/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:10.194 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36650241/36667084/36667084/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:10.189 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36650241/36667084/36667084/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:10.792 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36650241/36667084/36667084/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:10.194 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36650241/36667084/36667084/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:10.830 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:19:13.572 TRACE GossipServiceBase ] Looks like node [192.168.33.12:2113] is DEAD (Gossip send failed). [PID:03068:010 2015.07.04 13:19:13.573 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip send failed to [192.168.33.12:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36650241/36667084/36667084/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:12.792 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36650241/36667084/36667084/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:12.197 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36650241/36667084/36667084/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:13.071 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36650241/36667084/36667084/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:12.792 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36650241/36667084/36667084/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:13.573 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36650241/36667084/36667084/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:13.071 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:19:13.577 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36650241/36667084/36667084/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:12.792 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36650241/36667084/36667084/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:13.573 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36650241/36667084/36667084/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:13.071 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36650241/36667084/36667084/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:12.792 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36650241/36667084/36667084/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:13.198 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36650241/36667084/36667084/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:13.575 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:19:18.932 TRACE GossipServiceBase ] Looks like node [192.168.33.12:2113] is DEAD (Gossip send failed). [PID:03068:010 2015.07.04 13:19:18.933 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip send failed to [192.168.33.12:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:17.796 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:17.201 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:18.076 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:17.796 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:18.932 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:18.076 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:19:18.941 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:17.796 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:18.932 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:18.076 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:17.796 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:18.202 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:18.941 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:19:23.939 TRACE GossipServiceBase ] Looks like node [192.168.33.13:2113] is DEAD (Gossip send failed). [PID:03068:010 2015.07.04 13:19:23.940 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip send failed to [192.168.33.13:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:22.801 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:22.205 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:23.081 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:23.939 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:22.205 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:23.081 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:19:25.088 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.13:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:23.939 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:24.095 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:25.083 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:25.051 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:24.851 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:25.088 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:19:28.952 TRACE GossipServiceBase ] Looks like node [192.168.33.12:2113] is DEAD (Gossip send failed). [PID:03068:010 2015.07.04 13:19:28.954 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip send failed to [192.168.33.12:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:27.054 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:27.210 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:28.086 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:27.054 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:28.953 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:28.086 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:19:29.952 TRACE GossipServiceBase ] Looks like node [192.168.33.13:2113] is DEAD (Gossip send failed). [PID:03068:010 2015.07.04 13:19:29.952 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip send failed to [192.168.33.13:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:27.054 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:28.953 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:29.087 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:29.952 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:28.953 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:29.087 -------------------------------------------------------------------------------- [PID:03068:219 2015.07.04 13:19:29.954 INFO TcpConnection ] ES TcpConnection closed [13:19:29.954: N192.168.33.12:38609, L192.168.33.11:1111, {f6710670-1730-49c6-b3ba-9fd60b480cfb}]: Received bytes: 3191, Sent bytes: 81862 Send calls: 114, callbacks: 114 Receive calls: 66, callbacks: 66 Close reason: [Success] Socket closed [PID:03068:219 2015.07.04 13:19:29.955 INFO TcpConnectionManager] Connection 'internal-normal' [192.168.33.12:38609, {f6710670-1730-49c6-b3ba-9fd60b480cfb}] closed: Success. [PID:03068:010 2015.07.04 13:19:29.955 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.13:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:29.952 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:28.953 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:29.087 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:28.807 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:28.953 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:29.955 -------------------------------------------------------------------------------- [PID:03068:202 2015.07.04 13:19:29.962 INFO TcpConnection ] ES TcpConnection closed [13:19:29.962: N192.168.33.13:32966, L192.168.33.11:1111, {a579d8ed-4512-4c80-a2b7-775aab971341}]: Received bytes: 8056, Sent bytes: 38518593 Send calls: 8795, callbacks: 8795 Receive calls: 166, callbacks: 166 Close reason: [Success] Socket closed [PID:03068:202 2015.07.04 13:19:29.962 INFO TcpConnectionManager] Connection 'internal-normal' [192.168.33.13:32966, {a579d8ed-4512-4c80-a2b7-775aab971341}] closed: Success. [PID:03068:010 2015.07.04 13:19:29.962 TRACE GossipServiceBase ] Looks like node [192.168.33.13:1111] is DEAD (TCP connection lost). [PID:03068:010 2015.07.04 13:19:29.964 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection lost to [192.168.33.13:1111]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:28.807 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:28.953 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:29.955 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:29.963 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:28.953 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:29.955 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:19:30.091 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.13:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Slave, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:29.963 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:28.953 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:30.087 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36685339/36685512/36685512/E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} | 2015-07-04 13:19:30.055 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:30.055 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:30.091 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:19:30.091 DEBUG ClusterVNodeControll] There are FEW MASTERS according to gossip, need to start elections. MASTER: [InstanceId: {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}, InternalTcp: 192.168.33.11:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.11:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.11:2113, ExternalHttp: 192.168.33.11:2114]. GOSSIP: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36685339/36685512/36685512/E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} | 2015-07-04 13:19:30.055 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:30.055 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Master, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:30.091 [PID:03068:010 2015.07.04 13:19:30.091 DEBUG ElectionsService ] ELECTIONS: STARTING ELECTIONS. [PID:03068:010 2015.07.04 13:19:30.091 DEBUG ElectionsService ] ELECTIONS: (V=44) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:30.091 DEBUG ElectionsService ] ELECTIONS: (V=44) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:19:30.111 DEBUG ElectionsService ] ELECTIONS: (V=45) VIEWCHANGE FROM [192.168.33.13:2113, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:19:30.113 DEBUG ElectionsService ] ELECTIONS: (V=45) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:30.113 DEBUG ElectionsService ] ELECTIONS: (V=45) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:19:30.113 DEBUG ElectionsService ] ELECTIONS: (V=45) MAJORITY OF VIEWCHANGE. [PID:03068:010 2015.07.04 13:19:30.121 DEBUG ElectionsService ] ELECTIONS: (V=45) PREPARE FROM [192.168.33.13:2113, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:19:30.121 DEBUG ElectionsService ] ELECTIONS: (V=45) SHIFT TO REG_NONLEADER. [PID:03068:010 2015.07.04 13:19:30.124 DEBUG ElectionsService ] ELECTIONS: (V=45) PROPOSAL FROM [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] M=[192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}](L=36685339,W=36685512,C=36685512,E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066}). ME=[192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}](L=36667084,W=36684018,C=36684018,E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7}). [PID:03068:010 2015.07.04 13:19:30.125 DEBUG ElectionsService ] ELECTIONS: (V=45) ACCEPT FROM [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] M=[192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]). [PID:03068:010 2015.07.04 13:19:30.125 DEBUG ElectionsService ] ELECTIONS: (V=45) ACCEPT FROM [192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}] M=[192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]). [PID:03068:010 2015.07.04 13:19:30.125 INFO ElectionsService ] ELECTIONS: (V=45) DONE. ELECTED MASTER = [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}](L=36685339,W=36685512,C=36685512,E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066}). ME=[192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}](L=36667084,W=36684018,C=36684018,E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7}). [PID:03068:010 2015.07.04 13:19:30.125 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] PRE-REPLICA STATE, WAITING FOR CHASER TO CATCH UP... MASTER IS [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] [PID:03068:014 2015.07.04 13:19:30.125 INFO ProjectionCoreServic] _projections is not empty after all the projections have been killed [PID:03068:013 2015.07.04 13:19:30.125 INFO ProjectionCoreServic] _projections is not empty after all the projections have been killed [PID:03068:012 2015.07.04 13:19:30.125 INFO ProjectionCoreServic] _projections is not empty after all the projections have been killed [PID:03068:012 2015.07.04 13:19:30.125 INFO ProjectionCoreServic] _subscriptions is not empty after all the projections have been killed [PID:03068:014 2015.07.04 13:19:30.125 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:03068:012 2015.07.04 13:19:30.125 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:03068:012 2015.07.04 13:19:30.125 INFO ProjectionCoreServic] _subscriptionEventReaders is not empty after all the projections have been killed [PID:03068:012 2015.07.04 13:19:30.127 ERROR QueuedHandlerAutoRes] Error while processing message EventStore.Projections.Core.Messages.ReaderSubscriptionManagement+Unsubscribe in queued handler 'Projection Core #0'. System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary. at System.Collections.Generic.Dictionary`2[System.Guid,System.Guid].get_Item (Guid key) [0x00000] in :0 at EventStore.Projections.Core.Services.Processing.EventReaderCoreService.Handle (EventStore.Projections.Core.Messages.Unsubscribe message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Projections.Core.Messages.ReaderSubscriptionManagement+Unsubscribe].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:013 2015.07.04 13:19:30.128 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:03068:220 2015.07.04 13:19:30.129 INFO TcpConnectionManager] Connection 'master-normal' ({c245f743-2f6a-4333-ac97-cead5cc9a080}) to [192.168.33.13:1111] established. [PID:03068:010 2015.07.04 13:19:30.129 INFO ReplicaService ] Subscribing at LogPosition: 36684018 (0x22FC0F2) to MASTER [192.168.33.13:1111, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] as replica with SubscriptionId: {d058a688-8f15-497f-8230-8d1bee26aeef}, ConnectionId: {c245f743-2f6a-4333-ac97-cead5cc9a080}, LocalEndPoint: [192.168.33.11:49825], Epochs: E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740} E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65} E30@310956:{26cd0216-4e18-43aa-a899-11057cb44b2f} E29@293661:{e83f9ab8-c0e1-4b7f-ad41-504a907b90d1} E28@286573:{219b52c3-b7dd-4a47-834a-730aa0e69066} E27@247371:{d635cf1b-4fac-4c60-b507-8ba566601c8f} E26@247151:{44eaa2de-fcaa-41db-b5b7-c0fb3add327d} E25@246931:{fcf0df26-2741-43a6-a8d6-3cdb1b0d9206} E24@229645:{1d2c878f-7071-4e70-9e12-bc951df9797f}... . [PID:03068:007 2015.07.04 13:19:30.140 INFO ClusterStorageWriter] === SUBSCRIBED to [192.168.33.13:1111,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] at 36684018 (0x22FC0F2). SubscriptionId: {d058a688-8f15-497f-8230-8d1bee26aeef}. [PID:03068:010 2015.07.04 13:19:30.140 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CATCHING UP!!! BANZAI!!! MASTER IS [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] [PID:03068:014 2015.07.04 13:19:30.142 FATAL ResponseWriter ] Cannot write commands to the stream $projections-$master. status: AccessDenied [PID:03068:012 2015.07.04 13:19:30.142 FATAL ResponseWriter ] Cannot write commands to the stream $projections-$master. status: AccessDenied [PID:03068:014 2015.07.04 13:19:30.142 ERROR QueuedHandlerAutoRes] Error while processing message WRITE COMPLETED: CorrelationId: f2ee4e3b-14a8-4fbc-a7f7-1216ee92a1b1, Result: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projection Core #2'. System.Exception: Cannot write commands to the stream $projections-$master. status: AccessDenied at EventStore.Projections.Core.Services.Management.ResponseWriter.m__0 (EventStore.Core.Messages.WriteEventsCompleted completed) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStorey11`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response, OperationResult result) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStoreyB+c__AnonStoreyC.<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].EventStore.Core.Bus.IHandle.Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:012 2015.07.04 13:19:30.142 ERROR QueuedHandlerAutoRes] Error while processing message WRITE COMPLETED: CorrelationId: 8d40a818-7107-49ba-9ee4-b1db01998df3, Result: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projection Core #0'. System.Exception: Cannot write commands to the stream $projections-$master. status: AccessDenied at EventStore.Projections.Core.Services.Management.ResponseWriter.m__0 (EventStore.Core.Messages.WriteEventsCompleted completed) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStorey11`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response, OperationResult result) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStoreyB+c__AnonStoreyC.<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].EventStore.Core.Bus.IHandle.Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:010 2015.07.04 13:19:30.609 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] CLONE ASSIGNMENT RECEIVED FROM [192.168.33.13:1111,n/a,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:19:30.609 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CLONE!!! SPARTA!!! MASTER IS [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] [PID:03068:013 2015.07.04 13:19:30.609 FATAL ResponseWriter ] Cannot write commands to the stream $projections-$master. status: AccessDenied [PID:03068:013 2015.07.04 13:19:30.611 ERROR QueuedHandlerAutoRes] Error while processing message WRITE COMPLETED: CorrelationId: 8f20fbe2-1c31-497e-8708-79c84e2ed74f, Result: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projection Core #1'. System.Exception: Cannot write commands to the stream $projections-$master. status: AccessDenied at EventStore.Projections.Core.Services.Management.ResponseWriter.m__0 (EventStore.Core.Messages.WriteEventsCompleted completed) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStorey11`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response, OperationResult result) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStoreyB+c__AnonStoreyC.<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].EventStore.Core.Bus.IHandle.Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:010 2015.07.04 13:19:30.612 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] SLAVE ASSIGNMENT RECEIVED FROM [192.168.33.13:1111,n/a,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:19:30.613 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS SLAVE!!! SPARTA!!! MASTER IS [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] [PID:03068:015 2015.07.04 13:19:30.613 DEBUG EpochManager ] === Update Last Epoch E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} (previous epoch at 36612193). [PID:03068:015 2015.07.04 13:19:30.616 DEBUG EpochManager ] === Update Last Epoch E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} (previous epoch at 36684018). [PID:03068:010 2015.07.04 13:19:31.095 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36685339/36685512/36685512/E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} | 2015-07-04 13:19:30.130 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36667084/36684018/36684018/E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} | 2015-07-04 13:19:30.055 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36685339/36685736/36685736/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:31.088 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36685339/36685512/36685512/E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} | 2015-07-04 13:19:31.094 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36685339/36685736/36685736/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:31.103 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36685339/36685736/36685736/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:31.094 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:19:32.020 DEBUG ElectionsService ] ELECTIONS: (V=46) VIEWCHANGE FROM [192.168.33.12:2113, {6648e0c4-835f-4575-a12e-0f82442ed57e}]. [PID:03068:010 2015.07.04 13:19:32.021 DEBUG ElectionsService ] ELECTIONS: (V=46) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:32.021 DEBUG ElectionsService ] ELECTIONS: (V=46) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:19:32.021 DEBUG ElectionsService ] ELECTIONS: (V=46) MAJORITY OF VIEWCHANGE. [PID:03068:010 2015.07.04 13:19:32.615 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.13:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36685339/36685512/36685512/E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} | 2015-07-04 13:19:32.210 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36685339/36685736/36685736/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:32.217 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36685339/36685736/36685736/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:32.210 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36685339/36685736/36685736/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:32.107 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36685339/36685736/36685736/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:32.217 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36685339/36685736/36685736/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:32.615 -------------------------------------------------------------------------------- [PID:03068:012 2015.07.04 13:19:32.637 FATAL ResponseWriter ] Cannot write commands to the stream $projections-$master. status: AccessDenied [PID:03068:012 2015.07.04 13:19:32.637 ERROR QueuedHandlerAutoRes] Error while processing message WRITE COMPLETED: CorrelationId: 85760d08-ca8c-40eb-8d5b-9d8fa12ad298, Result: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projection Core #0'. System.Exception: Cannot write commands to the stream $projections-$master. status: AccessDenied at EventStore.Projections.Core.Services.Management.ResponseWriter.m__0 (EventStore.Core.Messages.WriteEventsCompleted completed) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStorey11`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response, OperationResult result) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStoreyB+c__AnonStoreyC.<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].EventStore.Core.Bus.IHandle.Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:014 2015.07.04 13:19:32.638 FATAL ResponseWriter ] Cannot write commands to the stream $projections-$master. status: AccessDenied [PID:03068:014 2015.07.04 13:19:32.638 ERROR QueuedHandlerAutoRes] Error while processing message WRITE COMPLETED: CorrelationId: 0914e2ac-325a-4afd-a2bb-4659eddee4ea, Result: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projection Core #2'. System.Exception: Cannot write commands to the stream $projections-$master. status: AccessDenied at EventStore.Projections.Core.Services.Management.ResponseWriter.m__0 (EventStore.Core.Messages.WriteEventsCompleted completed) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStorey11`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response, OperationResult result) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStoreyB+c__AnonStoreyC.<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].EventStore.Core.Bus.IHandle.Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:010 2015.07.04 13:19:32.641 DEBUG ElectionsService ] ELECTIONS: (V=47) VIEWCHANGE FROM [192.168.33.13:2113, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:19:32.641 DEBUG ElectionsService ] ELECTIONS: (V=47) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:32.641 DEBUG ElectionsService ] ELECTIONS: (V=47) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:19:32.641 DEBUG ElectionsService ] ELECTIONS: (V=47) MAJORITY OF VIEWCHANGE. [PID:03068:010 2015.07.04 13:19:32.641 DEBUG ElectionsService ] ELECTIONS: (V=47) SHIFT TO PREPARE PHASE. [PID:03068:010 2015.07.04 13:19:32.641 DEBUG ElectionsService ] ELECTIONS: (V=47) PREPARE_OK FROM [192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}](L=36685736,W=36685869,C=36685869,E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f}). [PID:03068:010 2015.07.04 13:19:33.643 DEBUG ElectionsService ] ELECTIONS: (V=47) TIMED OUT! (S=ElectingLeader, M=). [PID:03068:010 2015.07.04 13:19:33.643 DEBUG ElectionsService ] ELECTIONS: (V=48) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:33.643 DEBUG ElectionsService ] ELECTIONS: (V=48) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:013 2015.07.04 13:19:33.957 FATAL ResponseWriter ] Cannot write commands to the stream $projections-$master. status: AccessDenied [PID:03068:013 2015.07.04 13:19:33.957 ERROR QueuedHandlerAutoRes] Error while processing message WRITE COMPLETED: CorrelationId: e338d874-924b-4202-80aa-e49b56442a97, Result: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projection Core #1'. System.Exception: Cannot write commands to the stream $projections-$master. status: AccessDenied at EventStore.Projections.Core.Services.Management.ResponseWriter.m__0 (EventStore.Core.Messages.WriteEventsCompleted completed) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStorey11`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response, OperationResult result) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStoreyB+c__AnonStoreyC.<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].EventStore.Core.Bus.IHandle.Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:010 2015.07.04 13:19:34.644 DEBUG ElectionsService ] ELECTIONS: (V=48) TIMED OUT! (S=ElectingLeader, M=). [PID:03068:010 2015.07.04 13:19:34.644 DEBUG ElectionsService ] ELECTIONS: (V=49) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:34.644 DEBUG ElectionsService ] ELECTIONS: (V=49) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:19:34.957 DEBUG ElectionsService ] ELECTIONS: (V=49) VIEWCHANGE FROM [192.168.33.13:2113, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:19:34.957 DEBUG ElectionsService ] ELECTIONS: (V=49) MAJORITY OF VIEWCHANGE. [PID:03068:010 2015.07.04 13:19:34.958 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.13:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36685339/36685736/36685736/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:32.107 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36685339/36685736/36685736/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:32.217 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36685736/36685869/36685869/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:34.091 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36685869/36686002/36686002/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:32.811 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36685339/36685736/36685736/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:32.602 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36685736/36685869/36685869/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:34.957 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:19:35.143 TRACE GossipServiceBase ] Looks like master [192.168.33.13:2113, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] is DEAD (Gossip send failed), though we wait for TCP to decide. [PID:03068:010 2015.07.04 13:19:35.458 DEBUG ElectionsService ] ELECTIONS: (V=49) VIEWCHANGE FROM [192.168.33.12:2113, {6648e0c4-835f-4575-a12e-0f82442ed57e}]. [PID:03068:010 2015.07.04 13:19:35.646 DEBUG ElectionsService ] ELECTIONS: (V=49) TIMED OUT! (S=ElectingLeader, M=). [PID:03068:010 2015.07.04 13:19:35.646 DEBUG ElectionsService ] ELECTIONS: (V=50) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:35.646 DEBUG ElectionsService ] ELECTIONS: (V=50) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:19:35.965 DEBUG ElectionsService ] ELECTIONS: (V=48) VIEWCHANGE FROM [192.168.33.12:2113, {6648e0c4-835f-4575-a12e-0f82442ed57e}]. [PID:03068:010 2015.07.04 13:19:35.965 DEBUG ElectionsService ] ELECTIONS: (V=48) MAJORITY OF VIEWCHANGE. [PID:03068:010 2015.07.04 13:19:36.459 DEBUG ElectionsService ] ELECTIONS: (V=50) VIEWCHANGE FROM [192.168.33.13:2113, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:19:36.647 DEBUG ElectionsService ] ELECTIONS: (V=50) TIMED OUT! (S=ElectingLeader, M=). [PID:03068:010 2015.07.04 13:19:36.647 DEBUG ElectionsService ] ELECTIONS: (V=51) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:36.647 DEBUG ElectionsService ] ELECTIONS: (V=51) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:012 2015.07.04 13:19:36.974 FATAL ResponseWriter ] Cannot write commands to the stream $projections-$master. status: AccessDenied [PID:03068:012 2015.07.04 13:19:36.974 ERROR QueuedHandlerAutoRes] Error while processing message WRITE COMPLETED: CorrelationId: b5edc632-2952-4ab6-a442-a633eeff8fe1, Result: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projection Core #0'. System.Exception: Cannot write commands to the stream $projections-$master. status: AccessDenied at EventStore.Projections.Core.Services.Management.ResponseWriter.m__0 (EventStore.Core.Messages.WriteEventsCompleted completed) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStorey11`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response, OperationResult result) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStoreyB+c__AnonStoreyC.<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].EventStore.Core.Bus.IHandle.Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:010 2015.07.04 13:19:36.975 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.13:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36685869/36686002/36686002/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:32.811 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36685339/36685736/36685736/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:32.602 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36699671/36700155/36700155/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:36.094 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36704680/36705219/36705219/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:34.813 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705219/36705219/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:34.033 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36699671/36705219/36701106/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:36.975 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:19:37.025 DEBUG ElectionsService ] ELECTIONS: (V=52) VIEWCHANGE FROM [192.168.33.12:2113, {6648e0c4-835f-4575-a12e-0f82442ed57e}]. [PID:03068:010 2015.07.04 13:19:37.025 DEBUG ElectionsService ] ELECTIONS: (V=52) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:37.025 DEBUG ElectionsService ] ELECTIONS: (V=52) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:19:37.033 DEBUG ElectionsService ] ELECTIONS: (V=52) MAJORITY OF VIEWCHANGE. [PID:03068:010 2015.07.04 13:19:37.525 DEBUG ElectionsService ] ELECTIONS: (V=50) VIEWCHANGE FROM [192.168.33.12:2113, {6648e0c4-835f-4575-a12e-0f82442ed57e}]. [PID:03068:010 2015.07.04 13:19:37.530 DEBUG ElectionsService ] ELECTIONS: (V=51) VIEWCHANGE FROM [192.168.33.12:2113, {6648e0c4-835f-4575-a12e-0f82442ed57e}]. [PID:03068:010 2015.07.04 13:19:37.648 DEBUG ElectionsService ] ELECTIONS: (V=53) VIEWCHANGE FROM [192.168.33.12:2113, {6648e0c4-835f-4575-a12e-0f82442ed57e}]. [PID:03068:010 2015.07.04 13:19:37.648 DEBUG ElectionsService ] ELECTIONS: (V=53) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:37.648 DEBUG ElectionsService ] ELECTIONS: (V=53) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:19:37.648 DEBUG ElectionsService ] ELECTIONS: (V=53) MAJORITY OF VIEWCHANGE. [PID:03068:010 2015.07.04 13:19:37.648 DEBUG ElectionsService ] ELECTIONS: (V=53) SHIFT TO PREPARE PHASE. [PID:03068:010 2015.07.04 13:19:37.648 DEBUG ElectionsService ] ELECTIONS: (V=53) PREPARE_OK FROM [192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}](L=36704680,W=36705219,C=36705219,E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f}). [PID:03068:010 2015.07.04 13:19:38.650 DEBUG ElectionsService ] ELECTIONS: (V=53) TIMED OUT! (S=ElectingLeader, M=). [PID:03068:010 2015.07.04 13:19:38.650 DEBUG ElectionsService ] ELECTIONS: (V=54) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:38.650 DEBUG ElectionsService ] ELECTIONS: (V=54) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:008 2015.07.04 13:19:38.742 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.13:1111, L192.168.33.11:49825, {c245f743-2f6a-4333-ac97-cead5cc9a080}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 14 [PID:03068:008 2015.07.04 13:19:38.742 INFO TcpConnection ] ES TcpConnection closed [13:19:38.742: N192.168.33.13:1111, L192.168.33.11:49825, {c245f743-2f6a-4333-ac97-cead5cc9a080}]: Received bytes: 24855, Sent bytes: 12387 Send calls: 24, callbacks: 23 Receive calls: 15, callbacks: 14 Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 14 [PID:03068:008 2015.07.04 13:19:38.742 INFO TcpConnectionManager] Connection 'master-normal' [192.168.33.13:1111, {c245f743-2f6a-4333-ac97-cead5cc9a080}] closed: Success. [PID:03068:010 2015.07.04 13:19:38.742 TRACE GossipServiceBase ] Looks like node [192.168.33.13:1111] is DEAD (TCP connection lost). [PID:03068:010 2015.07.04 13:19:38.742 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection lost to [192.168.33.13:1111]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36704680/36705219/36705219/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:37.529 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705219/36705219/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:37.536 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36704680/36705219/36705219/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:38.095 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36704680/36705219/36705219/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:38.743 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705219/36705219/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:37.536 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36704680/36705219/36705219/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:38.095 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:19:38.743 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}, InternalTcp: 192.168.33.13:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.13:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.13:2113, ExternalHttp: 192.168.33.13:2114]. [PID:03068:010 2015.07.04 13:19:39.244 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] PRE-REPLICA STATE, WAITING FOR CHASER TO CATCH UP... MASTER IS [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] [PID:03068:010 2015.07.04 13:19:39.244 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.13:1111, L192.168.33.11:49825, {c245f743-2f6a-4333-ac97-cead5cc9a080}] cleanly. Reason: Reconnecting from old master [192.168.33.13:1111] to new master: [192.168.33.13:1111]. [PID:03068:010 2015.07.04 13:19:39.244 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}, InternalTcp: 192.168.33.13:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.13:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.13:2113, ExternalHttp: 192.168.33.13:2114]. [PID:03068:012 2015.07.04 13:19:39.246 INFO ProjectionCoreServic] _projections is not empty after all the projections have been killed [PID:03068:012 2015.07.04 13:19:39.246 INFO ProjectionCoreServic] _subscriptions is not empty after all the projections have been killed [PID:03068:012 2015.07.04 13:19:39.246 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:03068:012 2015.07.04 13:19:39.246 INFO ProjectionCoreServic] _subscriptionEventReaders is not empty after all the projections have been killed [PID:03068:012 2015.07.04 13:19:39.246 ERROR QueuedHandlerAutoRes] Error while processing message EventStore.Projections.Core.Messages.ReaderSubscriptionManagement+Unsubscribe in queued handler 'Projection Core #0'. System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary. at System.Collections.Generic.Dictionary`2[System.Guid,System.Guid].get_Item (Guid key) [0x00000] in :0 at EventStore.Projections.Core.Services.Processing.EventReaderCoreService.Handle (EventStore.Projections.Core.Messages.Unsubscribe message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Projections.Core.Messages.ReaderSubscriptionManagement+Unsubscribe].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:013 2015.07.04 13:19:39.248 INFO ProjectionCoreServic] _projections is not empty after all the projections have been killed [PID:03068:013 2015.07.04 13:19:39.248 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:03068:014 2015.07.04 13:19:39.248 INFO ProjectionCoreServic] _projections is not empty after all the projections have been killed [PID:03068:014 2015.07.04 13:19:39.250 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:03068:010 2015.07.04 13:19:39.251 INFO ReplicaService ] Subscribing at LogPosition: 36705219 (0x23013C3) to MASTER [192.168.33.13:1111, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] as replica with SubscriptionId: {64f582fa-cdbc-4435-b02d-787fb9d0d48d}, ConnectionId: {decdcdb2-44a1-4a2f-86dc-caa41fd0e17b}, LocalEndPoint: [], Epochs: E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740} E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65} E30@310956:{26cd0216-4e18-43aa-a899-11057cb44b2f} E29@293661:{e83f9ab8-c0e1-4b7f-ad41-504a907b90d1} E28@286573:{219b52c3-b7dd-4a47-834a-730aa0e69066} E27@247371:{d635cf1b-4fac-4c60-b507-8ba566601c8f} E26@247151:{44eaa2de-fcaa-41db-b5b7-c0fb3add327d}... . [PID:03068:010 2015.07.04 13:19:39.651 DEBUG ElectionsService ] ELECTIONS: (V=54) TIMED OUT! (S=ElectingLeader, M=). [PID:03068:010 2015.07.04 13:19:39.652 DEBUG ElectionsService ] ELECTIONS: (V=55) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:39.652 DEBUG ElectionsService ] ELECTIONS: (V=55) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:19:40.252 INFO ReplicaService ] Subscribing at LogPosition: 36705219 (0x23013C3) to MASTER [192.168.33.13:1111, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] as replica with SubscriptionId: {22bd095a-9130-4551-9e31-df182a42d7f1}, ConnectionId: {decdcdb2-44a1-4a2f-86dc-caa41fd0e17b}, LocalEndPoint: [], Epochs: E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740} E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65} E30@310956:{26cd0216-4e18-43aa-a899-11057cb44b2f} E29@293661:{e83f9ab8-c0e1-4b7f-ad41-504a907b90d1} E28@286573:{219b52c3-b7dd-4a47-834a-730aa0e69066} E27@247371:{d635cf1b-4fac-4c60-b507-8ba566601c8f} E26@247151:{44eaa2de-fcaa-41db-b5b7-c0fb3add327d}... . [PID:03068:230 2015.07.04 13:19:40.464 INFO TcpConnection ] ES TcpConnection closed [13:19:40.464: N192.168.33.13:1111, L, {decdcdb2-44a1-4a2f-86dc-caa41fd0e17b}]: Received bytes: 0, Sent bytes: 0 Send calls: 0, callbacks: 0 Receive calls: 0, callbacks: 0 Close reason: [Success] Connection establishment timeout. [PID:03068:230 2015.07.04 13:19:40.464 INFO TcpConnectionManager] Connection 'master-normal' [192.168.33.13:1111, {decdcdb2-44a1-4a2f-86dc-caa41fd0e17b}] closed: Success. [PID:03068:010 2015.07.04 13:19:40.653 DEBUG ElectionsService ] ELECTIONS: (V=55) TIMED OUT! (S=ElectingLeader, M=). [PID:03068:010 2015.07.04 13:19:40.653 DEBUG ElectionsService ] ELECTIONS: (V=56) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:40.653 DEBUG ElectionsService ] ELECTIONS: (V=56) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:19:40.965 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.13:1111, L, {decdcdb2-44a1-4a2f-86dc-caa41fd0e17b}] cleanly. Reason: Reconnecting from old master [192.168.33.13:1111] to new master: [192.168.33.13:1111]. [PID:03068:010 2015.07.04 13:19:41.254 INFO ReplicaService ] Subscribing at LogPosition: 36705219 (0x23013C3) to MASTER [192.168.33.13:1111, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] as replica with SubscriptionId: {4730a06f-8e1e-494e-b1c1-0d2d319c81a5}, ConnectionId: {80431a5e-ea9b-49da-baae-63451ac93206}, LocalEndPoint: [], Epochs: E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740} E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65} E30@310956:{26cd0216-4e18-43aa-a899-11057cb44b2f} E29@293661:{e83f9ab8-c0e1-4b7f-ad41-504a907b90d1} E28@286573:{219b52c3-b7dd-4a47-834a-730aa0e69066} E27@247371:{d635cf1b-4fac-4c60-b507-8ba566601c8f} E26@247151:{44eaa2de-fcaa-41db-b5b7-c0fb3add327d}... . [PID:03068:010 2015.07.04 13:19:41.656 DEBUG ElectionsService ] ELECTIONS: (V=56) TIMED OUT! (S=ElectingLeader, M=). [PID:03068:010 2015.07.04 13:19:41.656 DEBUG ElectionsService ] ELECTIONS: (V=57) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:41.656 DEBUG ElectionsService ] ELECTIONS: (V=57) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:220 2015.07.04 13:19:42.152 INFO TcpConnection ] ES TcpConnection closed [13:19:42.152: N192.168.33.13:1111, L, {80431a5e-ea9b-49da-baae-63451ac93206}]: Received bytes: 0, Sent bytes: 0 Send calls: 0, callbacks: 0 Receive calls: 0, callbacks: 0 Close reason: [Success] Connection establishment timeout. [PID:03068:220 2015.07.04 13:19:42.152 INFO TcpConnectionManager] Connection 'master-normal' [192.168.33.13:1111, {80431a5e-ea9b-49da-baae-63451ac93206}] closed: Success. [PID:03068:010 2015.07.04 13:19:42.158 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.13:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36704680/36705219/36705219/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:38.743 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705219/36705219/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:37.536 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36704680/36705219/36705219/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:42.099 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36704680/36705219/36705219/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:40.818 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705219/36705219/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:38.222 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36704680/36705219/36705219/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:42.157 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:19:42.255 INFO ReplicaService ] Subscribing at LogPosition: 36705219 (0x23013C3) to MASTER [192.168.33.13:1111, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] as replica with SubscriptionId: {e40e8c0f-91ca-4189-830b-56fe87ace8ed}, ConnectionId: {80431a5e-ea9b-49da-baae-63451ac93206}, LocalEndPoint: [], Epochs: E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740} E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65} E30@310956:{26cd0216-4e18-43aa-a899-11057cb44b2f} E29@293661:{e83f9ab8-c0e1-4b7f-ad41-504a907b90d1} E28@286573:{219b52c3-b7dd-4a47-834a-730aa0e69066} E27@247371:{d635cf1b-4fac-4c60-b507-8ba566601c8f} E26@247151:{44eaa2de-fcaa-41db-b5b7-c0fb3add327d}... . [PID:03068:010 2015.07.04 13:19:42.658 TRACE GossipServiceBase ] Looks like master [192.168.33.13:2113, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] is DEAD (Gossip send failed), though we wait for TCP to decide. [PID:03068:010 2015.07.04 13:19:42.658 DEBUG ElectionsService ] ELECTIONS: (V=54) VIEWCHANGE FROM [192.168.33.12:2113, {6648e0c4-835f-4575-a12e-0f82442ed57e}]. [PID:03068:010 2015.07.04 13:19:42.658 DEBUG ElectionsService ] ELECTIONS: (V=54) MAJORITY OF VIEWCHANGE. [PID:03068:010 2015.07.04 13:19:42.664 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.13:1111, L, {80431a5e-ea9b-49da-baae-63451ac93206}] cleanly. Reason: Reconnecting from old master [192.168.33.13:1111] to new master: [192.168.33.13:1111]. [PID:03068:010 2015.07.04 13:19:42.666 DEBUG ElectionsService ] ELECTIONS: (V=57) TIMED OUT! (S=ElectingLeader, M=). [PID:03068:010 2015.07.04 13:19:42.666 DEBUG ElectionsService ] ELECTIONS: (V=58) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:42.666 DEBUG ElectionsService ] ELECTIONS: (V=58) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:19:42.679 DEBUG ElectionsService ] ELECTIONS: (V=54) VIEWCHANGE FROM [192.168.33.13:2113, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:19:42.679 DEBUG ElectionsService ] ELECTIONS: (V=54) MAJORITY OF VIEWCHANGE. [PID:03068:010 2015.07.04 13:19:42.681 DEBUG ElectionsService ] ELECTIONS: (V=55) VIEWCHANGE FROM [192.168.33.12:2113, {6648e0c4-835f-4575-a12e-0f82442ed57e}]. [PID:03068:216 2015.07.04 13:19:42.681 INFO TcpConnectionManager] Connection 'master-normal' ({2dc9dd75-c25f-465c-9480-bc32422ffdc5}) to [192.168.33.13:1111] established. [PID:03068:010 2015.07.04 13:19:42.971 TRACE GossipServiceBase ] Looks like node [192.168.33.12:2113] is DEAD (Gossip send failed). [PID:03068:010 2015.07.04 13:19:42.972 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip send failed to [192.168.33.12:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36704680/36705219/36705219/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:42.681 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705219/36705219/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:38.222 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36704680/36705219/36705219/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:42.157 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36704680/36705219/36705219/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:42.681 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705219/36705219/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:42.971 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36704680/36705219/36705219/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:42.157 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:19:43.131 DEBUG ElectionsService ] ELECTIONS: (V=58) VIEWCHANGE FROM [192.168.33.13:2113, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:19:43.257 INFO ReplicaService ] Subscribing at LogPosition: 36705219 (0x23013C3) to MASTER [192.168.33.13:1111, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] as replica with SubscriptionId: {aaef096e-2435-45d8-aa2d-53be02593887}, ConnectionId: {2dc9dd75-c25f-465c-9480-bc32422ffdc5}, LocalEndPoint: [192.168.33.11:49840], Epochs: E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740} E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65} E30@310956:{26cd0216-4e18-43aa-a899-11057cb44b2f} E29@293661:{e83f9ab8-c0e1-4b7f-ad41-504a907b90d1} E28@286573:{219b52c3-b7dd-4a47-834a-730aa0e69066} E27@247371:{d635cf1b-4fac-4c60-b507-8ba566601c8f} E26@247151:{44eaa2de-fcaa-41db-b5b7-c0fb3add327d}... . [PID:03068:010 2015.07.04 13:19:43.261 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CATCHING UP!!! BANZAI!!! MASTER IS [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] [PID:03068:007 2015.07.04 13:19:43.261 INFO ClusterStorageWriter] === SUBSCRIBED to [192.168.33.13:1111,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] at 36705219 (0x23013C3). SubscriptionId: {aaef096e-2435-45d8-aa2d-53be02593887}. [PID:03068:010 2015.07.04 13:19:43.262 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] CLONE ASSIGNMENT RECEIVED FROM [192.168.33.13:1111,n/a,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:19:43.262 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CLONE!!! SPARTA!!! MASTER IS [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] [PID:03068:010 2015.07.04 13:19:43.262 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] SLAVE ASSIGNMENT RECEIVED FROM [192.168.33.13:1111,n/a,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:19:43.262 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS SLAVE!!! SPARTA!!! MASTER IS [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] [PID:03068:015 2015.07.04 13:19:43.262 DEBUG EpochManager ] === Update Last Epoch E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} (previous epoch at 36685512). [PID:03068:010 2015.07.04 13:19:43.667 DEBUG ElectionsService ] ELECTIONS: (V=58) TIMED OUT! (S=ElectingLeader, M=). [PID:03068:010 2015.07.04 13:19:43.667 DEBUG ElectionsService ] ELECTIONS: (V=59) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:43.667 DEBUG ElectionsService ] ELECTIONS: (V=59) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:19:44.630 DEBUG ElectionsService ] ELECTIONS: (V=59) VIEWCHANGE FROM [192.168.33.13:2113, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:19:44.630 DEBUG ElectionsService ] ELECTIONS: (V=59) MAJORITY OF VIEWCHANGE. [PID:03068:010 2015.07.04 13:19:44.630 DEBUG ElectionsService ] ELECTIONS: (V=59) SHIFT TO PREPARE PHASE. [PID:03068:010 2015.07.04 13:19:44.630 DEBUG ElectionsService ] ELECTIONS: (V=59) PREPARE_OK FROM [192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}](L=36704680,W=36705443,C=36705443,E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3}). [PID:03068:010 2015.07.04 13:19:44.632 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.13:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36704680/36705219/36705219/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:39.818 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705219/36705219/E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} | 2015-07-04 13:19:42.971 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36704680/36705443/36705443/E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} | 2015-07-04 13:19:44.101 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36704680/36705443/36705443/E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} | 2015-07-04 13:19:44.594 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705443/36705443/E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} | 2015-07-04 13:19:44.592 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36704680/36705443/36705443/E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} | 2015-07-04 13:19:44.631 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:19:44.641 DEBUG ElectionsService ] ELECTIONS: (V=59) PREPARE_OK FROM [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}](L=36704680,W=36705443,C=36705443,E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3}). [PID:03068:010 2015.07.04 13:19:44.641 DEBUG ElectionsService ] ELECTIONS: (V=59) SHIFT TO REG_LEADER. [PID:03068:010 2015.07.04 13:19:44.641 DEBUG ElectionsService ] ELECTIONS: (V=59) SENDING PROPOSAL CANDIDATE: [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}](L=36704680,W=36705443,C=36705443,E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3}), ME: [192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}](L=36704680,W=36705443,C=36705443,E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3}). [PID:03068:010 2015.07.04 13:19:44.641 DEBUG ElectionsService ] ELECTIONS: (V=59) ACCEPT FROM [192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}] M=[192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]). [PID:03068:013 2015.07.04 13:19:44.656 FATAL ResponseWriter ] Cannot write commands to the stream $projections-$master. status: AccessDenied [PID:03068:012 2015.07.04 13:19:44.656 FATAL ResponseWriter ] Cannot write commands to the stream $projections-$master. status: AccessDenied [PID:03068:013 2015.07.04 13:19:44.656 ERROR QueuedHandlerAutoRes] Error while processing message WRITE COMPLETED: CorrelationId: 7166e9d6-b5d7-422a-bed0-84adfecce8a6, Result: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projection Core #1'. System.Exception: Cannot write commands to the stream $projections-$master. status: AccessDenied at EventStore.Projections.Core.Services.Management.ResponseWriter.m__0 (EventStore.Core.Messages.WriteEventsCompleted completed) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStorey11`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response, OperationResult result) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStoreyB+c__AnonStoreyC.<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].EventStore.Core.Bus.IHandle.Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:014 2015.07.04 13:19:44.656 FATAL ResponseWriter ] Cannot write commands to the stream $projections-$master. status: AccessDenied [PID:03068:014 2015.07.04 13:19:44.656 ERROR QueuedHandlerAutoRes] Error while processing message WRITE COMPLETED: CorrelationId: dda87daf-481a-4058-a6fe-a037ec6b5487, Result: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projection Core #2'. System.Exception: Cannot write commands to the stream $projections-$master. status: AccessDenied at EventStore.Projections.Core.Services.Management.ResponseWriter.m__0 (EventStore.Core.Messages.WriteEventsCompleted completed) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStorey11`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response, OperationResult result) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStoreyB+c__AnonStoreyC.<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].EventStore.Core.Bus.IHandle.Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:012 2015.07.04 13:19:44.656 ERROR QueuedHandlerAutoRes] Error while processing message WRITE COMPLETED: CorrelationId: 7c591e13-d856-4b96-ab23-64d5cfafc560, Result: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projection Core #0'. System.Exception: Cannot write commands to the stream $projections-$master. status: AccessDenied at EventStore.Projections.Core.Services.Management.ResponseWriter.m__0 (EventStore.Core.Messages.WriteEventsCompleted completed) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStorey11`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response, OperationResult result) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStoreyB+c__AnonStoreyC.<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].EventStore.Core.Bus.IHandle.Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:010 2015.07.04 13:19:44.656 DEBUG ElectionsService ] ELECTIONS: (V=59) ACCEPT FROM [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] M=[192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]). [PID:03068:010 2015.07.04 13:19:44.656 INFO ElectionsService ] ELECTIONS: (V=59) DONE. ELECTED MASTER = [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}](L=36704680,W=36705443,C=36705443,E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3}). ME=[192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}](L=36704680,W=36705443,C=36705443,E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3}). [PID:03068:015 2015.07.04 13:19:44.658 DEBUG EpochManager ] === Update Last Epoch E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} (previous epoch at 36705219). [PID:03068:235 2015.07.04 13:19:44.997 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:03068:010 2015.07.04 13:19:46.034 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36704680/36705443/36705443/E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} | 2015-07-04 13:19:44.594 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705443/36705443/E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} | 2015-07-04 13:19:44.592 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:19:45.101 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36704680/36705443/36705443/E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} | 2015-07-04 13:19:46.034 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705443/36705443/E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} | 2015-07-04 13:19:45.503 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:19:46.034 -------------------------------------------------------------------------------- [PID:03068:235 2015.07.04 13:19:46.550 DEBUG GossipController ] Error while reading request (gossip): EndRead failure [PID:03068:010 2015.07.04 13:19:46.977 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.13:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36704680/36705443/36705443/E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} | 2015-07-04 13:19:46.034 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705443/36705443/E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} | 2015-07-04 13:19:45.503 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:19:46.102 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:19:46.825 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705443/36705443/E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} | 2015-07-04 13:19:46.512 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:19:46.977 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:19:48.043 DEBUG ElectionsService ] ELECTIONS: (V=61) VIEWCHANGE FROM [192.168.33.12:2113, {6648e0c4-835f-4575-a12e-0f82442ed57e}]. [PID:03068:010 2015.07.04 13:19:48.044 DEBUG ElectionsService ] ELECTIONS: (V=61) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:48.044 DEBUG ElectionsService ] ELECTIONS: (V=61) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:19:48.044 DEBUG ElectionsService ] ELECTIONS: (V=61) MAJORITY OF VIEWCHANGE. [PID:03068:010 2015.07.04 13:19:48.478 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:19:47.073 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [Slave, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705443/36705443/E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} | 2015-07-04 13:19:46.512 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:19:48.105 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:19:47.073 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705443/36705443/E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} | 2015-07-04 13:19:48.050 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:19:48.477 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:19:49.045 DEBUG ElectionsService ] ELECTIONS: (V=61) TIMED OUT! (S=ElectingLeader, M=). [PID:03068:010 2015.07.04 13:19:49.045 DEBUG ElectionsService ] ELECTIONS: (V=62) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:49.045 DEBUG ElectionsService ] ELECTIONS: (V=62) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:008 2015.07.04 13:19:49.193 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.13:1111, L192.168.33.11:49840, {2dc9dd75-c25f-465c-9480-bc32422ffdc5}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 11 [PID:03068:008 2015.07.04 13:19:49.193 INFO TcpConnection ] ES TcpConnection closed [13:19:49.193: N192.168.33.13:1111, L192.168.33.11:49840, {2dc9dd75-c25f-465c-9480-bc32422ffdc5}]: Received bytes: 1463, Sent bytes: 24189 Send calls: 11, callbacks: 10 Receive calls: 12, callbacks: 11 Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 11 [PID:03068:008 2015.07.04 13:19:49.193 INFO TcpConnectionManager] Connection 'master-normal' [192.168.33.13:1111, {2dc9dd75-c25f-465c-9480-bc32422ffdc5}] closed: Success. [PID:03068:010 2015.07.04 13:19:49.193 TRACE GossipServiceBase ] Looks like node [192.168.33.13:1111] is DEAD (TCP connection lost). [PID:03068:010 2015.07.04 13:19:49.193 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection lost to [192.168.33.13:1111]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:19:47.073 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705443/36705443/E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} | 2015-07-04 13:19:48.065 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:19:49.107 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:19:49.194 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705443/36705443/E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} | 2015-07-04 13:19:48.065 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:19:49.107 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:19:49.193 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}, InternalTcp: 192.168.33.13:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.13:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.13:2113, ExternalHttp: 192.168.33.13:2114]. [PID:03068:010 2015.07.04 13:19:49.694 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] PRE-REPLICA STATE, WAITING FOR CHASER TO CATCH UP... MASTER IS [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] [PID:03068:010 2015.07.04 13:19:49.695 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.13:1111, L192.168.33.11:49840, {2dc9dd75-c25f-465c-9480-bc32422ffdc5}] cleanly. Reason: Reconnecting from old master [192.168.33.13:1111] to new master: [192.168.33.13:1111]. [PID:03068:010 2015.07.04 13:19:49.696 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}, InternalTcp: 192.168.33.13:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.13:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.13:2113, ExternalHttp: 192.168.33.13:2114]. [PID:03068:010 2015.07.04 13:19:49.696 INFO ReplicaService ] Subscribing at LogPosition: 36705667 (0x2301583) to MASTER [192.168.33.13:1111, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] as replica with SubscriptionId: {f84397b2-7226-4540-9463-5b9c38882610}, ConnectionId: {47809dc3-19bb-4b65-810a-0579aae1c2d7}, LocalEndPoint: [], Epochs: E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740} E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65} E30@310956:{26cd0216-4e18-43aa-a899-11057cb44b2f} E29@293661:{e83f9ab8-c0e1-4b7f-ad41-504a907b90d1} E28@286573:{219b52c3-b7dd-4a47-834a-730aa0e69066}... . [PID:03068:014 2015.07.04 13:19:49.696 INFO ProjectionCoreServic] _projections is not empty after all the projections have been killed [PID:03068:013 2015.07.04 13:19:49.696 INFO ProjectionCoreServic] _projections is not empty after all the projections have been killed [PID:03068:012 2015.07.04 13:19:49.697 INFO ProjectionCoreServic] _projections is not empty after all the projections have been killed [PID:03068:013 2015.07.04 13:19:49.697 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:03068:012 2015.07.04 13:19:49.697 INFO ProjectionCoreServic] _subscriptions is not empty after all the projections have been killed [PID:03068:012 2015.07.04 13:19:49.697 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:03068:012 2015.07.04 13:19:49.697 INFO ProjectionCoreServic] _subscriptionEventReaders is not empty after all the projections have been killed [PID:03068:012 2015.07.04 13:19:49.699 ERROR QueuedHandlerAutoRes] Error while processing message EventStore.Projections.Core.Messages.ReaderSubscriptionManagement+Unsubscribe in queued handler 'Projection Core #0'. System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary. at System.Collections.Generic.Dictionary`2[System.Guid,System.Guid].get_Item (Guid key) [0x00000] in :0 at EventStore.Projections.Core.Services.Processing.EventReaderCoreService.Handle (EventStore.Projections.Core.Messages.Unsubscribe message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Projections.Core.Messages.ReaderSubscriptionManagement+Unsubscribe].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:014 2015.07.04 13:19:49.699 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:03068:010 2015.07.04 13:19:49.982 TRACE GossipServiceBase ] Looks like node [192.168.33.12:2113] is DEAD (Gossip send failed). [PID:03068:010 2015.07.04 13:19:49.984 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip send failed to [192.168.33.12:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:19:49.194 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705443/36705443/E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} | 2015-07-04 13:19:48.065 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:19:49.696 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:19:49.194 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705443/36705443/E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} | 2015-07-04 13:19:49.984 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:19:49.696 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:19:49.984 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}, InternalTcp: 192.168.33.13:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.13:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.13:2113, ExternalHttp: 192.168.33.13:2114]. [PID:03068:010 2015.07.04 13:19:50.045 DEBUG ElectionsService ] ELECTIONS: (V=62) TIMED OUT! (S=ElectingLeader, M=). [PID:03068:010 2015.07.04 13:19:50.045 DEBUG ElectionsService ] ELECTIONS: (V=63) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:50.045 DEBUG ElectionsService ] ELECTIONS: (V=63) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:231 2015.07.04 13:19:50.047 INFO TcpConnectionManager] Connection 'master-normal' ({47809dc3-19bb-4b65-810a-0579aae1c2d7}) to [192.168.33.13:1111] established. [PID:03068:010 2015.07.04 13:19:50.047 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection established to [192.168.33.13:1111]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:19:49.194 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705443/36705443/E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} | 2015-07-04 13:19:49.984 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:19:49.696 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:19:50.047 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705443/36705443/E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} | 2015-07-04 13:19:49.984 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:19:49.696 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:19:50.047 DEBUG ElectionsService ] ELECTIONS: (V=64) VIEWCHANGE FROM [192.168.33.12:2113, {6648e0c4-835f-4575-a12e-0f82442ed57e}]. [PID:03068:010 2015.07.04 13:19:50.047 DEBUG ElectionsService ] ELECTIONS: (V=64) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:50.047 DEBUG ElectionsService ] ELECTIONS: (V=64) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:19:50.047 DEBUG ElectionsService ] ELECTIONS: (V=64) MAJORITY OF VIEWCHANGE. [PID:03068:010 2015.07.04 13:19:50.702 INFO ReplicaService ] Subscribing at LogPosition: 36705667 (0x2301583) to MASTER [192.168.33.13:1111, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] as replica with SubscriptionId: {e67b5652-2e73-4b7f-b062-5dc17f1fb59c}, ConnectionId: {47809dc3-19bb-4b65-810a-0579aae1c2d7}, LocalEndPoint: [192.168.33.11:49847], Epochs: E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740} E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65} E30@310956:{26cd0216-4e18-43aa-a899-11057cb44b2f} E29@293661:{e83f9ab8-c0e1-4b7f-ad41-504a907b90d1} E28@286573:{219b52c3-b7dd-4a47-834a-730aa0e69066}... . [PID:03068:010 2015.07.04 13:19:51.049 DEBUG ElectionsService ] ELECTIONS: (V=64) TIMED OUT! (S=ElectingLeader, M=). [PID:03068:010 2015.07.04 13:19:51.049 DEBUG ElectionsService ] ELECTIONS: (V=65) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:51.049 DEBUG ElectionsService ] ELECTIONS: (V=65) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:19:51.703 INFO ReplicaService ] Subscribing at LogPosition: 36705667 (0x2301583) to MASTER [192.168.33.13:1111, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] as replica with SubscriptionId: {9332be89-603c-401d-8fb0-3ddcd9bbf6c7}, ConnectionId: {47809dc3-19bb-4b65-810a-0579aae1c2d7}, LocalEndPoint: [192.168.33.11:49847], Epochs: E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740} E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65} E30@310956:{26cd0216-4e18-43aa-a899-11057cb44b2f} E29@293661:{e83f9ab8-c0e1-4b7f-ad41-504a907b90d1} E28@286573:{219b52c3-b7dd-4a47-834a-730aa0e69066}... . [PID:03068:010 2015.07.04 13:19:52.050 DEBUG ElectionsService ] ELECTIONS: (V=65) TIMED OUT! (S=ElectingLeader, M=). [PID:03068:010 2015.07.04 13:19:52.050 DEBUG ElectionsService ] ELECTIONS: (V=66) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:52.050 DEBUG ElectionsService ] ELECTIONS: (V=66) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:19:52.643 TRACE ClusterVNodeControll] Ignoring ReplicaSubscribed because SubscriptionId {f84397b2-7226-4540-9463-5b9c38882610} is wrong. Current SubscriptionId is {9332be89-603c-401d-8fb0-3ddcd9bbf6c7}. [PID:03068:010 2015.07.04 13:19:52.644 TRACE ClusterVNodeControll] Ignoring ReplicaSubscribed because SubscriptionId {e67b5652-2e73-4b7f-b062-5dc17f1fb59c} is wrong. Current SubscriptionId is {9332be89-603c-401d-8fb0-3ddcd9bbf6c7}. [PID:03068:010 2015.07.04 13:19:52.704 INFO ReplicaService ] Subscribing at LogPosition: 36705667 (0x2301583) to MASTER [192.168.33.13:1111, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] as replica with SubscriptionId: {0161fa18-bae6-4266-85c1-fb206e6eeaed}, ConnectionId: {47809dc3-19bb-4b65-810a-0579aae1c2d7}, LocalEndPoint: [192.168.33.11:49847], Epochs: E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740} E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65} E30@310956:{26cd0216-4e18-43aa-a899-11057cb44b2f} E29@293661:{e83f9ab8-c0e1-4b7f-ad41-504a907b90d1} E28@286573:{219b52c3-b7dd-4a47-834a-730aa0e69066}... . [PID:03068:010 2015.07.04 13:19:53.051 DEBUG ElectionsService ] ELECTIONS: (V=66) TIMED OUT! (S=ElectingLeader, M=). [PID:03068:010 2015.07.04 13:19:53.051 DEBUG ElectionsService ] ELECTIONS: (V=67) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:53.051 DEBUG ElectionsService ] ELECTIONS: (V=67) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:013 2015.07.04 13:19:53.174 FATAL ResponseWriter ] Cannot write commands to the stream $projections-$master. status: AccessDenied [PID:03068:012 2015.07.04 13:19:53.174 FATAL ResponseWriter ] Cannot write commands to the stream $projections-$master. status: AccessDenied [PID:03068:013 2015.07.04 13:19:53.174 ERROR QueuedHandlerAutoRes] Error while processing message WRITE COMPLETED: CorrelationId: ee9870c0-e00f-4bb4-a568-3321d1a0277b, Result: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projection Core #1'. System.Exception: Cannot write commands to the stream $projections-$master. status: AccessDenied at EventStore.Projections.Core.Services.Management.ResponseWriter.m__0 (EventStore.Core.Messages.WriteEventsCompleted completed) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStorey11`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response, OperationResult result) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStoreyB+c__AnonStoreyC.<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].EventStore.Core.Bus.IHandle.Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:014 2015.07.04 13:19:53.174 FATAL ResponseWriter ] Cannot write commands to the stream $projections-$master. status: AccessDenied [PID:03068:012 2015.07.04 13:19:53.174 ERROR QueuedHandlerAutoRes] Error while processing message WRITE COMPLETED: CorrelationId: dd7cb799-c084-4d17-879a-181bf3b88af0, Result: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projection Core #0'. System.Exception: Cannot write commands to the stream $projections-$master. status: AccessDenied at EventStore.Projections.Core.Services.Management.ResponseWriter.m__0 (EventStore.Core.Messages.WriteEventsCompleted completed) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStorey11`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response, OperationResult result) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStoreyB+c__AnonStoreyC.<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].EventStore.Core.Bus.IHandle.Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:014 2015.07.04 13:19:53.175 ERROR QueuedHandlerAutoRes] Error while processing message WRITE COMPLETED: CorrelationId: 1192d3c2-c556-43d0-b37b-6b3587da48b9, Result: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projection Core #2'. System.Exception: Cannot write commands to the stream $projections-$master. status: AccessDenied at EventStore.Projections.Core.Services.Management.ResponseWriter.m__0 (EventStore.Core.Messages.WriteEventsCompleted completed) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStorey11`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response, OperationResult result) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStoreyB+c__AnonStoreyC.<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].EventStore.Core.Bus.IHandle.Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:010 2015.07.04 13:19:53.551 TRACE ClusterVNodeControll] Ignoring ReplicaSubscribed because SubscriptionId {9332be89-603c-401d-8fb0-3ddcd9bbf6c7} is wrong. Current SubscriptionId is {0161fa18-bae6-4266-85c1-fb206e6eeaed}. [PID:03068:010 2015.07.04 13:19:53.552 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CATCHING UP!!! BANZAI!!! MASTER IS [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] [PID:03068:007 2015.07.04 13:19:53.552 INFO ClusterStorageWriter] === SUBSCRIBED to [192.168.33.13:1111,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] at 36705667 (0x2301583). SubscriptionId: {0161fa18-bae6-4266-85c1-fb206e6eeaed}. [PID:03068:010 2015.07.04 13:19:53.642 DEBUG ElectionsService ] ELECTIONS: (V=68) VIEWCHANGE FROM [192.168.33.13:2113, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:19:53.642 DEBUG ElectionsService ] ELECTIONS: (V=68) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:53.642 DEBUG ElectionsService ] ELECTIONS: (V=68) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:19:53.648 DEBUG ElectionsService ] ELECTIONS: (V=68) MAJORITY OF VIEWCHANGE. [PID:03068:010 2015.07.04 13:19:53.651 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] CLONE ASSIGNMENT RECEIVED FROM [192.168.33.13:1111,n/a,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:19:53.651 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CLONE!!! SPARTA!!! MASTER IS [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] [PID:03068:010 2015.07.04 13:19:53.651 DEBUG ElectionsService ] ELECTIONS: (V=68) PREPARE FROM [192.168.33.13:2113, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:19:53.651 DEBUG ElectionsService ] ELECTIONS: (V=68) SHIFT TO REG_NONLEADER. [PID:03068:010 2015.07.04 13:19:54.485 DEBUG ElectionsService ] ELECTIONS: (V=69) VIEWCHANGE FROM [192.168.33.13:2113, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:19:54.485 DEBUG ElectionsService ] ELECTIONS: (V=69) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:19:54.485 DEBUG ElectionsService ] ELECTIONS: (V=69) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:19:54.485 DEBUG ElectionsService ] ELECTIONS: (V=69) MAJORITY OF VIEWCHANGE. [PID:03068:010 2015.07.04 13:19:54.485 DEBUG ElectionsService ] ELECTIONS: (V=69) SHIFT TO PREPARE PHASE. [PID:03068:010 2015.07.04 13:19:54.485 DEBUG ElectionsService ] ELECTIONS: (V=69) PREPARE_OK FROM [192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}](L=36705667,W=36722781,C=36722781,E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce}). [PID:03068:010 2015.07.04 13:19:54.984 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] SLAVE ASSIGNMENT RECEIVED FROM [192.168.33.13:1111,n/a,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:19:54.984 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS SLAVE!!! SPARTA!!! MASTER IS [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] [PID:03068:010 2015.07.04 13:19:54.990 DEBUG ElectionsService ] ELECTIONS: (V=69) PREPARE_OK FROM [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}](L=36705667,W=36722781,C=36722781,E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce}). [PID:03068:010 2015.07.04 13:19:54.990 DEBUG ElectionsService ] ELECTIONS: (V=69) SHIFT TO REG_LEADER. [PID:03068:010 2015.07.04 13:19:54.990 DEBUG ElectionsService ] ELECTIONS: (V=69) SENDING PROPOSAL CANDIDATE: [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}](L=36705667,W=36722781,C=36722781,E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce}), ME: [192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}](L=36705667,W=36722781,C=36722781,E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce}). [PID:03068:010 2015.07.04 13:19:54.990 DEBUG ElectionsService ] ELECTIONS: (V=69) ACCEPT FROM [192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}] M=[192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]). [PID:03068:013 2015.07.04 13:19:54.994 FATAL ResponseWriter ] Cannot write commands to the stream $projections-$master. status: AccessDenied [PID:03068:013 2015.07.04 13:19:54.994 ERROR QueuedHandlerAutoRes] Error while processing message WRITE COMPLETED: CorrelationId: 48b62c55-5e01-449a-afde-72b5b0094a5e, Result: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projection Core #1'. System.Exception: Cannot write commands to the stream $projections-$master. status: AccessDenied at EventStore.Projections.Core.Services.Management.ResponseWriter.m__0 (EventStore.Core.Messages.WriteEventsCompleted completed) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStorey11`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response, OperationResult result) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStoreyB+c__AnonStoreyC.<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].EventStore.Core.Bus.IHandle.Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:014 2015.07.04 13:19:54.994 FATAL ResponseWriter ] Cannot write commands to the stream $projections-$master. status: AccessDenied [PID:03068:014 2015.07.04 13:19:54.994 ERROR QueuedHandlerAutoRes] Error while processing message WRITE COMPLETED: CorrelationId: 0af24f65-bbd3-4598-9638-b0ae98ff5307, Result: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projection Core #2'. System.Exception: Cannot write commands to the stream $projections-$master. status: AccessDenied at EventStore.Projections.Core.Services.Management.ResponseWriter.m__0 (EventStore.Core.Messages.WriteEventsCompleted completed) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStorey11`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response, OperationResult result) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStoreyB+c__AnonStoreyC.<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].EventStore.Core.Bus.IHandle.Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:012 2015.07.04 13:19:55.013 FATAL ResponseWriter ] Cannot write commands to the stream $projections-$master. status: AccessDenied [PID:03068:012 2015.07.04 13:19:55.013 ERROR QueuedHandlerAutoRes] Error while processing message WRITE COMPLETED: CorrelationId: 4fbb329d-cb24-4ff5-a12a-4eaf7a8c9f29, Result: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projection Core #0'. System.Exception: Cannot write commands to the stream $projections-$master. status: AccessDenied at EventStore.Projections.Core.Services.Management.ResponseWriter.m__0 (EventStore.Core.Messages.WriteEventsCompleted completed) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStorey11`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response, OperationResult result) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStoreyB+c__AnonStoreyC.<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].EventStore.Core.Bus.IHandle.Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:010 2015.07.04 13:19:55.013 DEBUG ElectionsService ] ELECTIONS: (V=69) ACCEPT FROM [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] M=[192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]). [PID:03068:010 2015.07.04 13:19:55.013 INFO ElectionsService ] ELECTIONS: (V=69) DONE. ELECTED MASTER = [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}](L=36705667,W=36722781,C=36722781,E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce}). ME=[192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}](L=36705667,W=36722781,C=36722781,E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce}). [PID:03068:015 2015.07.04 13:19:55.014 DEBUG EpochManager ] === Update Last Epoch E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} (previous epoch at 36705443). [PID:03068:010 2015.07.04 13:19:55.118 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.13:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36705667/36722781/36722781/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:19:54.952 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705443/36705443/E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} | 2015-07-04 13:19:49.984 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:19:55.112 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:19:55.080 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705443/36705443/E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} | 2015-07-04 13:19:49.984 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:19:55.118 -------------------------------------------------------------------------------- [PID:03068:012 2015.07.04 13:19:55.249 FATAL ResponseWriter ] Cannot write commands to the stream $projections-$master. status: AccessDenied [PID:03068:012 2015.07.04 13:19:55.249 ERROR QueuedHandlerAutoRes] Error while processing message WRITE COMPLETED: CorrelationId: 9b1c7556-55ee-47a9-99ec-82a835a01f6f, Result: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projection Core #0'. System.Exception: Cannot write commands to the stream $projections-$master. status: AccessDenied at EventStore.Projections.Core.Services.Management.ResponseWriter.m__0 (EventStore.Core.Messages.WriteEventsCompleted completed) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStorey11`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response, OperationResult result) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStoreyB+c__AnonStoreyC.<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].EventStore.Core.Bus.IHandle.Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:244 2015.07.04 13:20:01.584 DEBUG GossipController ] Error while reading request (gossip): EndRead failure [PID:03068:238 2015.07.04 13:20:01.584 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:03068:244 2015.07.04 13:20:07.205 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:03068:244 2015.07.04 13:20:07.206 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:03068:244 2015.07.04 13:20:07.206 DEBUG GossipController ] Error while reading request (gossip): EndRead failure [PID:03068:244 2015.07.04 13:20:07.711 DEBUG GossipController ] Error while reading request (gossip): EndRead failure [PID:03068:010 2015.07.04 13:20:07.718 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:05.840 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705443/36705443/E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} | 2015-07-04 13:19:49.984 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:07.124 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:05.840 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:20:05.248 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:07.717 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:20:07.718 DEBUG ElectionsService ] ELECTIONS: (V=73) VIEWCHANGE FROM [192.168.33.12:2113, {6648e0c4-835f-4575-a12e-0f82442ed57e}]. [PID:03068:010 2015.07.04 13:20:07.718 DEBUG ElectionsService ] ELECTIONS: (V=73) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:20:07.718 DEBUG ElectionsService ] ELECTIONS: (V=73) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:20:07.718 DEBUG ElectionsService ] ELECTIONS: (V=73) MAJORITY OF VIEWCHANGE. [PID:03068:010 2015.07.04 13:20:07.722 DEBUG ElectionsService ] ELECTIONS: (V=73) SHIFT TO PREPARE PHASE. [PID:03068:010 2015.07.04 13:20:07.722 DEBUG ElectionsService ] ELECTIONS: (V=73) PREPARE_OK FROM [192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}](L=36705667,W=36723005,C=36723005,E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2}). [PID:03068:010 2015.07.04 13:20:08.723 DEBUG ElectionsService ] ELECTIONS: (V=73) TIMED OUT! (S=ElectingLeader, M=). [PID:03068:010 2015.07.04 13:20:08.723 DEBUG ElectionsService ] ELECTIONS: (V=74) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:20:08.723 DEBUG ElectionsService ] ELECTIONS: (V=74) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:20:08.998 TRACE GossipServiceBase ] Looks like master [192.168.33.13:2113, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] is DEAD (Gossip send failed), though we wait for TCP to decide. [PID:03068:008 2015.07.04 13:20:09.466 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.13:1111, L192.168.33.11:49847, {47809dc3-19bb-4b65-810a-0579aae1c2d7}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 28 [PID:03068:008 2015.07.04 13:20:09.466 INFO TcpConnection ] ES TcpConnection closed [13:20:09.467: N192.168.33.13:1111, L192.168.33.11:49847, {47809dc3-19bb-4b65-810a-0579aae1c2d7}]: Received bytes: 74912, Sent bytes: 25869 Send calls: 22, callbacks: 21 Receive calls: 29, callbacks: 28 Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 28 [PID:03068:008 2015.07.04 13:20:09.466 INFO TcpConnectionManager] Connection 'master-normal' [192.168.33.13:1111, {47809dc3-19bb-4b65-810a-0579aae1c2d7}] closed: Success. [PID:03068:010 2015.07.04 13:20:09.467 TRACE GossipServiceBase ] Looks like node [192.168.33.13:1111] is DEAD (TCP connection lost). [PID:03068:010 2015.07.04 13:20:09.467 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection lost to [192.168.33.13:1111]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:07.842 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:20:07.759 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:09.126 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:09.467 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:20:07.759 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:09.126 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:20:09.468 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}, InternalTcp: 192.168.33.13:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.13:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.13:2113, ExternalHttp: 192.168.33.13:2114]. [PID:03068:010 2015.07.04 13:20:09.724 DEBUG ElectionsService ] ELECTIONS: (V=74) TIMED OUT! (S=ElectingLeader, M=). [PID:03068:010 2015.07.04 13:20:09.724 DEBUG ElectionsService ] ELECTIONS: (V=75) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:20:09.724 DEBUG ElectionsService ] ELECTIONS: (V=75) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:20:09.970 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] PRE-REPLICA STATE, WAITING FOR CHASER TO CATCH UP... MASTER IS [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] [PID:03068:010 2015.07.04 13:20:09.970 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.13:1111, L192.168.33.11:49847, {47809dc3-19bb-4b65-810a-0579aae1c2d7}] cleanly. Reason: Reconnecting from old master [192.168.33.13:1111] to new master: [192.168.33.13:1111]. [PID:03068:010 2015.07.04 13:20:09.971 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}, InternalTcp: 192.168.33.13:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.13:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.13:2113, ExternalHttp: 192.168.33.13:2114]. [PID:03068:012 2015.07.04 13:20:09.971 INFO ProjectionCoreServic] _projections is not empty after all the projections have been killed [PID:03068:014 2015.07.04 13:20:09.971 INFO ProjectionCoreServic] _projections is not empty after all the projections have been killed [PID:03068:012 2015.07.04 13:20:09.971 INFO ProjectionCoreServic] _subscriptions is not empty after all the projections have been killed [PID:03068:014 2015.07.04 13:20:09.971 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:03068:012 2015.07.04 13:20:09.971 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:03068:012 2015.07.04 13:20:09.971 INFO ProjectionCoreServic] _subscriptionEventReaders is not empty after all the projections have been killed [PID:03068:013 2015.07.04 13:20:09.971 INFO ProjectionCoreServic] _projections is not empty after all the projections have been killed [PID:03068:012 2015.07.04 13:20:09.971 ERROR QueuedHandlerAutoRes] Error while processing message EventStore.Projections.Core.Messages.ReaderSubscriptionManagement+Unsubscribe in queued handler 'Projection Core #0'. System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary. at System.Collections.Generic.Dictionary`2[System.Guid,System.Guid].get_Item (Guid key) [0x00000] in :0 at EventStore.Projections.Core.Services.Processing.EventReaderCoreService.Handle (EventStore.Projections.Core.Messages.Unsubscribe message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Projections.Core.Messages.ReaderSubscriptionManagement+Unsubscribe].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:013 2015.07.04 13:20:09.971 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:03068:010 2015.07.04 13:20:09.973 INFO ReplicaService ] Subscribing at LogPosition: 36723005 (0x230593D) to MASTER [192.168.33.13:1111, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] as replica with SubscriptionId: {53c99298-1b83-45cd-9f38-f3b4815d1d8c}, ConnectionId: {189d8b0b-ecae-47bc-b4c5-2adcce7ffa22}, LocalEndPoint: [], Epochs: E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740} E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65} E30@310956:{26cd0216-4e18-43aa-a899-11057cb44b2f} E29@293661:{e83f9ab8-c0e1-4b7f-ad41-504a907b90d1}... . [PID:03068:010 2015.07.04 13:20:10.003 TRACE GossipServiceBase ] Looks like node [192.168.33.12:2113] is DEAD (Gossip send failed). [PID:03068:010 2015.07.04 13:20:10.004 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip send failed to [192.168.33.12:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:09.467 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:20:07.759 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:09.970 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:09.467 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:20:10.003 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:09.970 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:20:10.004 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}, InternalTcp: 192.168.33.13:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.13:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.13:2113, ExternalHttp: 192.168.33.13:2114]. [PID:03068:249 2015.07.04 13:20:10.008 INFO TcpConnectionManager] Connection 'master-normal' ({189d8b0b-ecae-47bc-b4c5-2adcce7ffa22}) to [192.168.33.13:1111] established. [PID:03068:010 2015.07.04 13:20:10.009 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection established to [192.168.33.13:1111]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:09.467 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:20:10.003 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:09.970 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:10.009 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:20:10.003 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:09.970 -------------------------------------------------------------------------------- [PID:03068:249 2015.07.04 13:20:10.009 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:03068:010 2015.07.04 13:20:10.011 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CATCHING UP!!! BANZAI!!! MASTER IS [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] [PID:03068:007 2015.07.04 13:20:10.011 INFO ClusterStorageWriter] === SUBSCRIBED to [192.168.33.13:1111,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] at 36723005 (0x230593D). SubscriptionId: {53c99298-1b83-45cd-9f38-f3b4815d1d8c}. [PID:03068:010 2015.07.04 13:20:10.013 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] CLONE ASSIGNMENT RECEIVED FROM [192.168.33.13:1111,n/a,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:20:10.013 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CLONE!!! SPARTA!!! MASTER IS [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] [PID:03068:010 2015.07.04 13:20:10.013 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] SLAVE ASSIGNMENT RECEIVED FROM [192.168.33.13:1111,n/a,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:20:10.013 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS SLAVE!!! SPARTA!!! MASTER IS [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] [PID:03068:013 2015.07.04 13:20:10.014 FATAL ResponseWriter ] Cannot write commands to the stream $projections-$master. status: AccessDenied [PID:03068:012 2015.07.04 13:20:10.014 FATAL ResponseWriter ] Cannot write commands to the stream $projections-$master. status: AccessDenied [PID:03068:013 2015.07.04 13:20:10.014 ERROR QueuedHandlerAutoRes] Error while processing message WRITE COMPLETED: CorrelationId: b332d61f-1f7f-40f2-9ed9-e21e9fa01f9f, Result: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projection Core #1'. System.Exception: Cannot write commands to the stream $projections-$master. status: AccessDenied at EventStore.Projections.Core.Services.Management.ResponseWriter.m__0 (EventStore.Core.Messages.WriteEventsCompleted completed) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStorey11`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response, OperationResult result) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStoreyB+c__AnonStoreyC.<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].EventStore.Core.Bus.IHandle.Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:012 2015.07.04 13:20:10.014 ERROR QueuedHandlerAutoRes] Error while processing message WRITE COMPLETED: CorrelationId: cd729f11-9f0b-49cc-aa2b-1e4152ec9ce8, Result: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projection Core #0'. System.Exception: Cannot write commands to the stream $projections-$master. status: AccessDenied at EventStore.Projections.Core.Services.Management.ResponseWriter.m__0 (EventStore.Core.Messages.WriteEventsCompleted completed) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStorey11`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response, OperationResult result) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStoreyB+c__AnonStoreyC.<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].EventStore.Core.Bus.IHandle.Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:014 2015.07.04 13:20:10.016 FATAL ResponseWriter ] Cannot write commands to the stream $projections-$master. status: AccessDenied [PID:03068:014 2015.07.04 13:20:10.017 ERROR QueuedHandlerAutoRes] Error while processing message WRITE COMPLETED: CorrelationId: dc37455c-376c-4181-a6c8-f32b77e88782, Result: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projection Core #2'. System.Exception: Cannot write commands to the stream $projections-$master. status: AccessDenied at EventStore.Projections.Core.Services.Management.ResponseWriter.m__0 (EventStore.Core.Messages.WriteEventsCompleted completed) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStorey11`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response, OperationResult result) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStoreyB+c__AnonStoreyC.<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].EventStore.Core.Bus.IHandle.Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:010 2015.07.04 13:20:10.726 DEBUG ElectionsService ] ELECTIONS: (V=75) TIMED OUT! (S=ElectingLeader, M=). [PID:03068:010 2015.07.04 13:20:10.726 DEBUG ElectionsService ] ELECTIONS: (V=76) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:20:10.726 DEBUG ElectionsService ] ELECTIONS: (V=76) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:20:11.727 DEBUG ElectionsService ] ELECTIONS: (V=76) TIMED OUT! (S=ElectingLeader, M=). [PID:03068:010 2015.07.04 13:20:11.727 DEBUG ElectionsService ] ELECTIONS: (V=77) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:20:11.727 DEBUG ElectionsService ] ELECTIONS: (V=77) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:008 2015.07.04 13:20:12.412 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.13:1111, L192.168.33.11:49859, {189d8b0b-ecae-47bc-b4c5-2adcce7ffa22}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 2 [PID:03068:008 2015.07.04 13:20:12.413 INFO TcpConnection ] ES TcpConnection closed [13:20:12.413: N192.168.33.13:1111, L192.168.33.11:49859, {189d8b0b-ecae-47bc-b4c5-2adcce7ffa22}]: Received bytes: 377, Sent bytes: 7215 Send calls: 6, callbacks: 6 Receive calls: 3, callbacks: 2 Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 2 [PID:03068:008 2015.07.04 13:20:12.413 INFO TcpConnectionManager] Connection 'master-normal' [192.168.33.13:1111, {189d8b0b-ecae-47bc-b4c5-2adcce7ffa22}] closed: Success. [PID:03068:010 2015.07.04 13:20:12.414 TRACE GossipServiceBase ] Looks like node [192.168.33.13:1111] is DEAD (TCP connection lost). [PID:03068:010 2015.07.04 13:20:12.414 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection lost to [192.168.33.13:1111]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:10.009 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:20:10.003 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:12.128 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:12.414 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:20:10.003 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:12.128 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:20:12.414 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}, InternalTcp: 192.168.33.13:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.13:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.13:2113, ExternalHttp: 192.168.33.13:2114]. [PID:03068:010 2015.07.04 13:20:12.705 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.13:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:12.414 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:20:10.003 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:12.128 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:12.165 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:20:10.003 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:12.704 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:20:12.727 DEBUG ElectionsService ] ELECTIONS: (V=77) TIMED OUT! (S=ElectingLeader, M=). [PID:03068:010 2015.07.04 13:20:12.727 DEBUG ElectionsService ] ELECTIONS: (V=78) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:20:12.727 DEBUG ElectionsService ] ELECTIONS: (V=78) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:20:12.915 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] PRE-REPLICA STATE, WAITING FOR CHASER TO CATCH UP... MASTER IS [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] [PID:03068:010 2015.07.04 13:20:12.916 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.13:1111, L192.168.33.11:49859, {189d8b0b-ecae-47bc-b4c5-2adcce7ffa22}] cleanly. Reason: Reconnecting from old master [192.168.33.13:1111] to new master: [192.168.33.13:1111]. [PID:03068:012 2015.07.04 13:20:12.917 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:03068:250 2015.07.04 13:20:12.918 INFO TcpConnectionManager] Connection 'master-normal' ({977aa412-2bbf-4dff-ae5a-97f2106686bb}) to [192.168.33.13:1111] established. [PID:03068:014 2015.07.04 13:20:12.918 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:03068:013 2015.07.04 13:20:12.918 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:03068:010 2015.07.04 13:20:12.918 INFO ReplicaService ] Subscribing at LogPosition: 36723005 (0x230593D) to MASTER [192.168.33.13:1111, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] as replica with SubscriptionId: {fde5afb5-d521-4836-b2ac-eae9b2064aef}, ConnectionId: {977aa412-2bbf-4dff-ae5a-97f2106686bb}, LocalEndPoint: [192.168.33.11:49865], Epochs: E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740} E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65} E30@310956:{26cd0216-4e18-43aa-a899-11057cb44b2f} E29@293661:{e83f9ab8-c0e1-4b7f-ad41-504a907b90d1}... . [PID:03068:010 2015.07.04 13:20:13.729 DEBUG ElectionsService ] ELECTIONS: (V=78) TIMED OUT! (S=ElectingLeader, M=). [PID:03068:010 2015.07.04 13:20:13.729 DEBUG ElectionsService ] ELECTIONS: (V=79) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:20:13.729 DEBUG ElectionsService ] ELECTIONS: (V=79) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:20:13.919 INFO ReplicaService ] Subscribing at LogPosition: 36723005 (0x230593D) to MASTER [192.168.33.13:1111, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] as replica with SubscriptionId: {a670e477-b4af-4746-921d-4d54b37453b9}, ConnectionId: {977aa412-2bbf-4dff-ae5a-97f2106686bb}, LocalEndPoint: [192.168.33.11:49865], Epochs: E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740} E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65} E30@310956:{26cd0216-4e18-43aa-a899-11057cb44b2f} E29@293661:{e83f9ab8-c0e1-4b7f-ad41-504a907b90d1}... . [PID:03068:249 2015.07.04 13:20:14.077 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:03068:250 2015.07.04 13:20:14.578 DEBUG GossipController ] Error while reading request (gossip): EndRead failure [PID:03068:008 2015.07.04 13:20:14.620 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.13:1111, L192.168.33.11:49865, {977aa412-2bbf-4dff-ae5a-97f2106686bb}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 0 [PID:03068:008 2015.07.04 13:20:14.621 INFO TcpConnection ] ES TcpConnection closed [13:20:14.621: N192.168.33.13:1111, L192.168.33.11:49865, {977aa412-2bbf-4dff-ae5a-97f2106686bb}]: Received bytes: 0, Sent bytes: 1260 Send calls: 3, callbacks: 3 Receive calls: 1, callbacks: 0 Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 0 [PID:03068:008 2015.07.04 13:20:14.621 INFO TcpConnectionManager] Connection 'master-normal' [192.168.33.13:1111, {977aa412-2bbf-4dff-ae5a-97f2106686bb}] closed: Success. [PID:03068:010 2015.07.04 13:20:14.621 TRACE GossipServiceBase ] Looks like node [192.168.33.13:1111] is DEAD (TCP connection lost). [PID:03068:010 2015.07.04 13:20:14.622 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection lost to [192.168.33.13:1111]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:12.919 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:20:10.003 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:14.131 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:14.622 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:20:10.003 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:14.131 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:20:14.622 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}, InternalTcp: 192.168.33.13:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.13:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.13:2113, ExternalHttp: 192.168.33.13:2114]. [PID:03068:010 2015.07.04 13:20:14.731 DEBUG ElectionsService ] ELECTIONS: (V=79) TIMED OUT! (S=ElectingLeader, M=). [PID:03068:010 2015.07.04 13:20:14.731 DEBUG ElectionsService ] ELECTIONS: (V=80) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:20:14.731 DEBUG ElectionsService ] ELECTIONS: (V=80) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:20:14.921 INFO ReplicaService ] Subscribing at LogPosition: 36723005 (0x230593D) to MASTER [192.168.33.13:1111, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] as replica with SubscriptionId: {65ac7de7-0049-4ef8-912f-bb0bc3640dc4}, ConnectionId: {977aa412-2bbf-4dff-ae5a-97f2106686bb}, LocalEndPoint: [192.168.33.11:49865], Epochs: E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740} E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65} E30@310956:{26cd0216-4e18-43aa-a899-11057cb44b2f} E29@293661:{e83f9ab8-c0e1-4b7f-ad41-504a907b90d1}... . [PID:03068:010 2015.07.04 13:20:15.123 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.13:1111, L192.168.33.11:49865, {977aa412-2bbf-4dff-ae5a-97f2106686bb}] cleanly. Reason: Reconnecting from old master [192.168.33.13:1111] to new master: [192.168.33.13:1111]. [PID:03068:249 2015.07.04 13:20:15.124 INFO TcpConnectionManager] Connection 'master-normal' ({bb02c484-3def-483d-ab12-d4f5f18bdd68}) to [192.168.33.13:1111] established. [PID:03068:010 2015.07.04 13:20:15.125 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection established to [192.168.33.13:1111]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:14.622 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:20:10.003 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:14.131 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:15.124 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:20:10.003 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:14.131 -------------------------------------------------------------------------------- [PID:03068:010 2015.07.04 13:20:15.211 DEBUG ElectionsService ] ELECTIONS: (V=79) VIEWCHANGE FROM [192.168.33.13:2113, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:20:15.211 DEBUG ElectionsService ] ELECTIONS: (V=79) MAJORITY OF VIEWCHANGE. [PID:03068:010 2015.07.04 13:20:15.211 DEBUG ElectionsService ] ELECTIONS: (V=78) VIEWCHANGE FROM [192.168.33.13:2113, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:20:15.476 DEBUG ElectionsService ] ELECTIONS: (IV=80) VIEWCHANGEPROOF FROM [192.168.33.13:2113, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. JUMPING TO NON-LEADER STATE. [PID:03068:010 2015.07.04 13:20:15.476 DEBUG ElectionsService ] ELECTIONS: (V=80) SHIFT TO REG_NONLEADER. [PID:03068:010 2015.07.04 13:20:15.734 DEBUG ElectionsService ] ELECTIONS: (V=80) TIMED OUT! (S=NonLeader, M=). [PID:03068:010 2015.07.04 13:20:15.734 DEBUG ElectionsService ] ELECTIONS: (V=81) SHIFT TO LEADER ELECTION. [PID:03068:010 2015.07.04 13:20:15.734 DEBUG ElectionsService ] ELECTIONS: (V=81) VIEWCHANGE FROM [192.168.33.11:2113, {7a43c5fe-02a4-4c29-a61e-2d8cfa201207}]. [PID:03068:010 2015.07.04 13:20:15.742 DEBUG ElectionsService ] ELECTIONS: (V=81) VIEWCHANGE FROM [192.168.33.13:2113, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:20:15.742 DEBUG ElectionsService ] ELECTIONS: (V=81) MAJORITY OF VIEWCHANGE. [PID:03068:010 2015.07.04 13:20:15.742 DEBUG ElectionsService ] ELECTIONS: (V=81) SHIFT TO PREPARE PHASE. [PID:03068:010 2015.07.04 13:20:15.742 DEBUG ElectionsService ] ELECTIONS: (V=81) PREPARE_OK FROM [192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}](L=36705667,W=36723005,C=36723005,E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2}). [PID:03068:010 2015.07.04 13:20:15.746 DEBUG ElectionsService ] ELECTIONS: (V=81) PREPARE_OK FROM [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}](L=36705667,W=36723005,C=36723005,E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2}). [PID:03068:010 2015.07.04 13:20:15.746 DEBUG ElectionsService ] ELECTIONS: (V=81) SHIFT TO REG_LEADER. [PID:03068:010 2015.07.04 13:20:15.746 DEBUG ElectionsService ] ELECTIONS: (V=81) SENDING PROPOSAL CANDIDATE: [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}](L=36705667,W=36723005,C=36723005,E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2}), ME: [192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}](L=36705667,W=36723005,C=36723005,E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2}). [PID:03068:010 2015.07.04 13:20:15.746 DEBUG ElectionsService ] ELECTIONS: (V=81) ACCEPT FROM [192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}] M=[192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]). [PID:03068:010 2015.07.04 13:20:15.751 DEBUG ElectionsService ] ELECTIONS: (V=81) ACCEPT FROM [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] M=[192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]). [PID:03068:010 2015.07.04 13:20:15.751 INFO ElectionsService ] ELECTIONS: (V=81) DONE. ELECTED MASTER = [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}](L=36705667,W=36723005,C=36723005,E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2}). ME=[192.168.33.11:2113,{7a43c5fe-02a4-4c29-a61e-2d8cfa201207}](L=36705667,W=36723005,C=36723005,E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2}). [PID:03068:010 2015.07.04 13:20:15.926 INFO ReplicaService ] Subscribing at LogPosition: 36723005 (0x230593D) to MASTER [192.168.33.13:1111, {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] as replica with SubscriptionId: {69e15707-cf0e-4412-8a52-0f071eacbf21}, ConnectionId: {bb02c484-3def-483d-ab12-d4f5f18bdd68}, LocalEndPoint: [192.168.33.11:49867], Epochs: E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740} E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65} E30@310956:{26cd0216-4e18-43aa-a899-11057cb44b2f} E29@293661:{e83f9ab8-c0e1-4b7f-ad41-504a907b90d1}... . [PID:03068:007 2015.07.04 13:20:15.930 INFO ClusterStorageWriter] === SUBSCRIBED to [192.168.33.13:1111,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] at 36723005 (0x230593D). SubscriptionId: {69e15707-cf0e-4412-8a52-0f071eacbf21}. [PID:03068:010 2015.07.04 13:20:15.930 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CATCHING UP!!! BANZAI!!! MASTER IS [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] [PID:03068:010 2015.07.04 13:20:15.931 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] CLONE ASSIGNMENT RECEIVED FROM [192.168.33.13:1111,n/a,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:20:15.931 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CLONE!!! SPARTA!!! MASTER IS [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] [PID:03068:010 2015.07.04 13:20:15.932 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] SLAVE ASSIGNMENT RECEIVED FROM [192.168.33.13:1111,n/a,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}]. [PID:03068:010 2015.07.04 13:20:15.932 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS SLAVE!!! SPARTA!!! MASTER IS [192.168.33.13:2113,{eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff}] [PID:03068:015 2015.07.04 13:20:15.937 DEBUG EpochManager ] === Update Last Epoch E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} (previous epoch at 36722781). [PID:03068:012 2015.07.04 13:20:15.941 FATAL ResponseWriter ] Cannot write commands to the stream $projections-$master. status: AccessDenied [PID:03068:012 2015.07.04 13:20:15.941 ERROR QueuedHandlerAutoRes] Error while processing message WRITE COMPLETED: CorrelationId: 9ef17d15-94bc-4724-b5e5-3913a4a90889, Result: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projection Core #0'. System.Exception: Cannot write commands to the stream $projections-$master. status: AccessDenied at EventStore.Projections.Core.Services.Management.ResponseWriter.m__0 (EventStore.Core.Messages.WriteEventsCompleted completed) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStorey11`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response, OperationResult result) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStoreyB+c__AnonStoreyC.<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].EventStore.Core.Bus.IHandle.Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:014 2015.07.04 13:20:15.944 FATAL ResponseWriter ] Cannot write commands to the stream $projections-$master. status: AccessDenied [PID:03068:014 2015.07.04 13:20:15.944 ERROR QueuedHandlerAutoRes] Error while processing message WRITE COMPLETED: CorrelationId: 1ef5bfa6-141c-4deb-bde2-00e08cfd8a06, Result: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projection Core #2'. System.Exception: Cannot write commands to the stream $projections-$master. status: AccessDenied at EventStore.Projections.Core.Services.Management.ResponseWriter.m__0 (EventStore.Core.Messages.WriteEventsCompleted completed) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStorey11`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response, OperationResult result) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStoreyB+c__AnonStoreyC.<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].EventStore.Core.Bus.IHandle.Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:013 2015.07.04 13:20:15.945 FATAL ResponseWriter ] Cannot write commands to the stream $projections-$master. status: AccessDenied [PID:03068:013 2015.07.04 13:20:15.945 ERROR QueuedHandlerAutoRes] Error while processing message WRITE COMPLETED: CorrelationId: 73a46845-0048-4b85-aff9-16e3b6ee8a54, Result: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projection Core #1'. System.Exception: Cannot write commands to the stream $projections-$master. status: AccessDenied at EventStore.Projections.Core.Services.Management.ResponseWriter.m__0 (EventStore.Core.Messages.WriteEventsCompleted completed) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStorey11`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response, OperationResult result) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStoreyB+c__AnonStoreyC.<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].EventStore.Core.Bus.IHandle.Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:010 2015.07.04 13:20:16.140 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.13:2113]) Old: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36705667/36723005/36723005/E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} | 2015-07-04 13:20:15.177 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:20:10.003 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723229/36723229/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:20:16.133 New: VND {eeba22ab-bd6e-4c4b-bcb9-5b883bf7c2ff} [Master, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36705667/36723229/36723229/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:20:16.102 VND {6648e0c4-835f-4575-a12e-0f82442ed57e} [PreReplica, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36704680/36705667/36705667/E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} | 2015-07-04 13:20:10.003 VND {7a43c5fe-02a4-4c29-a61e-2d8cfa201207} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723229/36723229/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:20:16.140 -------------------------------------------------------------------------------- [PID:03068:012 2015.07.04 13:20:16.200 FATAL ResponseWriter ] Cannot write commands to the stream $projections-$master. status: AccessDenied [PID:03068:012 2015.07.04 13:20:16.200 ERROR QueuedHandlerAutoRes] Error while processing message WRITE COMPLETED: CorrelationId: 6833ab21-87e9-4cdd-95b9-7ffd49397fe1, Result: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projection Core #0'. System.Exception: Cannot write commands to the stream $projections-$master. status: AccessDenied at EventStore.Projections.Core.Services.Management.ResponseWriter.m__0 (EventStore.Core.Messages.WriteEventsCompleted completed) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStorey11`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response, OperationResult result) [0x00000] in :0 at EventStore.Core.Helpers.IODispatcherAsync+c__AnonStoreyB+c__AnonStoreyC.<>m__0 (EventStore.Core.Messages.WriteEventsCompleted response) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+WriteEvents,EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].EventStore.Core.Bus.IHandle.Handle (EventStore.Core.Messages.WriteEventsCompleted message) [0x00000] in :0 at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+WriteEventsCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0 [PID:03068:249 2015.07.04 13:20:19.058 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:10352:001 2015.07.04 13:20:49.788 INFO ProgramBase`1 ] ES VERSION: 3.0.5.0 (release-v3.0.5/a9a82d955a0ac1db548b1873fa3be66923dc6032, Tue, 26 May 2015 18:51:16 +0000) OS: Unknown (Unix 3.13.0.55) RUNTIME: 3.12.0 (tarball) (64-bit) GC: 2 GENERATIONS LOGS: /tmp/log HELP: False () VERSION: False () LOG: /tmp/log (Command Line) CONFIG: () DEFINES: () WHAT IF: True (Command Line) INT IP: 192.168.33.11 (Command Line) EXT IP: 192.168.33.11 (Command Line) INT HTTP PORT: 2113 (Command Line) EXT HTTP PORT: 2114 (Command Line) INT TCP PORT: 1111 (Command Line) INT SECURE TCP PORT: 0 () EXT TCP PORT: 1112 (Command Line) EXT SECURE TCP PORT: 0 () INT TCP HEARTBEAT TIMEOUT: 700 () EXT TCP HEARTBEAT TIMEOUT: 1000 () INT TCP HEARTBEAT INTERVAL: 700 () EXT TCP HEARTBEAT INTERVAL: 2000 () FORCE: False () CLUSTER SIZE: 3 (Command Line) NODE PRIORITY: 0 () MIN FLUSH DELAY MS: 2 () COMMIT COUNT: -1 () PREPARE COUNT: -1 () ADMIN ON EXT: True () STATS ON EXT: True () GOSSIP ON EXT: True () DISABLE SCAVENGE MERGING: False () DISCOVER VIA DNS: True () CLUSTER DNS: escluster.mooo.com (Command Line) CLUSTER GOSSIP PORT: 2113 (Command Line) GOSSIP SEED: () STATS PERIOD SEC: 30 () CACHED CHUNKS: -1 () CHUNKS CACHE SIZE: 536871424 () MAX MEM TABLE SIZE: 1000000 () DB: /tmp/db (Command Line) MEM DB: False () SKIP DB VERIFY: False () RUN PROJECTIONS: All (Command Line) PROJECTION THREADS: 3 () WORKER THREADS: 5 () HTTP PREFIXES: () ENABLE TRUSTED AUTH: False () CERTIFICATE STORE LOCATION: () CERTIFICATE STORE NAME: () CERTIFICATE SUBJECT NAME: () CERTIFICATE THUMBPRINT: () CERTIFICATE FILE: () CERTIFICATE PASSWORD: () USE INTERNAL SSL: False () SSL TARGET HOST: n/a () SSL VALIDATE SERVER: True () AUTHENTICATION TYPE: internal () PREPARE TIMEOUT MS: 2000 () COMMIT TIMEOUT MS: 2000 () UNSAFE DISABLE FLUSH TO DISK: False () GOSSIP INTERVAL MS: 1000 () GOSSIP ALLOWED DIFFERENCE MS: 60000 () GOSSIP TIMEOUT MS: 500 () [PID:10352:001 2015.07.04 13:20:49.804 INFO Application ] Exiting with exit code: 0. Exit reason: WhatIf option specified [PID:10368:001 2015.07.04 13:22:45.429 INFO ProgramBase`1 ] ES VERSION: 3.0.5.0 (release-v3.0.5/a9a82d955a0ac1db548b1873fa3be66923dc6032, Tue, 26 May 2015 18:51:16 +0000) OS: Unknown (Unix 3.13.0.55) RUNTIME: 3.12.0 (tarball) (64-bit) GC: 2 GENERATIONS LOGS: /tmp/log HELP: False () VERSION: False () LOG: /tmp/log (Command Line) CONFIG: () DEFINES: () WHAT IF: False () INT IP: 192.168.33.11 (Command Line) EXT IP: 192.168.33.11 (Command Line) INT HTTP PORT: 2113 (Command Line) EXT HTTP PORT: 2114 (Command Line) INT TCP PORT: 1111 (Command Line) INT SECURE TCP PORT: 0 () EXT TCP PORT: 1112 (Command Line) EXT SECURE TCP PORT: 0 () INT TCP HEARTBEAT TIMEOUT: 700 () EXT TCP HEARTBEAT TIMEOUT: 1000 () INT TCP HEARTBEAT INTERVAL: 700 () EXT TCP HEARTBEAT INTERVAL: 2000 () FORCE: False () CLUSTER SIZE: 3 (Command Line) NODE PRIORITY: 0 () MIN FLUSH DELAY MS: 2 () COMMIT COUNT: -1 () PREPARE COUNT: -1 () ADMIN ON EXT: True () STATS ON EXT: True () GOSSIP ON EXT: True () DISABLE SCAVENGE MERGING: False () DISCOVER VIA DNS: True () CLUSTER DNS: escluster.mooo.com (Command Line) CLUSTER GOSSIP PORT: 2113 (Command Line) GOSSIP SEED: () STATS PERIOD SEC: 30 () CACHED CHUNKS: -1 () CHUNKS CACHE SIZE: 536871424 () MAX MEM TABLE SIZE: 1000000 () DB: /tmp/db (Command Line) MEM DB: False () SKIP DB VERIFY: False () RUN PROJECTIONS: All (Command Line) PROJECTION THREADS: 3 () WORKER THREADS: 5 () HTTP PREFIXES: () ENABLE TRUSTED AUTH: False () CERTIFICATE STORE LOCATION: () CERTIFICATE STORE NAME: () CERTIFICATE SUBJECT NAME: () CERTIFICATE THUMBPRINT: () CERTIFICATE FILE: () CERTIFICATE PASSWORD: () USE INTERNAL SSL: False () SSL TARGET HOST: n/a () SSL VALIDATE SERVER: True () AUTHENTICATION TYPE: internal () PREPARE TIMEOUT MS: 2000 () COMMIT TIMEOUT MS: 2000 () UNSAFE DISABLE FLUSH TO DISK: False () GOSSIP INTERVAL MS: 2000 (Command Line) GOSSIP ALLOWED DIFFERENCE MS: 60000 () GOSSIP TIMEOUT MS: 3000 (Command Line) [PID:10368:001 2015.07.04 13:22:45.445 INFO ProgramBase`1 ] Quorum size set to 2 [PID:10368:001 2015.07.04 13:22:45.447 INFO ProgramBase`1 ] Can't find plugins path: plugins [PID:10368:001 2015.07.04 13:22:45.462 INFO ProgramBase`1 ] INSTANCE ID: 843c5aa9-0910-4d75-9747-251d784ce634 DATABASE: /tmp/db WRITER CHECKPOINT: 36723229 (0x2305A1D) CHASER CHECKPOINT: 36723229 (0x2305A1D) EPOCH CHECKPOINT: 36723005 (0x230593D) TRUNCATE CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF) [PID:10368:001 2015.07.04 13:22:45.556 TRACE MessageHierarchy ] MessageHierarchy initialization took 00:00:00.0710183. [PID:10368:001 2015.07.04 13:22:45.766 TRACE TFChunk ] CACHED TFChunk #0-0 (chunk-000000.000000) in 00:00:00.0104186. [PID:10368:001 2015.07.04 13:22:45.893 INFO MiniWeb ] Starting MiniWeb for /web/es/js/projections ==> /home/vagrant/EventStore-OSS-Linux-v3.0.5/projections [PID:10368:001 2015.07.04 13:22:45.895 INFO MiniWeb ] Starting MiniWeb for /web/es/js/projections/v8/Prelude ==> /home/vagrant/EventStore-OSS-Linux-v3.0.5/Prelude [PID:10368:001 2015.07.04 13:22:45.895 INFO MiniWeb ] Starting MiniWeb for /web/es/js/projections/resources ==> /home/vagrant/EventStore-OSS-Linux-v3.0.5/web-resources/js [PID:10368:001 2015.07.04 13:22:45.896 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/{*remaining_path} [PID:10368:001 2015.07.04 13:22:45.896 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/v8/Prelude/{*remaining_path} [PID:10368:001 2015.07.04 13:22:45.896 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/resources/{*remaining_path} [PID:10368:001 2015.07.04 13:22:45.897 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/{*remaining_path} [PID:10368:001 2015.07.04 13:22:45.897 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/v8/Prelude/{*remaining_path} [PID:10368:001 2015.07.04 13:22:45.897 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/resources/{*remaining_path} [PID:10368:001 2015.07.04 13:22:45.902 INFO MiniWeb ] Starting MiniWeb for /web ==> /home/vagrant/EventStore-OSS-Linux-v3.0.5/clusternode-web [PID:10368:001 2015.07.04 13:22:45.902 TRACE MiniWeb ] Binding MiniWeb to /web/{*remaining_path} [PID:10368:001 2015.07.04 13:22:45.902 INFO MiniWeb ] Starting MiniWeb for /web/users ==> /home/vagrant/EventStore-OSS-Linux-v3.0.5/Users/web [PID:10368:001 2015.07.04 13:22:45.902 TRACE MiniWeb ] Binding MiniWeb to /web/users/{*remaining_path} [PID:10368:001 2015.07.04 13:22:45.902 INFO MiniWeb ] Starting MiniWeb for /web ==> /home/vagrant/EventStore-OSS-Linux-v3.0.5/clusternode-web [PID:10368:001 2015.07.04 13:22:45.902 TRACE MiniWeb ] Binding MiniWeb to /web/{*remaining_path} [PID:10368:001 2015.07.04 13:22:45.903 INFO MiniWeb ] Starting MiniWeb for /web/users ==> /home/vagrant/EventStore-OSS-Linux-v3.0.5/Users/web [PID:10368:001 2015.07.04 13:22:45.903 TRACE MiniWeb ] Binding MiniWeb to /web/users/{*remaining_path} [PID:10368:010 2015.07.04 13:22:45.926 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] SYSTEM INIT... [PID:10368:010 2015.07.04 13:22:45.940 INFO TcpServerListener ] Starting Normal TCP listening on TCP endpoint: 192.168.33.11:1112. [PID:10368:010 2015.07.04 13:22:45.944 INFO TcpServerListener ] Starting Normal TCP listening on TCP endpoint: 192.168.33.11:1111. [PID:10368:015 2015.07.04 13:22:45.949 INFO IndexCommitter ] TableIndex initialization... [PID:10368:015 2015.07.04 13:22:45.952 INFO IndexCommitter ] ReadIndex building... [PID:10368:010 2015.07.04 13:22:45.963 INFO HttpAsyncServer ] Starting HTTP server on [http://192.168.33.11:2114/]... [PID:10368:010 2015.07.04 13:22:45.965 INFO HttpAsyncServer ] HTTP server is up and listening on [http://192.168.33.11:2114/] [PID:10368:010 2015.07.04 13:22:45.965 INFO HttpAsyncServer ] Starting HTTP server on [http://192.168.33.11:2113/]... [PID:10368:010 2015.07.04 13:22:45.965 INFO HttpAsyncServer ] HTTP server is up and listening on [http://192.168.33.11:2113/] [PID:10368:010 2015.07.04 13:22:45.981 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [MainQueue]: SystemInit - 62ms. Q: 0/596. [PID:10368:010 2015.07.04 13:22:45.981 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] Service 'StorageWriter' initialized. [PID:10368:010 2015.07.04 13:22:45.981 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] Service 'StorageReader' initialized. [PID:10368:015 2015.07.04 13:22:46.004 DEBUG IndexCommitter ] ReadIndex rebuilding done: total processed 2829 records, time elapsed: 00:00:00.0517210. [PID:10368:010 2015.07.04 13:22:46.014 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] Service 'StorageChaser' initialized. [PID:10368:010 2015.07.04 13:22:46.014 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] SYSTEM START... [PID:10368:010 2015.07.04 13:22:46.016 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS UNKNOWN!!! WHOA!!! [PID:10368:010 2015.07.04 13:22:46.026 DEBUG ElectionsService ] ELECTIONS: STARTING ELECTIONS. [PID:10368:010 2015.07.04 13:22:46.028 DEBUG ElectionsService ] ELECTIONS: (V=0) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:22:46.028 DEBUG ElectionsService ] ELECTIONS: (V=0) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:22:46.165 TRACE GossipServiceBase ] CLUSTER HAS CHANGED Old: VND {843c5aa9-0910-4d75-9747-251d784ce634} [Unknown, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723229/36723229/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:22:46.018 New: MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.33.13:2113, 192.168.33.13:2113] | 2015-07-04 13:22:46.151 MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.33.12:2113, 192.168.33.12:2113] | 2015-07-04 13:22:46.151 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Unknown, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723229/36723229/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:22:46.018 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:22:47.031 DEBUG ElectionsService ] ELECTIONS: (V=0) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:22:47.031 DEBUG ElectionsService ] ELECTIONS: (V=1) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:22:47.031 DEBUG ElectionsService ] ELECTIONS: (V=1) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:22:48.032 DEBUG ElectionsService ] ELECTIONS: (V=1) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:22:48.032 DEBUG ElectionsService ] ELECTIONS: (V=2) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:22:48.032 DEBUG ElectionsService ] ELECTIONS: (V=2) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:22:49.033 DEBUG ElectionsService ] ELECTIONS: (V=2) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:22:49.033 DEBUG ElectionsService ] ELECTIONS: (V=3) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:22:49.033 DEBUG ElectionsService ] ELECTIONS: (V=3) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:22:49.210 TRACE GossipServiceBase ] Looks like node [192.168.33.13:2113] is DEAD (Gossip send failed). [PID:10368:010 2015.07.04 13:22:49.211 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip send failed to [192.168.33.13:2113]) Old: MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.33.13:2113, 192.168.33.13:2113] | 2015-07-04 13:22:46.151 MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.33.12:2113, 192.168.33.12:2113] | 2015-07-04 13:22:46.151 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Unknown, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723229/36723229/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:22:48.284 New: MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.33.13:2113, 192.168.33.13:2113] | 2015-07-04 13:22:49.210 MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.33.12:2113, 192.168.33.12:2113] | 2015-07-04 13:22:46.151 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Unknown, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723229/36723229/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:22:48.284 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:22:49.977 TRACE GossipServiceBase ] Looks like node [192.168.33.12:2113] is DEAD (Gossip send failed). [PID:10368:010 2015.07.04 13:22:49.977 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip send failed to [192.168.33.12:2113]) Old: MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.33.13:2113, 192.168.33.13:2113] | 2015-07-04 13:22:49.210 MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.33.12:2113, 192.168.33.12:2113] | 2015-07-04 13:22:46.151 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Unknown, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723229/36723229/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:22:48.284 New: MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.33.13:2113, 192.168.33.13:2113] | 2015-07-04 13:22:49.210 MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.33.12:2113, 192.168.33.12:2113] | 2015-07-04 13:22:49.977 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Unknown, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723229/36723229/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:22:48.284 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:22:50.034 DEBUG ElectionsService ] ELECTIONS: (V=3) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:22:50.034 DEBUG ElectionsService ] ELECTIONS: (V=4) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:22:50.034 DEBUG ElectionsService ] ELECTIONS: (V=4) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:22:51.036 DEBUG ElectionsService ] ELECTIONS: (V=4) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:22:51.036 DEBUG ElectionsService ] ELECTIONS: (V=5) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:22:51.036 DEBUG ElectionsService ] ELECTIONS: (V=5) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:22:52.038 DEBUG ElectionsService ] ELECTIONS: (V=5) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:22:52.038 DEBUG ElectionsService ] ELECTIONS: (V=6) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:22:52.038 DEBUG ElectionsService ] ELECTIONS: (V=6) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:22:53.039 DEBUG ElectionsService ] ELECTIONS: (V=6) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:22:53.039 DEBUG ElectionsService ] ELECTIONS: (V=7) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:22:53.039 DEBUG ElectionsService ] ELECTIONS: (V=7) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:22:54.040 DEBUG ElectionsService ] ELECTIONS: (V=7) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:22:54.040 DEBUG ElectionsService ] ELECTIONS: (V=8) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:22:54.040 DEBUG ElectionsService ] ELECTIONS: (V=8) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:22:55.041 DEBUG ElectionsService ] ELECTIONS: (V=8) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:22:55.041 DEBUG ElectionsService ] ELECTIONS: (V=9) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:22:55.041 DEBUG ElectionsService ] ELECTIONS: (V=9) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:22:56.042 DEBUG ElectionsService ] ELECTIONS: (V=9) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:22:56.042 DEBUG ElectionsService ] ELECTIONS: (V=10) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:22:56.042 DEBUG ElectionsService ] ELECTIONS: (V=10) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:22:57.044 DEBUG ElectionsService ] ELECTIONS: (V=10) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:22:57.044 DEBUG ElectionsService ] ELECTIONS: (V=11) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:22:57.044 DEBUG ElectionsService ] ELECTIONS: (V=11) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:22:57.781 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.33.13:2113, 192.168.33.13:2113] | 2015-07-04 13:22:49.210 MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.33.12:2113, 192.168.33.12:2113] | 2015-07-04 13:22:49.977 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Unknown, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723229/36723229/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:22:56.287 New: MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.33.13:2113, 192.168.33.13:2113] | 2015-07-04 13:22:57.493 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Unknown, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36723229/36740334/36740334/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:22:57.737 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Unknown, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36705667/36723229/36723229/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:22:57.779 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:22:58.045 DEBUG ElectionsService ] ELECTIONS: (V=11) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:22:58.045 DEBUG ElectionsService ] ELECTIONS: (V=12) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:22:58.045 DEBUG ElectionsService ] ELECTIONS: (V=12) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:22:59.047 DEBUG ElectionsService ] ELECTIONS: (V=12) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:22:59.047 DEBUG ElectionsService ] ELECTIONS: (V=13) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:22:59.047 DEBUG ElectionsService ] ELECTIONS: (V=13) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:00.048 DEBUG ElectionsService ] ELECTIONS: (V=13) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:23:00.048 DEBUG ElectionsService ] ELECTIONS: (V=14) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:00.048 DEBUG ElectionsService ] ELECTIONS: (V=14) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:01.050 DEBUG ElectionsService ] ELECTIONS: (V=14) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:23:01.050 DEBUG ElectionsService ] ELECTIONS: (V=15) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:01.050 DEBUG ElectionsService ] ELECTIONS: (V=15) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:01.887 DEBUG ElectionsService ] ELECTIONS: (V=13) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:01.887 DEBUG ElectionsService ] ELECTIONS: (V=13) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:23:01.887 DEBUG ElectionsService ] ELECTIONS: (V=15) SHIFT TO PREPARE PHASE. [PID:10368:010 2015.07.04 13:23:01.888 DEBUG ElectionsService ] ELECTIONS: (V=15) PREPARE_OK FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36705667,W=36723229,C=36723229,E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2}). [PID:10368:010 2015.07.04 13:23:01.901 DEBUG ElectionsService ] ELECTIONS: (V=15) PREPARE_OK FROM [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}](L=36723229,W=36740334,C=36740334,E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2}). [PID:10368:010 2015.07.04 13:23:01.901 DEBUG ElectionsService ] ELECTIONS: (V=15) SHIFT TO REG_LEADER. [PID:10368:010 2015.07.04 13:23:01.905 DEBUG ElectionsService ] ELECTIONS: (V=15) SENDING PROPOSAL CANDIDATE: [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}](L=36723229,W=36740334,C=36740334,E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2}), ME: [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36705667,W=36723229,C=36723229,E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2}). [PID:10368:010 2015.07.04 13:23:01.905 DEBUG ElectionsService ] ELECTIONS: (V=15) ACCEPT FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}] M=[192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]). [PID:10368:010 2015.07.04 13:23:01.929 DEBUG ElectionsService ] ELECTIONS: (V=15) ACCEPT FROM [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] M=[192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]). [PID:10368:010 2015.07.04 13:23:01.929 INFO ElectionsService ] ELECTIONS: (V=15) DONE. ELECTED MASTER = [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}](L=36723229,W=36740334,C=36740334,E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2}). ME=[192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36705667,W=36723229,C=36723229,E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2}). [PID:10368:010 2015.07.04 13:23:01.929 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] PRE-REPLICA STATE, WAITING FOR CHASER TO CATCH UP... MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:23:01.958 INFO ReplicaService ] Subscribing at LogPosition: 36723229 (0x2305A1D) to MASTER [192.168.33.12:1111, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}] as replica with SubscriptionId: {7880b7c0-b274-4f92-8065-3a330ebf00c7}, ConnectionId: {17132809-7c34-455d-80ed-920ca1e7193e}, LocalEndPoint: [], Epochs: E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740} E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65} E30@310956:{26cd0216-4e18-43aa-a899-11057cb44b2f}... . [PID:10368:025 2015.07.04 13:23:01.985 INFO TcpConnectionManager] Connection 'master-normal' ({17132809-7c34-455d-80ed-920ca1e7193e}) to [192.168.33.12:1111] established. [PID:10368:010 2015.07.04 13:23:02.094 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CATCHING UP!!! BANZAI!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:23:02.107 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] CLONE ASSIGNMENT RECEIVED FROM [192.168.33.12:1111,n/a,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:02.107 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CLONE!!! SPARTA!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:007 2015.07.04 13:23:02.109 INFO ClusterStorageWriter] === SUBSCRIBED to [192.168.33.12:1111,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] at 36723229 (0x2305A1D). SubscriptionId: {7880b7c0-b274-4f92-8065-3a330ebf00c7}. [PID:10368:015 2015.07.04 13:23:02.134 DEBUG EpochManager ] === Update Last Epoch E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} (previous epoch at 36723005). [PID:10368:010 2015.07.04 13:23:03.083 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] SLAVE ASSIGNMENT RECEIVED FROM [192.168.33.12:1111,n/a,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:03.084 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS SLAVE!!! SPARTA!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:23:04.298 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.33.13:2113, 192.168.33.13:2113] | 2015-07-04 13:22:57.493 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Unknown, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36723229/36740334/36740334/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:23:01.986 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36741515/36741688/36741688/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:04.292 New: MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.33.13:2113, 192.168.33.13:2113] | 2015-07-04 13:22:57.493 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36741515/36741688/36741688/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:04.305 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36741515/36741688/36741688/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:04.298 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:23:28.961 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.13:2113]) Old: MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.33.13:2113, 192.168.33.13:2113] | 2015-07-04 13:22:57.493 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761243/36761782/36761782/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:25.466 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:28.303 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [Unknown, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36723229/36740334/36740334/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:23:27.954 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761243/36761782/36761782/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:25.466 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:28.961 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:23:31.979 DEBUG ElectionsService ] ELECTIONS: (V=15) PREPARE FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. [PID:10368:010 2015.07.04 13:23:32.546 DEBUG ElectionsService ] ELECTIONS: (V=16) VIEWCHANGE FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. [PID:10368:010 2015.07.04 13:23:32.547 DEBUG ElectionsService ] ELECTIONS: (V=16) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:32.547 DEBUG ElectionsService ] ELECTIONS: (V=16) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:32.547 DEBUG ElectionsService ] ELECTIONS: (V=16) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:23:33.548 DEBUG ElectionsService ] ELECTIONS: (V=16) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:23:33.548 DEBUG ElectionsService ] ELECTIONS: (V=17) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:33.548 DEBUG ElectionsService ] ELECTIONS: (V=17) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:34.550 DEBUG ElectionsService ] ELECTIONS: (V=17) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:23:34.550 DEBUG ElectionsService ] ELECTIONS: (V=18) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:34.550 DEBUG ElectionsService ] ELECTIONS: (V=18) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:008 2015.07.04 13:23:34.727 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L192.168.33.11:53212, {17132809-7c34-455d-80ed-920ca1e7193e}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 65 [PID:10368:008 2015.07.04 13:23:34.742 INFO TcpConnection ] ES TcpConnection closed [13:23:34.729: N192.168.33.12:1111, L192.168.33.11:53212, {17132809-7c34-455d-80ed-920ca1e7193e}]: Received bytes: 61762, Sent bytes: 4085 Send calls: 67, callbacks: 67 Receive calls: 71, callbacks: 70 Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 65 [PID:10368:008 2015.07.04 13:23:34.746 INFO TcpConnectionManager] Connection 'master-normal' [192.168.33.12:1111, {17132809-7c34-455d-80ed-920ca1e7193e}] closed: Success. [PID:10368:010 2015.07.04 13:23:34.746 TRACE GossipServiceBase ] Looks like node [192.168.33.12:1111] is DEAD (TCP connection lost). [PID:10368:010 2015.07.04 13:23:34.747 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection lost to [192.168.33.12:1111]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [Unknown, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36723229/36740334/36740334/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:23:32.438 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:31.473 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:34.306 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [Unknown, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36723229/36740334/36740334/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:23:32.438 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:34.747 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:34.306 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:23:34.747 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {fe562867-3182-4fcd-a3e5-6bb9c60441e8}, InternalTcp: 192.168.33.12:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.12:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.12:2113, ExternalHttp: 192.168.33.12:2114]. [PID:10368:010 2015.07.04 13:23:35.249 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] PRE-REPLICA STATE, WAITING FOR CHASER TO CATCH UP... MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:23:35.249 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L192.168.33.11:53212, {17132809-7c34-455d-80ed-920ca1e7193e}] cleanly. Reason: Reconnecting from old master [192.168.33.12:1111] to new master: [192.168.33.12:1111]. [PID:10368:010 2015.07.04 13:23:35.250 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {fe562867-3182-4fcd-a3e5-6bb9c60441e8}, InternalTcp: 192.168.33.12:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.12:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.12:2113, ExternalHttp: 192.168.33.12:2114]. [PID:10368:028 2015.07.04 13:23:35.251 INFO TcpConnectionManager] Connection 'master-normal' ({9f66d78c-6f26-4db1-b969-2097b6432371}) to [192.168.33.12:1111] established. [PID:10368:010 2015.07.04 13:23:35.251 INFO ReplicaService ] Subscribing at LogPosition: 36778896 (0x2313390) to MASTER [192.168.33.12:1111, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}] as replica with SubscriptionId: {f4b5e2bd-0c9e-4385-bc05-b9381bdb2d50}, ConnectionId: {9f66d78c-6f26-4db1-b969-2097b6432371}, LocalEndPoint: [192.168.33.11:53223], Epochs: E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740} E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65}... . [PID:10368:010 2015.07.04 13:23:35.253 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection established to [192.168.33.12:1111]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [Unknown, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36723229/36740334/36740334/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:23:32.438 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:34.747 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:35.250 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [Unknown, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36723229/36740334/36740334/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:23:32.438 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:35.252 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:35.250 -------------------------------------------------------------------------------- [PID:10368:014 2015.07.04 13:23:35.266 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:013 2015.07.04 13:23:35.266 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:012 2015.07.04 13:23:35.266 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:010 2015.07.04 13:23:35.550 DEBUG ElectionsService ] ELECTIONS: (V=18) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:23:35.550 DEBUG ElectionsService ] ELECTIONS: (V=19) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:35.550 DEBUG ElectionsService ] ELECTIONS: (V=19) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:36.037 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CATCHING UP!!! BANZAI!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:007 2015.07.04 13:23:36.037 INFO ClusterStorageWriter] === SUBSCRIBED to [192.168.33.12:1111,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] at 36778896 (0x2313390). SubscriptionId: {f4b5e2bd-0c9e-4385-bc05-b9381bdb2d50}. [PID:10368:010 2015.07.04 13:23:36.552 DEBUG ElectionsService ] ELECTIONS: (V=19) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:23:36.552 DEBUG ElectionsService ] ELECTIONS: (V=20) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:36.552 DEBUG ElectionsService ] ELECTIONS: (V=20) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:37.554 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] CLONE ASSIGNMENT RECEIVED FROM [192.168.33.12:1111,n/a,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:37.554 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CLONE!!! SPARTA!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:23:37.555 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] SLAVE ASSIGNMENT RECEIVED FROM [192.168.33.12:1111,n/a,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:37.555 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS SLAVE!!! SPARTA!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:23:37.558 DEBUG ElectionsService ] ELECTIONS: (V=20) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:23:37.558 DEBUG ElectionsService ] ELECTIONS: (V=21) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:37.558 DEBUG ElectionsService ] ELECTIONS: (V=21) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:38.043 DEBUG ElectionsService ] ELECTIONS: (V=19) VIEWCHANGE FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. [PID:10368:010 2015.07.04 13:23:38.043 DEBUG ElectionsService ] ELECTIONS: (V=19) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:23:38.043 DEBUG ElectionsService ] ELECTIONS: (V=18) VIEWCHANGE FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. [PID:10368:010 2015.07.04 13:23:38.047 DEBUG ElectionsService ] ELECTIONS: (V=17) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:38.048 DEBUG ElectionsService ] ELECTIONS: (V=20) VIEWCHANGE FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. [PID:10368:010 2015.07.04 13:23:38.560 DEBUG ElectionsService ] ELECTIONS: (V=21) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:23:38.560 DEBUG ElectionsService ] ELECTIONS: (V=22) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:38.560 DEBUG ElectionsService ] ELECTIONS: (V=22) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:38.923 DEBUG ElectionsService ] ELECTIONS: (V=20) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:38.923 DEBUG ElectionsService ] ELECTIONS: (V=20) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:23:38.923 DEBUG ElectionsService ] ELECTIONS: (IV=19) VIEWCHANGEPROOF FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. JUMPING TO NON-LEADER STATE. [PID:10368:010 2015.07.04 13:23:38.923 DEBUG ElectionsService ] ELECTIONS: (V=19) SHIFT TO REG_NONLEADER. [PID:10368:010 2015.07.04 13:23:39.005 DEBUG ElectionsService ] ELECTIONS: (V=22) VIEWCHANGE FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. [PID:10368:010 2015.07.04 13:23:39.005 DEBUG ElectionsService ] ELECTIONS: (V=22) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:39.005 DEBUG ElectionsService ] ELECTIONS: (V=22) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:39.005 DEBUG ElectionsService ] ELECTIONS: (V=22) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:23:39.008 DEBUG ElectionsService ] ELECTIONS: (V=22) VIEWCHANGE FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. [PID:10368:010 2015.07.04 13:23:39.561 DEBUG ElectionsService ] ELECTIONS: (V=22) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:23:39.561 DEBUG ElectionsService ] ELECTIONS: (V=23) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:39.561 DEBUG ElectionsService ] ELECTIONS: (V=23) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:40.043 DEBUG ElectionsService ] ELECTIONS: (V=23) VIEWCHANGE FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. [PID:10368:010 2015.07.04 13:23:40.047 DEBUG ElectionsService ] ELECTIONS: (V=23) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:23:40.047 DEBUG ElectionsService ] ELECTIONS: (V=23) SHIFT TO PREPARE PHASE. [PID:10368:010 2015.07.04 13:23:40.047 DEBUG ElectionsService ] ELECTIONS: (V=23) PREPARE_OK FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36761782,W=36778896,C=36778896,E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b}). [PID:10368:010 2015.07.04 13:23:40.052 DEBUG ElectionsService ] ELECTIONS: (V=23) PREPARE_OK FROM [192.168.33.13:2113,{6bba87a7-5f19-443e-b978-c86c62e4973c}](L=36723229,W=36740334,C=36740334,E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2}). [PID:10368:010 2015.07.04 13:23:40.052 DEBUG ElectionsService ] ELECTIONS: (V=23) SHIFT TO REG_LEADER. [PID:10368:010 2015.07.04 13:23:40.054 DEBUG ElectionsService ] ELECTIONS: (V=23) SENDING PROPOSAL CANDIDATE: [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}](L=36761782,W=36778896,C=36778896,E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b}), ME: [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36761782,W=36778896,C=36778896,E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b}). [PID:10368:010 2015.07.04 13:23:40.054 DEBUG ElectionsService ] ELECTIONS: (V=23) ACCEPT FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}] M=[192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]). [PID:10368:010 2015.07.04 13:23:40.562 DEBUG ElectionsService ] ELECTIONS: (V=23) TIMED OUT! (S=Leader, M=). [PID:10368:010 2015.07.04 13:23:40.562 DEBUG ElectionsService ] ELECTIONS: (V=24) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:40.562 DEBUG ElectionsService ] ELECTIONS: (V=24) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:41.563 DEBUG ElectionsService ] ELECTIONS: (V=24) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:23:41.563 DEBUG ElectionsService ] ELECTIONS: (V=25) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:41.563 DEBUG ElectionsService ] ELECTIONS: (V=25) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:008 2015.07.04 13:23:42.460 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L192.168.33.11:53223, {9f66d78c-6f26-4db1-b969-2097b6432371}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 8 [PID:10368:008 2015.07.04 13:23:42.460 INFO TcpConnection ] ES TcpConnection closed [13:23:42.460: N192.168.33.12:1111, L192.168.33.11:53223, {9f66d78c-6f26-4db1-b969-2097b6432371}]: Received bytes: 817, Sent bytes: 1910 Send calls: 8, callbacks: 7 Receive calls: 9, callbacks: 8 Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 8 [PID:10368:008 2015.07.04 13:23:42.460 INFO TcpConnectionManager] Connection 'master-normal' [192.168.33.12:1111, {9f66d78c-6f26-4db1-b969-2097b6432371}] closed: Success. [PID:10368:010 2015.07.04 13:23:42.460 TRACE GossipServiceBase ] Looks like node [192.168.33.12:1111] is DEAD (TCP connection lost). [PID:10368:010 2015.07.04 13:23:42.460 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection lost to [192.168.33.12:1111]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [Unknown, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36723229/36740334/36740334/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:23:39.187 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:39.226 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:42.309 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [Unknown, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36723229/36740334/36740334/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:23:39.187 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:42.461 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:42.309 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:23:42.460 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {fe562867-3182-4fcd-a3e5-6bb9c60441e8}, InternalTcp: 192.168.33.12:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.12:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.12:2113, ExternalHttp: 192.168.33.12:2114]. [PID:10368:010 2015.07.04 13:23:42.565 DEBUG ElectionsService ] ELECTIONS: (V=25) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:23:42.565 DEBUG ElectionsService ] ELECTIONS: (V=26) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:42.565 DEBUG ElectionsService ] ELECTIONS: (V=26) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:42.962 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] PRE-REPLICA STATE, WAITING FOR CHASER TO CATCH UP... MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:23:42.962 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L192.168.33.11:53223, {9f66d78c-6f26-4db1-b969-2097b6432371}] cleanly. Reason: Reconnecting from old master [192.168.33.12:1111] to new master: [192.168.33.12:1111]. [PID:10368:010 2015.07.04 13:23:42.963 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {fe562867-3182-4fcd-a3e5-6bb9c60441e8}, InternalTcp: 192.168.33.12:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.12:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.12:2113, ExternalHttp: 192.168.33.12:2114]. [PID:10368:012 2015.07.04 13:23:42.963 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:013 2015.07.04 13:23:42.964 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:014 2015.07.04 13:23:42.964 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:010 2015.07.04 13:23:42.966 INFO ReplicaService ] Subscribing at LogPosition: 36778896 (0x2313390) to MASTER [192.168.33.12:1111, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}] as replica with SubscriptionId: {ef466778-0434-4931-b222-9bbef7f98117}, ConnectionId: {0025599f-4b9f-4306-b602-073ee0e3d17a}, LocalEndPoint: [], Epochs: E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740} E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65}... . [PID:10368:010 2015.07.04 13:23:43.045 DEBUG ElectionsService ] ELECTIONS: (V=24) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:43.045 DEBUG ElectionsService ] ELECTIONS: (V=24) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:23:43.045 DEBUG ElectionsService ] ELECTIONS: (V=24) VIEWCHANGE FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. [PID:10368:042 2015.07.04 13:23:43.051 INFO TcpConnectionManager] Connection 'master-normal' ({0025599f-4b9f-4306-b602-073ee0e3d17a}) to [192.168.33.12:1111] established. [PID:10368:010 2015.07.04 13:23:43.052 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection established to [192.168.33.12:1111]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [Unknown, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36723229/36740334/36740334/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:23:39.187 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:42.461 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:42.963 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [Unknown, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36723229/36740334/36740334/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:23:39.187 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:43.051 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:42.963 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:23:43.054 DEBUG ElectionsService ] ELECTIONS: (V=25) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:43.054 DEBUG ElectionsService ] ELECTIONS: (V=25) VIEWCHANGE FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. [PID:10368:007 2015.07.04 13:23:43.054 INFO ClusterStorageWriter] === SUBSCRIBED to [192.168.33.12:1111,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] at 36778896 (0x2313390). SubscriptionId: {ef466778-0434-4931-b222-9bbef7f98117}. [PID:10368:010 2015.07.04 13:23:43.055 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CATCHING UP!!! BANZAI!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:23:43.056 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] CLONE ASSIGNMENT RECEIVED FROM [192.168.33.12:1111,n/a,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:43.056 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CLONE!!! SPARTA!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:23:43.056 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] SLAVE ASSIGNMENT RECEIVED FROM [192.168.33.12:1111,n/a,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:43.057 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS SLAVE!!! SPARTA!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:23:43.566 DEBUG ElectionsService ] ELECTIONS: (V=26) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:23:43.566 DEBUG ElectionsService ] ELECTIONS: (V=27) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:43.566 DEBUG ElectionsService ] ELECTIONS: (V=27) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:43.920 DEBUG ElectionsService ] ELECTIONS: (V=27) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:43.920 DEBUG ElectionsService ] ELECTIONS: (V=27) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:23:44.547 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.13:2113]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [Unknown, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36723229/36740334/36740334/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:23:39.187 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:43.051 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:44.309 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36723229/36740334/36740334/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:23:44.510 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:44.547 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:44.547 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:23:44.567 DEBUG ElectionsService ] ELECTIONS: (V=27) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:23:44.567 DEBUG ElectionsService ] ELECTIONS: (V=28) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:44.567 DEBUG ElectionsService ] ELECTIONS: (V=28) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:45.003 DEBUG ElectionsService ] ELECTIONS: (V=27) VIEWCHANGE FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. [PID:10368:010 2015.07.04 13:23:45.003 DEBUG ElectionsService ] ELECTIONS: (V=27) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:23:45.014 DEBUG ElectionsService ] ELECTIONS: (V=28) VIEWCHANGE FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. [PID:10368:010 2015.07.04 13:23:45.014 DEBUG ElectionsService ] ELECTIONS: (V=26) VIEWCHANGE FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. [PID:10368:010 2015.07.04 13:23:45.014 DEBUG ElectionsService ] ELECTIONS: (IV=25) VIEWCHANGEPROOF FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. JUMPING TO NON-LEADER STATE. [PID:10368:010 2015.07.04 13:23:45.014 DEBUG ElectionsService ] ELECTIONS: (V=25) SHIFT TO REG_NONLEADER. [PID:10368:010 2015.07.04 13:23:46.006 DEBUG ElectionsService ] ELECTIONS: (V=29) VIEWCHANGE FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. [PID:10368:010 2015.07.04 13:23:46.007 DEBUG ElectionsService ] ELECTIONS: (V=29) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:46.007 DEBUG ElectionsService ] ELECTIONS: (V=29) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:46.007 DEBUG ElectionsService ] ELECTIONS: (V=29) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:23:46.007 DEBUG ElectionsService ] ELECTIONS: (V=29) SHIFT TO PREPARE PHASE. [PID:10368:010 2015.07.04 13:23:46.007 DEBUG ElectionsService ] ELECTIONS: (V=29) PREPARE_OK FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36761782,W=36778896,C=36778896,E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b}). [PID:10368:010 2015.07.04 13:23:47.009 DEBUG ElectionsService ] ELECTIONS: (V=29) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:23:47.009 DEBUG ElectionsService ] ELECTIONS: (V=30) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:47.009 DEBUG ElectionsService ] ELECTIONS: (V=30) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:008 2015.07.04 13:23:47.156 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L192.168.33.11:53227, {0025599f-4b9f-4306-b602-073ee0e3d17a}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 9 [PID:10368:008 2015.07.04 13:23:47.156 INFO TcpConnection ] ES TcpConnection closed [13:23:47.157: N192.168.33.12:1111, L192.168.33.11:53227, {0025599f-4b9f-4306-b602-073ee0e3d17a}]: Received bytes: 795, Sent bytes: 1888 Send calls: 9, callbacks: 8 Receive calls: 10, callbacks: 9 Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 9 [PID:10368:008 2015.07.04 13:23:47.156 INFO TcpConnectionManager] Connection 'master-normal' [192.168.33.12:1111, {0025599f-4b9f-4306-b602-073ee0e3d17a}] closed: Success. [PID:10368:010 2015.07.04 13:23:47.158 TRACE GossipServiceBase ] Looks like node [192.168.33.12:1111] is DEAD (TCP connection lost). [PID:10368:010 2015.07.04 13:23:47.158 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection lost to [192.168.33.12:1111]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36723229/36740334/36740334/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:23:45.189 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:45.002 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:46.309 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36723229/36740334/36740334/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:23:45.189 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:47.158 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:46.309 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:23:47.158 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {fe562867-3182-4fcd-a3e5-6bb9c60441e8}, InternalTcp: 192.168.33.12:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.12:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.12:2113, ExternalHttp: 192.168.33.12:2114]. [PID:10368:010 2015.07.04 13:23:47.659 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] PRE-REPLICA STATE, WAITING FOR CHASER TO CATCH UP... MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:23:47.659 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L192.168.33.11:53227, {0025599f-4b9f-4306-b602-073ee0e3d17a}] cleanly. Reason: Reconnecting from old master [192.168.33.12:1111] to new master: [192.168.33.12:1111]. [PID:10368:014 2015.07.04 13:23:47.660 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:013 2015.07.04 13:23:47.660 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:010 2015.07.04 13:23:47.660 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {fe562867-3182-4fcd-a3e5-6bb9c60441e8}, InternalTcp: 192.168.33.12:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.12:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.12:2113, ExternalHttp: 192.168.33.12:2114]. [PID:10368:012 2015.07.04 13:23:47.660 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:010 2015.07.04 13:23:47.662 INFO ReplicaService ] Subscribing at LogPosition: 36778896 (0x2313390) to MASTER [192.168.33.12:1111, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}] as replica with SubscriptionId: {f57b8857-a390-49cb-8401-360d705d5e0d}, ConnectionId: {306248a8-424c-43a9-996c-6188041eaf9e}, LocalEndPoint: [], Epochs: E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740} E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65}... . [PID:10368:010 2015.07.04 13:23:48.011 DEBUG ElectionsService ] ELECTIONS: (V=30) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:23:48.011 DEBUG ElectionsService ] ELECTIONS: (V=31) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:48.011 DEBUG ElectionsService ] ELECTIONS: (V=31) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:043 2015.07.04 13:23:48.061 INFO TcpConnectionManager] Connection 'master-normal' ({306248a8-424c-43a9-996c-6188041eaf9e}) to [192.168.33.12:1111] established. [PID:10368:010 2015.07.04 13:23:48.062 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection established to [192.168.33.12:1111]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36723229/36740334/36740334/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:23:45.189 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:47.158 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:47.660 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36723229/36740334/36740334/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:23:45.189 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:48.061 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:47.660 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:23:48.062 DEBUG ElectionsService ] ELECTIONS: (V=30) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:48.062 DEBUG ElectionsService ] ELECTIONS: (V=30) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:23:48.062 DEBUG ElectionsService ] ELECTIONS: (V=31) SHIFT TO PREPARE PHASE. [PID:10368:010 2015.07.04 13:23:48.062 DEBUG ElectionsService ] ELECTIONS: (V=31) PREPARE_OK FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36761782,W=36778896,C=36778896,E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b}). [PID:10368:010 2015.07.04 13:23:48.563 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CATCHING UP!!! BANZAI!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:007 2015.07.04 13:23:48.564 INFO ClusterStorageWriter] === SUBSCRIBED to [192.168.33.12:1111,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] at 36778896 (0x2313390). SubscriptionId: {f57b8857-a390-49cb-8401-360d705d5e0d}. [PID:10368:010 2015.07.04 13:23:48.566 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] CLONE ASSIGNMENT RECEIVED FROM [192.168.33.12:1111,n/a,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:48.566 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CLONE!!! SPARTA!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:23:48.567 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] SLAVE ASSIGNMENT RECEIVED FROM [192.168.33.12:1111,n/a,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:48.567 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS SLAVE!!! SPARTA!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:23:49.012 DEBUG ElectionsService ] ELECTIONS: (V=31) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:23:49.012 DEBUG ElectionsService ] ELECTIONS: (V=32) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:49.012 DEBUG ElectionsService ] ELECTIONS: (V=32) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:49.274 DEBUG ElectionsService ] ELECTIONS: (IV=32) VIEWCHANGEPROOF FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. JUMPING TO LEADER STATE. [PID:10368:010 2015.07.04 13:23:49.274 DEBUG ElectionsService ] ELECTIONS: (V=32) SHIFT TO PREPARE PHASE. [PID:10368:010 2015.07.04 13:23:49.274 DEBUG ElectionsService ] ELECTIONS: (V=32) PREPARE_OK FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36761782,W=36778896,C=36778896,E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b}). [PID:10368:010 2015.07.04 13:23:49.573 DEBUG ElectionsService ] ELECTIONS: (V=32) PREPARE_OK FROM [192.168.33.13:2113,{6bba87a7-5f19-443e-b978-c86c62e4973c}](L=36723229,W=36740334,C=36740334,E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2}). [PID:10368:010 2015.07.04 13:23:49.573 DEBUG ElectionsService ] ELECTIONS: (V=32) SHIFT TO REG_LEADER. [PID:10368:010 2015.07.04 13:23:49.573 DEBUG ElectionsService ] ELECTIONS: (V=32) SENDING PROPOSAL CANDIDATE: [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}](L=36761782,W=36778896,C=36778896,E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b}), ME: [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36761782,W=36778896,C=36778896,E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b}). [PID:10368:010 2015.07.04 13:23:49.573 DEBUG ElectionsService ] ELECTIONS: (V=32) ACCEPT FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}] M=[192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]). [PID:10368:010 2015.07.04 13:23:50.002 DEBUG ElectionsService ] ELECTIONS: (V=32) ACCEPT FROM [192.168.33.13:2113,{6bba87a7-5f19-443e-b978-c86c62e4973c}] M=[192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]). [PID:10368:010 2015.07.04 13:23:50.002 INFO ElectionsService ] ELECTIONS: (V=32) DONE. ELECTED MASTER = [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}](L=36761782,W=36778896,C=36778896,E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b}). ME=[192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36761782,W=36778896,C=36778896,E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b}). [PID:10368:010 2015.07.04 13:23:51.439 DEBUG ElectionsService ] ELECTIONS: (V=33) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:51.439 DEBUG ElectionsService ] ELECTIONS: (V=33) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:51.439 DEBUG ElectionsService ] ELECTIONS: (V=33) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:51.439 DEBUG ElectionsService ] ELECTIONS: (V=33) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:23:51.439 DEBUG ElectionsService ] ELECTIONS: (V=34) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:51.439 DEBUG ElectionsService ] ELECTIONS: (V=34) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:51.439 DEBUG ElectionsService ] ELECTIONS: (V=34) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:51.439 DEBUG ElectionsService ] ELECTIONS: (V=34) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:23:52.440 DEBUG ElectionsService ] ELECTIONS: (V=34) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:23:52.441 DEBUG ElectionsService ] ELECTIONS: (V=35) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:52.441 DEBUG ElectionsService ] ELECTIONS: (V=35) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:53.440 DEBUG ElectionsService ] ELECTIONS: (V=35) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:53.440 DEBUG ElectionsService ] ELECTIONS: (V=35) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:23:53.440 DEBUG ElectionsService ] ELECTIONS: (V=35) SHIFT TO PREPARE PHASE. [PID:10368:010 2015.07.04 13:23:53.441 DEBUG ElectionsService ] ELECTIONS: (V=35) PREPARE_OK FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36761782,W=36778896,C=36778896,E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b}). [PID:10368:010 2015.07.04 13:23:53.444 DEBUG ElectionsService ] ELECTIONS: (V=35) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:23:53.444 DEBUG ElectionsService ] ELECTIONS: (V=36) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:53.444 DEBUG ElectionsService ] ELECTIONS: (V=36) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:54.446 DEBUG ElectionsService ] ELECTIONS: (V=36) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:23:54.446 DEBUG ElectionsService ] ELECTIONS: (V=37) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:54.446 DEBUG ElectionsService ] ELECTIONS: (V=37) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:008 2015.07.04 13:23:54.568 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L192.168.33.11:53236, {306248a8-424c-43a9-996c-6188041eaf9e}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 9 [PID:10368:008 2015.07.04 13:23:54.568 INFO TcpConnection ] ES TcpConnection closed [13:23:54.568: N192.168.33.12:1111, L192.168.33.11:53236, {306248a8-424c-43a9-996c-6188041eaf9e}]: Received bytes: 817, Sent bytes: 1932 Send calls: 8, callbacks: 8 Receive calls: 10, callbacks: 9 Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 9 [PID:10368:008 2015.07.04 13:23:54.568 INFO TcpConnectionManager] Connection 'master-normal' [192.168.33.12:1111, {306248a8-424c-43a9-996c-6188041eaf9e}] closed: Success. [PID:10368:010 2015.07.04 13:23:54.568 TRACE GossipServiceBase ] Looks like node [192.168.33.12:1111] is DEAD (TCP connection lost). [PID:10368:010 2015.07.04 13:23:54.570 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection lost to [192.168.33.12:1111]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36723229/36740334/36740334/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:23:51.191 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:49.481 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:54.314 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36723229/36740334/36740334/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:23:51.191 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:54.569 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:54.314 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:23:54.570 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {fe562867-3182-4fcd-a3e5-6bb9c60441e8}, InternalTcp: 192.168.33.12:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.12:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.12:2113, ExternalHttp: 192.168.33.12:2114]. [PID:10368:010 2015.07.04 13:23:54.579 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36723229/36740334/36740334/E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} | 2015-07-04 13:23:51.191 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:54.569 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:54.314 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [Clone, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:54.490 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:54.491 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:54.578 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:23:54.579 DEBUG ElectionsService ] ELECTIONS: (V=37) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:54.579 DEBUG ElectionsService ] ELECTIONS: (V=37) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:23:54.579 DEBUG ElectionsService ] ELECTIONS: (V=37) SHIFT TO PREPARE PHASE. [PID:10368:010 2015.07.04 13:23:54.579 DEBUG ElectionsService ] ELECTIONS: (V=37) PREPARE_OK FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36761782,W=36778896,C=36778896,E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b}). [PID:10368:010 2015.07.04 13:23:55.070 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] PRE-REPLICA STATE, WAITING FOR CHASER TO CATCH UP... MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:23:55.070 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L192.168.33.11:53236, {306248a8-424c-43a9-996c-6188041eaf9e}] cleanly. Reason: Reconnecting from old master [192.168.33.12:1111] to new master: [192.168.33.12:1111]. [PID:10368:012 2015.07.04 13:23:55.072 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:013 2015.07.04 13:23:55.072 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:014 2015.07.04 13:23:55.072 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:010 2015.07.04 13:23:55.072 INFO ReplicaService ] Subscribing at LogPosition: 36778896 (0x2313390) to MASTER [192.168.33.12:1111, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}] as replica with SubscriptionId: {76454c72-03f6-43c9-8e67-d4830af64194}, ConnectionId: {d0d36984-f0fc-4f24-bd86-6a103b0000d3}, LocalEndPoint: [], Epochs: E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740} E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65}... . [PID:10368:055 2015.07.04 13:23:55.076 INFO TcpConnectionManager] Connection 'master-normal' ({d0d36984-f0fc-4f24-bd86-6a103b0000d3}) to [192.168.33.12:1111] established. [PID:10368:010 2015.07.04 13:23:55.077 DEBUG ElectionsService ] ELECTIONS: (V=38) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:55.077 DEBUG ElectionsService ] ELECTIONS: (V=38) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:55.077 DEBUG ElectionsService ] ELECTIONS: (V=38) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:55.077 DEBUG ElectionsService ] ELECTIONS: (V=38) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:23:55.077 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CATCHING UP!!! BANZAI!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:007 2015.07.04 13:23:55.077 INFO ClusterStorageWriter] === SUBSCRIBED to [192.168.33.12:1111,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] at 36778896 (0x2313390). SubscriptionId: {76454c72-03f6-43c9-8e67-d4830af64194}. [PID:10368:010 2015.07.04 13:23:55.078 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] CLONE ASSIGNMENT RECEIVED FROM [192.168.33.12:1111,n/a,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:55.078 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CLONE!!! SPARTA!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:23:55.078 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] SLAVE ASSIGNMENT RECEIVED FROM [192.168.33.12:1111,n/a,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:55.078 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS SLAVE!!! SPARTA!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:23:55.479 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [Clone, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:54.490 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:55.076 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:55.079 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [Clone, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:55.035 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:55.484 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:55.479 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:23:55.938 DEBUG ElectionsService ] ELECTIONS: (V=39) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:55.938 DEBUG ElectionsService ] ELECTIONS: (V=39) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:55.938 DEBUG ElectionsService ] ELECTIONS: (V=39) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:55.938 DEBUG ElectionsService ] ELECTIONS: (V=39) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:23:56.939 DEBUG ElectionsService ] ELECTIONS: (V=39) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:23:56.939 DEBUG ElectionsService ] ELECTIONS: (V=40) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:56.939 DEBUG ElectionsService ] ELECTIONS: (V=40) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:008 2015.07.04 13:23:57.478 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L192.168.33.11:53242, {d0d36984-f0fc-4f24-bd86-6a103b0000d3}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 6 [PID:10368:008 2015.07.04 13:23:57.478 INFO TcpConnection ] ES TcpConnection closed [13:23:57.478: N192.168.33.12:1111, L192.168.33.11:53242, {d0d36984-f0fc-4f24-bd86-6a103b0000d3}]: Received bytes: 773, Sent bytes: 1866 Send calls: 8, callbacks: 7 Receive calls: 7, callbacks: 6 Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 6 [PID:10368:008 2015.07.04 13:23:57.478 INFO TcpConnectionManager] Connection 'master-normal' [192.168.33.12:1111, {d0d36984-f0fc-4f24-bd86-6a103b0000d3}] closed: Success. [PID:10368:010 2015.07.04 13:23:57.478 TRACE GossipServiceBase ] Looks like node [192.168.33.12:1111] is DEAD (TCP connection lost). [PID:10368:010 2015.07.04 13:23:57.478 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection lost to [192.168.33.12:1111]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [Clone, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:55.035 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:55.484 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:56.314 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [Clone, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:55.035 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:57.479 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:56.314 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:23:57.478 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {fe562867-3182-4fcd-a3e5-6bb9c60441e8}, InternalTcp: 192.168.33.12:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.12:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.12:2113, ExternalHttp: 192.168.33.12:2114]. [PID:10368:010 2015.07.04 13:23:57.947 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.13:2113]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [Clone, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:55.035 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:57.479 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:56.314 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:55.471 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:57.479 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:57.947 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:23:57.947 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {fe562867-3182-4fcd-a3e5-6bb9c60441e8}, InternalTcp: 192.168.33.12:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.12:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.12:2113, ExternalHttp: 192.168.33.12:2114]. [PID:10368:010 2015.07.04 13:23:57.948 DEBUG ElectionsService ] ELECTIONS: (V=40) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:23:57.948 DEBUG ElectionsService ] ELECTIONS: (V=41) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:57.948 DEBUG ElectionsService ] ELECTIONS: (V=41) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:57.948 DEBUG ElectionsService ] ELECTIONS: (V=40) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:57.948 DEBUG ElectionsService ] ELECTIONS: (V=40) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:23:57.948 DEBUG ElectionsService ] ELECTIONS: (V=41) SHIFT TO PREPARE PHASE. [PID:10368:010 2015.07.04 13:23:57.948 DEBUG ElectionsService ] ELECTIONS: (V=41) PREPARE_OK FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36761782,W=36778896,C=36778896,E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b}). [PID:10368:010 2015.07.04 13:23:57.980 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] PRE-REPLICA STATE, WAITING FOR CHASER TO CATCH UP... MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:23:57.980 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L192.168.33.11:53242, {d0d36984-f0fc-4f24-bd86-6a103b0000d3}] cleanly. Reason: Reconnecting from old master [192.168.33.12:1111] to new master: [192.168.33.12:1111]. [PID:10368:010 2015.07.04 13:23:57.980 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {fe562867-3182-4fcd-a3e5-6bb9c60441e8}, InternalTcp: 192.168.33.12:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.12:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.12:2113, ExternalHttp: 192.168.33.12:2114]. [PID:10368:012 2015.07.04 13:23:57.980 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:013 2015.07.04 13:23:57.980 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:014 2015.07.04 13:23:57.980 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:010 2015.07.04 13:23:57.982 INFO ReplicaService ] Subscribing at LogPosition: 36778896 (0x2313390) to MASTER [192.168.33.12:1111, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}] as replica with SubscriptionId: {f9552a6e-b334-4863-96c2-13c6ec73fd51}, ConnectionId: {abfbb023-5aaa-46ea-8bd7-bac362dffbd1}, LocalEndPoint: [], Epochs: E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740} E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65}... . [PID:10368:059 2015.07.04 13:23:57.982 INFO TcpConnectionManager] Connection 'master-normal' ({abfbb023-5aaa-46ea-8bd7-bac362dffbd1}) to [192.168.33.12:1111] established. [PID:10368:010 2015.07.04 13:23:57.983 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection established to [192.168.33.12:1111]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:55.471 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:57.479 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:57.980 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:55.471 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:57.982 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:57.980 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:23:58.950 DEBUG ElectionsService ] ELECTIONS: (V=41) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:23:58.950 DEBUG ElectionsService ] ELECTIONS: (V=42) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:23:58.950 DEBUG ElectionsService ] ELECTIONS: (V=42) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:23:58.983 INFO ReplicaService ] Subscribing at LogPosition: 36778896 (0x2313390) to MASTER [192.168.33.12:1111, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}] as replica with SubscriptionId: {78e91b51-421e-49aa-835e-9766fa4c1c81}, ConnectionId: {abfbb023-5aaa-46ea-8bd7-bac362dffbd1}, LocalEndPoint: [192.168.33.11:53246], Epochs: E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740} E31@311176:{712b1924-5a5b-48c9-a3cc-41e2a4c35c65}... . [PID:10368:062 2015.07.04 13:23:59.099 ERROR GossipController ] Received as POST invalid ClusterInfo from [http://192.168.33.11:2113/gossip]. Content-Type: application/json, Body: . [PID:10368:010 2015.07.04 13:23:59.450 TRACE ClusterVNodeControll] Ignoring ReplicaSubscribed because SubscriptionId {f9552a6e-b334-4863-96c2-13c6ec73fd51} is wrong. Current SubscriptionId is {78e91b51-421e-49aa-835e-9766fa4c1c81}. [PID:10368:010 2015.07.04 13:23:59.452 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:55.471 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:57.982 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:58.315 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [Clone, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:58.954 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36778896/36795971/36795971/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:59.458 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:59.451 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:23:59.452 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CATCHING UP!!! BANZAI!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:007 2015.07.04 13:23:59.452 INFO ClusterStorageWriter] === SUBSCRIBED to [192.168.33.12:1111,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] at 36778896 (0x2313390). SubscriptionId: {78e91b51-421e-49aa-835e-9766fa4c1c81}. [PID:10368:010 2015.07.04 13:23:59.482 DEBUG ElectionsService ] ELECTIONS: (V=42) PREPARE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:59.483 DEBUG ElectionsService ] ELECTIONS: (V=42) SHIFT TO REG_NONLEADER. [PID:10368:010 2015.07.04 13:23:59.483 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] CLONE ASSIGNMENT RECEIVED FROM [192.168.33.12:1111,n/a,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:59.483 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CLONE!!! SPARTA!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:23:59.493 DEBUG ElectionsService ] ELECTIONS: (V=42) PROPOSAL FROM [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] M=[192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}](L=36778896,W=36795971,C=36795971,E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b}). ME=[192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36778896,W=36795971,C=36795971,E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b}). [PID:10368:010 2015.07.04 13:23:59.493 DEBUG ElectionsService ] ELECTIONS: (V=42) ACCEPT FROM [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] M=[192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]). [PID:10368:010 2015.07.04 13:23:59.493 DEBUG ElectionsService ] ELECTIONS: (V=42) ACCEPT FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}] M=[192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]). [PID:10368:010 2015.07.04 13:23:59.493 INFO ElectionsService ] ELECTIONS: (V=42) DONE. ELECTED MASTER = [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}](L=36778896,W=36795971,C=36795971,E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b}). ME=[192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36778896,W=36795971,C=36795971,E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b}). [PID:10368:015 2015.07.04 13:23:59.505 DEBUG EpochManager ] === Update Last Epoch E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} (previous epoch at 36740334). [PID:10368:010 2015.07.04 13:23:59.524 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] SLAVE ASSIGNMENT RECEIVED FROM [192.168.33.12:1111,n/a,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:23:59.524 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS SLAVE!!! SPARTA!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:23:59.533 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.13:2113]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [Clone, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:58.954 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36778896/36795971/36795971/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:59.458 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:23:59.525 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:59.488 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36778896/36795971/36795971/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:59.458 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:23:59.533 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:24:00.021 DEBUG ElectionsService ] ELECTIONS: (V=42) PROPOSAL FROM [192.168.33.13:2113,{6bba87a7-5f19-443e-b978-c86c62e4973c}] M=[192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}](L=36778896,W=36796195,C=36796195,E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd}). ME=[192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36778896,W=36796195,C=36796195,E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd}). [PID:10368:010 2015.07.04 13:24:00.021 DEBUG ElectionsService ] ELECTIONS: (V=42) ACCEPT FROM [192.168.33.13:2113,{6bba87a7-5f19-443e-b978-c86c62e4973c}] M=[192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]). [PID:10368:010 2015.07.04 13:24:00.021 DEBUG ElectionsService ] ELECTIONS: (V=42) ACCEPT FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}] M=[192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]). [PID:10368:010 2015.07.04 13:24:00.084 DEBUG ElectionsService ] ELECTIONS: (V=43) VIEWCHANGE FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. [PID:10368:010 2015.07.04 13:24:00.084 DEBUG ElectionsService ] ELECTIONS: (V=43) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:24:00.084 DEBUG ElectionsService ] ELECTIONS: (V=43) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:24:00.084 DEBUG ElectionsService ] ELECTIONS: (V=43) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:24:01.086 DEBUG ElectionsService ] ELECTIONS: (V=43) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:24:01.086 DEBUG ElectionsService ] ELECTIONS: (V=44) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:24:01.086 DEBUG ElectionsService ] ELECTIONS: (V=44) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:008 2015.07.04 13:24:01.385 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L192.168.33.11:53246, {abfbb023-5aaa-46ea-8bd7-bac362dffbd1}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 13 [PID:10368:008 2015.07.04 13:24:01.385 INFO TcpConnection ] ES TcpConnection closed [13:24:01.385: N192.168.33.12:1111, L192.168.33.11:53246, {abfbb023-5aaa-46ea-8bd7-bac362dffbd1}]: Received bytes: 19345, Sent bytes: 2531 Send calls: 12, callbacks: 12 Receive calls: 14, callbacks: 13 Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 13 [PID:10368:008 2015.07.04 13:24:01.385 INFO TcpConnectionManager] Connection 'master-normal' [192.168.33.12:1111, {abfbb023-5aaa-46ea-8bd7-bac362dffbd1}] closed: Success. [PID:10368:010 2015.07.04 13:24:01.386 TRACE GossipServiceBase ] Looks like node [192.168.33.12:1111] is DEAD (TCP connection lost). [PID:10368:010 2015.07.04 13:24:01.386 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection lost to [192.168.33.12:1111]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:57.195 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36778896/36795971/36795971/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:59.458 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:00.316 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:57.195 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36778896/36795971/36795971/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:24:01.387 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:00.316 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:24:01.386 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {fe562867-3182-4fcd-a3e5-6bb9c60441e8}, InternalTcp: 192.168.33.12:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.12:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.12:2113, ExternalHttp: 192.168.33.12:2114]. [PID:10368:010 2015.07.04 13:24:01.888 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] PRE-REPLICA STATE, WAITING FOR CHASER TO CATCH UP... MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:24:01.888 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L192.168.33.11:53246, {abfbb023-5aaa-46ea-8bd7-bac362dffbd1}] cleanly. Reason: Reconnecting from old master [192.168.33.12:1111] to new master: [192.168.33.12:1111]. [PID:10368:010 2015.07.04 13:24:01.888 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {fe562867-3182-4fcd-a3e5-6bb9c60441e8}, InternalTcp: 192.168.33.12:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.12:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.12:2113, ExternalHttp: 192.168.33.12:2114]. [PID:10368:012 2015.07.04 13:24:01.889 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:013 2015.07.04 13:24:01.889 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:014 2015.07.04 13:24:01.889 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:010 2015.07.04 13:24:01.890 INFO ReplicaService ] Subscribing at LogPosition: 36796195 (0x2317723) to MASTER [192.168.33.12:1111, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}] as replica with SubscriptionId: {a3a2accb-7b92-4e98-991f-eb11f97090a3}, ConnectionId: {07eece71-daef-4d44-be9e-af2b8c596410}, LocalEndPoint: [], Epochs: E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740}... . [PID:10368:010 2015.07.04 13:24:02.087 DEBUG ElectionsService ] ELECTIONS: (V=44) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:24:02.087 DEBUG ElectionsService ] ELECTIONS: (V=45) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:24:02.087 DEBUG ElectionsService ] ELECTIONS: (V=45) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:055 2015.07.04 13:24:02.101 INFO TcpConnectionManager] Connection 'master-normal' ({07eece71-daef-4d44-be9e-af2b8c596410}) to [192.168.33.12:1111] established. [PID:10368:010 2015.07.04 13:24:02.101 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection established to [192.168.33.12:1111]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:57.195 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36778896/36795971/36795971/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:24:01.387 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:01.889 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:57.195 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36778896/36795971/36795971/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:24:02.102 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:01.889 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:24:02.101 DEBUG ElectionsService ] ELECTIONS: (V=44) VIEWCHANGE FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. [PID:10368:010 2015.07.04 13:24:02.101 DEBUG ElectionsService ] ELECTIONS: (V=44) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:24:02.892 INFO ReplicaService ] Subscribing at LogPosition: 36796195 (0x2317723) to MASTER [192.168.33.12:1111, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}] as replica with SubscriptionId: {4622cbf1-e521-4297-8586-33fb5ed17718}, ConnectionId: {07eece71-daef-4d44-be9e-af2b8c596410}, LocalEndPoint: [192.168.33.11:53249], Epochs: E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740}... . [PID:10368:010 2015.07.04 13:24:03.088 DEBUG ElectionsService ] ELECTIONS: (V=45) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:24:03.088 DEBUG ElectionsService ] ELECTIONS: (V=46) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:24:03.088 DEBUG ElectionsService ] ELECTIONS: (V=46) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:24:03.586 TRACE GossipServiceBase ] Looks like node [192.168.33.13:2113] is DEAD (Gossip send failed). [PID:10368:010 2015.07.04 13:24:03.586 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip send failed to [192.168.33.13:2113]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:23:57.195 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36778896/36795971/36795971/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:24:02.102 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:02.318 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:24:03.586 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36778896/36795971/36795971/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:24:02.102 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:02.318 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:24:03.586 DEBUG ElectionsService ] ELECTIONS: (V=45) VIEWCHANGE FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. [PID:10368:010 2015.07.04 13:24:03.586 DEBUG ElectionsService ] ELECTIONS: (V=45) MAJORITY OF VIEWCHANGE. [PID:10368:008 2015.07.04 13:24:03.803 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L192.168.33.11:53249, {07eece71-daef-4d44-be9e-af2b8c596410}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 0 [PID:10368:008 2015.07.04 13:24:03.803 INFO TcpConnection ] ES TcpConnection closed [13:24:03.803: N192.168.33.12:1111, L192.168.33.11:53249, {07eece71-daef-4d44-be9e-af2b8c596410}]: Received bytes: 0, Sent bytes: 622 Send calls: 2, callbacks: 1 Receive calls: 1, callbacks: 0 Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 0 [PID:10368:008 2015.07.04 13:24:03.803 INFO TcpConnectionManager] Connection 'master-normal' [192.168.33.12:1111, {07eece71-daef-4d44-be9e-af2b8c596410}] closed: Success. [PID:10368:010 2015.07.04 13:24:03.804 TRACE GossipServiceBase ] Looks like node [192.168.33.12:1111] is DEAD (TCP connection lost). [PID:10368:010 2015.07.04 13:24:03.805 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection lost to [192.168.33.12:1111]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:24:03.586 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36778896/36795971/36795971/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:24:02.102 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:02.318 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:24:03.586 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36778896/36795971/36795971/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:24:03.805 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:02.318 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:24:03.805 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {fe562867-3182-4fcd-a3e5-6bb9c60441e8}, InternalTcp: 192.168.33.12:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.12:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.12:2113, ExternalHttp: 192.168.33.12:2114]. [PID:10368:010 2015.07.04 13:24:03.897 INFO ReplicaService ] Subscribing at LogPosition: 36796195 (0x2317723) to MASTER [192.168.33.12:1111, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}] as replica with SubscriptionId: {606a4a9f-c86a-4937-8609-90488644cf11}, ConnectionId: {07eece71-daef-4d44-be9e-af2b8c596410}, LocalEndPoint: [192.168.33.11:53249], Epochs: E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740}... . [PID:10368:010 2015.07.04 13:24:04.090 DEBUG ElectionsService ] ELECTIONS: (V=46) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:24:04.091 DEBUG ElectionsService ] ELECTIONS: (V=47) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:24:04.091 DEBUG ElectionsService ] ELECTIONS: (V=47) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:24:04.306 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L192.168.33.11:53249, {07eece71-daef-4d44-be9e-af2b8c596410}] cleanly. Reason: Reconnecting from old master [192.168.33.12:1111] to new master: [192.168.33.12:1111]. [PID:10368:068 2015.07.04 13:24:04.307 INFO TcpConnectionManager] Connection 'master-normal' ({206ecac0-8062-4738-9c4b-98c22ca4532f}) to [192.168.33.12:1111] established. [PID:10368:010 2015.07.04 13:24:04.308 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection established to [192.168.33.12:1111]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:24:03.586 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36778896/36795971/36795971/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:24:03.805 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:02.318 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:24:03.586 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36778896/36795971/36795971/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:24:04.308 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:02.318 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:24:04.456 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36761782/36778896/36778896/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:24:03.586 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36778896/36795971/36795971/E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} | 2015-07-04 13:24:04.308 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:04.318 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [Clone, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:01.197 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:04.464 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:04.456 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:24:04.463 DEBUG ElectionsService ] ELECTIONS: (IV=45) VIEWCHANGEPROOF FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. JUMPING TO NON-LEADER STATE. [PID:10368:010 2015.07.04 13:24:04.463 DEBUG ElectionsService ] ELECTIONS: (V=45) SHIFT TO REG_NONLEADER. [PID:10368:010 2015.07.04 13:24:04.463 DEBUG ElectionsService ] ELECTIONS: (V=47) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:24:04.463 DEBUG ElectionsService ] ELECTIONS: (V=47) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:24:04.463 DEBUG ElectionsService ] ELECTIONS: (V=47) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:24:04.463 DEBUG ElectionsService ] ELECTIONS: (V=47) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:24:04.463 DEBUG ElectionsService ] ELECTIONS: (V=47) SHIFT TO PREPARE PHASE. [PID:10368:010 2015.07.04 13:24:04.463 DEBUG ElectionsService ] ELECTIONS: (V=47) PREPARE_OK FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36778896,W=36796195,C=36796195,E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd}). [PID:10368:010 2015.07.04 13:24:04.899 INFO ReplicaService ] Subscribing at LogPosition: 36796195 (0x2317723) to MASTER [192.168.33.12:1111, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}] as replica with SubscriptionId: {24d31d7d-7172-48fa-8f67-f710c2d8a9ce}, ConnectionId: {206ecac0-8062-4738-9c4b-98c22ca4532f}, LocalEndPoint: [192.168.33.11:53251], Epochs: E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740}... . [PID:10368:010 2015.07.04 13:24:05.093 DEBUG ElectionsService ] ELECTIONS: (V=47) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:24:05.093 DEBUG ElectionsService ] ELECTIONS: (V=48) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:24:05.093 DEBUG ElectionsService ] ELECTIONS: (V=48) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:24:05.900 INFO ReplicaService ] Subscribing at LogPosition: 36796195 (0x2317723) to MASTER [192.168.33.12:1111, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}] as replica with SubscriptionId: {a2bef483-749d-4a9e-ad35-c3b660e22c2f}, ConnectionId: {206ecac0-8062-4738-9c4b-98c22ca4532f}, LocalEndPoint: [192.168.33.11:53251], Epochs: E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740}... . [PID:10368:010 2015.07.04 13:24:06.095 DEBUG ElectionsService ] ELECTIONS: (V=48) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:24:06.095 DEBUG ElectionsService ] ELECTIONS: (V=49) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:24:06.095 DEBUG ElectionsService ] ELECTIONS: (V=49) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:24:06.465 DEBUG ElectionsService ] ELECTIONS: (V=48) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:24:06.465 DEBUG ElectionsService ] ELECTIONS: (V=48) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:24:06.465 TRACE ClusterVNodeControll] Ignoring ReplicaSubscribed because SubscriptionId {24d31d7d-7172-48fa-8f67-f710c2d8a9ce} is wrong. Current SubscriptionId is {a2bef483-749d-4a9e-ad35-c3b660e22c2f}. [PID:10368:010 2015.07.04 13:24:06.902 INFO ReplicaService ] Subscribing at LogPosition: 36796195 (0x2317723) to MASTER [192.168.33.12:1111, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}] as replica with SubscriptionId: {6ce035b9-6b64-46df-9959-7c91538b095a}, ConnectionId: {206ecac0-8062-4738-9c4b-98c22ca4532f}, LocalEndPoint: [192.168.33.11:53251], Epochs: E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740}... . [PID:10368:010 2015.07.04 13:24:07.097 DEBUG ElectionsService ] ELECTIONS: (V=49) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:24:07.097 DEBUG ElectionsService ] ELECTIONS: (V=50) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:24:07.097 DEBUG ElectionsService ] ELECTIONS: (V=50) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:24:07.097 DEBUG ElectionsService ] ELECTIONS: (V=50) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:24:07.097 DEBUG ElectionsService ] ELECTIONS: (V=50) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:24:07.097 DEBUG ElectionsService ] ELECTIONS: (V=50) SHIFT TO PREPARE PHASE. [PID:10368:010 2015.07.04 13:24:07.097 DEBUG ElectionsService ] ELECTIONS: (V=50) PREPARE_OK FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36778896,W=36796195,C=36796195,E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd}). [PID:10368:010 2015.07.04 13:24:07.097 TRACE ClusterVNodeControll] Ignoring ReplicaSubscribed because SubscriptionId {a2bef483-749d-4a9e-ad35-c3b660e22c2f} is wrong. Current SubscriptionId is {6ce035b9-6b64-46df-9959-7c91538b095a}. [PID:10368:010 2015.07.04 13:24:07.903 INFO ReplicaService ] Subscribing at LogPosition: 36796195 (0x2317723) to MASTER [192.168.33.12:1111, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}] as replica with SubscriptionId: {d0bd288a-67ca-41ff-b683-eb01955c76aa}, ConnectionId: {206ecac0-8062-4738-9c4b-98c22ca4532f}, LocalEndPoint: [192.168.33.11:53251], Epochs: E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff} E32@328469:{363e4dfe-ca06-421a-9cb6-ff883e3c4740}... . [PID:10368:010 2015.07.04 13:24:08.096 TRACE ClusterVNodeControll] Ignoring ReplicaSubscribed because SubscriptionId {6ce035b9-6b64-46df-9959-7c91538b095a} is wrong. Current SubscriptionId is {d0bd288a-67ca-41ff-b683-eb01955c76aa}. [PID:10368:010 2015.07.04 13:24:08.099 DEBUG ElectionsService ] ELECTIONS: (V=50) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:24:08.099 DEBUG ElectionsService ] ELECTIONS: (V=51) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:24:08.100 DEBUG ElectionsService ] ELECTIONS: (V=51) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:24:08.107 DEBUG ElectionsService ] ELECTIONS: (V=51) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:24:08.107 DEBUG ElectionsService ] ELECTIONS: (V=51) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:24:08.597 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [Clone, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:01.197 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:01.486 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:08.319 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [Clone, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:04.839 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:07.489 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:08.592 -------------------------------------------------------------------------------- [PID:10368:007 2015.07.04 13:24:08.598 INFO ClusterStorageWriter] === SUBSCRIBED to [192.168.33.12:1111,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] at 36796195 (0x2317723). SubscriptionId: {d0bd288a-67ca-41ff-b683-eb01955c76aa}. [PID:10368:010 2015.07.04 13:24:08.599 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CATCHING UP!!! BANZAI!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:24:08.605 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] CLONE ASSIGNMENT RECEIVED FROM [192.168.33.12:1111,n/a,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:24:08.605 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CLONE!!! SPARTA!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:24:08.605 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] SLAVE ASSIGNMENT RECEIVED FROM [192.168.33.12:1111,n/a,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:24:08.605 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS SLAVE!!! SPARTA!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:24:09.058 DEBUG ElectionsService ] ELECTIONS: (V=52) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:24:09.058 DEBUG ElectionsService ] ELECTIONS: (V=52) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:24:09.058 DEBUG ElectionsService ] ELECTIONS: (V=52) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:24:09.058 DEBUG ElectionsService ] ELECTIONS: (V=52) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:24:09.063 DEBUG ElectionsService ] ELECTIONS: (V=52) PREPARE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:24:09.063 DEBUG ElectionsService ] ELECTIONS: (V=52) SHIFT TO REG_NONLEADER. [PID:10368:010 2015.07.04 13:24:09.068 DEBUG ElectionsService ] ELECTIONS: (V=52) PROPOSAL FROM [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] M=[192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}](L=36778896,W=36796195,C=36796195,E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd}). ME=[192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36778896,W=36796195,C=36796195,E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd}). [PID:10368:010 2015.07.04 13:24:09.068 DEBUG ElectionsService ] ELECTIONS: (V=52) ACCEPT FROM [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] M=[192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]). [PID:10368:010 2015.07.04 13:24:09.068 DEBUG ElectionsService ] ELECTIONS: (V=52) ACCEPT FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}] M=[192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]). [PID:10368:010 2015.07.04 13:24:09.068 INFO ElectionsService ] ELECTIONS: (V=52) DONE. ELECTED MASTER = [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}](L=36778896,W=36796195,C=36796195,E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd}). ME=[192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36778896,W=36796195,C=36796195,E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd}). [PID:10368:015 2015.07.04 13:24:09.075 DEBUG EpochManager ] === Update Last Epoch E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} (previous epoch at 36795971). [PID:10368:010 2015.07.04 13:24:10.328 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [Clone, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:04.839 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:08.611 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:24:10.320 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [Clone, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:04.839 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:24:10.335 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:24:10.328 -------------------------------------------------------------------------------- [PID:10368:055 2015.07.04 13:24:13.049 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:10368:073 2015.07.04 13:24:15.032 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:10368:055 2015.07.04 13:24:20.033 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:10368:073 2015.07.04 13:24:20.036 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:10368:010 2015.07.04 13:24:23.489 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [Clone, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:04.839 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:24:17.493 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:24:22.325 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:23.019 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:24:23.497 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:24:23.489 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:24:29.283 DEBUG ElectionsService ] ELECTIONS: (IV=65) VIEWCHANGEPROOF FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. JUMPING TO LEADER STATE. [PID:10368:010 2015.07.04 13:24:29.283 DEBUG ElectionsService ] ELECTIONS: (V=65) SHIFT TO PREPARE PHASE. [PID:10368:010 2015.07.04 13:24:30.579 DEBUG ElectionsService ] ELECTIONS: (V=73) VIEWCHANGE FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. [PID:10368:010 2015.07.04 13:24:30.579 DEBUG ElectionsService ] ELECTIONS: (V=73) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:24:30.579 DEBUG ElectionsService ] ELECTIONS: (V=73) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:24:30.579 DEBUG ElectionsService ] ELECTIONS: (V=73) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:24:31.580 DEBUG ElectionsService ] ELECTIONS: (V=73) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:24:31.580 DEBUG ElectionsService ] ELECTIONS: (V=74) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:24:31.580 DEBUG ElectionsService ] ELECTIONS: (V=74) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:008 2015.07.04 13:24:31.681 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L192.168.33.11:53251, {206ecac0-8062-4738-9c4b-98c22ca4532f}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 29 [PID:10368:008 2015.07.04 13:24:31.681 INFO TcpConnection ] ES TcpConnection closed [13:24:31.681: N192.168.33.12:1111, L192.168.33.11:53251, {206ecac0-8062-4738-9c4b-98c22ca4532f}]: Received bytes: 20117, Sent bytes: 4079 Send calls: 25, callbacks: 25 Receive calls: 30, callbacks: 29 Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 29 [PID:10368:008 2015.07.04 13:24:31.681 INFO TcpConnectionManager] Connection 'master-normal' [192.168.33.12:1111, {206ecac0-8062-4738-9c4b-98c22ca4532f}] closed: Success. [PID:10368:010 2015.07.04 13:24:31.685 TRACE GossipServiceBase ] Looks like node [192.168.33.12:1111] is DEAD (TCP connection lost). [PID:10368:010 2015.07.04 13:24:31.685 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection lost to [192.168.33.12:1111]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:30.542 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36796419/36813528/36813528/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:24:30.579 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36796419/36813528/36813528/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:24:30.579 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:30.542 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36796419/36813528/36813528/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:24:31.685 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36796419/36813528/36813528/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:24:30.579 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:24:31.685 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {fe562867-3182-4fcd-a3e5-6bb9c60441e8}, InternalTcp: 192.168.33.12:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.12:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.12:2113, ExternalHttp: 192.168.33.12:2114]. [PID:10368:010 2015.07.04 13:24:32.186 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] PRE-REPLICA STATE, WAITING FOR CHASER TO CATCH UP... MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:24:32.186 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L192.168.33.11:53251, {206ecac0-8062-4738-9c4b-98c22ca4532f}] cleanly. Reason: Reconnecting from old master [192.168.33.12:1111] to new master: [192.168.33.12:1111]. [PID:10368:010 2015.07.04 13:24:32.186 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {fe562867-3182-4fcd-a3e5-6bb9c60441e8}, InternalTcp: 192.168.33.12:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.12:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.12:2113, ExternalHttp: 192.168.33.12:2114]. [PID:10368:012 2015.07.04 13:24:32.187 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:013 2015.07.04 13:24:32.187 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:014 2015.07.04 13:24:32.187 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:081 2015.07.04 13:24:32.189 INFO TcpConnectionManager] Connection 'master-normal' ({ab79fee4-944c-419a-810c-c60422d88163}) to [192.168.33.12:1111] established. [PID:10368:010 2015.07.04 13:24:32.189 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection established to [192.168.33.12:1111]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:30.542 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36796419/36813528/36813528/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:24:31.685 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36796419/36813528/36813528/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:24:32.187 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:30.542 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36796419/36813528/36813528/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:24:32.189 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36796419/36813528/36813528/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:24:32.187 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:24:32.189 INFO ReplicaService ] Subscribing at LogPosition: 36813528 (0x231BAD8) to MASTER [192.168.33.12:1111, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}] as replica with SubscriptionId: {c5283ddd-1e60-47d6-b0ee-ce865897858a}, ConnectionId: {ab79fee4-944c-419a-810c-c60422d88163}, LocalEndPoint: [192.168.33.11:53269], Epochs: E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff}... . [PID:10368:010 2015.07.04 13:24:32.581 DEBUG ElectionsService ] ELECTIONS: (V=74) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:24:32.582 DEBUG ElectionsService ] ELECTIONS: (V=75) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:24:32.582 DEBUG ElectionsService ] ELECTIONS: (V=75) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:24:33.191 INFO ReplicaService ] Subscribing at LogPosition: 36813528 (0x231BAD8) to MASTER [192.168.33.12:1111, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}] as replica with SubscriptionId: {f98f8ebe-818d-459b-b7e8-cbbc8104023b}, ConnectionId: {ab79fee4-944c-419a-810c-c60422d88163}, LocalEndPoint: [192.168.33.11:53269], Epochs: E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0} E33@328689:{523b0b34-29b0-41d5-b169-e3d0d44900ff}... . [PID:10368:010 2015.07.04 13:24:33.481 TRACE ClusterVNodeControll] Ignoring ReplicaSubscribed because SubscriptionId {c5283ddd-1e60-47d6-b0ee-ce865897858a} is wrong. Current SubscriptionId is {f98f8ebe-818d-459b-b7e8-cbbc8104023b}. [PID:10368:010 2015.07.04 13:24:33.493 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:30.542 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36796419/36813528/36813528/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:24:32.189 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36796419/36813528/36813528/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:24:32.332 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:33.490 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36796419/36813528/36813528/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:24:33.500 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36796419/36813528/36813528/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:24:33.493 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:24:33.494 DEBUG ElectionsService ] ELECTIONS: (V=74) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:24:33.494 DEBUG ElectionsService ] ELECTIONS: (V=74) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:24:33.494 DEBUG ElectionsService ] ELECTIONS: (V=75) SHIFT TO PREPARE PHASE. [PID:10368:010 2015.07.04 13:24:33.494 DEBUG ElectionsService ] ELECTIONS: (V=75) PREPARE_OK FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36796419,W=36813528,C=36813528,E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f}). [PID:10368:010 2015.07.04 13:24:33.495 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CATCHING UP!!! BANZAI!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:24:33.495 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] CLONE ASSIGNMENT RECEIVED FROM [192.168.33.12:1111,n/a,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:24:33.495 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CLONE!!! SPARTA!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:24:33.495 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] SLAVE ASSIGNMENT RECEIVED FROM [192.168.33.12:1111,n/a,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:24:33.495 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS SLAVE!!! SPARTA!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:007 2015.07.04 13:24:33.496 INFO ClusterStorageWriter] === SUBSCRIBED to [192.168.33.12:1111,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] at 36813528 (0x231BAD8). SubscriptionId: {f98f8ebe-818d-459b-b7e8-cbbc8104023b}. [PID:10368:010 2015.07.04 13:24:33.500 DEBUG ElectionsService ] ELECTIONS: (V=75) PREPARE_OK FROM [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}](L=36796419,W=36813528,C=36813528,E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f}). [PID:10368:010 2015.07.04 13:24:33.500 DEBUG ElectionsService ] ELECTIONS: (V=75) SHIFT TO REG_LEADER. [PID:10368:010 2015.07.04 13:24:33.500 DEBUG ElectionsService ] ELECTIONS: (V=75) SENDING PROPOSAL CANDIDATE: [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}](L=36796419,W=36813528,C=36813528,E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f}), ME: [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36796419,W=36813528,C=36813528,E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f}). [PID:10368:010 2015.07.04 13:24:33.500 DEBUG ElectionsService ] ELECTIONS: (V=75) ACCEPT FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}] M=[192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]). [PID:10368:010 2015.07.04 13:24:33.505 DEBUG ElectionsService ] ELECTIONS: (V=75) ACCEPT FROM [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] M=[192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]). [PID:10368:010 2015.07.04 13:24:33.505 INFO ElectionsService ] ELECTIONS: (V=75) DONE. ELECTED MASTER = [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}](L=36796419,W=36813528,C=36813528,E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f}). ME=[192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36796419,W=36813528,C=36813528,E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f}). [PID:10368:015 2015.07.04 13:24:33.508 DEBUG EpochManager ] === Update Last Epoch E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} (previous epoch at 36796195). [PID:10368:010 2015.07.04 13:24:34.339 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:33.490 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36796419/36813528/36813528/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:24:33.504 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36796419/36813752/36813752/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:24:34.333 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:33.490 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36796419/36813752/36813752/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:24:34.347 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36796419/36813752/36813752/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:24:34.339 -------------------------------------------------------------------------------- [PID:10368:080 2015.07.04 13:24:47.102 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:10368:080 2015.07.04 13:24:50.094 DEBUG GossipController ] Error while reading request (gossip): EndRead failure [PID:10368:083 2015.07.04 13:24:50.102 DEBUG GossipController ] Error while reading request (gossip): EndRead failure [PID:10368:010 2015.07.04 13:24:50.103 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.13:2113]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796195/36796195/E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} | 2015-07-04 13:24:33.490 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36796419/36813752/36813752/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:24:49.517 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36796419/36813752/36813752/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:24:49.510 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:24:47.057 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36796419/36813752/36813752/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:24:49.517 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36796419/36813752/36813752/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:24:50.103 -------------------------------------------------------------------------------- [PID:10368:080 2015.07.04 13:24:54.104 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:10368:080 2015.07.04 13:24:54.110 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:10368:086 2015.07.04 13:25:05.138 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:10368:085 2015.07.04 13:25:07.601 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:10368:088 2015.07.04 13:25:12.157 DEBUG GossipController ] Error while reading request (gossip): EndRead failure [PID:10368:088 2015.07.04 13:25:12.157 DEBUG GossipController ] Error while reading request (gossip): EndRead failure [PID:10368:087 2015.07.04 13:25:20.609 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:10368:088 2015.07.04 13:25:22.611 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:10368:088 2015.07.04 13:25:24.612 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:10368:010 2015.07.04 13:25:25.112 DEBUG ElectionsService ] ELECTIONS: (IV=87) VIEWCHANGEPROOF FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. JUMPING TO NON-LEADER STATE. [PID:10368:010 2015.07.04 13:25:25.112 DEBUG ElectionsService ] ELECTIONS: (V=87) SHIFT TO REG_NONLEADER. [PID:10368:091 2015.07.04 13:25:26.614 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:10368:092 2015.07.04 13:25:40.123 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:10368:093 2015.07.04 13:25:40.637 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:10368:094 2015.07.04 13:25:45.641 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:10368:010 2015.07.04 13:25:48.376 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:25:43.094 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36830869/36847988/36847988/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:25:43.143 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36830869/36847988/36847988/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:25:48.369 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:25:44.351 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36830869/36847988/36847988/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:25:48.383 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36830869/36847988/36847988/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:25:48.375 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:25:51.136 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.13:2113]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:25:44.351 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36830869/36847988/36847988/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:25:49.543 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36830869/36847988/36847988/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:25:50.369 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:25:48.598 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36830869/36847988/36847988/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:25:49.543 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36830869/36847988/36847988/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:25:51.136 -------------------------------------------------------------------------------- [PID:10368:095 2015.07.04 13:25:53.137 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:10368:093 2015.07.04 13:25:53.644 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:10368:010 2015.07.04 13:25:54.216 DEBUG ElectionsService ] ELECTIONS: (V=157) VIEWCHANGE FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. [PID:10368:010 2015.07.04 13:25:54.217 DEBUG ElectionsService ] ELECTIONS: (V=157) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:25:54.217 DEBUG ElectionsService ] ELECTIONS: (V=157) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:25:54.217 DEBUG ElectionsService ] ELECTIONS: (V=157) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:25:55.219 DEBUG ElectionsService ] ELECTIONS: (V=157) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:25:55.219 DEBUG ElectionsService ] ELECTIONS: (V=158) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:25:55.219 DEBUG ElectionsService ] ELECTIONS: (V=158) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:008 2015.07.04 13:25:55.565 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L192.168.33.11:53269, {ab79fee4-944c-419a-810c-c60422d88163}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 66 [PID:10368:008 2015.07.04 13:25:55.565 INFO TcpConnection ] ES TcpConnection closed [13:25:55.565: N192.168.33.12:1111, L192.168.33.11:53269, {ab79fee4-944c-419a-810c-c60422d88163}]: Received bytes: 38778, Sent bytes: 3645 Send calls: 58, callbacks: 58 Receive calls: 67, callbacks: 66 Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 66 [PID:10368:008 2015.07.04 13:25:55.566 INFO TcpConnectionManager] Connection 'master-normal' [192.168.33.12:1111, {ab79fee4-944c-419a-810c-c60422d88163}] closed: Success. [PID:10368:010 2015.07.04 13:25:55.566 TRACE GossipServiceBase ] Looks like node [192.168.33.12:1111] is DEAD (TCP connection lost). [PID:10368:010 2015.07.04 13:25:55.566 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection lost to [192.168.33.12:1111]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:25:55.159 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36830869/36847988/36847988/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:25:55.196 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36830869/36847988/36847988/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:25:55.196 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:25:55.159 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36830869/36847988/36847988/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:25:55.566 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36830869/36847988/36847988/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:25:55.196 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:25:55.566 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {fe562867-3182-4fcd-a3e5-6bb9c60441e8}, InternalTcp: 192.168.33.12:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.12:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.12:2113, ExternalHttp: 192.168.33.12:2114]. [PID:10368:010 2015.07.04 13:25:56.068 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] PRE-REPLICA STATE, WAITING FOR CHASER TO CATCH UP... MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:25:56.068 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L192.168.33.11:53269, {ab79fee4-944c-419a-810c-c60422d88163}] cleanly. Reason: Reconnecting from old master [192.168.33.12:1111] to new master: [192.168.33.12:1111]. [PID:10368:010 2015.07.04 13:25:56.069 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {fe562867-3182-4fcd-a3e5-6bb9c60441e8}, InternalTcp: 192.168.33.12:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.12:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.12:2113, ExternalHttp: 192.168.33.12:2114]. [PID:10368:012 2015.07.04 13:25:56.070 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:013 2015.07.04 13:25:56.070 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:014 2015.07.04 13:25:56.070 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:097 2015.07.04 13:25:56.070 INFO TcpConnectionManager] Connection 'master-normal' ({ef70187d-0f25-49f0-b731-20a1ef8387f6}) to [192.168.33.12:1111] established. [PID:10368:010 2015.07.04 13:25:56.070 INFO ReplicaService ] Subscribing at LogPosition: 36847988 (0x2324174) to MASTER [192.168.33.12:1111, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}] as replica with SubscriptionId: {a2caa8e6-725d-4848-b5ba-6353ae3c74d6}, ConnectionId: {ef70187d-0f25-49f0-b731-20a1ef8387f6}, LocalEndPoint: [192.168.33.11:53281], Epochs: E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0}... . [PID:10368:010 2015.07.04 13:25:56.072 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection established to [192.168.33.12:1111]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:25:55.159 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36830869/36847988/36847988/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:25:55.566 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36830869/36847988/36847988/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:25:56.068 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:25:55.159 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36830869/36847988/36847988/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:25:56.071 VND {843c5aa9-0910-4d75-9747-251d784ce634} [PreReplica, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36830869/36847988/36847988/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:25:56.068 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:25:56.154 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CATCHING UP!!! BANZAI!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:007 2015.07.04 13:25:56.154 INFO ClusterStorageWriter] === SUBSCRIBED to [192.168.33.12:1111,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] at 36847988 (0x2324174). SubscriptionId: {a2caa8e6-725d-4848-b5ba-6353ae3c74d6}. [PID:10368:010 2015.07.04 13:25:56.155 DEBUG ElectionsService ] ELECTIONS: (V=158) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:25:56.155 DEBUG ElectionsService ] ELECTIONS: (V=158) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:25:56.155 DEBUG ElectionsService ] ELECTIONS: (V=158) SHIFT TO PREPARE PHASE. [PID:10368:010 2015.07.04 13:25:56.155 DEBUG ElectionsService ] ELECTIONS: (V=158) PREPARE_OK FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36830869,W=36847988,C=36847988,E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93}). [PID:10368:010 2015.07.04 13:25:56.157 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] CLONE ASSIGNMENT RECEIVED FROM [192.168.33.12:1111,n/a,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:25:56.157 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CLONE!!! SPARTA!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:25:56.158 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] SLAVE ASSIGNMENT RECEIVED FROM [192.168.33.12:1111,n/a,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:25:56.158 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS SLAVE!!! SPARTA!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:25:56.199 DEBUG ElectionsService ] ELECTIONS: (V=158) PREPARE_OK FROM [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}](L=36830869,W=36847988,C=36847988,E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93}). [PID:10368:010 2015.07.04 13:25:56.199 DEBUG ElectionsService ] ELECTIONS: (V=158) SHIFT TO REG_LEADER. [PID:10368:010 2015.07.04 13:25:56.199 DEBUG ElectionsService ] ELECTIONS: (V=158) SENDING PROPOSAL CANDIDATE: [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}](L=36830869,W=36847988,C=36847988,E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93}), ME: [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36830869,W=36847988,C=36847988,E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93}). [PID:10368:010 2015.07.04 13:25:56.199 DEBUG ElectionsService ] ELECTIONS: (V=158) ACCEPT FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}] M=[192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]). [PID:10368:010 2015.07.04 13:25:56.221 DEBUG ElectionsService ] ELECTIONS: (V=158) TIMED OUT! (S=Leader, M=). [PID:10368:010 2015.07.04 13:25:56.221 DEBUG ElectionsService ] ELECTIONS: (V=159) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:25:56.221 DEBUG ElectionsService ] ELECTIONS: (V=159) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:25:57.221 DEBUG ElectionsService ] ELECTIONS: (V=159) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:25:57.221 DEBUG ElectionsService ] ELECTIONS: (V=160) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:25:57.221 DEBUG ElectionsService ] ELECTIONS: (V=160) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:25:58.201 DEBUG ElectionsService ] ELECTIONS: (V=159) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:25:58.201 DEBUG ElectionsService ] ELECTIONS: (V=159) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:25:58.223 DEBUG ElectionsService ] ELECTIONS: (V=160) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:25:58.223 DEBUG ElectionsService ] ELECTIONS: (V=161) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:25:58.223 DEBUG ElectionsService ] ELECTIONS: (V=161) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:25:58.649 DEBUG ElectionsService ] ELECTIONS: (V=159) VIEWCHANGE FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. [PID:10368:010 2015.07.04 13:25:58.649 DEBUG ElectionsService ] ELECTIONS: (V=159) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:25:58.649 DEBUG ElectionsService ] ELECTIONS: (V=161) SHIFT TO PREPARE PHASE. [PID:10368:010 2015.07.04 13:25:58.649 DEBUG ElectionsService ] ELECTIONS: (V=161) PREPARE_OK FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36847988,W=36865031,C=36865031,E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93}). [PID:10368:010 2015.07.04 13:25:59.224 DEBUG ElectionsService ] ELECTIONS: (V=161) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:25:59.224 DEBUG ElectionsService ] ELECTIONS: (V=162) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:25:59.224 DEBUG ElectionsService ] ELECTIONS: (V=162) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:26:00.068 DEBUG ElectionsService ] ELECTIONS: (V=162) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:26:00.068 DEBUG ElectionsService ] ELECTIONS: (V=162) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:26:00.225 DEBUG ElectionsService ] ELECTIONS: (V=162) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:26:00.225 DEBUG ElectionsService ] ELECTIONS: (V=163) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:26:00.225 DEBUG ElectionsService ] ELECTIONS: (V=163) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:106 2015.07.04 13:26:01.200 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:10368:010 2015.07.04 13:26:01.226 DEBUG ElectionsService ] ELECTIONS: (V=163) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:26:01.226 DEBUG ElectionsService ] ELECTIONS: (V=164) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:26:01.226 DEBUG ElectionsService ] ELECTIONS: (V=164) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:008 2015.07.04 13:26:01.877 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L192.168.33.11:53281, {ef70187d-0f25-49f0-b731-20a1ef8387f6}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 11 [PID:10368:008 2015.07.04 13:26:01.877 INFO TcpConnection ] ES TcpConnection closed [13:26:01.877: N192.168.33.12:1111, L192.168.33.11:53281, {ef70187d-0f25-49f0-b731-20a1ef8387f6}]: Received bytes: 18982, Sent bytes: 1973 Send calls: 10, callbacks: 10 Receive calls: 12, callbacks: 11 Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 11 [PID:10368:008 2015.07.04 13:26:01.877 INFO TcpConnectionManager] Connection 'master-normal' [192.168.33.12:1111, {ef70187d-0f25-49f0-b731-20a1ef8387f6}] closed: Success. [PID:10368:010 2015.07.04 13:26:01.879 TRACE GossipServiceBase ] Looks like node [192.168.33.12:1111] is DEAD (TCP connection lost). [PID:10368:010 2015.07.04 13:26:01.879 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection lost to [192.168.33.12:1111]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:25:59.503 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36847988/36865031/36865031/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:25:59.547 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36847988/36865031/36865031/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:26:00.373 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:25:59.503 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36847988/36865031/36865031/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:26:01.879 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36847988/36865031/36865031/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:26:00.373 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:26:01.880 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {fe562867-3182-4fcd-a3e5-6bb9c60441e8}, InternalTcp: 192.168.33.12:1111, InternalSecureTcp: , ExternalTcp: 192.168.33.12:1112, ExternalSecureTcp: , InternalHttp: 192.168.33.12:2113, ExternalHttp: 192.168.33.12:2114]. [PID:10368:010 2015.07.04 13:26:02.228 DEBUG ElectionsService ] ELECTIONS: (V=164) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:26:02.228 DEBUG ElectionsService ] ELECTIONS: (V=165) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:26:02.228 DEBUG ElectionsService ] ELECTIONS: (V=165) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:26:02.240 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:25:59.503 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36847988/36865031/36865031/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:26:01.879 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36847988/36865031/36865031/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:26:00.373 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:25:59.503 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36847988/36865255/36865255/E54@36865031:{ad3728ee-29b2-49c3-8908-c9f564c87313} | 2015-07-04 13:26:01.548 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36847988/36865031/36865031/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:26:02.231 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:26:02.381 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] PRE-REPLICA STATE, WAITING FOR CHASER TO CATCH UP... MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:26:02.381 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.33.12:1111, L192.168.33.11:53281, {ef70187d-0f25-49f0-b731-20a1ef8387f6}] cleanly. Reason: Reconnecting from old master [192.168.33.12:1111] to new master: [192.168.33.12:1111]. [PID:10368:012 2015.07.04 13:26:02.382 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:014 2015.07.04 13:26:02.382 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:013 2015.07.04 13:26:02.382 INFO ProjectionCoreServic] _eventReaders is not empty after all the projections have been killed [PID:10368:010 2015.07.04 13:26:02.382 INFO ReplicaService ] Subscribing at LogPosition: 36865031 (0x2328407) to MASTER [192.168.33.12:1111, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}] as replica with SubscriptionId: {c0b1190e-5513-4556-a425-385d813885b1}, ConnectionId: {b6a7167b-bf51-4bd9-b3bc-d297a5089efb}, LocalEndPoint: [], Epochs: E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} E51@36795971:{23204d5b-7782-4056-81f3-1073a2cea3fd} E50@36740334:{433628f5-9c65-4ccb-863b-5cc63ca2e39b} E49@36723005:{02ba809d-c9b9-47db-9d62-0766d8a586a2} E48@36722781:{1bcf5086-32d3-4810-abe0-02e912f737e2} E47@36705443:{e2991cb1-75fa-4aa3-83cf-50399778d2ce} E46@36705219:{ae8ac01f-a53a-4ebb-ac61-89f1637caef3} E45@36685512:{6106ceb9-ac27-4132-acf8-e412f92fac8f} E44@36684018:{2fa110fd-87c8-4586-ab3f-66e9ab851066} E43@36612193:{06838ed7-91ee-41c8-952d-f5853b93e6b7} E42@36610231:{8d23f58b-97d5-43db-afb8-34a7405622c5} E41@499692:{43dff52a-ad0d-4f88-ba68-9e3f4022da5b} E40@482483:{9e41cc83-f2f2-488c-b64d-32d9d3339f76} E39@482263:{75a2a8b6-9bb1-4e65-ad89-d2d47ceed812} E38@482043:{fb7b6022-bf5d-40b3-bd61-1f84534e3ac9} E37@425855:{8b099dbf-0662-44e5-be66-1367d4d02845} E36@423894:{3d8361ff-1a7a-4194-95fc-afc41a16e864} E35@406608:{f2801d1b-ed4e-4185-8c8c-bc6972a01fed} E34@367354:{6a3268cd-fe42-4e2e-b66b-445c6bfcbfb0}... . [PID:10368:010 2015.07.04 13:26:02.702 DEBUG ElectionsService ] ELECTIONS: (V=163) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:26:02.703 DEBUG ElectionsService ] ELECTIONS: (V=163) MAJORITY OF VIEWCHANGE. [PID:10368:106 2015.07.04 13:26:03.202 INFO TcpConnectionManager] Connection 'master-normal' ({b6a7167b-bf51-4bd9-b3bc-d297a5089efb}) to [192.168.33.12:1111] established. [PID:10368:010 2015.07.04 13:26:03.207 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CATCHING UP!!! BANZAI!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:007 2015.07.04 13:26:03.208 INFO ClusterStorageWriter] === SUBSCRIBED to [192.168.33.12:1111,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] at 36865031 (0x2328407). SubscriptionId: {c0b1190e-5513-4556-a425-385d813885b1}. [PID:10368:010 2015.07.04 13:26:03.211 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] CLONE ASSIGNMENT RECEIVED FROM [192.168.33.12:1111,n/a,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:26:03.211 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS CLONE!!! SPARTA!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:26:03.212 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] SLAVE ASSIGNMENT RECEIVED FROM [192.168.33.12:1111,n/a,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:26:03.212 INFO ClusterVNodeControll] ========== [192.168.33.11:2113] IS SLAVE!!! SPARTA!!! MASTER IS [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] [PID:10368:010 2015.07.04 13:26:03.214 DEBUG ElectionsService ] ELECTIONS: (IV=163) VIEWCHANGEPROOF FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. JUMPING TO NON-LEADER STATE. [PID:10368:010 2015.07.04 13:26:03.214 DEBUG ElectionsService ] ELECTIONS: (V=163) SHIFT TO REG_NONLEADER. [PID:10368:015 2015.07.04 13:26:03.217 DEBUG EpochManager ] === Update Last Epoch E54@36865031:{ad3728ee-29b2-49c3-8908-c9f564c87313} (previous epoch at 36813528). [PID:10368:010 2015.07.04 13:26:03.219 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:26:01.256 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36847988/36865255/36865255/E54@36865031:{ad3728ee-29b2-49c3-8908-c9f564c87313} | 2015-07-04 13:26:03.202 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36847988/36865255/36865031/E53@36813528:{de6c4666-6991-41b0-9a39-bb891b775b93} | 2015-07-04 13:26:03.214 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:26:01.592 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36847988/36865255/36865255/E54@36865031:{ad3728ee-29b2-49c3-8908-c9f564c87313} | 2015-07-04 13:26:03.221 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36847988/36865255/36865255/E54@36865031:{ad3728ee-29b2-49c3-8908-c9f564c87313} | 2015-07-04 13:26:03.219 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:26:04.073 DEBUG ElectionsService ] ELECTIONS: (V=164) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:26:04.074 DEBUG ElectionsService ] ELECTIONS: (V=164) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:26:04.074 DEBUG ElectionsService ] ELECTIONS: (V=164) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:26:04.074 DEBUG ElectionsService ] ELECTIONS: (V=164) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:26:04.074 DEBUG ElectionsService ] ELECTIONS: (V=164) SHIFT TO PREPARE PHASE. [PID:10368:010 2015.07.04 13:26:04.074 DEBUG ElectionsService ] ELECTIONS: (V=164) PREPARE_OK FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36847988,W=36865255,C=36865255,E54@36865031:{ad3728ee-29b2-49c3-8908-c9f564c87313}). [PID:10368:010 2015.07.04 13:26:05.075 DEBUG ElectionsService ] ELECTIONS: (V=164) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:26:05.075 DEBUG ElectionsService ] ELECTIONS: (V=165) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:26:05.075 DEBUG ElectionsService ] ELECTIONS: (V=165) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:26:06.076 DEBUG ElectionsService ] ELECTIONS: (V=165) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:26:06.076 DEBUG ElectionsService ] ELECTIONS: (V=166) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:26:06.076 DEBUG ElectionsService ] ELECTIONS: (V=166) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:26:06.206 DEBUG ElectionsService ] ELECTIONS: (V=165) VIEWCHANGE FROM [192.168.33.13:2113, {6bba87a7-5f19-443e-b978-c86c62e4973c}]. [PID:10368:010 2015.07.04 13:26:06.206 DEBUG ElectionsService ] ELECTIONS: (V=165) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:26:07.078 DEBUG ElectionsService ] ELECTIONS: (V=166) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:26:07.078 DEBUG ElectionsService ] ELECTIONS: (V=167) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:26:07.078 DEBUG ElectionsService ] ELECTIONS: (V=167) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:26:07.209 DEBUG ElectionsService ] ELECTIONS: (V=165) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:26:07.209 DEBUG ElectionsService ] ELECTIONS: (V=165) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:26:07.210 DEBUG ElectionsService ] ELECTIONS: (V=167) SHIFT TO PREPARE PHASE. [PID:10368:010 2015.07.04 13:26:07.210 DEBUG ElectionsService ] ELECTIONS: (V=167) PREPARE_OK FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36847988,W=36865255,C=36865255,E54@36865031:{ad3728ee-29b2-49c3-8908-c9f564c87313}). [PID:10368:010 2015.07.04 13:26:08.079 DEBUG ElectionsService ] ELECTIONS: (V=167) TIMED OUT! (S=ElectingLeader, M=). [PID:10368:010 2015.07.04 13:26:08.079 DEBUG ElectionsService ] ELECTIONS: (V=168) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:26:08.079 DEBUG ElectionsService ] ELECTIONS: (V=168) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:26:08.709 TRACE GossipServiceBase ] Looks like node [192.168.33.13:2113] is DEAD (Gossip send failed). [PID:10368:010 2015.07.04 13:26:08.709 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip send failed to [192.168.33.13:2113]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:26:01.592 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36847988/36865255/36865255/E54@36865031:{ad3728ee-29b2-49c3-8908-c9f564c87313} | 2015-07-04 13:26:03.548 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36847988/36865255/36865255/E54@36865031:{ad3728ee-29b2-49c3-8908-c9f564c87313} | 2015-07-04 13:26:08.376 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:26:08.709 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36847988/36865255/36865255/E54@36865031:{ad3728ee-29b2-49c3-8908-c9f564c87313} | 2015-07-04 13:26:03.548 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36847988/36865255/36865255/E54@36865031:{ad3728ee-29b2-49c3-8908-c9f564c87313} | 2015-07-04 13:26:08.376 -------------------------------------------------------------------------------- [PID:10368:010 2015.07.04 13:26:08.709 DEBUG ElectionsService ] ELECTIONS: (V=168) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:26:08.709 DEBUG ElectionsService ] ELECTIONS: (V=168) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:26:08.712 DEBUG ElectionsService ] ELECTIONS: (V=169) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:26:08.712 DEBUG ElectionsService ] ELECTIONS: (V=169) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:26:08.712 DEBUG ElectionsService ] ELECTIONS: (V=169) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:26:08.712 DEBUG ElectionsService ] ELECTIONS: (V=169) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:26:08.713 DEBUG ElectionsService ] ELECTIONS: (V=169) SHIFT TO PREPARE PHASE. [PID:10368:010 2015.07.04 13:26:08.713 DEBUG ElectionsService ] ELECTIONS: (V=169) PREPARE_OK FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36847988,W=36865255,C=36865255,E54@36865031:{ad3728ee-29b2-49c3-8908-c9f564c87313}). [PID:10368:010 2015.07.04 13:26:09.719 DEBUG ElectionsService ] ELECTIONS: (V=170) VIEWCHANGE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:26:09.720 DEBUG ElectionsService ] ELECTIONS: (V=170) SHIFT TO LEADER ELECTION. [PID:10368:010 2015.07.04 13:26:09.720 DEBUG ElectionsService ] ELECTIONS: (V=170) VIEWCHANGE FROM [192.168.33.11:2113, {843c5aa9-0910-4d75-9747-251d784ce634}]. [PID:10368:010 2015.07.04 13:26:09.720 DEBUG ElectionsService ] ELECTIONS: (V=170) MAJORITY OF VIEWCHANGE. [PID:10368:010 2015.07.04 13:26:09.721 DEBUG ElectionsService ] ELECTIONS: (V=170) PREPARE FROM [192.168.33.12:2113, {fe562867-3182-4fcd-a3e5-6bb9c60441e8}]. [PID:10368:010 2015.07.04 13:26:09.721 DEBUG ElectionsService ] ELECTIONS: (V=170) SHIFT TO REG_NONLEADER. [PID:10368:010 2015.07.04 13:26:09.729 DEBUG ElectionsService ] ELECTIONS: (V=170) PROPOSAL FROM [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] M=[192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}](L=36847988,W=36865255,C=36865255,E54@36865031:{ad3728ee-29b2-49c3-8908-c9f564c87313}). ME=[192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36847988,W=36865255,C=36865255,E54@36865031:{ad3728ee-29b2-49c3-8908-c9f564c87313}). [PID:10368:010 2015.07.04 13:26:09.729 DEBUG ElectionsService ] ELECTIONS: (V=170) ACCEPT FROM [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}] M=[192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]). [PID:10368:010 2015.07.04 13:26:09.729 DEBUG ElectionsService ] ELECTIONS: (V=170) ACCEPT FROM [192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}] M=[192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}]). [PID:10368:010 2015.07.04 13:26:09.729 INFO ElectionsService ] ELECTIONS: (V=170) DONE. ELECTED MASTER = [192.168.33.12:2113,{fe562867-3182-4fcd-a3e5-6bb9c60441e8}](L=36847988,W=36865255,C=36865255,E54@36865031:{ad3728ee-29b2-49c3-8908-c9f564c87313}). ME=[192.168.33.11:2113,{843c5aa9-0910-4d75-9747-251d784ce634}](L=36847988,W=36865255,C=36865255,E54@36865031:{ad3728ee-29b2-49c3-8908-c9f564c87313}). [PID:10368:015 2015.07.04 13:26:09.734 DEBUG EpochManager ] === Update Last Epoch E55@36865255:{48a0c614-9b8f-4b70-8e27-4ec9c1589d26} (previous epoch at 36865031). [PID:10368:010 2015.07.04 13:26:10.384 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.33.12:2113]) Old: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:26:08.709 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36847988/36865255/36865255/E54@36865031:{ad3728ee-29b2-49c3-8908-c9f564c87313} | 2015-07-04 13:26:03.548 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36847988/36865479/36865479/E55@36865255:{48a0c614-9b8f-4b70-8e27-4ec9c1589d26} | 2015-07-04 13:26:10.378 New: VND {6bba87a7-5f19-443e-b978-c86c62e4973c} [PreReplica, 192.168.33.13:1111, n/a, 192.168.33.13:1112, n/a, 192.168.33.13:2113, 192.168.33.13:2114] 36778896/36796419/36796419/E52@36796195:{ad4d768f-a5e8-418e-b09f-95bba05df14f} | 2015-07-04 13:26:08.709 VND {fe562867-3182-4fcd-a3e5-6bb9c60441e8} [Master, 192.168.33.12:1111, n/a, 192.168.33.12:1112, n/a, 192.168.33.12:2113, 192.168.33.12:2114] 36847988/36865479/36865479/E55@36865255:{48a0c614-9b8f-4b70-8e27-4ec9c1589d26} | 2015-07-04 13:26:10.392 VND {843c5aa9-0910-4d75-9747-251d784ce634} [Slave, 192.168.33.11:1111, n/a, 192.168.33.11:1112, n/a, 192.168.33.11:2113, 192.168.33.11:2114] 36847988/36865479/36865479/E55@36865255:{48a0c614-9b8f-4b70-8e27-4ec9c1589d26} | 2015-07-04 13:26:10.383 -------------------------------------------------------------------------------- [PID:10368:117 2015.07.04 13:26:28.200 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:10368:114 2015.07.04 13:26:29.201 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:10368:117 2015.07.04 13:26:29.670 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:10368:114 2015.07.04 13:26:29.671 DEBUG ElectController ] Error while reading request: EndRead failure. [PID:10368:117 2015.07.04 13:26:35.207 DEBUG GossipController ] Error while reading request (gossip): EndRead failure [PID:10368:117 2015.07.04 13:26:35.673 DEBUG GossipController ] Error while reading request (gossip): EndRead failure [PID:10368:117 2015.07.04 13:26:36.318 DEBUG ElectController ] Error while reading request: EndRead failure.