[PID:04255:001 2018.12.12 21:12:53.474 INFO ProgramBase`1 ] ES VERSION: 4.1.1.0 (release-ha-v4.1.1/ab47b09405f57c0dbb9fd567de276ad886d6cc2c, Mon, 7 May 2018 13:53:35 +0400) [PID:04255:001 2018.12.12 21:12:53.573 INFO ProgramBase`1 ] OS: Linux (Unix 4.15.0.1035) [PID:04255:001 2018.12.12 21:12:53.583 INFO ProgramBase`1 ] RUNTIME: 4.6.2 (Stable 4.6.2.16/ac9e222) (64-bit) [PID:04255:001 2018.12.12 21:12:53.583 INFO ProgramBase`1 ] GC: 2 GENERATIONS [PID:04255:001 2018.12.12 21:12:53.583 INFO ProgramBase`1 ] LOGS: /datadrive0/log/eventstore [PID:04255:001 2018.12.12 21:12:53.604 INFO ProgramBase`1 ] MODIFIED OPTIONS: CONFIG: /etc/eventstore/eventstore0.conf (Command Line) START STANDARD PROJECTIONS: True (Config File) RUN PROJECTIONS: ALL (Config File) DB: /datadrive0/db/eventstore (Config File) LOG: /datadrive0/log/eventstore (Config File) STATS PERIOD SEC: 3600 (Config File) MAX MEM TABLE SIZE: 500000 (Config File) INT IP: 192.168.2.4 (Config File) EXT IP: 192.168.2.4 (Config File) INT HTTP PORT: 1113 (Config File) EXT HTTP PORT: 1114 (Config File) INT TCP PORT: 1111 (Config File) EXT TCP PORT: 1112 (Config File) INT HTTP PREFIXES: http://*:1113/ (Config File) EXT HTTP PREFIXES: http://*:1114/ (Config File) ADD INTERFACE PREFIXES: false (Config File) DISCOVER VIA DNS: false (Config File) PREPARE TIMEOUT MS: 7500 (Config File) COMMIT TIMEOUT MS: 7500 (Config File) SKIP DB VERIFY: true (Config File) SKIP INDEX VERIFY: true (Config File) EXT TCP HEARTBEAT TIMEOUT: 6000 (Config File) EXT TCP HEARTBEAT INTERVAL: 3000 (Config File) GOSSIP INTERVAL MS: 1000 (Config File) GOSSIP TIMEOUT MS: 2000 (Config File) GOSSIP SEED: 192.168.2.4:2113,192.168.2.4:3113 (Config File) CLUSTER SIZE: 3 (Config File) DEFAULT OPTIONS: HELP: False () VERSION: False () DEFINES: () WHAT IF: False () DISABLE HTTP CACHING: False () MONO MIN THREADPOOL SIZE: 10 () INT SECURE TCP PORT: 0 () EXT SECURE TCP PORT ADVERTISE AS: 0 () EXT SECURE TCP PORT: 0 () EXT IP ADVERTISE AS: () EXT TCP PORT ADVERTISE AS: 0 () EXT HTTP PORT ADVERTISE AS: 0 () INT IP ADVERTISE AS: () INT SECURE TCP PORT ADVERTISE AS: 0 () INT TCP PORT ADVERTISE AS: 0 () INT HTTP PORT ADVERTISE AS: 0 () INT TCP HEARTBEAT TIMEOUT: 700 () INT TCP HEARTBEAT INTERVAL: 700 () GOSSIP ON SINGLE NODE: False () CONNECTION PENDING SEND BYTES THRESHOLD: 10485760 () FORCE: False () NODE PRIORITY: 0 () MIN FLUSH DELAY MS: 2 () COMMIT COUNT: -1 () PREPARE COUNT: -1 () ADMIN ON EXT: True () STATS ON EXT: True () GOSSIP ON EXT: True () DISABLE SCAVENGE MERGING: False () SCAVENGE HISTORY MAX AGE: 30 () CLUSTER DNS: fake.dns () CLUSTER GOSSIP PORT: 30777 () CACHED CHUNKS: -1 () READER THREADS COUNT: 4 () CHUNKS CACHE SIZE: 536871424 () HASH COLLISION READ LIMIT: 100 () INDEX: () MEM DB: False () WRITE THROUGH: False () UNBUFFERED: False () CHUNK INITIAL READER COUNT: 5 () PROJECTION THREADS: 3 () WORKER THREADS: 5 () PROJECTIONS QUERY EXPIRY: 0 () ENABLE TRUSTED AUTH: False () CERTIFICATE STORE LOCATION: () CERTIFICATE STORE NAME: () CERTIFICATE SUBJECT NAME: () CERTIFICATE THUMBPRINT: () CERTIFICATE FILE: () CERTIFICATE PASSWORD: () USE INTERNAL SSL: False () DISABLE INSECURE TCP: False () SSL TARGET HOST: n/a () SSL VALIDATE SERVER: True () AUTHENTICATION TYPE: internal () AUTHENTICATION CONFIG: () UNSAFE DISABLE FLUSH TO DISK: False () BETTER ORDERING: False () UNSAFE IGNORE HARD DELETE: False () INDEX CACHE DEPTH: 16 () OPTIMIZE INDEX MERGE: False () GOSSIP ALLOWED DIFFERENCE MS: 60000 () ENABLE HISTOGRAMS: False () LOG HTTP REQUESTS: False () ALWAYS KEEP SCAVENGED: False () SKIP INDEX SCAN ON READS: False () [PID:04255:001 2018.12.12 21:12:53.624 INFO ProgramBase`1 ] Quorum size set to 2 [PID:04255:001 2018.12.12 21:12:53.649 INFO ProgramBase`1 ] Cannot find plugins path: plugins [PID:04255:001 2018.12.12 21:12:58.703 INFO VNodeBuilder ] INSTANCE ID: fafb112d-9dd8-40dd-bb2b-8676b4a329de [PID:04255:001 2018.12.12 21:12:58.706 INFO VNodeBuilder ] DATABASE: /datadrive0/db/eventstore [PID:04255:001 2018.12.12 21:12:58.706 INFO VNodeBuilder ] WRITER CHECKPOINT: 0 (0x0) [PID:04255:001 2018.12.12 21:12:58.706 INFO VNodeBuilder ] CHASER CHECKPOINT: 0 (0x0) [PID:04255:001 2018.12.12 21:12:58.707 INFO VNodeBuilder ] EPOCH CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF) [PID:04255:001 2018.12.12 21:12:58.707 INFO VNodeBuilder ] TRUNCATE CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF) [PID:04255:001 2018.12.12 21:12:59.174 TRACE MessageHierarchy ] MessageHierarchy initialization took 00:00:00.3832914. [PID:04255:001 2018.12.12 21:13:01.012 TRACE TFChunk ] CACHED TFChunk #0-0 (chunk-000000.000000) in 00:00:00.0025444. [PID:04255:001 2018.12.12 21:13:01.138 INFO MiniWeb ] Starting MiniWeb for /web/es/js/projections ==> /usr/share/eventstore/projections [PID:04255:001 2018.12.12 21:13:01.138 INFO MiniWeb ] Starting MiniWeb for /web/es/js/projections/v8/Prelude ==> /usr/share/eventstore/Prelude [PID:04255:001 2018.12.12 21:13:01.139 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/{*remaining_path} [PID:04255:001 2018.12.12 21:13:01.139 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/v8/Prelude/{*remaining_path} [PID:04255:001 2018.12.12 21:13:01.140 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/{*remaining_path} [PID:04255:001 2018.12.12 21:13:01.140 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/v8/Prelude/{*remaining_path} [PID:04255:001 2018.12.12 21:13:01.150 INFO MiniWeb ] Starting MiniWeb for /web ==> /usr/share/eventstore/clusternode-web [PID:04255:001 2018.12.12 21:13:01.150 TRACE MiniWeb ] Binding MiniWeb to /web/{*remaining_path} [PID:04255:001 2018.12.12 21:13:01.150 INFO MiniWeb ] Starting MiniWeb for /web ==> /usr/share/eventstore/clusternode-web [PID:04255:001 2018.12.12 21:13:01.150 TRACE MiniWeb ] Binding MiniWeb to /web/{*remaining_path} [PID:04255:012 2018.12.12 21:13:01.170 INFO ClusterVNodeControll] ========== [192.168.2.4:1113] SYSTEM INIT... [PID:04255:012 2018.12.12 21:13:01.192 INFO TcpServerListener ] Starting Normal TCP listening on TCP endpoint: 192.168.2.4:1112. [PID:04255:012 2018.12.12 21:13:01.199 INFO TcpServerListener ] Starting Normal TCP listening on TCP endpoint: 192.168.2.4:1111. [PID:04255:013 2018.12.12 21:13:01.211 INFO IndexCommitter ] TableIndex initialization... [PID:04255:013 2018.12.12 21:13:01.221 INFO IndexCommitter ] ReadIndex building... [PID:04255:013 2018.12.12 21:13:01.222 DEBUG IndexCommitter ] ReadIndex rebuilding done: total processed 0 records, time elapsed: 00:00:00.0014280. [PID:04255:012 2018.12.12 21:13:01.239 INFO HttpAsyncServer ] Starting HTTP server on [http://*:1114/]... [PID:04255:012 2018.12.12 21:13:01.242 INFO HttpAsyncServer ] HTTP server is up and listening on [http://*:1114/] [PID:04255:012 2018.12.12 21:13:01.242 INFO HttpAsyncServer ] Starting HTTP server on [http://*:1113/]... [PID:04255:012 2018.12.12 21:13:01.242 INFO HttpAsyncServer ] HTTP server is up and listening on [http://*:1113/] [PID:04255:012 2018.12.12 21:13:01.258 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [MainQueue]: SystemInit - 92ms. Q: 0/8. [PID:04255:012 2018.12.12 21:13:01.258 INFO ClusterVNodeControll] ========== [192.168.2.4:1113] Service 'StorageWriter' initialized. [PID:04255:012 2018.12.12 21:13:01.258 INFO ClusterVNodeControll] ========== [192.168.2.4:1113] Service 'StorageReader' initialized. [PID:04255:012 2018.12.12 21:13:01.260 INFO ClusterVNodeControll] ========== [192.168.2.4:1113] Service 'StorageChaser' initialized. [PID:04255:012 2018.12.12 21:13:01.279 TRACE GossipServiceBase ] CLUSTER HAS CHANGED [PID:04255:012 2018.12.12 21:13:01.279 TRACE GossipServiceBase ] Old: [PID:04255:012 2018.12.12 21:13:01.280 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [Unknown, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:01.244 [PID:04255:012 2018.12.12 21:13:01.280 TRACE GossipServiceBase ] New: [PID:04255:012 2018.12.12 21:13:01.280 TRACE GossipServiceBase ] MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.2.4:3113, 192.168.2.4:3113] | 2018-12-12 21:13:01.261 [PID:04255:012 2018.12.12 21:13:01.280 TRACE GossipServiceBase ] MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.2.4:2113, 192.168.2.4:2113] | 2018-12-12 21:13:01.261 [PID:04255:012 2018.12.12 21:13:01.280 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [Unknown, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:01.244 [PID:04255:012 2018.12.12 21:13:01.280 TRACE GossipServiceBase ] -------------------------------------------------------------------------------- [PID:04255:012 2018.12.12 21:13:01.283 INFO ClusterVNodeControll] ========== [192.168.2.4:1113] SYSTEM START... [PID:04255:012 2018.12.12 21:13:01.288 INFO ClusterVNodeControll] ========== [192.168.2.4:1113] IS UNKNOWN... [PID:04255:006 2018.12.12 21:13:01.330 DEBUG PersistentSubscripti] Subscriptions received state change to Unknown stopping listening. [PID:04255:012 2018.12.12 21:13:01.363 DEBUG ElectionsService ] ELECTIONS: STARTING ELECTIONS. [PID:04255:012 2018.12.12 21:13:01.363 DEBUG ElectionsService ] ELECTIONS: (V=0) SHIFT TO LEADER ELECTION. [PID:04255:012 2018.12.12 21:13:01.363 DEBUG ElectionsService ] ELECTIONS: (V=0) VIEWCHANGE FROM [192.168.2.4:1113, {fafb112d-9dd8-40dd-bb2b-8676b4a329de}]. [PID:04255:012 2018.12.12 21:13:01.396 TRACE InMemoryBus ] SLOW BUS MSG [MainBus]: StartElections - 64ms. Handler: ElectionsService. [PID:04255:012 2018.12.12 21:13:01.396 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [MainQueue]: StartElections - 96ms. Q: 0/3. [PID:04255:008 2018.12.12 21:13:01.532 TRACE InMemoryBus ] SLOW BUS MSG [Worker #1 Bus]: SendOverHttp - 236ms. Handler: HttpSendService. [PID:04255:012 2018.12.12 21:13:01.559 TRACE GossipServiceBase ] Looks like node [192.168.2.4:3113] is DEAD (Gossip send failed). [PID:04255:012 2018.12.12 21:13:01.560 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip send failed to [192.168.2.4:3113]) [PID:04255:012 2018.12.12 21:13:01.560 TRACE GossipServiceBase ] Old: [PID:04255:012 2018.12.12 21:13:01.560 TRACE GossipServiceBase ] MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.2.4:3113, 192.168.2.4:3113] | 2018-12-12 21:13:01.261 [PID:04255:012 2018.12.12 21:13:01.561 TRACE GossipServiceBase ] MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.2.4:2113, 192.168.2.4:2113] | 2018-12-12 21:13:01.261 [PID:04255:012 2018.12.12 21:13:01.561 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [Unknown, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:01.505 [PID:04255:012 2018.12.12 21:13:01.561 TRACE GossipServiceBase ] New: [PID:04255:012 2018.12.12 21:13:01.562 TRACE GossipServiceBase ] MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.2.4:3113, 192.168.2.4:3113] | 2018-12-12 21:13:01.560 [PID:04255:012 2018.12.12 21:13:01.562 TRACE GossipServiceBase ] MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.2.4:2113, 192.168.2.4:2113] | 2018-12-12 21:13:01.261 [PID:04255:012 2018.12.12 21:13:01.562 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [Unknown, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:01.505 [PID:04255:012 2018.12.12 21:13:01.562 TRACE GossipServiceBase ] -------------------------------------------------------------------------------- [PID:04255:012 2018.12.12 21:13:01.562 TRACE GossipServiceBase ] Looks like node [192.168.2.4:2113] is DEAD (Gossip send failed). [PID:04255:012 2018.12.12 21:13:01.563 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip send failed to [192.168.2.4:2113]) [PID:04255:012 2018.12.12 21:13:01.563 TRACE GossipServiceBase ] Old: [PID:04255:012 2018.12.12 21:13:01.563 TRACE GossipServiceBase ] MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.2.4:3113, 192.168.2.4:3113] | 2018-12-12 21:13:01.560 [PID:04255:012 2018.12.12 21:13:01.563 TRACE GossipServiceBase ] MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.2.4:2113, 192.168.2.4:2113] | 2018-12-12 21:13:01.261 [PID:04255:012 2018.12.12 21:13:01.564 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [Unknown, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:01.505 [PID:04255:012 2018.12.12 21:13:01.564 TRACE GossipServiceBase ] New: [PID:04255:012 2018.12.12 21:13:01.564 TRACE GossipServiceBase ] MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.2.4:3113, 192.168.2.4:3113] | 2018-12-12 21:13:01.560 [PID:04255:012 2018.12.12 21:13:01.564 TRACE GossipServiceBase ] MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.2.4:2113, 192.168.2.4:2113] | 2018-12-12 21:13:01.563 [PID:04255:012 2018.12.12 21:13:01.565 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [Unknown, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:01.505 [PID:04255:012 2018.12.12 21:13:01.565 TRACE GossipServiceBase ] -------------------------------------------------------------------------------- [PID:04255:008 2018.12.12 21:13:01.569 TRACE QueuedHandlerThreadP] SLOW QUEUE MSG [Worker #1]: SendOverHttp - 237ms. Q: 0/0. [PID:04255:012 2018.12.12 21:13:02.405 DEBUG ElectionsService ] ELECTIONS: (V=0) TIMED OUT! (S=ElectingLeader, M=). [PID:04255:012 2018.12.12 21:13:02.405 DEBUG ElectionsService ] ELECTIONS: (V=1) SHIFT TO LEADER ELECTION. [PID:04255:012 2018.12.12 21:13:02.405 DEBUG ElectionsService ] ELECTIONS: (V=1) VIEWCHANGE FROM [192.168.2.4:1113, {fafb112d-9dd8-40dd-bb2b-8676b4a329de}]. [PID:04255:012 2018.12.12 21:13:03.408 DEBUG ElectionsService ] ELECTIONS: (V=1) TIMED OUT! (S=ElectingLeader, M=). [PID:04255:012 2018.12.12 21:13:03.408 DEBUG ElectionsService ] ELECTIONS: (V=2) SHIFT TO LEADER ELECTION. [PID:04255:012 2018.12.12 21:13:03.408 DEBUG ElectionsService ] ELECTIONS: (V=2) VIEWCHANGE FROM [192.168.2.4:1113, {fafb112d-9dd8-40dd-bb2b-8676b4a329de}]. [PID:04255:012 2018.12.12 21:13:04.416 DEBUG ElectionsService ] ELECTIONS: (V=2) TIMED OUT! (S=ElectingLeader, M=). [PID:04255:012 2018.12.12 21:13:04.416 DEBUG ElectionsService ] ELECTIONS: (V=3) SHIFT TO LEADER ELECTION. [PID:04255:012 2018.12.12 21:13:04.416 DEBUG ElectionsService ] ELECTIONS: (V=3) VIEWCHANGE FROM [192.168.2.4:1113, {fafb112d-9dd8-40dd-bb2b-8676b4a329de}]. [PID:04255:018 2018.12.12 21:13:04.590 DEBUG BufferManager ] Segments count: 1, buffers count: 20, should be when full: 20 [PID:04255:012 2018.12.12 21:13:04.913 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.2.4:2113]) [PID:04255:012 2018.12.12 21:13:04.925 TRACE GossipServiceBase ] Old: [PID:04255:012 2018.12.12 21:13:04.926 TRACE GossipServiceBase ] MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.2.4:3113, 192.168.2.4:3113] | 2018-12-12 21:13:01.560 [PID:04255:012 2018.12.12 21:13:04.934 TRACE GossipServiceBase ] MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.2.4:2113, 192.168.2.4:2113] | 2018-12-12 21:13:01.563 [PID:04255:012 2018.12.12 21:13:04.934 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [Unknown, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:04.332 [PID:04255:012 2018.12.12 21:13:04.944 TRACE GossipServiceBase ] New: [PID:04255:012 2018.12.12 21:13:04.944 TRACE GossipServiceBase ] MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.2.4:3113, 192.168.2.4:3113] | 2018-12-12 21:13:03.879 [PID:04255:012 2018.12.12 21:13:04.946 TRACE GossipServiceBase ] VND {e6f6caf5-4888-4852-bf6d-725d94802897} [Unknown, 192.168.2.4:2111, n/a, 192.168.2.4:2112, n/a, 192.168.2.4:2113, 192.168.2.4:2114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:04.807 [PID:04255:012 2018.12.12 21:13:04.950 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [Unknown, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:04.885 [PID:04255:012 2018.12.12 21:13:04.950 TRACE GossipServiceBase ] -------------------------------------------------------------------------------- [PID:04255:012 2018.12.12 21:13:04.956 TRACE InMemoryBus ] SLOW BUS MSG [MainBus]: GossipReceived - 71ms. Handler: NodeGossipService. [PID:04255:012 2018.12.12 21:13:04.962 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [MainQueue]: GossipReceived - 77ms. Q: 0/10. [PID:04255:012 2018.12.12 21:13:04.962 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.2.4:3113]) [PID:04255:012 2018.12.12 21:13:04.968 TRACE GossipServiceBase ] Old: [PID:04255:012 2018.12.12 21:13:04.968 TRACE GossipServiceBase ] MAN {00000000-0000-0000-0000-000000000000} [Manager, 192.168.2.4:3113, 192.168.2.4:3113] | 2018-12-12 21:13:03.879 [PID:04255:012 2018.12.12 21:13:04.970 TRACE GossipServiceBase ] VND {e6f6caf5-4888-4852-bf6d-725d94802897} [Unknown, 192.168.2.4:2111, n/a, 192.168.2.4:2112, n/a, 192.168.2.4:2113, 192.168.2.4:2114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:04.591 [PID:04255:012 2018.12.12 21:13:04.970 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [Unknown, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:04.963 [PID:04255:012 2018.12.12 21:13:04.970 TRACE GossipServiceBase ] New: [PID:04255:012 2018.12.12 21:13:04.974 TRACE GossipServiceBase ] VND {15230862-8531-4aeb-893f-68528714e7c3} [Initializing, 192.168.2.4:3111, n/a, 192.168.2.4:3112, n/a, 192.168.2.4:3113, 192.168.2.4:3114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:03.923 [PID:04255:012 2018.12.12 21:13:04.979 TRACE GossipServiceBase ] VND {e6f6caf5-4888-4852-bf6d-725d94802897} [Unknown, 192.168.2.4:2111, n/a, 192.168.2.4:2112, n/a, 192.168.2.4:2113, 192.168.2.4:2114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:04.591 [PID:04255:012 2018.12.12 21:13:04.979 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [Unknown, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:04.963 [PID:04255:012 2018.12.12 21:13:04.982 TRACE GossipServiceBase ] -------------------------------------------------------------------------------- [PID:04255:012 2018.12.12 21:13:04.982 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.2.4:3113]) [PID:04255:012 2018.12.12 21:13:04.988 TRACE GossipServiceBase ] Old: [PID:04255:012 2018.12.12 21:13:04.988 TRACE GossipServiceBase ] VND {15230862-8531-4aeb-893f-68528714e7c3} [Initializing, 192.168.2.4:3111, n/a, 192.168.2.4:3112, n/a, 192.168.2.4:3113, 192.168.2.4:3114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:03.923 [PID:04255:012 2018.12.12 21:13:04.988 TRACE GossipServiceBase ] VND {e6f6caf5-4888-4852-bf6d-725d94802897} [Unknown, 192.168.2.4:2111, n/a, 192.168.2.4:2112, n/a, 192.168.2.4:2113, 192.168.2.4:2114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:04.280 [PID:04255:012 2018.12.12 21:13:04.988 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [Unknown, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:04.982 [PID:04255:012 2018.12.12 21:13:04.989 TRACE GossipServiceBase ] New: [PID:04255:012 2018.12.12 21:13:04.989 TRACE GossipServiceBase ] VND {15230862-8531-4aeb-893f-68528714e7c3} [Unknown, 192.168.2.4:3111, n/a, 192.168.2.4:3112, n/a, 192.168.2.4:3113, 192.168.2.4:3114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:04.144 [PID:04255:012 2018.12.12 21:13:04.991 TRACE GossipServiceBase ] VND {e6f6caf5-4888-4852-bf6d-725d94802897} [Unknown, 192.168.2.4:2111, n/a, 192.168.2.4:2112, n/a, 192.168.2.4:2113, 192.168.2.4:2114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:04.280 [PID:04255:012 2018.12.12 21:13:04.992 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [Unknown, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:04.983 [PID:04255:012 2018.12.12 21:13:04.999 TRACE GossipServiceBase ] -------------------------------------------------------------------------------- [PID:04255:012 2018.12.12 21:13:05.109 DEBUG ElectionsService ] ELECTIONS: (V=1) VIEWCHANGE FROM [192.168.2.4:2113, {e6f6caf5-4888-4852-bf6d-725d94802897}]. [PID:04255:012 2018.12.12 21:13:05.112 DEBUG ElectionsService ] ELECTIONS: (V=1) MAJORITY OF VIEWCHANGE. [PID:04255:012 2018.12.12 21:13:05.420 DEBUG ElectionsService ] ELECTIONS: (V=3) TIMED OUT! (S=ElectingLeader, M=). [PID:04255:012 2018.12.12 21:13:05.420 DEBUG ElectionsService ] ELECTIONS: (V=4) SHIFT TO LEADER ELECTION. [PID:04255:012 2018.12.12 21:13:05.420 DEBUG ElectionsService ] ELECTIONS: (V=4) VIEWCHANGE FROM [192.168.2.4:1113, {fafb112d-9dd8-40dd-bb2b-8676b4a329de}]. [PID:04255:012 2018.12.12 21:13:05.443 DEBUG ElectionsService ] ELECTIONS: (V=4) VIEWCHANGE FROM [192.168.2.4:3113, {15230862-8531-4aeb-893f-68528714e7c3}]. [PID:04255:012 2018.12.12 21:13:05.443 DEBUG ElectionsService ] ELECTIONS: (V=4) MAJORITY OF VIEWCHANGE. [PID:04255:012 2018.12.12 21:13:05.446 DEBUG ElectionsService ] ELECTIONS: (V=4) VIEWCHANGE FROM [192.168.2.4:2113, {e6f6caf5-4888-4852-bf6d-725d94802897}]. [PID:04255:012 2018.12.12 21:13:05.461 DEBUG ElectionsService ] ELECTIONS: (V=4) PREPARE FROM [192.168.2.4:2113, {e6f6caf5-4888-4852-bf6d-725d94802897}]. [PID:04255:012 2018.12.12 21:13:05.461 DEBUG ElectionsService ] ELECTIONS: (V=4) SHIFT TO REG_NONLEADER. [PID:04255:012 2018.12.12 21:13:05.495 DEBUG ElectionsService ] ELECTIONS: (V=4) PROPOSAL FROM [192.168.2.4:2113,{e6f6caf5-4888-4852-bf6d-725d94802897}] M=[192.168.2.4:2113,{e6f6caf5-4888-4852-bf6d-725d94802897}](L=-1,W=0,C=0,E-1@-1:{00000000-0000-0000-0000-000000000000}). ME=[192.168.2.4:1113,{fafb112d-9dd8-40dd-bb2b-8676b4a329de}](L=-1,W=0,C=0,E-1@-1:{00000000-0000-0000-0000-000000000000}). [PID:04255:012 2018.12.12 21:13:05.496 DEBUG ElectionsService ] ELECTIONS: (V=4) ACCEPT FROM [192.168.2.4:2113,{e6f6caf5-4888-4852-bf6d-725d94802897}] M=[192.168.2.4:2113,{e6f6caf5-4888-4852-bf6d-725d94802897}]). [PID:04255:012 2018.12.12 21:13:05.496 DEBUG ElectionsService ] ELECTIONS: (V=4) ACCEPT FROM [192.168.2.4:1113,{fafb112d-9dd8-40dd-bb2b-8676b4a329de}] M=[192.168.2.4:2113,{e6f6caf5-4888-4852-bf6d-725d94802897}]). [PID:04255:012 2018.12.12 21:13:05.496 INFO ElectionsService ] ELECTIONS: (V=4) DONE. ELECTED MASTER = [192.168.2.4:2113,{e6f6caf5-4888-4852-bf6d-725d94802897}](L=-1,W=0,C=0,E-1@-1:{00000000-0000-0000-0000-000000000000}). ME=[192.168.2.4:1113,{fafb112d-9dd8-40dd-bb2b-8676b4a329de}](L=-1,W=0,C=0,E-1@-1:{00000000-0000-0000-0000-000000000000}). [PID:04255:012 2018.12.12 21:13:05.500 INFO ClusterVNodeControll] ========== [192.168.2.4:1113] PRE-REPLICA STATE, WAITING FOR CHASER TO CATCH UP... MASTER IS [192.168.2.4:2113,{e6f6caf5-4888-4852-bf6d-725d94802897}] [PID:04255:023 2018.12.12 21:13:05.500 DEBUG PersistentSubscripti] Subscriptions received state change to PreReplica stopping listening. [PID:04255:012 2018.12.12 21:13:05.503 DEBUG BufferManager ] Segments count: 1, buffers count: 512, should be when full: 512 [PID:04255:012 2018.12.12 21:13:05.549 TRACE InMemoryBus ] SLOW BUS MSG [MainBus]: BecomePreReplica - 48ms. Handler: ReplicaService. [PID:04255:012 2018.12.12 21:13:05.549 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [MainQueue]: ElectionsDone - 50ms. Q: 2/4. [PID:04255:006 2018.12.12 21:13:05.646 INFO TcpConnectionManager] Connection 'master-normal' ({7671915a-593f-48a9-9aec-80994115d775}) to [192.168.2.4:2111] established. [PID:04255:012 2018.12.12 21:13:05.683 INFO ReplicaService ] Subscribing at LogPosition: 0 (0x0) to MASTER [192.168.2.4:2111, {e6f6caf5-4888-4852-bf6d-725d94802897}] as replica with SubscriptionId: {34863166-9d35-4ab3-bf32-24b988ae188d}, ConnectionId: {7671915a-593f-48a9-9aec-80994115d775}, LocalEndPoint: [192.168.2.4:37254], Epochs: ... . [PID:04255:012 2018.12.12 21:13:05.699 TRACE InMemoryBus ] SLOW BUS MSG [MainBus]: SubscribeToMaster - 134ms. Handler: ReplicaService. [PID:04255:012 2018.12.12 21:13:05.704 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [MainQueue]: ChaserCaughtUp - 139ms. Q: 0/6. [PID:04255:012 2018.12.12 21:13:05.704 DEBUG ElectionsService ] ELECTIONS: (V=4) ACCEPT FROM [192.168.2.4:3113,{15230862-8531-4aeb-893f-68528714e7c3}] M=[192.168.2.4:2113,{e6f6caf5-4888-4852-bf6d-725d94802897}]). [PID:04255:012 2018.12.12 21:13:05.706 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.2.4:3113]) [PID:04255:012 2018.12.12 21:13:05.715 TRACE GossipServiceBase ] Old: [PID:04255:012 2018.12.12 21:13:05.715 TRACE GossipServiceBase ] VND {15230862-8531-4aeb-893f-68528714e7c3} [Unknown, 192.168.2.4:3111, n/a, 192.168.2.4:3112, n/a, 192.168.2.4:3113, 192.168.2.4:3114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:05.303 [PID:04255:012 2018.12.12 21:13:05.715 TRACE GossipServiceBase ] VND {e6f6caf5-4888-4852-bf6d-725d94802897} [Unknown, 192.168.2.4:2111, n/a, 192.168.2.4:2112, n/a, 192.168.2.4:2113, 192.168.2.4:2114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:05.341 [PID:04255:012 2018.12.12 21:13:05.715 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [PreReplica, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:05.549 [PID:04255:012 2018.12.12 21:13:05.716 TRACE GossipServiceBase ] New: [PID:04255:012 2018.12.12 21:13:05.716 TRACE GossipServiceBase ] VND {15230862-8531-4aeb-893f-68528714e7c3} [PreReplica, 192.168.2.4:3111, n/a, 192.168.2.4:3112, n/a, 192.168.2.4:3113, 192.168.2.4:3114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:05.569 [PID:04255:012 2018.12.12 21:13:05.717 TRACE GossipServiceBase ] VND {e6f6caf5-4888-4852-bf6d-725d94802897} [Unknown, 192.168.2.4:2111, n/a, 192.168.2.4:2112, n/a, 192.168.2.4:2113, 192.168.2.4:2114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:05.706 [PID:04255:012 2018.12.12 21:13:05.720 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [PreReplica, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:05.706 [PID:04255:012 2018.12.12 21:13:05.720 TRACE GossipServiceBase ] -------------------------------------------------------------------------------- [PID:04255:025 2018.12.12 21:13:05.890 TRACE InMemoryBus ] SLOW BUS MSG [Worker #4 Bus]: TcpSend - 204ms. Handler: TcpSendService. [PID:04255:025 2018.12.12 21:13:05.894 TRACE QueuedHandlerThreadP] SLOW QUEUE MSG [Worker #4]: TcpSend - 208ms. Q: 0/0. [PID:04255:012 2018.12.12 21:13:06.020 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.2.4:2113]) [PID:04255:012 2018.12.12 21:13:06.020 TRACE GossipServiceBase ] Old: [PID:04255:012 2018.12.12 21:13:06.020 TRACE GossipServiceBase ] VND {15230862-8531-4aeb-893f-68528714e7c3} [PreReplica, 192.168.2.4:3111, n/a, 192.168.2.4:3112, n/a, 192.168.2.4:3113, 192.168.2.4:3114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:05.850 [PID:04255:012 2018.12.12 21:13:06.020 TRACE GossipServiceBase ] VND {e6f6caf5-4888-4852-bf6d-725d94802897} [Unknown, 192.168.2.4:2111, n/a, 192.168.2.4:2112, n/a, 192.168.2.4:2113, 192.168.2.4:2114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:05.923 [PID:04255:012 2018.12.12 21:13:06.020 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [PreReplica, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:05.923 [PID:04255:012 2018.12.12 21:13:06.021 TRACE GossipServiceBase ] New: [PID:04255:012 2018.12.12 21:13:06.021 TRACE GossipServiceBase ] VND {15230862-8531-4aeb-893f-68528714e7c3} [PreReplica, 192.168.2.4:3111, n/a, 192.168.2.4:3112, n/a, 192.168.2.4:3113, 192.168.2.4:3114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:05.850 [PID:04255:012 2018.12.12 21:13:06.021 TRACE GossipServiceBase ] VND {e6f6caf5-4888-4852-bf6d-725d94802897} [Master, 192.168.2.4:2111, n/a, 192.168.2.4:2112, n/a, 192.168.2.4:2113, 192.168.2.4:2114] -1/168/0/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:06.000 [PID:04255:012 2018.12.12 21:13:06.022 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [PreReplica, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:06.020 [PID:04255:012 2018.12.12 21:13:06.022 TRACE GossipServiceBase ] -------------------------------------------------------------------------------- [PID:04255:012 2018.12.12 21:13:06.139 INFO ClusterVNodeControll] ========== [192.168.2.4:1113] IS CATCHING UP... MASTER IS [192.168.2.4:2113,{e6f6caf5-4888-4852-bf6d-725d94802897}] [PID:04255:017 2018.12.12 21:13:06.148 DEBUG PersistentSubscripti] Subscriptions received state change to CatchingUp stopping listening. [PID:04255:009 2018.12.12 21:13:06.240 INFO ClusterStorageWriter] === SUBSCRIBED to [192.168.2.4:2111,{e6f6caf5-4888-4852-bf6d-725d94802897}] at 0 (0x0). SubscriptionId: {34863166-9d35-4ab3-bf32-24b988ae188d}. [PID:04255:012 2018.12.12 21:13:06.284 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.2.4:3113]) [PID:04255:012 2018.12.12 21:13:06.284 TRACE GossipServiceBase ] Old: [PID:04255:012 2018.12.12 21:13:06.284 TRACE GossipServiceBase ] VND {15230862-8531-4aeb-893f-68528714e7c3} [PreReplica, 192.168.2.4:3111, n/a, 192.168.2.4:3112, n/a, 192.168.2.4:3113, 192.168.2.4:3114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:06.125 [PID:04255:012 2018.12.12 21:13:06.284 TRACE GossipServiceBase ] VND {e6f6caf5-4888-4852-bf6d-725d94802897} [Master, 192.168.2.4:2111, n/a, 192.168.2.4:2112, n/a, 192.168.2.4:2113, 192.168.2.4:2114] -1/644/310/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:06.125 [PID:04255:012 2018.12.12 21:13:06.284 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [CatchingUp, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:06.146 [PID:04255:012 2018.12.12 21:13:06.284 TRACE GossipServiceBase ] New: [PID:04255:012 2018.12.12 21:13:06.285 TRACE GossipServiceBase ] VND {15230862-8531-4aeb-893f-68528714e7c3} [CatchingUp, 192.168.2.4:3111, n/a, 192.168.2.4:3112, n/a, 192.168.2.4:3113, 192.168.2.4:3114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:06.282 [PID:04255:012 2018.12.12 21:13:06.285 TRACE GossipServiceBase ] VND {e6f6caf5-4888-4852-bf6d-725d94802897} [Master, 192.168.2.4:2111, n/a, 192.168.2.4:2112, n/a, 192.168.2.4:2113, 192.168.2.4:2114] -1/644/310/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:06.284 [PID:04255:012 2018.12.12 21:13:06.285 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [CatchingUp, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:06.284 [PID:04255:012 2018.12.12 21:13:06.285 TRACE GossipServiceBase ] -------------------------------------------------------------------------------- [PID:04255:012 2018.12.12 21:13:07.221 INFO ClusterVNodeControll] ========== [192.168.2.4:1113] CLONE ASSIGNMENT RECEIVED FROM [192.168.2.4:2111,n/a,{e6f6caf5-4888-4852-bf6d-725d94802897}]. [PID:04255:012 2018.12.12 21:13:07.221 INFO ClusterVNodeControll] ========== [192.168.2.4:1113] IS CLONE... MASTER IS [192.168.2.4:2113,{e6f6caf5-4888-4852-bf6d-725d94802897}] [PID:04255:016 2018.12.12 21:13:07.222 DEBUG PersistentSubscripti] Subscriptions received state change to Clone stopping listening. [PID:04255:012 2018.12.12 21:13:07.226 INFO ClusterVNodeControll] ========== [192.168.2.4:1113] SLAVE ASSIGNMENT RECEIVED FROM [192.168.2.4:2111,n/a,{e6f6caf5-4888-4852-bf6d-725d94802897}]. [PID:04255:012 2018.12.12 21:13:07.227 INFO ClusterVNodeControll] ========== [192.168.2.4:1113] IS SLAVE... MASTER IS [192.168.2.4:2113,{e6f6caf5-4888-4852-bf6d-725d94802897}] [PID:04255:019 2018.12.12 21:13:07.227 DEBUG PersistentSubscripti] Subscriptions received state change to Slave stopping listening. [PID:04255:012 2018.12.12 21:13:07.351 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.2.4:2113]) [PID:04255:012 2018.12.12 21:13:07.351 TRACE GossipServiceBase ] Old: [PID:04255:012 2018.12.12 21:13:07.352 TRACE GossipServiceBase ] VND {15230862-8531-4aeb-893f-68528714e7c3} [CatchingUp, 192.168.2.4:3111, n/a, 192.168.2.4:3112, n/a, 192.168.2.4:3113, 192.168.2.4:3114] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:06.383 [PID:04255:012 2018.12.12 21:13:07.353 TRACE GossipServiceBase ] VND {e6f6caf5-4888-4852-bf6d-725d94802897} [Master, 192.168.2.4:2111, n/a, 192.168.2.4:2112, n/a, 192.168.2.4:2113, 192.168.2.4:2114] -1/644/310/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:07.197 [PID:04255:012 2018.12.12 21:13:07.353 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [Slave, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] -1/310/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:07.349 [PID:04255:012 2018.12.12 21:13:07.353 TRACE GossipServiceBase ] New: [PID:04255:012 2018.12.12 21:13:07.354 TRACE GossipServiceBase ] VND {15230862-8531-4aeb-893f-68528714e7c3} [Slave, 192.168.2.4:3111, n/a, 192.168.2.4:3112, n/a, 192.168.2.4:3113, 192.168.2.4:3114] -1/310/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:07.300 [PID:04255:012 2018.12.12 21:13:07.354 TRACE GossipServiceBase ] VND {e6f6caf5-4888-4852-bf6d-725d94802897} [Master, 192.168.2.4:2111, n/a, 192.168.2.4:2112, n/a, 192.168.2.4:2113, 192.168.2.4:2114] -1/644/310/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:07.350 [PID:04255:012 2018.12.12 21:13:07.354 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [Slave, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] -1/310/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:07.351 [PID:04255:012 2018.12.12 21:13:07.354 TRACE GossipServiceBase ] -------------------------------------------------------------------------------- [PID:04255:023 2018.12.12 21:13:10.122 TRACE MonitoringService ] Created stats stream '$stats-192.168.2.4:1114', code = WrongExpectedVersion [PID:04255:023 2018.12.12 21:13:10.123 TRACE MonitoringService ] Created stats stream '$stats-192.168.2.4:1114', code = WrongExpectedVersion [PID:04255:009 2018.12.12 21:13:10.641 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [StorageWriterQueue]: DataChunkBulk - 3429ms. Q: 0/3. [PID:04255:013 2018.12.12 21:13:11.213 DEBUG EpochManager ] === Update Last Epoch E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} (previous epoch at -1). [PID:04255:012 2018.12.12 21:13:11.372 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.2.4:3113]) [PID:04255:012 2018.12.12 21:13:11.372 TRACE GossipServiceBase ] Old: [PID:04255:012 2018.12.12 21:13:11.374 TRACE GossipServiceBase ] VND {15230862-8531-4aeb-893f-68528714e7c3} [Slave, 192.168.2.4:3111, n/a, 192.168.2.4:3112, n/a, 192.168.2.4:3113, 192.168.2.4:3114] -1/310/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-12-12 21:13:10.422 [PID:04255:012 2018.12.12 21:13:11.374 TRACE GossipServiceBase ] VND {e6f6caf5-4888-4852-bf6d-725d94802897} [Master, 192.168.2.4:2111, n/a, 192.168.2.4:2112, n/a, 192.168.2.4:2113, 192.168.2.4:2114] -1/928/478/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:10.423 [PID:04255:012 2018.12.12 21:13:11.374 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [Slave, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] -1/644/168/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:11.368 [PID:04255:012 2018.12.12 21:13:11.374 TRACE GossipServiceBase ] New: [PID:04255:012 2018.12.12 21:13:11.374 TRACE GossipServiceBase ] VND {15230862-8531-4aeb-893f-68528714e7c3} [Slave, 192.168.2.4:3111, n/a, 192.168.2.4:3112, n/a, 192.168.2.4:3113, 192.168.2.4:3114] -1/644/168/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:11.370 [PID:04255:012 2018.12.12 21:13:11.374 TRACE GossipServiceBase ] VND {e6f6caf5-4888-4852-bf6d-725d94802897} [Master, 192.168.2.4:2111, n/a, 192.168.2.4:2112, n/a, 192.168.2.4:2113, 192.168.2.4:2114] -1/928/478/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:11.371 [PID:04255:012 2018.12.12 21:13:11.375 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [Slave, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] -1/644/168/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:11.371 [PID:04255:012 2018.12.12 21:13:11.375 TRACE GossipServiceBase ] -------------------------------------------------------------------------------- [PID:04255:009 2018.12.12 21:13:11.489 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [StorageWriterQueue]: DataChunkBulk - 847ms. Q: 0/1. [PID:04255:009 2018.12.12 21:13:12.139 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [StorageWriterQueue]: DataChunkBulk - 648ms. Q: 0/1. [PID:04255:024 2018.12.12 21:13:13.024 TRACE InMemoryBus ] SLOW BUS MSG [Worker #1 Bus]: SendOverHttp - 652ms. Handler: HttpSendService. [PID:04255:024 2018.12.12 21:13:13.024 TRACE QueuedHandlerThreadP] SLOW QUEUE MSG [Worker #1]: SendOverHttp - 652ms. Q: 0/0. [PID:04255:009 2018.12.12 21:13:13.155 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [StorageWriterQueue]: DataChunkBulk - 1015ms. Q: 0/0. [PID:04255:009 2018.12.12 21:13:14.729 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [StorageWriterQueue]: DataChunkBulk - 888ms. Q: 0/1. [PID:04255:009 2018.12.12 21:13:15.550 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [StorageWriterQueue]: DataChunkBulk - 821ms. Q: 0/0. [PID:04255:009 2018.12.12 21:13:18.790 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [StorageWriterQueue]: DataChunkBulk - 728ms. Q: 0/0. [PID:04255:009 2018.12.12 21:13:21.175 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [StorageWriterQueue]: DataChunkBulk - 1783ms. Q: 0/1. [PID:04255:009 2018.12.12 21:13:22.039 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [StorageWriterQueue]: DataChunkBulk - 863ms. Q: 0/0. [PID:04255:009 2018.12.12 21:13:25.788 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [StorageWriterQueue]: DataChunkBulk - 2946ms. Q: 0/0. [PID:04255:007 2018.12.12 21:13:30.614 INFO TcpConnection ] ES TcpConnection closed [21:13:30.595: N192.168.2.4:2111, L192.168.2.4:37254, {7671915a-593f-48a9-9aec-80994115d775}]:Received bytes: 5481, Sent bytes: 1340 [PID:04255:007 2018.12.12 21:13:30.614 INFO TcpConnection ] ES TcpConnection closed [21:13:30.614: N192.168.2.4:2111, L192.168.2.4:37254, {7671915a-593f-48a9-9aec-80994115d775}]:Send calls: 35, callbacks: 35 [PID:04255:007 2018.12.12 21:13:30.614 INFO TcpConnection ] ES TcpConnection closed [21:13:30.615: N192.168.2.4:2111, L192.168.2.4:37254, {7671915a-593f-48a9-9aec-80994115d775}]:Receive calls: 32, callbacks: 32 [PID:04255:007 2018.12.12 21:13:30.614 INFO TcpConnection ] ES TcpConnection closed [21:13:30.615: N192.168.2.4:2111, L192.168.2.4:37254, {7671915a-593f-48a9-9aec-80994115d775}]:Close reason: [Success] Socket closed [PID:04255:007 2018.12.12 21:13:30.617 INFO TcpConnectionManager] Connection 'master-normal' [192.168.2.4:2111, {7671915a-593f-48a9-9aec-80994115d775}] closed: Success. [PID:04255:012 2018.12.12 21:13:30.618 TRACE GossipServiceBase ] Looks like node [192.168.2.4:2111] is DEAD (TCP connection lost). [PID:04255:012 2018.12.12 21:13:30.618 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection lost to [192.168.2.4:2111]) [PID:04255:012 2018.12.12 21:13:30.618 TRACE GossipServiceBase ] Old: [PID:04255:012 2018.12.12 21:13:30.618 TRACE GossipServiceBase ] VND {15230862-8531-4aeb-893f-68528714e7c3} [Slave, 192.168.2.4:3111, n/a, 192.168.2.4:3112, n/a, 192.168.2.4:3113, 192.168.2.4:3114] 3617/3747/3747/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:30.542 [PID:04255:012 2018.12.12 21:13:30.618 TRACE GossipServiceBase ] VND {e6f6caf5-4888-4852-bf6d-725d94802897} [Master, 192.168.2.4:2111, n/a, 192.168.2.4:2112, n/a, 192.168.2.4:2113, 192.168.2.4:2114] 3617/4300/3880/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:30.595 [PID:04255:012 2018.12.12 21:13:30.618 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [Slave, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] 3617/3880/3747/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:30.600 [PID:04255:012 2018.12.12 21:13:30.619 TRACE GossipServiceBase ] New: [PID:04255:012 2018.12.12 21:13:30.619 TRACE GossipServiceBase ] VND {15230862-8531-4aeb-893f-68528714e7c3} [Slave, 192.168.2.4:3111, n/a, 192.168.2.4:3112, n/a, 192.168.2.4:3113, 192.168.2.4:3114] 3617/3747/3747/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:30.542 [PID:04255:012 2018.12.12 21:13:30.619 TRACE GossipServiceBase ] VND {e6f6caf5-4888-4852-bf6d-725d94802897} [Master, 192.168.2.4:2111, n/a, 192.168.2.4:2112, n/a, 192.168.2.4:2113, 192.168.2.4:2114] 3617/4300/3880/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:30.618 [PID:04255:012 2018.12.12 21:13:30.619 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [Slave, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] 3617/3880/3747/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:30.600 [PID:04255:012 2018.12.12 21:13:30.619 TRACE GossipServiceBase ] -------------------------------------------------------------------------------- [PID:04255:012 2018.12.12 21:13:30.619 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {e6f6caf5-4888-4852-bf6d-725d94802897}, InternalTcp: 192.168.2.4:2111, InternalSecureTcp: , ExternalTcp: 192.168.2.4:2112, ExternalSecureTcp: , InternalHttp: 192.168.2.4:2113, ExternalHttp: 192.168.2.4:2114]. [PID:04255:012 2018.12.12 21:13:30.620 DEBUG ElectionsService ] ELECTIONS: STARTING ELECTIONS. [PID:04255:012 2018.12.12 21:13:30.620 DEBUG ElectionsService ] ELECTIONS: (V=5) SHIFT TO LEADER ELECTION. [PID:04255:012 2018.12.12 21:13:30.620 DEBUG ElectionsService ] ELECTIONS: (V=5) VIEWCHANGE FROM [192.168.2.4:1113, {fafb112d-9dd8-40dd-bb2b-8676b4a329de}]. [PID:04255:012 2018.12.12 21:13:30.624 DEBUG ElectionsService ] ELECTIONS: (V=5) VIEWCHANGE FROM [192.168.2.4:3113, {15230862-8531-4aeb-893f-68528714e7c3}]. [PID:04255:012 2018.12.12 21:13:30.624 DEBUG ElectionsService ] ELECTIONS: (V=5) MAJORITY OF VIEWCHANGE. [PID:04255:012 2018.12.12 21:13:30.624 DEBUG ElectionsService ] ELECTIONS: (V=5) SHIFT TO PREPARE PHASE. [PID:04255:012 2018.12.12 21:13:30.626 DEBUG ElectionsService ] ELECTIONS: (V=5) PREPARE_OK FROM [192.168.2.4:1113,{fafb112d-9dd8-40dd-bb2b-8676b4a329de}](L=3617,W=3747,C=3747,E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3}). [PID:04255:012 2018.12.12 21:13:30.632 DEBUG ElectionsService ] ELECTIONS: (V=5) PREPARE_OK FROM [192.168.2.4:3113,{15230862-8531-4aeb-893f-68528714e7c3}](L=3617,W=3747,C=3747,E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3}). [PID:04255:012 2018.12.12 21:13:30.632 DEBUG ElectionsService ] ELECTIONS: (V=5) SHIFT TO REG_LEADER. [PID:04255:012 2018.12.12 21:13:30.642 DEBUG ElectionsService ] ELECTIONS: (V=5) SENDING PROPOSAL CANDIDATE: [192.168.2.4:1113,{fafb112d-9dd8-40dd-bb2b-8676b4a329de}](L=3617,W=3747,C=3747,E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3}), ME: [192.168.2.4:1113,{fafb112d-9dd8-40dd-bb2b-8676b4a329de}](L=3617,W=3747,C=3747,E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3}). [PID:04255:012 2018.12.12 21:13:30.642 DEBUG ElectionsService ] ELECTIONS: (V=5) ACCEPT FROM [192.168.2.4:1113,{fafb112d-9dd8-40dd-bb2b-8676b4a329de}] M=[192.168.2.4:1113,{fafb112d-9dd8-40dd-bb2b-8676b4a329de}]). [PID:04255:012 2018.12.12 21:13:31.129 INFO ClusterVNodeControll] ========== [192.168.2.4:1113] PRE-REPLICA STATE, WAITING FOR CHASER TO CATCH UP... MASTER IS [192.168.2.4:2113,{e6f6caf5-4888-4852-bf6d-725d94802897}] [PID:04255:022 2018.12.12 21:13:31.129 DEBUG PersistentSubscripti] Subscriptions received state change to PreReplica stopping listening. [PID:04255:012 2018.12.12 21:13:31.131 TRACE TcpConnectionManager] Closing connection 'master-normal' [192.168.2.4:2111, L192.168.2.4:37254, {7671915a-593f-48a9-9aec-80994115d775}] cleanly. Reason: Reconnecting from old master [192.168.2.4:2111] to new master: [192.168.2.4:2111]. [PID:04255:012 2018.12.12 21:13:31.131 DEBUG ClusterVNodeControll] There is NO MASTER or MASTER is DEAD according to GOSSIP. Starting new elections. MASTER: [InstanceId: {e6f6caf5-4888-4852-bf6d-725d94802897}, InternalTcp: 192.168.2.4:2111, InternalSecureTcp: , ExternalTcp: 192.168.2.4:2112, ExternalSecureTcp: , InternalHttp: 192.168.2.4:2113, ExternalHttp: 192.168.2.4:2114]. [PID:04255:018 2018.12.12 21:13:31.131 INFO TcpConnectionManager] Connection 'master-normal' ({8f869617-0227-4101-bc4a-b328b6ec1dd0}) to [192.168.2.4:2111] established. [PID:04255:012 2018.12.12 21:13:31.131 DEBUG ElectionsService ] ELECTIONS: STARTING ELECTIONS. [PID:04255:012 2018.12.12 21:13:31.132 DEBUG ElectionsService ] ELECTIONS: (V=6) SHIFT TO LEADER ELECTION. [PID:04255:012 2018.12.12 21:13:31.132 DEBUG ElectionsService ] ELECTIONS: (V=6) VIEWCHANGE FROM [192.168.2.4:1113, {fafb112d-9dd8-40dd-bb2b-8676b4a329de}]. [PID:04255:012 2018.12.12 21:13:31.132 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (TCP connection established to [192.168.2.4:2111]) [PID:04255:012 2018.12.12 21:13:31.132 TRACE GossipServiceBase ] Old: [PID:04255:012 2018.12.12 21:13:31.132 TRACE GossipServiceBase ] VND {15230862-8531-4aeb-893f-68528714e7c3} [Slave, 192.168.2.4:3111, n/a, 192.168.2.4:3112, n/a, 192.168.2.4:3113, 192.168.2.4:3114] 3617/3747/3747/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:30.542 [PID:04255:012 2018.12.12 21:13:31.133 TRACE GossipServiceBase ] VND {e6f6caf5-4888-4852-bf6d-725d94802897} [Master, 192.168.2.4:2111, n/a, 192.168.2.4:2112, n/a, 192.168.2.4:2113, 192.168.2.4:2114] 3617/4300/3880/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:30.618 [PID:04255:012 2018.12.12 21:13:31.133 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [PreReplica, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] 3617/3880/3747/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:31.132 [PID:04255:012 2018.12.12 21:13:31.133 TRACE GossipServiceBase ] New: [PID:04255:012 2018.12.12 21:13:31.133 TRACE GossipServiceBase ] VND {15230862-8531-4aeb-893f-68528714e7c3} [Slave, 192.168.2.4:3111, n/a, 192.168.2.4:3112, n/a, 192.168.2.4:3113, 192.168.2.4:3114] 3617/3747/3747/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:30.542 [PID:04255:012 2018.12.12 21:13:31.133 TRACE GossipServiceBase ] VND {e6f6caf5-4888-4852-bf6d-725d94802897} [Master, 192.168.2.4:2111, n/a, 192.168.2.4:2112, n/a, 192.168.2.4:2113, 192.168.2.4:2114] 3617/4300/3880/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:31.133 [PID:04255:012 2018.12.12 21:13:31.133 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [PreReplica, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] 3617/3880/3747/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:31.132 [PID:04255:012 2018.12.12 21:13:31.134 TRACE GossipServiceBase ] -------------------------------------------------------------------------------- [PID:04255:012 2018.12.12 21:13:31.139 DEBUG ElectionsService ] ELECTIONS: (V=6) PREPARE FROM [192.168.2.4:3113, {15230862-8531-4aeb-893f-68528714e7c3}]. [PID:04255:012 2018.12.12 21:13:31.139 DEBUG ElectionsService ] ELECTIONS: (V=6) SHIFT TO REG_NONLEADER. [PID:04255:012 2018.12.12 21:13:31.150 DEBUG ElectionsService ] ELECTIONS: (V=6) PROPOSAL FROM [192.168.2.4:3113,{15230862-8531-4aeb-893f-68528714e7c3}] M=[192.168.2.4:2113,{e6f6caf5-4888-4852-bf6d-725d94802897}](L=3617,W=3880,C=3747,E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3}). ME=[192.168.2.4:1113,{fafb112d-9dd8-40dd-bb2b-8676b4a329de}](L=3617,W=3747,C=3747,E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3}). [PID:04255:012 2018.12.12 21:13:31.150 DEBUG ElectionsService ] ELECTIONS: (V=6) ACCEPT FROM [192.168.2.4:3113,{15230862-8531-4aeb-893f-68528714e7c3}] M=[192.168.2.4:2113,{e6f6caf5-4888-4852-bf6d-725d94802897}]). [PID:04255:012 2018.12.12 21:13:31.150 DEBUG ElectionsService ] ELECTIONS: (V=6) ACCEPT FROM [192.168.2.4:1113,{fafb112d-9dd8-40dd-bb2b-8676b4a329de}] M=[192.168.2.4:2113,{e6f6caf5-4888-4852-bf6d-725d94802897}]). [PID:04255:012 2018.12.12 21:13:31.150 INFO ElectionsService ] ELECTIONS: (V=6) DONE. ELECTED MASTER = [192.168.2.4:2113,{e6f6caf5-4888-4852-bf6d-725d94802897}](L=3617,W=3880,C=3747,E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3}). ME=[192.168.2.4:1113,{fafb112d-9dd8-40dd-bb2b-8676b4a329de}](L=3617,W=3747,C=3747,E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3}). [PID:04255:009 2018.12.12 21:13:35.439 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [StorageWriterQueue]: DataChunkBulk - 4885ms. Q: 0/2. [PID:04255:009 2018.12.12 21:13:35.945 DEBUG StorageWriterService] Still waiting for chaser to catch up already for 00:00:00.2000886... [PID:04255:009 2018.12.12 21:13:35.945 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [StorageWriterQueue]: WaitForChaserToCatchUp - 505ms. Q: 0/0. [PID:04255:012 2018.12.12 21:13:35.946 INFO ReplicaService ] Subscribing at LogPosition: 3880 (0xF28) to MASTER [192.168.2.4:2111, {e6f6caf5-4888-4852-bf6d-725d94802897}] as replica with SubscriptionId: {ea202c1b-37f9-4176-b990-60ecca2b5836}, ConnectionId: {8f869617-0227-4101-bc4a-b328b6ec1dd0}, LocalEndPoint: [192.168.2.4:37318], Epochs: E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3}... . [PID:04255:012 2018.12.12 21:13:35.950 INFO ClusterVNodeControll] ========== [192.168.2.4:1113] IS CATCHING UP... MASTER IS [192.168.2.4:2113,{e6f6caf5-4888-4852-bf6d-725d94802897}] [PID:04255:009 2018.12.12 21:13:35.950 INFO ClusterStorageWriter] === SUBSCRIBED to [192.168.2.4:2111,{e6f6caf5-4888-4852-bf6d-725d94802897}] at 3880 (0xF28). SubscriptionId: {ea202c1b-37f9-4176-b990-60ecca2b5836}. [PID:04255:006 2018.12.12 21:13:35.950 DEBUG PersistentSubscripti] Subscriptions received state change to CatchingUp stopping listening. [PID:04255:012 2018.12.12 21:13:36.140 INFO ClusterVNodeControll] ========== [192.168.2.4:1113] CLONE ASSIGNMENT RECEIVED FROM [192.168.2.4:2111,n/a,{e6f6caf5-4888-4852-bf6d-725d94802897}]. [PID:04255:012 2018.12.12 21:13:36.140 INFO ClusterVNodeControll] ========== [192.168.2.4:1113] IS CLONE... MASTER IS [192.168.2.4:2113,{e6f6caf5-4888-4852-bf6d-725d94802897}] [PID:04255:008 2018.12.12 21:13:36.140 DEBUG PersistentSubscripti] Subscriptions received state change to Clone stopping listening. [PID:04255:012 2018.12.12 21:13:36.140 INFO ClusterVNodeControll] ========== [192.168.2.4:1113] SLAVE ASSIGNMENT RECEIVED FROM [192.168.2.4:2111,n/a,{e6f6caf5-4888-4852-bf6d-725d94802897}]. [PID:04255:012 2018.12.12 21:13:36.141 INFO ClusterVNodeControll] ========== [192.168.2.4:1113] IS SLAVE... MASTER IS [192.168.2.4:2113,{e6f6caf5-4888-4852-bf6d-725d94802897}] [PID:04255:017 2018.12.12 21:13:36.141 DEBUG PersistentSubscripti] Subscriptions received state change to Slave stopping listening. [PID:04255:009 2018.12.12 21:13:37.239 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [StorageWriterQueue]: DataChunkBulk - 1098ms. Q: 0/2. [PID:04255:012 2018.12.12 21:13:37.868 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.2.4:2113]) [PID:04255:012 2018.12.12 21:13:37.868 TRACE GossipServiceBase ] Old: [PID:04255:012 2018.12.12 21:13:37.868 TRACE GossipServiceBase ] VND {15230862-8531-4aeb-893f-68528714e7c3} [Slave, 192.168.2.4:3111, n/a, 192.168.2.4:3112, n/a, 192.168.2.4:3113, 192.168.2.4:3114] 3880/4486/4024/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:37.592 [PID:04255:012 2018.12.12 21:13:37.868 TRACE GossipServiceBase ] VND {e6f6caf5-4888-4852-bf6d-725d94802897} [Master, 192.168.2.4:2111, n/a, 192.168.2.4:2112, n/a, 192.168.2.4:2113, 192.168.2.4:2114] 3617/4300/3880/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:37.593 [PID:04255:012 2018.12.12 21:13:37.868 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [Slave, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] 4161/4486/4300/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:37.593 [PID:04255:012 2018.12.12 21:13:37.869 TRACE GossipServiceBase ] New: [PID:04255:012 2018.12.12 21:13:37.869 TRACE GossipServiceBase ] VND {15230862-8531-4aeb-893f-68528714e7c3} [Slave, 192.168.2.4:3111, n/a, 192.168.2.4:3112, n/a, 192.168.2.4:3113, 192.168.2.4:3114] 3880/4486/4024/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:37.592 [PID:04255:012 2018.12.12 21:13:37.869 TRACE GossipServiceBase ] VND {e6f6caf5-4888-4852-bf6d-725d94802897} [Master, 192.168.2.4:2111, n/a, 192.168.2.4:2112, n/a, 192.168.2.4:2113, 192.168.2.4:2114] 3747/4486/4486/E1@4300:{e83633b0-5028-4442-84d9-51f75163d7aa} | 2018-12-12 21:13:37.848 [PID:04255:012 2018.12.12 21:13:37.870 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [Slave, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] 4161/4486/4300/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:37.868 [PID:04255:012 2018.12.12 21:13:37.870 TRACE GossipServiceBase ] -------------------------------------------------------------------------------- [PID:04255:009 2018.12.12 21:13:39.739 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [StorageWriterQueue]: DataChunkBulk - 2448ms. Q: 0/0. [PID:04255:013 2018.12.12 21:13:40.339 DEBUG EpochManager ] === Update Last Epoch E1@4300:{e83633b0-5028-4442-84d9-51f75163d7aa} (previous epoch at 0). [PID:04255:012 2018.12.12 21:13:40.339 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip received from [192.168.2.4:3113]) [PID:04255:012 2018.12.12 21:13:40.340 TRACE GossipServiceBase ] Old: [PID:04255:012 2018.12.12 21:13:40.340 TRACE GossipServiceBase ] VND {15230862-8531-4aeb-893f-68528714e7c3} [Slave, 192.168.2.4:3111, n/a, 192.168.2.4:3112, n/a, 192.168.2.4:3113, 192.168.2.4:3114] 4161/4486/4300/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:38.874 [PID:04255:012 2018.12.12 21:13:40.340 TRACE GossipServiceBase ] VND {e6f6caf5-4888-4852-bf6d-725d94802897} [Master, 192.168.2.4:2111, n/a, 192.168.2.4:2112, n/a, 192.168.2.4:2113, 192.168.2.4:2114] 4161/6477/4486/E1@4300:{e83633b0-5028-4442-84d9-51f75163d7aa} | 2018-12-12 21:13:39.605 [PID:04255:012 2018.12.12 21:13:40.340 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [Slave, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] 4161/4486/4300/E0@0:{b11386f1-3101-4990-99b5-1a2f9f7221d3} | 2018-12-12 21:13:39.606 [PID:04255:012 2018.12.12 21:13:40.340 TRACE GossipServiceBase ] New: [PID:04255:012 2018.12.12 21:13:40.340 TRACE GossipServiceBase ] VND {15230862-8531-4aeb-893f-68528714e7c3} [Slave, 192.168.2.4:3111, n/a, 192.168.2.4:3112, n/a, 192.168.2.4:3113, 192.168.2.4:3114] 4161/4486/4486/E1@4300:{e83633b0-5028-4442-84d9-51f75163d7aa} | 2018-12-12 21:13:39.998 [PID:04255:012 2018.12.12 21:13:40.341 TRACE GossipServiceBase ] VND {e6f6caf5-4888-4852-bf6d-725d94802897} [Master, 192.168.2.4:2111, n/a, 192.168.2.4:2112, n/a, 192.168.2.4:2113, 192.168.2.4:2114] 4161/6477/4486/E1@4300:{e83633b0-5028-4442-84d9-51f75163d7aa} | 2018-12-12 21:13:39.605 [PID:04255:012 2018.12.12 21:13:40.341 TRACE GossipServiceBase ] VND {fafb112d-9dd8-40dd-bb2b-8676b4a329de} [Slave, 192.168.2.4:1111, 192.168.2.4:0, 192.168.2.4:1112, 192.168.2.4:0, 192.168.2.4:1113, 192.168.2.4:1114] 4161/4486/4486/E1@4300:{e83633b0-5028-4442-84d9-51f75163d7aa} | 2018-12-12 21:13:40.340 [PID:04255:012 2018.12.12 21:13:40.341 TRACE GossipServiceBase ] -------------------------------------------------------------------------------- [PID:04255:012 2018.12.12 21:13:40.341 TRACE InMemoryBus ] SLOW BUS MSG [MainBus]: GossipReceived - 340ms. Handler: NodeGossipService. [PID:04255:012 2018.12.12 21:13:40.341 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [MainQueue]: GossipReceived - 340ms. Q: 0/3. [PID:04255:009 2018.12.12 21:13:41.540 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [StorageWriterQueue]: DataChunkBulk - 1196ms. Q: 0/0. [PID:04255:009 2018.12.12 21:13:48.739 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [StorageWriterQueue]: DataChunkBulk - 1034ms. Q: 0/0.