Starting apexeventseventstoretest_eventstore3_1 ... Starting apexeventseventstoretest_eventstore3_1 Attaching to apexeventseventstoretest_eventstore3_1, apexeventseventstoretest_eventstore2_1, apexeventseventstoretest_eventstore1_1 eventstore3_1 | [00001,01,02:26:26.698] eventstore3_1 | ES VERSION: 4.1.0.0 (HEAD/daddeca886f43e40489efd7d9f45cb6ddd4aed8a, Tue, 23 Jan 2018 14:12:35 +0200) eventstore3_1 | [00001,01,02:26:26.717] OS: Linux (Unix 4.9.49.0) eventstore3_1 | [00001,01,02:26:26.720] RUNTIME: 4.6.2 (Stable 4.6.2.16/ac9e222) (64-bit) eventstore3_1 | [00001,01,02:26:26.721] GC: 2 GENERATIONS eventstore3_1 | [00001,01,02:26:26.721] LOGS: /var/log/eventstore eventstore3_1 | [00001,01,02:26:26.727] MODIFIED OPTIONS: eventstore3_1 | eventstore3_1 | START STANDARD PROJECTIONS: True (Environment Variable) eventstore3_1 | INT IP: 172.20.0.2 (Environment Variable) eventstore3_1 | INT HTTP PORT: 31003 (Environment Variable) eventstore3_1 | EXT HTTP PORT: 31001 (Environment Variable) eventstore3_1 | INT TCP PORT: 31004 (Environment Variable) eventstore3_1 | EXT TCP PORT: 31002 (Environment Variable) eventstore3_1 | EXT TCP PORT ADVERTISE AS: 31002 (Environment Variable) eventstore3_1 | EXT HTTP PORT ADVERTISE AS: 31001 (Environment Variable) eventstore3_1 | INT TCP PORT ADVERTISE AS: 31004 (Environment Variable) eventstore3_1 | INT HTTP PORT ADVERTISE AS: 31003 (Environment Variable) eventstore3_1 | CLUSTER SIZE: 3 (Environment Variable) eventstore3_1 | CLUSTER DNS: eventstore1 (Environment Variable) eventstore3_1 | CLUSTER GOSSIP PORT: 31000 (Environment Variable) eventstore3_1 | RUN PROJECTIONS: All (Environment Variable) eventstore3_1 | INT HTTP PREFIXES: http://*:31003/ (Environment Variable) eventstore3_1 | EXT HTTP PREFIXES: http://*:31001/ (Environment Variable) eventstore3_1 | EXT IP: 0.0.0.0 (Config File) eventstore3_1 | ADD INTERFACE PREFIXES: false (Config File) eventstore3_1 | eventstore3_1 | DEFAULT OPTIONS: eventstore3_1 | eventstore3_1 | CONFIG: /etc/eventstore/eventstore.conf () eventstore3_1 | HELP: False () eventstore3_1 | VERSION: False () eventstore3_1 | LOG: /var/log/eventstore () eventstore3_1 | DEFINES: () eventstore3_1 | WHAT IF: False () eventstore3_1 | DISABLE HTTP CACHING: False () eventstore3_1 | MONO MIN THREADPOOL SIZE: 10 () eventstore3_1 | INT SECURE TCP PORT: 0 () eventstore3_1 | EXT SECURE TCP PORT ADVERTISE AS: 0 () eventstore3_1 | EXT SECURE TCP PORT: 0 () eventstore3_1 | EXT IP ADVERTISE AS: () eventstore3_1 | INT IP ADVERTISE AS: () eventstore3_1 | INT SECURE TCP PORT ADVERTISE AS: 0 () eventstore3_1 | INT TCP HEARTBEAT TIMEOUT: 700 () eventstore3_1 | EXT TCP HEARTBEAT TIMEOUT: 1000 () eventstore3_1 | INT TCP HEARTBEAT INTERVAL: 700 () eventstore3_1 | EXT TCP HEARTBEAT INTERVAL: 2000 () eventstore3_1 | GOSSIP ON SINGLE NODE: False () eventstore3_1 | CONNECTION PENDING SEND BYTES THRESHOLD: 10485760 () eventstore3_1 | FORCE: False () eventstore3_1 | NODE PRIORITY: 0 () eventstore3_1 | MIN FLUSH DELAY MS: 2 () eventstore3_1 | COMMIT COUNT: -1 () eventstore3_1 | PREPARE COUNT: -1 () eventstore3_1 | ADMIN ON EXT: True () eventstore3_1 | STATS ON EXT: True () eventstore3_1 | GOSSIP ON EXT: True () eventstore3_1 | DISABLE SCAVENGE MERGING: False () eventstore3_1 | SCAVENGE HISTORY MAX AGE: 30 () eventstore3_1 | DISCOVER VIA DNS: True () eventstore3_1 | GOSSIP SEED: () eventstore3_1 | STATS PERIOD SEC: 30 () eventstore3_1 | CACHED CHUNKS: -1 () eventstore3_1 | READER THREADS COUNT: 4 () eventstore3_1 | CHUNKS CACHE SIZE: 536871424 () eventstore3_1 | MAX MEM TABLE SIZE: 1000000 () eventstore3_1 | HASH COLLISION READ LIMIT: 100 () eventstore3_1 | DB: /var/lib/eventstore () eventstore3_1 | INDEX: () eventstore3_1 | MEM DB: False () eventstore3_1 | SKIP DB VERIFY: False () eventstore3_1 | WRITE THROUGH: False () eventstore3_1 | UNBUFFERED: False () eventstore3_1 | CHUNK INITIAL READER COUNT: 5 () eventstore3_1 | PROJECTION THREADS: 3 () eventstore3_1 | WORKER THREADS: 5 () eventstore3_1 | PROJECTIONS QUERY EXPIRY: 0 () eventstore3_1 | ENABLE TRUSTED AUTH: False () eventstore3_1 | CERTIFICATE STORE LOCATION: () eventstore3_1 | CERTIFICATE STORE NAME: () eventstore3_1 | CERTIFICATE SUBJECT NAME: () eventstore3_1 | CERTIFICATE THUMBPRINT: () eventstore3_1 | CERTIFICATE FILE: () eventstore3_1 | CERTIFICATE PASSWORD: () eventstore3_1 | USE INTERNAL SSL: False () eventstore3_1 | DISABLE INSECURE TCP: False () eventstore3_1 | SSL TARGET HOST: n/a () eventstore3_1 | SSL VALIDATE SERVER: True () eventstore3_1 | AUTHENTICATION TYPE: internal () eventstore3_1 | AUTHENTICATION CONFIG: () eventstore3_1 | PREPARE TIMEOUT MS: 2000 () eventstore3_1 | COMMIT TIMEOUT MS: 2000 () eventstore3_1 | UNSAFE DISABLE FLUSH TO DISK: False () eventstore3_1 | BETTER ORDERING: False () eventstore3_1 | UNSAFE IGNORE HARD DELETE: False () eventstore3_1 | SKIP INDEX VERIFY: False () eventstore3_1 | INDEX CACHE DEPTH: 16 () eventstore3_1 | GOSSIP INTERVAL MS: 1000 () eventstore3_1 | GOSSIP ALLOWED DIFFERENCE MS: 60000 () eventstore3_1 | GOSSIP TIMEOUT MS: 500 () eventstore3_1 | ENABLE HISTOGRAMS: False () eventstore3_1 | LOG HTTP REQUESTS: False () eventstore3_1 | ALWAYS KEEP SCAVENGED: False () eventstore3_1 | SKIP INDEX SCAN ON READS: False () eventstore3_1 | eventstore3_1 | [00001,01,02:26:26.732] Quorum size set to 2 eventstore3_1 | [00001,01,02:26:26.758] Cannot find plugins path: plugins eventstore3_1 | [00001,01,02:26:26.807] INSTANCE ID: 90f57b38-10ce-443c-ad92-cd465e024c51 eventstore3_1 | [00001,01,02:26:26.808] DATABASE: /var/lib/eventstore eventstore3_1 | [00001,01,02:26:26.808] WRITER CHECKPOINT: 0 (0x0) eventstore3_1 | [00001,01,02:26:26.808] CHASER CHECKPOINT: 0 (0x0) eventstore3_1 | [00001,01,02:26:26.808] EPOCH CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF) eventstore3_1 | [00001,01,02:26:26.808] TRUNCATE CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF) eventstore3_1 | [00001,01,02:26:26.978] MessageHierarchy initialization took 00:00:00.1381117. eventstore3_1 | [00001,01,02:26:27.068] Opened ongoing /var/lib/eventstore/chunk-000000.000000 as version 3 eventstore3_1 | [00001,01,02:26:27.076] CACHED TFChunk #0-0 (chunk-000000.000000) in 00:00:00.0025996. eventstore3_1 | [00001,01,02:26:27.211] Starting MiniWeb for /web/es/js/projections ==> /usr/share/eventstore/projections eventstore3_1 | [00001,01,02:26:27.211] Starting MiniWeb for /web/es/js/projections/v8/Prelude ==> /usr/share/eventstore/Prelude eventstore3_1 | [00001,01,02:26:27.212] Binding MiniWeb to /web/es/js/projections/{*remaining_path} eventstore3_1 | [00001,01,02:26:27.212] Binding MiniWeb to /web/es/js/projections/v8/Prelude/{*remaining_path} eventstore3_1 | [00001,01,02:26:27.213] Binding MiniWeb to /web/es/js/projections/{*remaining_path} eventstore3_1 | [00001,01,02:26:27.213] Binding MiniWeb to /web/es/js/projections/v8/Prelude/{*remaining_path} eventstore3_1 | [00001,01,02:26:27.227] Starting MiniWeb for /web ==> /usr/share/eventstore/clusternode-web eventstore3_1 | [00001,01,02:26:27.227] Binding MiniWeb to /web/{*remaining_path} eventstore3_1 | [00001,01,02:26:27.228] Starting MiniWeb for /web ==> /usr/share/eventstore/clusternode-web eventstore3_1 | [00001,01,02:26:27.228] Binding MiniWeb to /web/{*remaining_path} eventstore3_1 | [00001,12,02:26:27.255] ========== [172.20.0.2:31003] SYSTEM INIT... eventstore3_1 | [00001,12,02:26:27.295] Starting Normal TCP listening on TCP endpoint: 0.0.0.0:31002. eventstore3_1 | [00001,13,02:26:27.306] TableIndex initialization... eventstore3_1 | [00001,13,02:26:27.335] ReadIndex building... eventstore3_1 | [00001,12,02:26:27.336] SLOW BUS MSG [MainBus]: SystemInit - 48ms. Handler: TcpService. eventstore3_1 | [00001,12,02:26:27.336] Starting Normal TCP listening on TCP endpoint: 172.20.0.2:31004. eventstore3_1 | [00001,13,02:26:27.337] ReadIndex rebuilding done: total processed 0 records, time elapsed: 00:00:00.0021040. eventstore3_1 | [00001,12,02:26:27.363] Starting HTTP server on [http://*:31001/]... eventstore3_1 | [00001,12,02:26:27.374] HTTP server is up and listening on [http://*:31001/] eventstore3_1 | [00001,12,02:26:27.374] Starting HTTP server on [http://*:31003/]... eventstore3_1 | [00001,12,02:26:27.374] HTTP server is up and listening on [http://*:31003/] eventstore3_1 | [00001,12,02:26:27.396] SLOW QUEUE MSG [MainQueue]: SystemInit - 149ms. Q: 0/7. eventstore3_1 | [00001,12,02:26:27.396] ========== [172.20.0.2:31003] Service 'StorageWriter' initialized. eventstore3_1 | [00001,12,02:26:27.396] ========== [172.20.0.2:31003] Service 'StorageReader' initialized. eventstore3_1 | [00001,12,02:26:27.397] ========== [172.20.0.2:31003] Service 'StorageChaser' initialized. eventstore3_1 | [00001,12,02:26:27.415] CLUSTER HAS CHANGED eventstore3_1 | [00001,12,02:26:27.415] Old: eventstore3_1 | [00001,12,02:26:27.419] VND {90f57b38-10ce-443c-ad92-cd465e024c51} [Unknown, 172.20.0.2:31004, 172.20.0.2:0, 0.0.0.0:31002, 0.0.0.0:0, 172.20.0.2:31003, 0.0.0.0:31001] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-07-25 02:26:27.375 eventstore3_1 | [00001,12,02:26:27.419] New: eventstore3_1 | [00001,12,02:26:27.420] MAN {00000000-0000-0000-0000-000000000000} [Manager, 172.20.0.3:31000, 172.20.0.3:31000] | 2018-07-25 02:26:27.398 eventstore3_1 | [00001,12,02:26:27.420] VND {90f57b38-10ce-443c-ad92-cd465e024c51} [Unknown, 172.20.0.2:31004, 172.20.0.2:0, 0.0.0.0:31002, 0.0.0.0:0, 172.20.0.2:31003, 0.0.0.0:31001] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-07-25 02:26:27.375 eventstore3_1 | [00001,12,02:26:27.420] -------------------------------------------------------------------------------- eventstore3_1 | [00001,12,02:26:27.423] ========== [172.20.0.2:31003] SYSTEM START... eventstore3_1 | [00001,12,02:26:27.428] ========== [172.20.0.2:31003] IS UNKNOWN... eventstore3_1 | [00001,12,02:26:27.481] SLOW QUEUE MSG [MainQueue]: ReadStreamEventsBackward - 50ms. Q: 4/4. eventstore3_1 | [00001,06,02:26:27.516] Subscriptions received state change to Unknown stopping listening. eventstore3_1 | [00001,06,02:26:27.518] SLOW BUS MSG [PersistentSubscriptionsBus]: BecomeUnknown - 86ms. Handler: PersistentSubscriptionService. eventstore3_1 | [00001,12,02:26:27.529] ELECTIONS: STARTING ELECTIONS. eventstore3_1 | [00001,12,02:26:27.529] ELECTIONS: (V=0) SHIFT TO LEADER ELECTION. eventstore3_1 | [00001,12,02:26:27.531] ELECTIONS: (V=0) VIEWCHANGE FROM [172.20.0.2:31003, {90f57b38-10ce-443c-ad92-cd465e024c51}]. eventstore3_1 | [00001,12,02:26:27.537] SLOW BUS MSG [MainBus]: StartElections - 55ms. Handler: ElectionsService. eventstore3_1 | [00001,12,02:26:27.537] SLOW QUEUE MSG [MainQueue]: StartElections - 55ms. Q: 0/3. eventstore3_1 | [00001,07,02:26:27.962] SLOW BUS MSG [Worker #1 Bus]: SendOverHttp - 529ms. Handler: HttpSendService. eventstore3_1 | [00001,19,02:26:27.963] SLOW BUS MSG [Worker #3 Bus]: SendOverHttp - 421ms. Handler: HttpSendService. eventstore3_1 | [00001,16,02:26:27.976] SLOW BUS MSG [Worker #5 Bus]: SendOverHttp - 313ms. Handler: HttpSendService. eventstore3_1 | [00001,12,02:26:28.036] Looks like node [172.20.0.3:31000] is DEAD (Gossip send failed). eventstore3_1 | [00001,12,02:26:28.036] CLUSTER HAS CHANGED (gossip send failed to [172.20.0.3:31000]) eventstore3_1 | [00001,12,02:26:28.036] Old: eventstore3_1 | [00001,12,02:26:28.037] MAN {00000000-0000-0000-0000-000000000000} [Manager, 172.20.0.3:31000, 172.20.0.3:31000] | 2018-07-25 02:26:27.398 eventstore3_1 | [00001,12,02:26:28.037] VND {90f57b38-10ce-443c-ad92-cd465e024c51} [Unknown, 172.20.0.2:31004, 172.20.0.2:0, 0.0.0.0:31002, 0.0.0.0:0, 172.20.0.2:31003, 0.0.0.0:31001] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-07-25 02:26:27.973 eventstore3_1 | [00001,12,02:26:28.037] New: eventstore3_1 | [00001,12,02:26:28.037] MAN {00000000-0000-0000-0000-000000000000} [Manager, 172.20.0.3:31000, 172.20.0.3:31000] | 2018-07-25 02:26:28.036 eventstore3_1 | [00001,12,02:26:28.057] VND {90f57b38-10ce-443c-ad92-cd465e024c51} [Unknown, 172.20.0.2:31004, 172.20.0.2:0, 0.0.0.0:31002, 0.0.0.0:0, 172.20.0.2:31003, 0.0.0.0:31001] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-07-25 02:26:27.973 eventstore3_1 | [00001,12,02:26:28.057] -------------------------------------------------------------------------------- eventstore3_1 | [00001,19,02:26:28.082] SLOW QUEUE MSG [Worker #3]: SendOverHttp - 421ms. Q: 0/0. eventstore3_1 | [00001,16,02:26:28.082] SLOW QUEUE MSG [Worker #5]: SendOverHttp - 328ms. Q: 0/0. eventstore3_1 | [00001,07,02:26:28.083] SLOW QUEUE MSG [Worker #1]: SendOverHttp - 529ms. Q: 0/1. eventstore3_1 | [00001,12,02:26:28.547] ELECTIONS: (V=0) TIMED OUT! (S=ElectingLeader, M=). eventstore3_1 | [00001,12,02:26:28.548] ELECTIONS: (V=1) SHIFT TO LEADER ELECTION. eventstore3_1 | [00001,12,02:26:28.548] ELECTIONS: (V=1) VIEWCHANGE FROM [172.20.0.2:31003, {90f57b38-10ce-443c-ad92-cd465e024c51}]. eventstore3_1 | [00001,12,02:26:29.549] ELECTIONS: (V=1) TIMED OUT! (S=ElectingLeader, M=). eventstore3_1 | [00001,12,02:26:29.549] ELECTIONS: (V=2) SHIFT TO LEADER ELECTION. eventstore3_1 | [00001,12,02:26:29.549] ELECTIONS: (V=2) VIEWCHANGE FROM [172.20.0.2:31003, {90f57b38-10ce-443c-ad92-cd465e024c51}]. eventstore3_1 | [00001,12,02:26:30.552] ELECTIONS: (V=2) TIMED OUT! (S=ElectingLeader, M=). eventstore3_1 | [00001,12,02:26:30.552] ELECTIONS: (V=3) SHIFT TO LEADER ELECTION. eventstore3_1 | [00001,12,02:26:30.552] ELECTIONS: (V=3) VIEWCHANGE FROM [172.20.0.2:31003, {90f57b38-10ce-443c-ad92-cd465e024c51}]. eventstore3_1 | [00001,12,02:26:31.556] ELECTIONS: (V=3) TIMED OUT! (S=ElectingLeader, M=). eventstore3_1 | [00001,12,02:26:31.556] ELECTIONS: (V=4) SHIFT TO LEADER ELECTION. eventstore3_1 | [00001,12,02:26:31.556] ELECTIONS: (V=4) VIEWCHANGE FROM [172.20.0.2:31003, {90f57b38-10ce-443c-ad92-cd465e024c51}]. eventstore3_1 | [00001,12,02:26:32.559] ELECTIONS: (V=4) TIMED OUT! (S=ElectingLeader, M=). eventstore3_1 | [00001,12,02:26:32.559] ELECTIONS: (V=5) SHIFT TO LEADER ELECTION. eventstore3_1 | [00001,12,02:26:32.559] ELECTIONS: (V=5) VIEWCHANGE FROM [172.20.0.2:31003, {90f57b38-10ce-443c-ad92-cd465e024c51}].