Starting apexeventseventstoretest_eventstore2_1 ... Starting apexeventseventstoretest_eventstore2_1 Attaching to apexeventseventstoretest_eventstore3_1, apexeventseventstoretest_eventstore2_1, apexeventseventstoretest_eventstore1_1 eventstore2_1 | [00001,01,02:26:26.901] eventstore2_1 | ES VERSION: 4.1.0.0 (HEAD/daddeca886f43e40489efd7d9f45cb6ddd4aed8a, Tue, 23 Jan 2018 14:12:35 +0200) eventstore2_1 | [00001,01,02:26:26.930] OS: Linux (Unix 4.9.49.0) eventstore2_1 | [00001,01,02:26:26.934] RUNTIME: 4.6.2 (Stable 4.6.2.16/ac9e222) (64-bit) eventstore2_1 | [00001,01,02:26:26.934] GC: 2 GENERATIONS eventstore2_1 | [00001,01,02:26:26.934] LOGS: /var/log/eventstore eventstore2_1 | [00001,01,02:26:26.946] MODIFIED OPTIONS: eventstore2_1 | eventstore2_1 | START STANDARD PROJECTIONS: True (Environment Variable) eventstore2_1 | INT IP: 172.20.0.4 (Environment Variable) eventstore2_1 | INT HTTP PORT: 21003 (Environment Variable) eventstore2_1 | EXT HTTP PORT: 21001 (Environment Variable) eventstore2_1 | INT TCP PORT: 21004 (Environment Variable) eventstore2_1 | EXT TCP PORT: 21002 (Environment Variable) eventstore2_1 | EXT TCP PORT ADVERTISE AS: 21002 (Environment Variable) eventstore2_1 | EXT HTTP PORT ADVERTISE AS: 21001 (Environment Variable) eventstore2_1 | INT TCP PORT ADVERTISE AS: 21004 (Environment Variable) eventstore2_1 | INT HTTP PORT ADVERTISE AS: 21003 (Environment Variable) eventstore2_1 | CLUSTER SIZE: 3 (Environment Variable) eventstore2_1 | CLUSTER DNS: eventstore1 (Environment Variable) eventstore2_1 | CLUSTER GOSSIP PORT: 21000 (Environment Variable) eventstore2_1 | RUN PROJECTIONS: All (Environment Variable) eventstore2_1 | INT HTTP PREFIXES: http://*:21003/ (Environment Variable) eventstore2_1 | EXT HTTP PREFIXES: http://*:21001/ (Environment Variable) eventstore2_1 | EXT IP: 0.0.0.0 (Config File) eventstore2_1 | ADD INTERFACE PREFIXES: false (Config File) eventstore2_1 | eventstore2_1 | DEFAULT OPTIONS: eventstore2_1 | eventstore2_1 | CONFIG: /etc/eventstore/eventstore.conf () eventstore2_1 | HELP: False () eventstore2_1 | VERSION: False () eventstore2_1 | LOG: /var/log/eventstore () eventstore2_1 | DEFINES: () eventstore2_1 | WHAT IF: False () eventstore2_1 | DISABLE HTTP CACHING: False () eventstore2_1 | MONO MIN THREADPOOL SIZE: 10 () eventstore2_1 | INT SECURE TCP PORT: 0 () eventstore2_1 | EXT SECURE TCP PORT ADVERTISE AS: 0 () eventstore2_1 | EXT SECURE TCP PORT: 0 () eventstore2_1 | EXT IP ADVERTISE AS: () eventstore2_1 | INT IP ADVERTISE AS: () eventstore2_1 | INT SECURE TCP PORT ADVERTISE AS: 0 () eventstore2_1 | INT TCP HEARTBEAT TIMEOUT: 700 () eventstore2_1 | EXT TCP HEARTBEAT TIMEOUT: 1000 () eventstore2_1 | INT TCP HEARTBEAT INTERVAL: 700 () eventstore2_1 | EXT TCP HEARTBEAT INTERVAL: 2000 () eventstore2_1 | GOSSIP ON SINGLE NODE: False () eventstore2_1 | CONNECTION PENDING SEND BYTES THRESHOLD: 10485760 () eventstore2_1 | FORCE: False () eventstore2_1 | NODE PRIORITY: 0 () eventstore2_1 | MIN FLUSH DELAY MS: 2 () eventstore2_1 | COMMIT COUNT: -1 () eventstore2_1 | PREPARE COUNT: -1 () eventstore2_1 | ADMIN ON EXT: True () eventstore2_1 | STATS ON EXT: True () eventstore2_1 | GOSSIP ON EXT: True () eventstore2_1 | DISABLE SCAVENGE MERGING: False () eventstore2_1 | SCAVENGE HISTORY MAX AGE: 30 () eventstore2_1 | DISCOVER VIA DNS: True () eventstore2_1 | GOSSIP SEED: () eventstore2_1 | STATS PERIOD SEC: 30 () eventstore2_1 | CACHED CHUNKS: -1 () eventstore2_1 | READER THREADS COUNT: 4 () eventstore2_1 | CHUNKS CACHE SIZE: 536871424 () eventstore2_1 | MAX MEM TABLE SIZE: 1000000 () eventstore2_1 | HASH COLLISION READ LIMIT: 100 () eventstore2_1 | DB: /var/lib/eventstore () eventstore2_1 | INDEX: () eventstore2_1 | MEM DB: False () eventstore2_1 | SKIP DB VERIFY: False () eventstore2_1 | WRITE THROUGH: False () eventstore2_1 | UNBUFFERED: False () eventstore2_1 | CHUNK INITIAL READER COUNT: 5 () eventstore2_1 | PROJECTION THREADS: 3 () eventstore2_1 | WORKER THREADS: 5 () eventstore2_1 | PROJECTIONS QUERY EXPIRY: 0 () eventstore2_1 | ENABLE TRUSTED AUTH: False () eventstore2_1 | CERTIFICATE STORE LOCATION: () eventstore2_1 | CERTIFICATE STORE NAME: () eventstore2_1 | CERTIFICATE SUBJECT NAME: () eventstore2_1 | CERTIFICATE THUMBPRINT: () eventstore2_1 | CERTIFICATE FILE: () eventstore2_1 | CERTIFICATE PASSWORD: () eventstore2_1 | USE INTERNAL SSL: False () eventstore2_1 | DISABLE INSECURE TCP: False () eventstore2_1 | SSL TARGET HOST: n/a () eventstore2_1 | SSL VALIDATE SERVER: True () eventstore2_1 | AUTHENTICATION TYPE: internal () eventstore2_1 | AUTHENTICATION CONFIG: () eventstore2_1 | PREPARE TIMEOUT MS: 2000 () eventstore2_1 | COMMIT TIMEOUT MS: 2000 () eventstore2_1 | UNSAFE DISABLE FLUSH TO DISK: False () eventstore2_1 | BETTER ORDERING: False () eventstore2_1 | UNSAFE IGNORE HARD DELETE: False () eventstore2_1 | SKIP INDEX VERIFY: False () eventstore2_1 | INDEX CACHE DEPTH: 16 () eventstore2_1 | GOSSIP INTERVAL MS: 1000 () eventstore2_1 | GOSSIP ALLOWED DIFFERENCE MS: 60000 () eventstore2_1 | GOSSIP TIMEOUT MS: 500 () eventstore2_1 | ENABLE HISTOGRAMS: False () eventstore2_1 | LOG HTTP REQUESTS: False () eventstore2_1 | ALWAYS KEEP SCAVENGED: False () eventstore2_1 | SKIP INDEX SCAN ON READS: False () eventstore2_1 | eventstore2_1 | [00001,01,02:26:26.951] Quorum size set to 2 eventstore2_1 | [00001,01,02:26:26.960] Cannot find plugins path: plugins eventstore2_1 | [00001,01,02:26:27.069] INSTANCE ID: 5f978c32-927d-400d-beff-7e13ec9791b9 eventstore2_1 | [00001,01,02:26:27.072] DATABASE: /var/lib/eventstore eventstore2_1 | [00001,01,02:26:27.072] WRITER CHECKPOINT: 0 (0x0) eventstore2_1 | [00001,01,02:26:27.072] CHASER CHECKPOINT: 0 (0x0) eventstore2_1 | [00001,01,02:26:27.072] EPOCH CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF) eventstore2_1 | [00001,01,02:26:27.072] TRUNCATE CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF) eventstore2_1 | [00001,01,02:26:27.292] MessageHierarchy initialization took 00:00:00.1714311. eventstore2_1 | [00001,01,02:26:27.355] Opened ongoing /var/lib/eventstore/chunk-000000.000000 as version 3 eventstore2_1 | [00001,01,02:26:27.365] CACHED TFChunk #0-0 (chunk-000000.000000) in 00:00:00.0039485. eventstore2_1 | [00001,01,02:26:27.491] Starting MiniWeb for /web/es/js/projections ==> /usr/share/eventstore/projections eventstore2_1 | [00001,01,02:26:27.491] Starting MiniWeb for /web/es/js/projections/v8/Prelude ==> /usr/share/eventstore/Prelude eventstore2_1 | [00001,01,02:26:27.492] Binding MiniWeb to /web/es/js/projections/{*remaining_path} eventstore2_1 | [00001,01,02:26:27.492] Binding MiniWeb to /web/es/js/projections/v8/Prelude/{*remaining_path} eventstore2_1 | [00001,01,02:26:27.492] Binding MiniWeb to /web/es/js/projections/{*remaining_path} eventstore2_1 | [00001,01,02:26:27.492] Binding MiniWeb to /web/es/js/projections/v8/Prelude/{*remaining_path} eventstore2_1 | [00001,01,02:26:27.503] Starting MiniWeb for /web ==> /usr/share/eventstore/clusternode-web eventstore2_1 | [00001,01,02:26:27.503] Binding MiniWeb to /web/{*remaining_path} eventstore2_1 | [00001,01,02:26:27.503] Starting MiniWeb for /web ==> /usr/share/eventstore/clusternode-web eventstore2_1 | [00001,01,02:26:27.503] Binding MiniWeb to /web/{*remaining_path} eventstore2_1 | [00001,13,02:26:27.544] ========== [172.20.0.4:21003] SYSTEM INIT... eventstore2_1 | [00001,13,02:26:27.581] Starting Normal TCP listening on TCP endpoint: 0.0.0.0:21002. eventstore2_1 | [00001,13,02:26:27.586] Starting Normal TCP listening on TCP endpoint: 172.20.0.4:21004. eventstore2_1 | [00001,14,02:26:27.611] TableIndex initialization... eventstore2_1 | [00001,14,02:26:27.619] ReadIndex building... eventstore2_1 | [00001,14,02:26:27.620] ReadIndex rebuilding done: total processed 0 records, time elapsed: 00:00:00.0009110. eventstore2_1 | [00001,13,02:26:27.634] Starting HTTP server on [http://*:21001/]... eventstore2_1 | [00001,13,02:26:27.639] HTTP server is up and listening on [http://*:21001/] eventstore2_1 | [00001,13,02:26:27.667] SLOW BUS MSG [MainBus]: SystemInit - 52ms. Handler: HttpService. eventstore2_1 | [00001,13,02:26:27.667] Starting HTTP server on [http://*:21003/]... eventstore2_1 | [00001,13,02:26:27.667] HTTP server is up and listening on [http://*:21003/] eventstore2_1 | [00001,13,02:26:27.696] SLOW QUEUE MSG [MainQueue]: SystemInit - 171ms. Q: 0/8. eventstore2_1 | [00001,13,02:26:27.696] ========== [172.20.0.4:21003] Service 'StorageWriter' initialized. eventstore2_1 | [00001,13,02:26:27.696] ========== [172.20.0.4:21003] Service 'StorageReader' initialized. eventstore2_1 | [00001,13,02:26:27.699] ========== [172.20.0.4:21003] Service 'StorageChaser' initialized. eventstore2_1 | [00001,13,02:26:27.736] CLUSTER HAS CHANGED eventstore2_1 | [00001,13,02:26:27.736] Old: eventstore2_1 | [00001,13,02:26:27.737] VND {5f978c32-927d-400d-beff-7e13ec9791b9} [Unknown, 172.20.0.4:21004, 172.20.0.4:0, 0.0.0.0:21002, 0.0.0.0:0, 172.20.0.4:21003, 0.0.0.0:21001] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-07-25 02:26:27.668 eventstore2_1 | [00001,13,02:26:27.737] New: eventstore2_1 | [00001,13,02:26:27.737] VND {5f978c32-927d-400d-beff-7e13ec9791b9} [Unknown, 172.20.0.4:21004, 172.20.0.4:0, 0.0.0.0:21002, 0.0.0.0:0, 172.20.0.4:21003, 0.0.0.0:21001] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-07-25 02:26:27.668 eventstore2_1 | [00001,13,02:26:27.737] MAN {00000000-0000-0000-0000-000000000000} [Manager, 172.20.0.3:21000, 172.20.0.3:21000] | 2018-07-25 02:26:27.699 eventstore2_1 | [00001,13,02:26:27.737] -------------------------------------------------------------------------------- eventstore2_1 | [00001,13,02:26:27.740] ========== [172.20.0.4:21003] SYSTEM START... eventstore2_1 | [00001,13,02:26:27.750] ========== [172.20.0.4:21003] IS UNKNOWN... eventstore2_1 | [00001,08,02:26:27.854] Subscriptions received state change to Unknown stopping listening. eventstore2_1 | [00001,08,02:26:27.854] SLOW BUS MSG [PersistentSubscriptionsBus]: BecomeUnknown - 98ms. Handler: PersistentSubscriptionService. eventstore2_1 | [00001,13,02:26:27.923] ELECTIONS: STARTING ELECTIONS. eventstore2_1 | [00001,13,02:26:27.925] ELECTIONS: (V=0) SHIFT TO LEADER ELECTION. eventstore2_1 | [00001,13,02:26:27.934] ELECTIONS: (V=0) VIEWCHANGE FROM [172.20.0.4:21003, {5f978c32-927d-400d-beff-7e13ec9791b9}]. eventstore2_1 | [00001,13,02:26:27.935] SLOW BUS MSG [MainBus]: StartElections - 140ms. Handler: ElectionsService. eventstore2_1 | [00001,13,02:26:27.942] SLOW QUEUE MSG [MainQueue]: StartElections - 147ms. Q: 0/4. eventstore2_1 | [00001,07,02:26:28.203] SLOW BUS MSG [Worker #4 Bus]: SendOverHttp - 249ms. Handler: HttpSendService. eventstore2_1 | [00001,06,02:26:28.219] SLOW BUS MSG [Worker #1 Bus]: SendOverHttp - 451ms. Handler: HttpSendService. eventstore2_1 | [00001,13,02:26:28.229] Looks like node [172.20.0.3:21000] is DEAD (Gossip send failed). eventstore2_1 | [00001,13,02:26:28.231] CLUSTER HAS CHANGED (gossip send failed to [172.20.0.3:21000]) eventstore2_1 | [00001,13,02:26:28.231] Old: eventstore2_1 | [00001,13,02:26:28.231] VND {5f978c32-927d-400d-beff-7e13ec9791b9} [Unknown, 172.20.0.4:21004, 172.20.0.4:0, 0.0.0.0:21002, 0.0.0.0:0, 172.20.0.4:21003, 0.0.0.0:21001] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-07-25 02:26:28.072 eventstore2_1 | [00001,13,02:26:28.231] MAN {00000000-0000-0000-0000-000000000000} [Manager, 172.20.0.3:21000, 172.20.0.3:21000] | 2018-07-25 02:26:27.699 eventstore2_1 | [00001,13,02:26:28.231] New: eventstore2_1 | [00001,13,02:26:28.231] VND {5f978c32-927d-400d-beff-7e13ec9791b9} [Unknown, 172.20.0.4:21004, 172.20.0.4:0, 0.0.0.0:21002, 0.0.0.0:0, 172.20.0.4:21003, 0.0.0.0:21001] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-07-25 02:26:28.072 eventstore2_1 | [00001,13,02:26:28.231] MAN {00000000-0000-0000-0000-000000000000} [Manager, 172.20.0.3:21000, 172.20.0.3:21000] | 2018-07-25 02:26:28.230 eventstore2_1 | [00001,13,02:26:28.231] -------------------------------------------------------------------------------- eventstore2_1 | [00001,06,02:26:28.245] SLOW QUEUE MSG [Worker #1]: SendOverHttp - 452ms. Q: 0/0. eventstore2_1 | [00001,07,02:26:28.245] SLOW QUEUE MSG [Worker #4]: SendOverHttp - 249ms. Q: 0/0. eventstore2_1 | [00001,13,02:26:28.954] ELECTIONS: (V=0) TIMED OUT! (S=ElectingLeader, M=). eventstore2_1 | [00001,13,02:26:28.955] ELECTIONS: (V=1) SHIFT TO LEADER ELECTION. eventstore2_1 | [00001,13,02:26:28.955] ELECTIONS: (V=1) VIEWCHANGE FROM [172.20.0.4:21003, {5f978c32-927d-400d-beff-7e13ec9791b9}]. eventstore2_1 | [00001,13,02:26:29.965] ELECTIONS: (V=1) TIMED OUT! (S=ElectingLeader, M=). eventstore2_1 | [00001,13,02:26:29.965] ELECTIONS: (V=2) SHIFT TO LEADER ELECTION. eventstore2_1 | [00001,13,02:26:29.965] ELECTIONS: (V=2) VIEWCHANGE FROM [172.20.0.4:21003, {5f978c32-927d-400d-beff-7e13ec9791b9}]. eventstore2_1 | [00001,13,02:26:30.968] ELECTIONS: (V=2) TIMED OUT! (S=ElectingLeader, M=). eventstore2_1 | [00001,13,02:26:30.968] ELECTIONS: (V=3) SHIFT TO LEADER ELECTION. eventstore2_1 | [00001,13,02:26:30.968] ELECTIONS: (V=3) VIEWCHANGE FROM [172.20.0.4:21003, {5f978c32-927d-400d-beff-7e13ec9791b9}]. eventstore2_1 | [00001,13,02:26:31.971] ELECTIONS: (V=3) TIMED OUT! (S=ElectingLeader, M=). eventstore2_1 | [00001,13,02:26:31.971] ELECTIONS: (V=4) SHIFT TO LEADER ELECTION. eventstore2_1 | [00001,13,02:26:31.971] ELECTIONS: (V=4) VIEWCHANGE FROM [172.20.0.4:21003, {5f978c32-927d-400d-beff-7e13ec9791b9}].