Starting apexeventseventstoretest_eventstore1_1 ... Starting apexeventseventstoretest_eventstore1_1 ... done Attaching to apexeventseventstoretest_eventstore3_1, apexeventseventstoretest_eventstore2_1, apexeventseventstoretest_eventstore1_1 eventstore1_1 | [00001,01,02:26:28.507] eventstore1_1 | ES VERSION: 4.1.0.0 (HEAD/daddeca886f43e40489efd7d9f45cb6ddd4aed8a, Tue, 23 Jan 2018 14:12:35 +0200) eventstore1_1 | [00001,01,02:26:28.540] OS: Linux (Unix 4.9.49.0) eventstore1_1 | [00001,01,02:26:28.544] RUNTIME: 4.6.2 (Stable 4.6.2.16/ac9e222) (64-bit) eventstore1_1 | [00001,01,02:26:28.545] GC: 2 GENERATIONS eventstore1_1 | [00001,01,02:26:28.545] LOGS: /var/log/eventstore eventstore1_1 | [00001,01,02:26:28.554] MODIFIED OPTIONS: eventstore1_1 | eventstore1_1 | START STANDARD PROJECTIONS: True (Environment Variable) eventstore1_1 | INT IP: 172.20.0.3 (Environment Variable) eventstore1_1 | INT HTTP PORT: 11003 (Environment Variable) eventstore1_1 | EXT HTTP PORT: 11001 (Environment Variable) eventstore1_1 | INT TCP PORT: 11004 (Environment Variable) eventstore1_1 | EXT TCP PORT: 11002 (Environment Variable) eventstore1_1 | EXT TCP PORT ADVERTISE AS: 11002 (Environment Variable) eventstore1_1 | EXT HTTP PORT ADVERTISE AS: 11001 (Environment Variable) eventstore1_1 | INT TCP PORT ADVERTISE AS: 11004 (Environment Variable) eventstore1_1 | INT HTTP PORT ADVERTISE AS: 11003 (Environment Variable) eventstore1_1 | CLUSTER SIZE: 3 (Environment Variable) eventstore1_1 | CLUSTER DNS: eventstore1 (Environment Variable) eventstore1_1 | CLUSTER GOSSIP PORT: 11000 (Environment Variable) eventstore1_1 | RUN PROJECTIONS: All (Environment Variable) eventstore1_1 | INT HTTP PREFIXES: http://*:11003/ (Environment Variable) eventstore1_1 | EXT HTTP PREFIXES: http://*:11001/ (Environment Variable) eventstore1_1 | EXT IP: 0.0.0.0 (Config File) eventstore1_1 | ADD INTERFACE PREFIXES: false (Config File) eventstore1_1 | eventstore1_1 | DEFAULT OPTIONS: eventstore1_1 | eventstore1_1 | CONFIG: /etc/eventstore/eventstore.conf () eventstore1_1 | HELP: False () eventstore1_1 | VERSION: False () eventstore1_1 | LOG: /var/log/eventstore () eventstore1_1 | DEFINES: () eventstore1_1 | WHAT IF: False () eventstore1_1 | DISABLE HTTP CACHING: False () eventstore1_1 | MONO MIN THREADPOOL SIZE: 10 () eventstore1_1 | INT SECURE TCP PORT: 0 () eventstore1_1 | EXT SECURE TCP PORT ADVERTISE AS: 0 () eventstore1_1 | EXT SECURE TCP PORT: 0 () eventstore1_1 | EXT IP ADVERTISE AS: () eventstore1_1 | INT IP ADVERTISE AS: () eventstore1_1 | INT SECURE TCP PORT ADVERTISE AS: 0 () eventstore1_1 | INT TCP HEARTBEAT TIMEOUT: 700 () eventstore1_1 | EXT TCP HEARTBEAT TIMEOUT: 1000 () eventstore1_1 | INT TCP HEARTBEAT INTERVAL: 700 () eventstore1_1 | EXT TCP HEARTBEAT INTERVAL: 2000 () eventstore1_1 | GOSSIP ON SINGLE NODE: False () eventstore1_1 | CONNECTION PENDING SEND BYTES THRESHOLD: 10485760 () eventstore1_1 | FORCE: False () eventstore1_1 | NODE PRIORITY: 0 () eventstore1_1 | MIN FLUSH DELAY MS: 2 () eventstore1_1 | COMMIT COUNT: -1 () eventstore1_1 | PREPARE COUNT: -1 () eventstore1_1 | ADMIN ON EXT: True () eventstore1_1 | STATS ON EXT: True () eventstore1_1 | GOSSIP ON EXT: True () eventstore1_1 | DISABLE SCAVENGE MERGING: False () eventstore1_1 | SCAVENGE HISTORY MAX AGE: 30 () eventstore1_1 | DISCOVER VIA DNS: True () eventstore1_1 | GOSSIP SEED: () eventstore1_1 | STATS PERIOD SEC: 30 () eventstore1_1 | CACHED CHUNKS: -1 () eventstore1_1 | READER THREADS COUNT: 4 () eventstore1_1 | CHUNKS CACHE SIZE: 536871424 () eventstore1_1 | MAX MEM TABLE SIZE: 1000000 () eventstore1_1 | HASH COLLISION READ LIMIT: 100 () eventstore1_1 | DB: /var/lib/eventstore () eventstore1_1 | INDEX: () eventstore1_1 | MEM DB: False () eventstore1_1 | SKIP DB VERIFY: False () eventstore1_1 | WRITE THROUGH: False () eventstore1_1 | UNBUFFERED: False () eventstore1_1 | CHUNK INITIAL READER COUNT: 5 () eventstore1_1 | PROJECTION THREADS: 3 () eventstore1_1 | WORKER THREADS: 5 () eventstore1_1 | PROJECTIONS QUERY EXPIRY: 0 () eventstore1_1 | ENABLE TRUSTED AUTH: False () eventstore1_1 | CERTIFICATE STORE LOCATION: () eventstore1_1 | CERTIFICATE STORE NAME: () eventstore1_1 | CERTIFICATE SUBJECT NAME: () eventstore1_1 | CERTIFICATE THUMBPRINT: () eventstore1_1 | CERTIFICATE FILE: () eventstore1_1 | CERTIFICATE PASSWORD: () eventstore1_1 | USE INTERNAL SSL: False () eventstore1_1 | DISABLE INSECURE TCP: False () eventstore1_1 | SSL TARGET HOST: n/a () eventstore1_1 | SSL VALIDATE SERVER: True () eventstore1_1 | AUTHENTICATION TYPE: internal () eventstore1_1 | AUTHENTICATION CONFIG: () eventstore1_1 | PREPARE TIMEOUT MS: 2000 () eventstore1_1 | COMMIT TIMEOUT MS: 2000 () eventstore1_1 | UNSAFE DISABLE FLUSH TO DISK: False () eventstore1_1 | BETTER ORDERING: False () eventstore1_1 | UNSAFE IGNORE HARD DELETE: False () eventstore1_1 | SKIP INDEX VERIFY: False () eventstore1_1 | INDEX CACHE DEPTH: 16 () eventstore1_1 | GOSSIP INTERVAL MS: 1000 () eventstore1_1 | GOSSIP ALLOWED DIFFERENCE MS: 60000 () eventstore1_1 | GOSSIP TIMEOUT MS: 500 () eventstore1_1 | ENABLE HISTOGRAMS: False () eventstore1_1 | LOG HTTP REQUESTS: False () eventstore1_1 | ALWAYS KEEP SCAVENGED: False () eventstore1_1 | SKIP INDEX SCAN ON READS: False () eventstore1_1 | eventstore1_1 | [00001,01,02:26:28.561] Quorum size set to 2 eventstore1_1 | [00001,01,02:26:28.573] Cannot find plugins path: plugins eventstore1_1 | [00001,01,02:26:28.658] INSTANCE ID: 26521389-4fdc-4bd6-bc99-1ff7ecdc9810 eventstore1_1 | [00001,01,02:26:28.659] DATABASE: /var/lib/eventstore eventstore1_1 | [00001,01,02:26:28.659] WRITER CHECKPOINT: 0 (0x0) eventstore1_1 | [00001,01,02:26:28.660] CHASER CHECKPOINT: 0 (0x0) eventstore1_1 | [00001,01,02:26:28.660] EPOCH CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF) eventstore1_1 | [00001,01,02:26:28.660] TRUNCATE CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF) eventstore1_1 | [00001,01,02:26:28.844] MessageHierarchy initialization took 00:00:00.1515999. eventstore1_1 | [00001,01,02:26:28.908] Opened ongoing /var/lib/eventstore/chunk-000000.000000 as version 3 eventstore1_1 | [00001,01,02:26:28.917] CACHED TFChunk #0-0 (chunk-000000.000000) in 00:00:00.0029420. eventstore1_1 | [00001,01,02:26:29.067] Starting MiniWeb for /web/es/js/projections ==> /usr/share/eventstore/projections eventstore1_1 | [00001,01,02:26:29.067] Starting MiniWeb for /web/es/js/projections/v8/Prelude ==> /usr/share/eventstore/Prelude eventstore1_1 | [00001,01,02:26:29.068] Binding MiniWeb to /web/es/js/projections/{*remaining_path} eventstore1_1 | [00001,01,02:26:29.068] Binding MiniWeb to /web/es/js/projections/v8/Prelude/{*remaining_path} eventstore1_1 | [00001,01,02:26:29.069] Binding MiniWeb to /web/es/js/projections/{*remaining_path} eventstore1_1 | [00001,01,02:26:29.069] Binding MiniWeb to /web/es/js/projections/v8/Prelude/{*remaining_path} eventstore1_1 | [00001,01,02:26:29.082] Starting MiniWeb for /web ==> /usr/share/eventstore/clusternode-web eventstore1_1 | [00001,01,02:26:29.082] Binding MiniWeb to /web/{*remaining_path} eventstore1_1 | [00001,01,02:26:29.083] Starting MiniWeb for /web ==> /usr/share/eventstore/clusternode-web eventstore1_1 | [00001,01,02:26:29.083] Binding MiniWeb to /web/{*remaining_path} eventstore1_1 | [00001,14,02:26:29.113] ========== [172.20.0.3:11003] SYSTEM INIT... eventstore1_1 | [00001,15,02:26:29.152] TableIndex initialization... eventstore1_1 | [00001,14,02:26:29.152] Starting Normal TCP listening on TCP endpoint: 0.0.0.0:11002. eventstore1_1 | [00001,14,02:26:29.156] Starting Normal TCP listening on TCP endpoint: 172.20.0.3:11004. eventstore1_1 | [00001,15,02:26:29.165] ReadIndex building... eventstore1_1 | [00001,15,02:26:29.166] ReadIndex rebuilding done: total processed 0 records, time elapsed: 00:00:00.0010430. eventstore1_1 | [00001,14,02:26:29.185] Starting HTTP server on [http://*:11001/]... eventstore1_1 | [00001,14,02:26:29.191] HTTP server is up and listening on [http://*:11001/] eventstore1_1 | [00001,14,02:26:29.191] Starting HTTP server on [http://*:11003/]... eventstore1_1 | [00001,14,02:26:29.191] HTTP server is up and listening on [http://*:11003/] eventstore1_1 | [00001,14,02:26:29.214] SLOW QUEUE MSG [MainQueue]: SystemInit - 112ms. Q: 0/7. eventstore1_1 | [00001,14,02:26:29.214] ========== [172.20.0.3:11003] Service 'StorageWriter' initialized. eventstore1_1 | [00001,14,02:26:29.214] ========== [172.20.0.3:11003] Service 'StorageReader' initialized. eventstore1_1 | [00001,14,02:26:29.215] ========== [172.20.0.3:11003] Service 'StorageChaser' initialized. eventstore1_1 | [00001,14,02:26:29.247] CLUSTER HAS CHANGED eventstore1_1 | [00001,14,02:26:29.247] Old: eventstore1_1 | [00001,14,02:26:29.247] VND {26521389-4fdc-4bd6-bc99-1ff7ecdc9810} [Unknown, 172.20.0.3:11004, 172.20.0.3:0, 0.0.0.0:11002, 0.0.0.0:0, 172.20.0.3:11003, 0.0.0.0:11001] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-07-25 02:26:29.192 eventstore1_1 | [00001,14,02:26:29.248] New: eventstore1_1 | [00001,14,02:26:29.248] VND {26521389-4fdc-4bd6-bc99-1ff7ecdc9810} [Unknown, 172.20.0.3:11004, 172.20.0.3:0, 0.0.0.0:11002, 0.0.0.0:0, 172.20.0.3:11003, 0.0.0.0:11001] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-07-25 02:26:29.192 eventstore1_1 | [00001,14,02:26:29.248] MAN {00000000-0000-0000-0000-000000000000} [Manager, 172.20.0.3:11000, 172.20.0.3:11000] | 2018-07-25 02:26:29.217 eventstore1_1 | [00001,14,02:26:29.248] -------------------------------------------------------------------------------- eventstore1_1 | [00001,14,02:26:29.251] ========== [172.20.0.3:11003] SYSTEM START... eventstore1_1 | [00001,14,02:26:29.261] ========== [172.20.0.3:11003] IS UNKNOWN... eventstore1_1 | [00001,06,02:26:29.323] Subscriptions received state change to Unknown stopping listening. eventstore1_1 | [00001,14,02:26:29.362] ELECTIONS: STARTING ELECTIONS. eventstore1_1 | [00001,14,02:26:29.373] ELECTIONS: (V=0) SHIFT TO LEADER ELECTION. eventstore1_1 | [00001,14,02:26:29.375] ELECTIONS: (V=0) VIEWCHANGE FROM [172.20.0.3:11003, {26521389-4fdc-4bd6-bc99-1ff7ecdc9810}]. eventstore1_1 | [00001,14,02:26:29.393] SLOW BUS MSG [MainBus]: StartElections - 111ms. Handler: ElectionsService. eventstore1_1 | [00001,14,02:26:29.394] SLOW QUEUE MSG [MainQueue]: StartElections - 125ms. Q: 0/4. eventstore1_1 | [00001,06,02:26:29.394] SLOW BUS MSG [PersistentSubscriptionsBus]: BecomeUnknown - 69ms. Handler: PersistentSubscriptionService. eventstore1_1 | [00001,08,02:26:29.543] SLOW BUS MSG [Worker #1 Bus]: SendOverHttp - 277ms. Handler: HttpSendService. eventstore1_1 | [00001,08,02:26:29.567] SLOW QUEUE MSG [Worker #1]: SendOverHttp - 278ms. Q: 0/0. eventstore1_1 | [00001,14,02:26:29.570] Looks like node [172.20.0.3:11000] is DEAD (Gossip send failed). eventstore1_1 | [00001,14,02:26:29.571] CLUSTER HAS CHANGED (gossip send failed to [172.20.0.3:11000]) eventstore1_1 | [00001,14,02:26:29.571] Old: eventstore1_1 | [00001,14,02:26:29.571] VND {26521389-4fdc-4bd6-bc99-1ff7ecdc9810} [Unknown, 172.20.0.3:11004, 172.20.0.3:0, 0.0.0.0:11002, 0.0.0.0:0, 172.20.0.3:11003, 0.0.0.0:11001] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-07-25 02:26:29.505 eventstore1_1 | [00001,14,02:26:29.572] MAN {00000000-0000-0000-0000-000000000000} [Manager, 172.20.0.3:11000, 172.20.0.3:11000] | 2018-07-25 02:26:29.217 eventstore1_1 | [00001,14,02:26:29.572] New: eventstore1_1 | [00001,14,02:26:29.572] VND {26521389-4fdc-4bd6-bc99-1ff7ecdc9810} [Unknown, 172.20.0.3:11004, 172.20.0.3:0, 0.0.0.0:11002, 0.0.0.0:0, 172.20.0.3:11003, 0.0.0.0:11001] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2018-07-25 02:26:29.505 eventstore1_1 | [00001,14,02:26:29.578] MAN {00000000-0000-0000-0000-000000000000} [Manager, 172.20.0.3:11000, 172.20.0.3:11000] | 2018-07-25 02:26:29.571 eventstore1_1 | [00001,14,02:26:29.578] -------------------------------------------------------------------------------- eventstore1_1 | [00001,14,02:26:30.398] ELECTIONS: (V=0) TIMED OUT! (S=ElectingLeader, M=). eventstore1_1 | [00001,14,02:26:30.398] ELECTIONS: (V=1) SHIFT TO LEADER ELECTION. eventstore1_1 | [00001,14,02:26:30.398] ELECTIONS: (V=1) VIEWCHANGE FROM [172.20.0.3:11003, {26521389-4fdc-4bd6-bc99-1ff7ecdc9810}]. eventstore1_1 | [00001,14,02:26:31.402] ELECTIONS: (V=1) TIMED OUT! (S=ElectingLeader, M=). eventstore1_1 | [00001,14,02:26:31.402] ELECTIONS: (V=2) SHIFT TO LEADER ELECTION. eventstore1_1 | [00001,14,02:26:31.402] ELECTIONS: (V=2) VIEWCHANGE FROM [172.20.0.3:11003, {26521389-4fdc-4bd6-bc99-1ff7ecdc9810}]. eventstore1_1 | [00001,14,02:26:32.404] ELECTIONS: (V=2) TIMED OUT! (S=ElectingLeader, M=). eventstore1_1 | [00001,14,02:26:32.404] ELECTIONS: (V=3) SHIFT TO LEADER ELECTION. eventstore1_1 | [00001,14,02:26:32.404] ELECTIONS: (V=3) VIEWCHANGE FROM [172.20.0.3:11003, {26521389-4fdc-4bd6-bc99-1ff7ecdc9810}].