Greetings - I’m trying to run EventStore so it will listen on the interface that is not the loopback interface. I’ve tried many difference configuration combinations, but I keep getting an “Address already in use” error. I need to set the ExtHttpPrefix because there is going to be a proxy/forwarding IP in front of the server. Here are some example configurations I’ve tried (see below) - any ideas on what I’m doing wrong? I’m using version EventStore v3.10 on Ubuntu 14.04.3.
Example Config #1
Capture of the error I’m seeing:
[27818,01,20:57:54.625] MessageHierarchy initialization took 00:00:00.0920889.
[27818,01,20:57:54.681] CACHED TFChunk #0-0 (chunk-000000.000000) in 00:00:00.0010202.
[27818,01,20:57:54.766] Starting MiniWeb for /web/es/js/projections ==> /usr/share/eventstore/projections
[27818,01,20:57:54.766] Starting MiniWeb for /web/es/js/projections/v8/Prelude ==> /usr/share/eventstore/Prelude
[27818,01,20:57:54.767] Binding MiniWeb to /web/es/js/projections/{*remaining_path}
[27818,01,20:57:54.767] Binding MiniWeb to /web/es/js/projections/v8/Prelude/{*remaining_path}
[27818,01,20:57:54.775] Starting MiniWeb for /web ==> /usr/share/eventstore/clusternode-web
[27818,01,20:57:54.776] Binding MiniWeb to /web/{*remaining_path}
[27818,23,20:57:54.789] ========== [127.0.0.1:2112] SYSTEM INIT…
Exiting with exit code: 1.
Exit reason: Address already in use
[27818,23,20:57:54.824] Starting Normal TCP listening on TCP endpoint: 10.8.80.6:1113.
[27818,23,20:57:54.836] Failed to listen on TCP endpoint: 10.8.80.6:1113.
[27818,26,20:57:54.849] TableIndex initialization…
[27818,23,20:57:54.867] Exiting with exit code: 1.
Exit reason: Address already in use
[27818,26,20:57:54.877] ReadIndex building…
[27818,26,20:57:54.878] ReadIndex rebuilding done: total processed 0 records, time elapsed: 00:00:00.0004740.
There is nothing else installed on this system.
Basically its saying that the OS reports Already Open when it tries
to open 10.8.80.6:1113
What does netstat --listen say?
root@vagrant-ubuntu-trusty-64:/home/vagrant# sudo netstat --listen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:ssh : LISTEN
tcp 0 0 *:33848 : LISTEN
tcp 0 0 *:sunrpc : LISTEN
tcp6 0 0 [::]:45297 [::]:* LISTEN
tcp6 0 0 [::]:ssh [::]:* LISTEN
tcp6 0 0 [::]:sunrpc [::]:* LISTEN
udp 0 0 *:6723 :
udp 0 0 *:43667 :
udp 0 0 *:775 :
udp 0 0 localhost:807 :
udp 0 0 *:43007 :
udp 0 0 *:bootpc :
udp 0 0 *:bootpc :
udp 0 0 *:sunrpc :
udp6 0 0 [::]:21059 [::]:*
udp6 0 0 [::]:775 [::]:*
udp6 0 0 [::]:56231 [::]:*
udp6 0 0 [::]:51187 [::]:*
udp6 0 0 [::]:sunrpc [::]:*
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] SEQPACKET LISTENING 7167 /run/udev/control
unix 2 [ ACC ] STREAM LISTENING 8462 /var/run/dbus/system_bus_socket
unix 2 [ ACC ] STREAM LISTENING 6531 @/com/ubuntu/upstart
unix 2 [ ACC ] STREAM LISTENING 7805 /run/rpcbind.sock
unix 2 [ ACC ] STREAM LISTENING 24784 /var/run/docker.sock
unix 2 [ ACC ] STREAM LISTENING 9180 /var/run/acpid.socket
And ifconfig?
When I removed the ExtHttpPrefixes config file setting, the service comes up fine - but then proxied requests return a “Bad Request (Invalid host)” error…
root@vagrant-ubuntu-trusty-64:/home/vagrant# ifconfig
docker0 Link encap:Ethernet HWaddr 02:42:ba:52:e6:e7
inet addr:172.17.42.1 Bcast:0.0.0.0 Mask:255.255.0.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
eth0 Link encap:Ethernet HWaddr 08:00:27:c1:1a:d2
inet addr:10.0.2.15 Bcast:10.0.2.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fec1:1ad2/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:128710 errors:0 dropped:0 overruns:0 frame:0
TX packets:57782 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:108471068 (108.4 MB) TX bytes:3663276 (3.6 MB)
eth1 Link encap:Ethernet HWaddr 08:00:27:ae:07:37
inet addr:10.8.80.6 Bcast:10.8.80.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:feae:737/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:5170 errors:0 dropped:0 overruns:0 frame:0
TX packets:513 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:437442 (437.4 KB) TX bytes:1121106 (1.1 MB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
@pieter is this involved with the new things added around register interfaces?
I dropped a message to pieter.
What happens if you come up as default? IIRC 3.1.0 default binds to
*.2113 now and 0.0.0.0 as default
Thanks!
Hi Jordan,
Sorry for the hassle that the new change might have caused you. We tried to make things easier by handling the special case of setting the Internal/External IP Addresses to 0.0.0.0.
By default when you set the Internal or External IP Addresses to 0.0.0.0.
We will attempt to setup prefixes for all the IPV4 addresses on the box.
e.g.
http://192.168.0.1:2113/, http://127.0.0.1:2113/, http://localhost:2113
will be added to the External Http Prefixes
What is happening here is that you are setting http://*:2113 in addition to us adding prefixes, which means that essentially the prefixes have a double entry…
To satisfy your use case, you just have to set 0.0.0.0 and we do the rest.
We added a new flag called “–add-interface-prefixes” which adds int/ext http prefixes, which is set to true by default which does the above mentioned.
Thanks for the details Pieter. That does solve the problem of getting the “Address already in use” error (indicated in a previous post), but it doesn’t solve the problem I was seeing that motivated me to set the ExtHttpPrefixes property to begin with. I’m accessing the event store through a proxy/forwarding IP address (ie an IP address that is not assigned to the system running EventStore). When I make a request to the EventStore via this proxied/forwarding IP address, I get a “Bad Request (Invalid host)” error message. In version 3.0.3 of EventStore, setting the ExtHttpPrefix to “http://*:2113” solved that problem. \
Basically, I want to allow ALL incoming HTTP prefixes - how can I do this with EventStore >= v3.1.0?
Thanks - Jordan
ExtHttpPrefix: http://*:2113
--add-interface-prefixes=false
IIRC
Your 2nd config would work if you add the AddInterfacePrefixes:false as Greg has mentioned.
I still get the same “Address already in use” error (full output included below):
[14635,01,17:40:16.822]
ES VERSION: 20 (HEAD/1d91d2553dd10f1a97cc4e8d3fae0046265e9a28, Thu, 6 Aug 2015 11:13:43 +0200)
OS: Unknown (Unix 3.13.0.62)
RUNTIME: 3.12.1 (tarball) (64-bit)
GC: 2 GENERATIONS
LOGS: /var/log/eventstore
MODIFIED OPTIONS:
CONFIG: /etc/eventstore/eventstore.conf (Config File)
RUN PROJECTIONS: None (Config File)
CLUSTER SIZE: 1 (Config File)
EXT IP: 0.0.0.0 (Config File)
EXT HTTP PREFIXES: http://*:2113/ (Config File)
ADD INTERFACE PREFIXES: False (Config File)
DEFAULT OPTIONS:
HELP: False ()
VERSION: False ()
LOG: /var/log/eventstore ()
DEFINES: ()
WHAT IF: False ()
DEVELOPMENT MODE: False ()
MONO MIN THREADPOOL SIZE: 10 ()
INT IP: 127.0.0.1 ()
INT HTTP PORT: 2112 ()
EXT HTTP PORT: 2113 ()
INT TCP PORT: 1112 ()
INT SECURE TCP PORT: 0 ()
EXT TCP PORT: 1113 ()
EXT SECURE TCP PORT: 0 ()
EXT IP ADVERTISE AS: ()
EXT TCP PORT ADVERTISE AS: 0 ()
EXT HTTP PORT ADVERTISE AS: 0 ()
INT TCP HEARTBEAT TIMEOUT: 700 ()
EXT TCP HEARTBEAT TIMEOUT: 1000 ()
INT TCP HEARTBEAT INTERVAL: 700 ()
EXT TCP HEARTBEAT INTERVAL: 2000 ()
FORCE: False ()
NODE PRIORITY: 0 ()
MIN FLUSH DELAY MS: 2 ()
COMMIT COUNT: -1 ()
PREPARE COUNT: -1 ()
ADMIN ON EXT: True ()
STATS ON EXT: True ()
GOSSIP ON EXT: True ()
DISABLE SCAVENGE MERGING: False ()
DISCOVER VIA DNS: True ()
CLUSTER DNS: fake.dns ()
CLUSTER GOSSIP PORT: 30777 ()
GOSSIP SEED: ()
STATS PERIOD SEC: 30 ()
CACHED CHUNKS: -1 ()
CHUNKS CACHE SIZE: 536871424 ()
MAX MEM TABLE SIZE: 1000000 ()
DB: /var/lib/eventstore ()
MEM DB: False ()
SKIP DB VERIFY: False ()
PROJECTION THREADS: 3 ()
WORKER THREADS: 5 ()
INT HTTP PREFIXES: ()
ENABLE TRUSTED AUTH: False ()
CERTIFICATE STORE LOCATION: ()
CERTIFICATE STORE NAME: ()
CERTIFICATE SUBJECT NAME: ()
CERTIFICATE THUMBPRINT: ()
CERTIFICATE FILE: ()
CERTIFICATE PASSWORD: ()
USE INTERNAL SSL: False ()
SSL TARGET HOST: n/a ()
SSL VALIDATE SERVER: True ()
AUTHENTICATION TYPE: internal ()
PREPARE TIMEOUT MS: 2000 ()
COMMIT TIMEOUT MS: 2000 ()
UNSAFE DISABLE FLUSH TO DISK: False ()
GOSSIP INTERVAL MS: 1000 ()
GOSSIP ALLOWED DIFFERENCE MS: 60000 ()
GOSSIP TIMEOUT MS: 500 ()
[14635,01,17:40:16.842] Quorum size set to 1
[14635,01,17:40:16.845] Can’t find plugins path: plugins
[14635,01,17:40:16.886]
INSTANCE ID: 763f615b-cd8b-4f1e-876c-dbae7756bddc
DATABASE: /var/lib/eventstore
WRITER CHECKPOINT: 73639 (0x11FA7)
CHASER CHECKPOINT: 73639 (0x11FA7)
EPOCH CHECKPOINT: 1046 (0x416)
TRUNCATE CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF)
[14635,01,17:40:16.999] MessageHierarchy initialization took 00:00:00.0812300.
[14635,01,17:40:17.055] CACHED TFChunk #0-0 (chunk-000000.000000) in 00:00:00.0014662.
[14635,01,17:40:17.216] Starting MiniWeb for /web/es/js/projections ==> /usr/share/eventstore/projections
[14635,01,17:40:17.216] Starting MiniWeb for /web/es/js/projections/v8/Prelude ==> /usr/share/eventstore/Prelude
[14635,01,17:40:17.217] Binding MiniWeb to /web/es/js/projections/{*remaining_path}
[14635,01,17:40:17.217] Binding MiniWeb to /web/es/js/projections/v8/Prelude/{*remaining_path}
[14635,01,17:40:17.229] Starting MiniWeb for /web ==> /usr/share/eventstore/clusternode-web
[14635,01,17:40:17.230] Binding MiniWeb to /web/{*remaining_path}
[14635,23,17:40:17.252] ========== [127.0.0.1:2112] SYSTEM INIT…
[14635,23,17:40:17.288] Starting Normal TCP listening on TCP endpoint: 0.0.0.0:1113.
[14635,26,17:40:17.307] TableIndex initialization…
[14635,26,17:40:17.318] ReadIndex building…
Exiting with exit code: 1.
Exit reason: HTTP async server failed to start listening at [http://*:2113/, http://localhost:2113/].
[14635,23,17:40:17.330] Starting HTTP server on [http://*:2113/,http://localhost:2113/]…
[14635,23,17:40:17.347] Failed to start http server
Address already in use
[14635,26,17:40:17.355] ReadIndex rebuilding done: total processed 13 records, time elapsed: 00:00:00.0363880.
[14635,23,17:40:17.381] Exiting with exit code: 1.
Exit reason: HTTP async server failed to start listening at [http://*:2113/, http://localhost:2113/].
@pieter - That worked (setting the IP to something other than 0.0.0.0). Although I can move forward now with these changes, it would be nice (for simplicity’s sake) to be able to set the IP to 0.0.0.0. Ultimately, my setup will go into a Docker container, so the IP address of the Docker container’s eth0 interface is not what will be used externally - the Docker host IP will.
Thanks Greg & Pieter for your help!
We will definitely take a look at this. I do apologise for the frustration in getting this to work.
No problem. Thanks again for the help!