Issues when slave is forwarding to master

EventStoreDB
1

Hi,

We are seeing some issues when we are hitting the slave and it tries to forward to master. We have configured ports as shown in this sample

[http://docs.geteventstore.com/configuration/cluster-without-manager-no](http://docs.geteventstore.com/configuration/cluster-without-manager-no)[des/](http://docs.geteventstore.com/configuration/cluster-without-manager-nodes/)

1:09 PM [PID:01021:025 2014.11.18 12:07:57.660 DEBUG HttpSendService ] Error while forwarding request body from ‘http://1.1.1.1:2114/streams/SystemEvents’ to ‘http://1.1.1.2:2113/streams/SystemEvents’ (POST): The object was used after being disposed…

-Svein Arne

2

Svein,

Are the requests failing after this message?

James

3

Is this on an occasional request or on all of them?

4

Semed like all request on slave

5

OK is it possible the slave cannot talk to the master?

6

They can talk. Although, got the same type of error when we accidentally connected to the wrong port (1111 or 1112) to our F# client just now. Maybe the config is off. Although we just followed the sample above.

7

I have the same error message when the slaves forward to master. Did you find the problem?

8

Put up config. My guess is they cant connect to master are you using virtual ips etc in azure aws etc?

9

the cluster nodes are started with

/opt/eventstore/clusternode --http-prefixes=http://evscluster01t.example.org:2113/,http://eventstore01t.example.org:2113/ --config=/etc/eventstore/eventstore.yml

The config file is as follows:

Log: /var/log/eventstore/

Db: /data/event-store

WorkerThreads: 16

IntIp: 10.3.0.36

ExtIp: 0.0.0.0

ClusterSize: 3

DiscoverViaDns: true

ClusterGossipPort: 2112

#ClusterDns:

ClusterDns: “evscluster01t.example.org”

it is mono on linux.

the error message is Error while forwarding request body from ‘http://eventstore01t.example.org:2113/streams/asdf45’ to ‘http://10.3.0.36:2112/streams/asdf45’ (POST): The object was used after being disposed…

I can curl existing streams curl http://10.3.0.35:2112/streams/asdf39, curl http://10.3.0.36:2112/streams/asdf39, http://10.3.0.37:2112/streams/asdf39 from each of the three nodes, it does return a atom feed.

From the nodes I can curl -i -d@/tmp/simple-event.txt “http://10.3.0.36:2112/streams/foo” -H “Content-Type:application/vnd.eventstore.events+json” to the master but not to the slaves. Slaves return 400 Bad request (Invalid Host), Master return 201 Created.

So, as far as I can tell, they are able to use DNS discovery to form the cluster. I can connect from slaves to master. But do I need to add the 10.3.0.xx to the http-prefixes?

10

Looking through this.

11

Got some more configuration from the log file:

ES VERSION: 3.0.1.0 (release-oss-v3.0.1/7fa876c111888dd5980dbd86d126e6abe13b05ab, Thu, 23 Oct 2014 22:27:04 +0100)

OS: Unknown (Unix 3.10.0.123)

RUNTIME: 3.8.0 ((detached/45d0ba1) (64-bit)

GC: 2 GENERATIONS

LOGS: /var/log/eventstore/

HELP: False ()

VERSION: False ()

LOG: /var/log/eventstore/ (Config File)

CONFIG: /etc/eventstore/eventstore.yml (Command Line)

DEFINES: ()

WHAT IF: False ()

INT IP: 10.3.0.35 (Config File)

EXT IP: 0.0.0.0 (Config File)

INT HTTP PORT: 2112 ()

EXT HTTP PORT: 2113 ()

INT TCP PORT: 1112 ()

INT SECURE TCP PORT: 0 ()

EXT TCP PORT: 1113 ()

EXT SECURE TCP PORT: 0 ()

INT TCP HEARTBEAT TIMEOUT: 700 ()

EXT TCP HEARTBEAT TIMEOUT: 1000 ()

INT TCP HEARTBEAT INTERVAL: 700 ()

EXT TCP HEARTBEAT INTERVAL: 2000 ()

FORCE: False ()

CLUSTER SIZE: 3 (Config File)

NODE PRIORITY: 0 ()

MIN FLUSH DELAY MS: 2 ()

COMMIT COUNT: -1 ()

PREPARE COUNT: -1 ()

ADMIN ON EXT: True ()

STATS ON EXT: True ()

GOSSIP ON EXT: True ()

DISABLE SCAVENGE MERGING: False ()

DISCOVER VIA DNS: true (Config File)

CLUSTER DNS: evscluster01t.bva.nu (Config File)

CLUSTER GOSSIP PORT: 2112 (Config File)

GOSSIP SEED: ()

STATS PERIOD SEC: 30 ()

CACHED CHUNKS: -1 ()

CHUNKS CACHE SIZE: 536871424 ()

MAX MEM TABLE SIZE: 1000000 ()

DB: /data/event-store (Config File)

MEM DB: False ()

SKIP DB VERIFY: False ()

RUN PROJECTIONS: System ()

PROJECTION THREADS: 3 ()

WORKER THREADS: 16 (Config File)

HTTP PREFIXES: [ http://evscluster01t.bva.nu:2113/, http://eventstore01t.bva.nu:2113/ ] (Command Line)

ENABLE TRUSTED AUTH: False ()

CERTIFICATE STORE LOCATION: ()

CERTIFICATE STORE NAME: ()

CERTIFICATE SUBJECT NAME: ()

CERTIFICATE THUMBPRINT: ()

CERTIFICATE FILE: ()

CERTIFICATE PASSWORD: ()

USE INTERNAL SSL: False ()

SSL TARGET HOST: n/a ()

SSL VALIDATE SERVER: True ()

AUTHENTICATION TYPE: internal ()

PREPARE TIMEOUT MS: 2000 ()

COMMIT TIMEOUT MS: 2000 ()

UNSAFE DISABLE FLUSH TO DISK: False ()

GOSSIP INTERVAL MS: 1000 ()

GOSSIP ALLOWED DIFFERENCE MS: 60000 ()

GOSSIP TIMEOUT MS: 500 ()

[40002,01,13:41:20.191] Quorum size set to 2

[40002,01,13:41:20.195] Can’t find plugins path: plugins

[40002,01,13:41:20.232]

INSTANCE ID: cf2a645e-eea9-4d34-a60b-d217632b5de3

DATABASE: /data/event-store

WRITER CHECKPOINT: 44287890 (0x2A3C792)

CHASER CHECKPOINT: 44287890 (0x2A3C792)

EPOCH CHECKPOINT: 44060655 (0x2A04FEF)

TRUNCATE CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF)

Later in the log I see messages that it has started the http://10.3.0.35:2112

[40002,10,13:41:20.997] Starting Normal TCP listening on TCP endpoint: 0.0.0.0:1113.

[40002,10,13:41:21.032] SLOW BUS MSG [MainBus]: SystemInit - 51ms. Handler: TcpService.

[40002,10,13:41:21.032] Starting Normal TCP listening on TCP endpoint: 10.3.0.35:1112.

[40002,10,13:41:21.061] Starting HTTP server on [http://evscluster01t.bva.nu:2113/,http://eventstore01t.bva.nu:2113/]…

[40002,10,13:41:21.082] HTTP server is up and listening on [http://evscluster01t.bva.nu:2113/,http://eventstore01t.bva.nu:2113/]

[40002,10,13:41:21.082] SLOW BUS MSG [MainBus]: SystemInit - 49ms. Handler: HttpService.

[40002,10,13:41:21.082] Starting HTTP server on [http://10.3.0.35:2112/]…

[40002,10,13:41:21.082] HTTP server is up and listening on [http://10.3.0.35:2112/]

Then later the outcome of the leader selection:

[40002,10,13:41:30.525] ========== [10.3.0.35:2112] SLAVE ASSIGNMENT RECEIVED FROM [10.3.0.36:1112,n/a,{a63f0a22-df89-469b-911e-f5d6990a952e}].

[40002,10,13:41:30.526] ========== [10.3.0.35:2112] IS SLAVE!!! SPARTA!!! MASTER IS [10.3.0.36:2112,{a63f0a22-df89-469b-911e-f5d6990a952e}]

Is a write from a client to master replicated using the InternalTcpPort?

Where as a client writing to a slave will forward the http request to the masterInternalIp:internalHttpPort?

Is there some way to increase the log level?

Regards,

Mats Svefors

12

ok, the problem was solved by adding http://hostname:gossipport to the http-prefix list.

13

Cool.

14

My solution has made it possible for non members of the cluster to write events to http://hostname:gossipport/streams/stream. Is this a concern?

15

Not sure what you mean? Normally any writer would write to the cluster?

16

Is there a difference in how the nodes handle a http request to the Internal vs External Port? My assumption was that the http requests to the internal port was handled in a different way, if not, why isn’t the slave forwarding the request to the external rather than the internal http port?

17

The external may not be reachable…