Framing Error

EventStoreDB
1

Why do I get this when I try to connect to a cluster with .Net Client api?

[21168,07,14:27:17.675] Internal TCP connection accepted: [Normal, 127.0.0.1:116
08, L127.0.0.1:2111, {e760b858-290c-40bb-aef3-212e05d0964c}].
[21168,26,14:27:17.691] FRAMING ERROR! Data:
000000: 47 45 54 20 2F 67 6F 73 73 69 70 3F 66 6F 72 6D | GET /gossip?form
000016: 61 74 3D 6A 73 6F 6E 20 48 54 54 50 2F 31 2E 31 | at=json HTTP/1.1
000032: 0D 0A 48 6F 73 74 3A 20 31 32 37 2E 30 2E 30 2E | …Host: 127.0.0.
000048: 31 3A 32 31 31 31 0D 0A 43 6F 6E 6E 65 63 74 69 | 1:2111…Connecti
000064: 6F 6E 3A 20 4B 65 65 70 2D 41 6C 69 76 65 0D 0A | on: Keep-Alive…
000080: 0D 0A | …

[21168,07,14:27:17.691] ES TcpConnection closed [14:27:17.691: N127.0.0.1:11608,
L127.0.0.1:2111, {e760b858-290c-40bb-aef3-212e05d0964c}]:
Received bytes: 82, Sent bytes: 0
Send calls: 1, callbacks: 0
Receive calls: 2, callbacks: 2
Close reason: [Success] Socket closed

[21168,07,14:27:17.691] Connection ‘internal-normal’ [127.0.0.1:11608, {e760b858
-290c-40bb-aef3-212e05d0964c}] closed: Success.

I send in the IPEndpoints of all nodes in the cluster:

IPAddress address;

IPAddress.TryParse(“127.0.0.1”, out address);

var ep1 = new IPEndPoint(address, 1111);

var ep2 = new IPEndPoint(address, 2111);

var ep3 = new IPEndPoint(address, 3111);

return new[] { ep1, ep2, ep3 };

then

var clusterSettings = ClusterSettings.Create()

.DiscoverClusterViaGossipSeeds()

.SetGossipSeedEndPoints(_endPoints)

.SetGossipTimeout(new TimeSpan(0, 0, 30))

.SetMaxDiscoverAttempts(10)

.Build();

var connectionSettings = ConnectionSettings.Create()

.SetDefaultUserCredentials(new UserCredentials(“admin”, “changeit”))

.SetHeartbeatInterval(new TimeSpan(0, 0, 0, 0, 500))

.SetHeartbeatTimeout(new TimeSpan(0, 0, 0, 0, 1000))

.SetReconnectionDelayTo(new TimeSpan(0, 0, 10))

.WithConnectionTimeoutOf(new TimeSpan(0, 0, 15))

.KeepReconnecting()

.KeepRetrying()

.PerformOnMasterOnly()

.Build();

var connection = EventStoreConnection.Create(connectionSettings, clusterSettings);

and then connectasync, and then the GetEventStoreRepository from common domain.

I get an InvalidOperationException and my nodes show the errors above.

2

correction: I get the invalidoperationexception when I try to save an aggregate with AppendToStreamAsync

3

Looks like you are connecting the tcp client to the http port :slight_smile:

4

No, they really are the tcp internal ports! This is what I use:

start EventStore.ClusterNode.exe --mem-db --log .\logs\log1 --int-ip 127.0.0.1 --ext-ip 127.0.0.1 --int-tcp-port=1111 --ext-tcp-port=1112 --int-http-port=1113 --ext-http-port=1114 --cluster-size=3 --discover-via-dns=false --gossip-seed=127.0.0.1:2113,127.0.0.1:3113

start EventStore.ClusterNode.exe --mem-db --log .\logs\log2 --int-ip 127.0.0.1 --ext-ip 127.0.0.1 --int-tcp-port=2111 --ext-tcp-port=2112 --int-http-port=2113 --ext-http-port=2114 --cluster-size=3 --discover-via-dns=false --gossip-seed=127.0.0.1:1113,127.0.0.1:3113

start EventStore.ClusterNode.exe --mem-db --log .\logs\log3 --int-ip 127.0.0.1 --ext-ip 127.0.0.1 --int-tcp-port=3111 --ext-tcp-port=3112 --int-http-port=3113 --ext-http-port=3114 --cluster-size=3 --discover-via-dns=false --gossip-seed=127.0.0.1:1113,127.0.0.1:2113

I turned on windows feature ‘Windows Communication Foundation HTTP Activation’ and ‘Windows Communication Foundation Non-HTTP Activation’ because they were turned off.

The nodes are responding to the attempts, and the client throws CannotEstablishConnectionexception with this innerexception message:

{“Failed to discover candidate in 10 attempts.”}

Nodes still say this

[06056,23,16:14:05.432] Internal TCP connection accepted: [Normal, 127.0.0.1:135
7, L127.0.0.1:3111, {8a63d29e-b5b7-472f-bb65-6b56dead00ec}].
[06056,23,16:14:05.432] FRAMING ERROR! Data:
000000: 47 45 54 20 2F 67 6F 73 73 69 70 3F 66 6F 72 6D | GET /gossip?form
000016: 61 74 3D 6A 73 6F 6E 20 48 54 54 50 2F 31 2E 31 | at=json HTTP/1.1
000032: 0D 0A 48 6F 73 74 3A 20 31 32 37 2E 30 2E 30 2E | …Host: 127.0.0.
000048: 31 3A 33 31 31 31 0D 0A 43 6F 6E 6E 65 63 74 69 | 1:3111…Connecti
000064: 6F 6E 3A 20 4B 65 65 70 2D 41 6C 69 76 65 0D 0A | on: Keep-Alive…
000080: 0D 0A | …

[06056,23,16:14:05.432] Closing connection ‘internal-normal’ [127.0.0.1:1357, L1
27.0.0.1:3111, {8a63d29e-b5b7-472f-bb65-6b56dead00ec}] due to error. Reason: Inv
alid TCP frame received. Error: Package size is out of bounds: 542393671 (max: 6
7108864)…
[06056,23,16:14:05.432] ES TcpConnection closed [16:14:05.432: N127.0.0.1:1357,
L127.0.0.1:3111, {8a63d29e-b5b7-472f-bb65-6b56dead00ec}]:
Received bytes: 82, Sent bytes: 0
Send calls: 1, callbacks: 0
Receive calls: 2, callbacks: 1
Close reason: [Success] Invalid TCP frame received. Error: Package size is out o
f bounds: 542393671 (max: 67108864)…

5

It’s weird that it is accepted, and then something is wrong.

The error is thrown upon ConnectAsyn.

Anyone know what might be wrong? (I assume it’s some windows thing…)

6

System.Net.WebException

“The server committed a protocol violation. Section=ResponseStatusLine”

7

does the 3.0.2 Client work with the 3.0.3 server?

8

Look at the data with the framing error its http no?

9

Btw i dont see why you are getting it will look more yhrough the config at home but thats rather obviously http data over tcp

10

yeah it’s true, but it’s very strange, because I use the tcp ports that I specify when I start the nodes. I tried with the http ports, just to see what it would look like, and there is no error thrown upon the connection then, only when I try to save an entity (the connection is not active).

11

As I said before you are talking HTTP to TCP ports or vice versa.

start EventStore.ClusterNode.exe --mem-db --log .\logs\log1 --int-ip
127.0.0.1 --ext-ip 127.0.0.1 --int-tcp-port=1111 --ext-tcp-port=1112
--int-http-port=1113 --ext-http-port=1114 --cluster-size=3
--discover-via-dns=false --gossip-seed=127.0.0.1:2113,127.0.0.1:3113
start EventStore.ClusterNode.exe --mem-db --log .\logs\log2 --int-ip
127.0.0.1 --ext-ip 127.0.0.1 --int-tcp-port=2111 --ext-tcp-port=2112
--int-http-port=2113 --ext-http-port=2114 --cluster-size=3
--discover-via-dns=false --gossip-seed=127.0.0.1:1113,127.0.0.1:3113
start EventStore.ClusterNode.exe --mem-db --log .\logs\log3 --int-ip
127.0.0.1 --ext-ip 127.0.0.1 --int-tcp-port=3111 --ext-tcp-port=3112
--int-http-port=3113 --ext-http-port=3114 --cluster-size=3
--discover-via-dns=false --gossip-seed=127.0.0.1:1113,127.0.0.1:2113

IPAddress address;

IPAddress.TryParse("127.0.0.1", out address);

var ep1 = new IPEndPoint(address, 1111);

var ep2 = new IPEndPoint(address, 2111);

var ep3 = new IPEndPoint(address, 3111);

return new[] { ep1, ep2, ep3 };

.SetGossipSeedEndPoints(_endPoints)

What are 1111,2111, 3111? According to config they are int-tcp ports.

Try:

var ep1 = new IPEndPoint(address, 1114); (or 3 will work in your config)
var ep2 = new IPEndPoint(address, 2114);
var ep3 = new IPEndPoint(address, 3114);

12

that’s right… it uses http for the gossip… I assumed it would use tcp.

Ok, but when I change to the http ports, its this instead heartbeat timeout. I raised hb timeout to 5000 ms, still get this.

[08948,18,19:12:50.272] External TCP connection accepted: [Normal, 127.0.0.1:186
9, L127.0.0.1:1112, {71dac8ef-0293-499b-ad24-2946849c1d0d}].
[08948,09,19:12:58.300] Closing connection ‘external-normal’ [127.0.0.1:1869, L1
27.0.0.1:1112, {71dac8ef-0293-499b-ad24-2946849c1d0d}] cleanly. Reason: HEARTBEA
T TIMEOUT at msgNum 2
[08948,09,19:12:58.300] ES TcpConnection closed [19:12:58.300: N127.0.0.1:1869,
L127.0.0.1:1112, {71dac8ef-0293-499b-ad24-2946849c1d0d}]:
Received bytes: 59, Sent bytes: 66
Send calls: 3, callbacks: 3
Receive calls: 3, callbacks: 2
Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 2

[08948,09,19:12:58.300] Connection ‘external-normal’ [127.0.0.1:1869, {71dac8ef-
0293-499b-ad24-2946849c1d0d}] closed: Success.

13

Are you debugging etc?

14

Yeah I saw that one too. I am in the debug mode, but I’m not stepping through, I just have a break point at the catch block.

I increased heartbeat interval to 5000 and hb timeout to 11000 and instead got this:

[08948,30,19:28:43.053] External TCP connection accepted: [Normal, 127.0.0.1:205
1, L127.0.0.1:1112, {6152fa1d-e8f5-4195-b27e-d5d775ff9c71}].
[08948,30,19:28:43.307] ES TcpConnection closed [19:28:43.307: N127.0.0.1:2051,
L127.0.0.1:1112, {6152fa1d-e8f5-4195-b27e-d5d775ff9c71}]:
Received bytes: 37, Sent bytes: 22
Send calls: 1, callbacks: 1
Receive calls: 2, callbacks: 2
Close reason: [ConnectionReset] Socket receive error

[08948,30,19:28:43.307] Connection ‘external-normal’ [127.0.0.1:2051, {6152fa1d-
e8f5-4195-b27e-d5d775ff9c71}] closed: ConnectionReset.

15

How are you setting timeout? there are two places to set it

16

I’ve played around with different numbers, from connection timeout values of 15ms up to 1500 ms, hb ntervall from 500 to 5000. hb timeout from 1000 to 11000. SetOperationTimeout from default to 15000.

I keep getting hb timeout and socket receive error, sometimes one sometimes both.

var clusterSettings = ClusterSettings.Create()
.DiscoverClusterViaGossipSeeds()
.SetGossipSeedEndPoints(_endPoints)
.SetGossipTimeout(new TimeSpan(0, 0, 60))
.SetMaxDiscoverAttempts(10)
.Build();

var connectionSettings = ConnectionSettings.Create()
.SetDefaultUserCredentials(new UserCredentials(“admin”, “changeit”))
.SetHeartbeatInterval(new TimeSpan(0, 0, 0, 0, 5000))
.SetHeartbeatTimeout(new TimeSpan(0, 0, 0, 0, 11000))
.SetReconnectionDelayTo(new TimeSpan(0, 0, 10))
.WithConnectionTimeoutOf(new TimeSpan(0, 0, 1500))
.SetOperationTimeoutTo(new TimeSpan(0, 0, 15000))
.SetTimeoutCheckPeriodTo(new TimeSpan(0, 0, 30000))
.KeepReconnecting()
.KeepRetrying()
.PerformOnMasterOnly()
.Build();

17

There is one on the client connection settings another is the command line …

18

I’m sorry… should’ve gotten that.

Anyway. I put 2 s interval and 5 s timeout on both int and ext tcp on command line at startup.

I still get Socket receive error in the server, and ‘EventStoreConnection [guid] is not active’-msg in the exeption in client. Sometimes the exception is caught in the catch block, sometimes it is not… and the debugger just stops.

Maybe I should go for http instead…

19

"Sometimes the exception is caught in the catch block, sometimes it is
not.. and the debugger just stops."

welcome to async code in the vs debugger. IIRC its unclick "on throw"
in options. Outside VS it debugger it will hit the exception handler.

in terms of your connection can you put your command line, your config
and a copy/paste of the log and client log? You can use
WithConsoleLogger when setting up the connection. My guess is you have
something funky in your code or a misconfiguration (noone else seems
to have such issues)

20

Sorry UseConsoleLogger

https://github.com/EventStore/EventStore/blob/dev/src/EventStore.ClientAPI/ConnectionSettingsBuilder.cs#L64