I am using a php tcp client (https://github.com/madkom/event-store-client) to connect to a persistent subscription on geteventstore 3.4.0 on linux. Everything works great and I am handling events as they come in. See example: https://github.com/madkom/event-store-client/blob/master/usage/persistentSubscription.php
There is however a problem when I kill the client before a message is ack-ed. Upon re-connecting the client. The server accepts the connection, and right before it re-tries the message (as expected), the connection is dropped. re-connecting the client again will result in the same.
When I restart the server and then restart the client, it works and the message is properly handled.
Example Log output:
[14579,18,17:17:49.313] External TCP connection accepted: [Normal, 192.168.10.10:56602, L192.168.10.10:1113, {da02a615-507c-4512-ad49-f277fca8151e}].
[14579,12,17:17:49.510] New connection to persistent subscription testSubscriptions.
[14579,17,17:17:53.328] ES TcpConnection closed [17:17:53.328: N192.168.10.10:56602, L192.168.10.10:1113, {da02a615-507c-4512-ad49-f277fca8151e}]:Received bytes: 107, Sent bytes: 188
[14579,17,17:17:53.329] ES TcpConnection closed [17:17:53.329: N192.168.10.10:56602, L192.168.10.10:1113, {da02a615-507c-4512-ad49-f277fca8151e}]:Send calls: 3, callbacks: 3
[14579,17,17:17:53.329] ES TcpConnection closed [17:17:53.329: N192.168.10.10:56602, L192.168.10.10:1113, {da02a615-507c-4512-ad49-f277fca8151e}]:Receive calls: 3, callbacks: 3
[14579,17,17:17:53.329] ES TcpConnection closed [17:17:53.329: N192.168.10.10:56602, L192.168.10.10:1113, {da02a615-507c-4512-ad49-f277fca8151e}]:Close reason: [Success] Socket closed
[14579,17,17:17:53.329] Connection 'external-normal' [192.168.10.10:56602, {da02a615-507c-4512-ad49-f277fca8151e}] closed: Success.
[14579,13,17:17:53.333] Lost connection from 192.168.10.10:56602
[14579,13,17:17:53.333] Retrying message testEvents::testSubscriptions testEvents/C:1390782/P:1390782
I have asked the author of the client the look into this as well, but wanted to see if it might be server related.
1. Does the server do anything different with tcp client connections when it needs to retry a message?
2. Is there a specific socket message type that needs to be handled when re-connecting to a persistent subscription with retries.
If it is helpful, the client is using the following known messageTypes:
https://github.com/madkom/event-store-client/blob/master/src/Domain/Socket/Message/MessageType.php
3. In some cases, the server log would show a HEARTBEAT TIMEOUT :
Closing connection 'external-normal' [192.168.10.10:55968, L192.168.10.10:1113, {13a329f9-bac1-4f95-ae54-b2ae6450a7e4}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 1
Therefore I played around with --int-tcp-heartbeat-timeout, --ext-tcp-heartbeat-timeout, --int-tcp-heartbeat-interval and --ext-tcp-heartbeat-interval without success. Are there any other settings that affect timeout of tcp clients?