Heartbeat Timeouts on Import

EventStoreDB
1

I’m pulling in a large dataset as a test (multiple GB). After a period of time, I am seeing a RetriesLimitReachedException in my client, and a HEARTBEAT TIMEOUT message on the server.

My code:

var file =

new System.IO.StreamReader(@“c:\users\jrivers\desktop\tsu.txt”);

string line;

var connectionSettings = ConnectionSettings.Create();

connectionSettings.SetHeartbeatTimeout(new TimeSpan(0, 3, 0));

connectionSettings.KeepReconnecting();

connectionSettings.KeepRetrying();

var connection = EventStoreConnection.Create(new IPEndPoint(IPAddress.Loopback, 1113));

connection.Connect();

var SerializerSettings = new JsonSerializerSettings { TypeNameHandling = TypeNameHandling.None };

var eventBatch = new List();

var allTasks = new List();

byte[] eventmetadata = Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(new { Something = “Nothing” }, SerializerSettings));

while ((line = file.ReadLine()) != null)

{

var strings = line.Split(’,’);

var tsuLine = new

{

update_id = Convert.ToInt64(strings[0]),

station_id = Convert.ToInt64(strings[1]),

entity_id = Convert.ToInt64(strings[2]),

created_date = DateTime.Parse(strings[6])

};

byte[] eventdata = Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(new { Time = DateTime.Now, Event = tsuline }, SerializerSettings));

eventBatch.Add(new EventData(Guid.NewGuid(), “trialevent”, true, eventdata, eventmetadata));

if (eventBatch.Count >= 20)

{

var task = connection.AppendToStreamAsync(“trialstream”, ExpectedVersion.Any, eventBatch);

allTasks.Add(task);

eventBatch = new List();

}

}

file.Close();

Task.WaitAll(allTasks.ToArray());

``

My questions:

  1. Why do you think I’m getting this error? Shouldn’t KeepRetrying solve this?

  2. Is this something I should need to guard against in production code? How do I guard this (I’m not sure if this is a connection failure that requires a rebuild of the connection, or if I just need to re-run the task, or what)?

  3. Is there some other way I’m failing here?

Thanks,

Josh

2

You are not bounding the number of concurrent operations. Try using a .Wait() on your async operation. Likely you have millions and millions of pending writes in your connection.

Cheers,

Greg

3

There’s also .LimitOperationsQueueTo on the connection settings

James

4

It might help if you actually pass the connectionSettings object when creating the connection :slight_smile:

5

This was the answer. Thanks!

6

On my Azure VM, just putting wait() inside the loop did not resolve the heartbeat timeouts. I tried doubling the size of the instance (2-proc to 4, more RAM), and I haven’t seen the failure, but it’s only been running 14 hours, so it’s not done yet (it
takes two hours with the tight loop and the connection settings working correctly.

I think I have the problem here licked (thanks to Robert), but I’d still like to understand what is going on here, since I can imagine it happening in situations that are less stable than single-machine-and-local.

Is the connection closing, or does the batch just need to be retried?

I assume that the correlation Ids are created when the EventData is instantiated, so retries should be idempotent?

Thanks,

Josh

7

What does this setting do? Is it client side, or server side? -Josh

8

Yes retries are idempotent. Heartbeat timeouts can be client timeouts or server timeouts. It basically says if the other side doesnt respond in time T consider them to be dead and force a reconnect/retry.

Limit Operations to discusses how many concurrent operations can be going at a time. You can imagine with your original code without any waits that you would go through the multiple gb file (creating operations). Obviously you can create them faster than the connection can process them. It determines how many can be in flight at a time vs just queued in the connection. The one James mentions bounds the size of the queue of pending operations (eg so you don’t blow up memory of your process).

Cheers,

Greg