RetriesLimitReachedException on large event

EventStoreDB
1

normally we don’t get events as large as this (~113mb), but we tried to push a big one in and got the retriesLimit exception. I understand that this would never be considered “good”, but should i expect this to fail?

2

Wait did you say a 113mb event? The eventstore if it ever actually received this would reject it. We should reject it sooner.

3

My mistake, I did my original test incorrectly. The error is actually a dump on GES:
[06448,42,12:23:07.193] Connection ‘external-normal’ [127.0.0.1:61567, {b5320abb-fffa-4667-94dc-30ef83ee392f}] closed: Success.

[06448,24,13:26:01.280] External TCP connection accepted: [Normal, 127.0.0.1:61933, L127.0.0.1:1113, {d55dfe95-bacb-4a24-a46a-8acae76e2c5d}].

[06448,14,13:26:08.313] Closing connection ‘external-normal’ [127.0.0.1:61933, L127.0.0.1:1113, {d55dfe95-bacb-4a24-a46a-8acae76e2c5d}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 1

[06448,14,13:26:08.313] ES TcpConnection closed [13:26:08.313: N127.0.0.1:61933, L127.0.0.1:1113, {d55dfe95-bacb-4a24-a46a-8acae76e2c5d}]:

Received bytes: 22, Sent bytes: 44

Send calls: 2, callbacks: 2

Receive calls: 2, callbacks: 1

Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 1

[06448,14,13:26:08.313] Connection ‘external-normal’ [127.0.0.1:61933, {d55dfe95-bacb-4a24-a46a-8acae76e2c5d}] closed: Success.

[06448,24,13:26:09.688] External TCP connection accepted: [Normal, 127.0.0.1:61934, L127.0.0.1:1113, {ac47efa3-0f4b-4fa3-a3cd-48f5adbe78a2}].

[06448,23,13:26:12.657] FRAMING ERROR! Data:

000000: 44 35 75 09 82 00 B4 AB 46 42 D4 A8 98 47 AB 5E | D5u…’«FBO".G«^

000016: B0 55 3E 2B E9 75 0A 44 58 69 3A 3A 4D 69 64 41 | °U>+éu.DXi::MidA

…

[06448,23,13:26:12.657] Closing connection ‘external-normal’ [127.0.0.1:61934, L127.0.0.1:1113, {ac47efa3-0f4b-4fa3-a3cd-48f5adbe78a2}] due to error.

Reason: Invalid TCP frame received. Error: Package size is out of bounds: 158676292 (max: 67108864)…

[06448,23,13:26:12.657] ES TcpConnection closed [13:26:12.657: N127.0.0.1:61934, L127.0.0.1:1113, {ac47efa3-0f4b-4fa3-a3cd-48f5adbe78a2}]:

Received bytes: 313324, Sent bytes: 0

Send calls: 1, callbacks: 0

Receive calls: 40, callbacks: 39

Close reason: [Success] Invalid TCP frame received. Error: Package size is out of bounds: 158676292 (max: 67108864)…

So, i guess the max size for an event is 64 mb ? Since we have the (VERY) occasional event larger, we’ll make a reference and store it outside of GES.

4

Why would you ever have a 64mb event?

5

This is actually something I have been thinking. Correct me if I’m wrong:

What of a system that, say, scours the internet for PDF files. An event such as “FileDownloaded(body)” would be a few MB in size (you can come up with examples of this that exceed 100MBs, e.g. videos).

I see few possible solutions:

  1. Don’t put data into an event (instead write it somewhere, and produce an event “FileDownload(link-to-file)”

  2. Don’t use event source for that part of the system.

Comments?

6

Typos:

  • “FileDownloaded(link-to-file)”
  • Don’t use event sourcing
7

Put it in a file system etc store and put a link to it in the event.
Its quite likely that you have consumers that are interested only in
metadata associated with the event. They can pull it if they want it