Write large number of events atomically

EventStoreDB
1

In posting https://groups.google.com/d/topic/event-store/UWuz5biqMME/discussion Greg & David talk about handling a large number of events atomically to a single stream.

We had not been using transactions and do not expect to have millions of events (expect to have 100-200k/write, measurements of electricity meters). So we were writing using AppendToStreamAsync. Initially with ~38k events in a write we seemed to be good. however now we’re starting to get a bit larger. This morning we started getting errors like:

[PID:04108:019 2015.07.15 14:52:29.011 INFO TcpService ] External TCP connection accepted: [Normal, 10.0.0.99:56940, L10.0.0.99:1113, {81062129-f370-4cd0-8779-00287ed9084a}].

[PID:04108:026 2015.07.15 14:52:31.168 ERROR LengthPrefixMessageF] FRAMING ERROR! Data:

000000: 74 7D 6F 0F 82 00 94 0F 56 06 D2 91 90 44 A7 0E | t}o…V.Ò…D§.

000016: 2A DF ED 8E C9 F9 0A 4F 58 69 3A 3A 52 65 63 65 | *ßí.Éù.OXi::Rece

000032: 69 76 65 64 50 61 63 6B 61 67 65 44 65 63 6F 64 | ivedPackageDecod

000048: 69 6E 67 53 65 74 2D 52 65 63 65 69 76 65 64 50 | ingSet-ReceivedP

000064: 61 63 6B 61 67 65 5F 35 31 37 34 65 33 34 38 33 | ackage_5174e3483

000080: 31 33 39 35 39 30 33 61 32 66 34 37 37 64 30 33 | 1395903a2f477d03

000096: 36 34 65 30 34 37 64 10 FE FF FF FF FF FF FF FF | 64e047d.þÿÿÿÿÿÿÿ

…

[PID:04108:026 2015.07.15 14:52:31.177 ERROR TcpConnectionManager] Closing connection ‘external-normal’ [10.0.0.99:56940, L10.0.0.99:1113, {81062129-f370-4cd0-8779-00287ed9084a}] due to error. Reason: Invalid TCP frame received. Error: Package size is out of bounds: 258964852 (max: 67108864)…

[PID:04108:026 2015.07.15 14:52:31.177 INFO TcpConnection ] ES TcpConnection closed [14:52:31.177: N10.0.0.99:56940, L10.0.0.99:1113, {81062129-f370-4cd0-8779-00287ed9084a}]:

Received bytes: 193287, Sent bytes: 2473

Send calls: 5, callbacks: 4

Receive calls: 25, callbacks: 24

Close reason: [Success] Invalid TCP frame received. Error: Package size is out of bounds: 258964852 (max: 67108864)…

[PID:04108:026 2015.07.15 14:52:31.204 INFO TcpConnectionManager] Connection ‘external-normal’ [10.0.0.99:56940, {81062129-f370-4cd0-8779-00287ed9084a}] closed: Success.

Is the best approach (short of not needing idempotency) to chunk and use the transactions as used by David?

Thanks,

Karl

2

Or to break apart differently and do many small writes. We have been
seriously discussing removing transaction support (it complicates many
things that could be made far simpler!)

258964852 bytes is a pretty big message to send over the network btw :slight_smile: