New Node.js driver for EventStore TCP interface

EventStoreDB
#1

Hey all,

I’ve been working on a Node.js implementation of the TCP interface, and it’s far enough along to share (I’m using it in a prototype to great success, but not in production yet).

Project is at https://github.com/kenpratt/nodejs-EventStore. See https://github.com/kenpratt/nodejs-EventStore/blob/master/examples/basic.js for an example.

It currently implements:

  • CreateStream

  • WriteEvents

  • ReadStreamEventsForward

  • SubscribeToStream

  • Heartbeat

As well as a couple of higher-level helpers:

  • ReadStream - like ReadStreamEventsForward, but a bit friendlier.

  • ReadAndSubscribeToStream - reads all the events in the stream, subscribes to the stream, and calls your callback for each existing and new event, making sure you don’t get any duplicates.

Comments & feedback welcome, and contributions encouraged. If Event Store wants to take over the project, I’m totally fine with that too :slight_smile:

Thanks!

-Ken

PS. I had to make a few changes to the protocol buffers file to get this working – it appears that some of the output from the TCP API doesn’t match the spec quite right and protobufs choked when decoding the packets:

EventLinkPair: changed link from required to optional

CreateStreamCompleted: changed error from required to optional

WriteEventsCompleted: changed error from required to optional

#2

This looks very cool. Have you done any perf testing by chance?

#3

I don’t have anything to compare this to, so these results probably aren’t meaningful, but on my MacBook in OS X 10.8, 2.6 GHz Intel Quad-Core i7 (of which EventStore seems to use 2-3 cores), I’m seeing:

Creating events: 3,500 - 5,000 events/second (single event per WriteEvents command)

Reading events: 10,000 - 11,000 events/second (using one big ReadStreamEventsForward command, 10,000 events)

After doing some benchmarking, EventStore tends to keep a CPU at 100%, so I’m thinking there are some OS X bugs factoring in somewhere. I pushed my benchmarking scripts (in examples dir) if you want to give them a shot in a proper EventStore dev env.

#4

Interesting that it runs on osx we don’t test there…

#5

Yeah, it semi-works on OS X. Projections are mostly broken, but basic event storage and retrieval (at least in-memory, haven’t tried to persist to disk) works. I also have everything running in an Ubuntu VM, but it works well enough on OS X that I can dev from there.

I think the main issue is the disk stats collection. Shouldn’t be too hard to fix. If you guys add support for it, that would be great, or else I could probably take a crack at it after the holidays.

#6

Hi Ken,

I am trying to get your node.js TCP driver to work with the current EventStore (Master), I am getting protobuf wiretype exceptions:

[PID:5760 2013.01.15 21:05:58.739 Info ProtobufExtensions 3]

Deserialization to EventStore.Core.Messages.TcpClientMessageDto+CreateStream failed

ProtoBuf.ProtoException: Invalid wire-type; this usually means you have over-written a file without truncating or setting the length; see http://stackoverflow.com/q/2152978/23354

at ProtoBuf.ProtoReader.ReadUInt32() in c:\Dev\protobuf-net\protobuf-net\ProtoReader.cs:line 192

at ProtoBuf.ProtoReader.ReadBoolean() in c:\Dev\protobuf-net\protobuf-net\ProtoReader.cs:line 842

at proto_2(Object , ProtoReader )

at ProtoBuf.Meta.TypeModel.DeserializeCore(ProtoReader reader, Type type, Object value, Boolean noAutoCreate) in c:\Dev\protobuf-net\protobuf-net\Meta\TypeModel.cs:line 679

at ProtoBuf.Meta.TypeModel.Deserialize(Stream source, Object value, Type type, SerializationContext context) in c:\Dev\protobuf-net\protobuf-net\Meta\TypeModel.cs:line 581

at ProtoBuf.Serializer.Deserialize[T](Stream source) in c:\Dev\protobuf-net\protobuf-net\Serializer.cs:line 77

at EventStore.Core.Services.Transport.Tcp.ProtobufExtensions.Deserialize[T](ArraySegment`1 data) in C:\Dev\eventstore.x64\EventStore\src\EventStore\EventStore.Core\Services\Transport\Tcp\ProtobufExtensions.cs:line 45

#7

Hey Matt,

Thanks for letting me know.

It looks like there’s been a bunch of changes to the protocol buffers API in the last couple weeks:

https://github.com/EventStore/EventStore/commits/master/src/EventStore/Protos/ClientAPI/ClientMessageDtos.proto

I’ll fix the node.js driver to work with trunk EventStore “soon” (may be as soon as tomorrow, but possibly next week). In the meantime, you could check out commit e06a2ad0, as I believe that’s the last version I tested it with (otherwise, a commit from dev branch from mid December should do it).

-Ken

#8

Yes this is in preparation for a backwards compatibility assurance.

#9

Okay, I bit the bullet and just did it now. https://github.com/kenpratt/nodejs-EventStore has been updated to work with EventStore trunk (Jan 16, commit 8d3c7450).

Not incredibly well tested, but my examples/basic script runs again. Let me know if you have any issues with it. Patches encouraged! :slight_smile:

-Ken

#10

The protobuf messages should now be stable for v1