Suitable for IoT / sensor data?

EventStoreDB
#1

I was wondering how suitable Event Store might be for a project I’m working on, which is basically to provide a dashboard for how a factory is doing. Many different data sources - periodic hardware status dumps, human activity (someone scanned a bar code, filled out a form, or otherwise triggered something). I’m less interested in replaying streams, I think my scenario would have many short (maybe 1 event) streams. But the possibility of a single API for dealing with past and new events is very appealing. Anyone have any thoughts?

J

#2

#of events/second?

How long to keep them?

Do you care about losing them?

#3

to clarify last one. Is missing one event in a million a big deal?

#4

#of events/second?

Highly variable, but not likely more than 4000 events/sec

How long to keep them?

Indefinitely, though it’d be acceptable to have something like online/archive environments

Do you care about losing them?

Generally no, but it depends entirely on the source. You can imagine some sources (especially hardware) being chatty/verbose where a dropped event might not even be noticed. But for relatively infrequent events like when a human triggers something or submits some data - those should be pretty reliable.

J

#5

For even 4000/sec not too big of a deal. I would wonder however about
your storage requirements over time. If you are archiving this would
obviously not be a problem.

#6

Its not that hard to abstract away the backend here so I would suggest
trying (you can always swap in something else later, you have 3-4
operations).

Cheers,

Greg

#7

Would it be a misuse of ES if I use it as a “temporal queue” and not so much for “entity composition by playback”?

#8

People do it, it can work in such circumstances (especially with
competing consumers now there)

#9

Thanks for the response Greg! I came across your 2014 Code on the Beach CQRS/Event Sourcing session yesterday and spent the rest of the day watching various sessions you’ve done. Great stuff, so thanks for that too!

Just curious, are things still moving toward Erlang?

Finally, do you know anything about this guy’s Eventful project on GitHub? It looks like a reasonably complete F# event sourcing library. I haven’t gotten around to reaching out to him to ask about it.

J

#10

Quick view of library looks reasonable.

Id like to see more the projection api and don't have a chance right now

#11

We’re currently running a manufacturing system on the Event Store, it records pretty much any human interactions with the system into Event Streams based on the batch currently being manufactured, as well as our access control system, which lets us see where people were in the building at certain times. As we get access to the data of more of our systems, we are adding them into Event Store.

As we have a lot of messages we can hit performance issues in the case of a cold boot (around 6 minutes to process all events to create view models in our system, I’m sure it could be optimised further), but it sounds like this wouldn’t be an issue for you if you aren’t going to be replaying streams.

We’ve yet to hit issues with storage, although have discussed archiving in the past as we need to keep data for a long time for regulatory reasons.

#12

It's also possible to batch your own events/messages inside an event store event, which sacrifices some latency, but increases throughput.. It's pretty simple to implement with something like reactive extensions.