So for kicks, I’m working on an exponential problem as a personal project. I’m processing chunks of binary data (at most 26 bytes), which spawns more chunks of binary data to process.
Right now I’m using MSSQL with a uniqueness constraint on the binary data. It’s working fine, but it’s a bit slow for what I need. The average is actually decent for SQL (400-500 writes per second), but the problem space is too large for that speed.
I had the idea that I could use EventStore to speed up the write process, but in order to do that I would need to use streams in … uh … unconventional ways. I wanted to do a quick boundary check with you guys.
There are too many results to fit in memory, which is why I put the unique constraint on the database. I need the uniqueness constraint on the binary data because it eliminates more than half of the results (and growing) at each step. The way I thought to have a unique constraint in event store is to embed it in the ID. I can do this pretty easily in .NET by using System.Data.Linq.Binary class. ToString() on that looks something like this:
example 1: vwodd9pR1Az+6k9H9LoMnQmZmDkk0t/wh3Q=
example 2: rxpq5UOFobWZA2V+
So my stream IDs would have to be like “state-rxpq5UOFobWZA2V+”. First off is that even doable (are special characters allowed)? I don’t think I will run into a length limitation (longest would be prefix + 36 characters).
Secondly, the sheer number of streams I would need to create is ridiculous. For instance, I looked at Neo4j but the billions of nodes limit is likely to block me eventually.
This is just a fun project for me, so no biggie. But I thought I would ask about known limitations before trying it.