Stopped Projections (ES 5.0.8)

EventStoreDB
#1

Hi

I’m in the middle of a large(ish?) migration of data from Azure Table Storage to Eventstore (5.0.8). Slow progress, but it seems to be progressing. One thing I’ve only just noticed though is that the by_category and by_event_type projections have stopped (this is via the web UI).

The messages they state are:

Failed to write events to $ce-UserPresence. Retry limit of 5 reached. Reason: CommitTimeout. Checkpoint: C:177694122231/P:177694122231.

and

Failed to write events to $et-$deleted. Retry limit of 5 reached. Reason: CommitTimeout. Checkpoint: C:177694122231/P:177694122231.

UserPresence is the stream that I’m performing the mass migration to.

I can’t say I’m knowledgeable enough to know if this is a critical issue and my data is lost (doesn’t appear to be… but its still migrating), or this is something that can just be safely ignored?

Does anyone have any suggestions/comments on this, since I’m definitely out of my ES comfort zone here.

EDIT:

Forgot to mention that this mass migration I’m doing is the second attempt. The first attempt had an error, so what I did was put a soft delete on all streams and then am reimporting. Unsure if the soft delete would impact what I described above. Just thought I should mention it.

Thanks

Ken

#2

Based on this question and the last it seems like you might be hitting the db a little too hard. Perhaps throttle it a little or give it a bigger VM to run on whilst you do the import

1 Like
#3

Thanks for that, the next time I do a mass migration I think I’ll scale the cluster VMs just for it.

One thing that blew up on us majorly was (I believe) the internal projections. The by_category and by_event_type ones had stopped. Restarted the ES service and they restarted, yay. Could see they were busy… but looked like it was just catching up. But… I go to sleep (end of a VERY long day) and I wake up to a completely broken cluster due to the harddrives being full. This VERY much confuses me.

All the events had been imported and processed the previous night. I was under the impression that these internal projections referenced existing events, and would not take up that much disk space. Overnight they had chewed up 40G worth of disk space.

Am I completely wrong in saying that these internal projections shouldn’t use up much disk space… (this is only a theory)… or is it a case of ES used disk space due to something else? (if so, what??.. I’m at a loss).

Any thoughts on why the disk space was suddenly chewed up after event ingestion is thoroughly welcome!.. :slight_smile:

#4

That does seem like a lot of disk space but without having better insight into the system its going to be difficult to work out.

projections can write both pointers (linkTo) to events and full events. It depends which method you use. See https://developers.eventstore.com/server/20.6/server/projections/user-defined-projections.html#functions

#5

Hi

Yeah, I was looking at that page earlier…but we’re purely using the built in projections (which I believe are linkTo events). But… one thing I noticed at https://developers.eventstore.com/server/5.0.8/server/projections/#continuous-querying was the comment:

" Some system projections emit link events to their streams for each event appended to the database. These projections are By Category, By Event Type and By Correlation Id. If all those three projections are enabled and started, adding one event to the database will, in fact, produce three additional events and, therefore, quadruples the number of write operations."

I’m trying to interpret if that means if “write operations” just means some internal process that is performed before actually hitting disk… OR… does it literally mean disk writes… and potentially more space?

Any experience around that? It seems like some documentation talks about linkTo and no extra events… but then the above seems to indicate otherwise. Basically I’m just trying to write an RCA for the business and to make sure we don’t hit this again :slight_smile:

Cheers

Ken

#6

So that bit is talking about the fact that any projection will involve some kind of write. the more projections you have writing events the more load there is on the server. Basically, if you don’t need the projection don’t run it.

Even the linkTo will result in a small amount of data being written to disk.

1 Like