Query1: https://pastebin.com/CHfYu8TJ
Query2: https://pastebin.com/Ly259PaA
Do you think the replication service may have tried to write to $ce-NotificationAccount and gummed it up? It currently isn’t running and it’s still showing that error.
Based on the query results from above, we would need to ask a few more questions to piece together the series of events here.
- Did you ever attempt to copy events from 1 db to another via a manual process? I see 2 $ProjectionsInitialized events in the projection registration stream. The 2nd one was written on the following date. 2017-03-12T22:58:50.6525235Z. Does that date have any signifcant meaning?
It doesn’t have any specific meaning that I can think of, however it may coincide with something I did around that time period that I mentioned in a reply to Greg above. Following a conversation around “how do I duplicate production events into dev for testing purposes”, the solution was to write a service that listened to everything on the prod instance and wrote them to the dev instance, which is what I did. I filtered out every event that began with ‘$’… but I think that was on the next day after I saw a bunch of diagnostic events coming through that we didn’t need in dev. Maybe that was enough to mess it up? None of it was manual in the sense of mucking with the files in the file system. It was all done through EventStore.Client.
Is this reversible?
Bueller?
Writing system events from one event store into another will certainly cause issues. Whether or not it is reversible would require figuring out what all has been pushed in. There are patterns to reverse such things whether or not it is worth doing vs re-running depends on how much is screwed up (if that system event is there, there are likely many others that will give bad results)
Luckily, this is only an issue in my dev environment, so I guess I can just delete the ES and redeploy fresh. Writing from one store to the other store was actually your suggestion. That being said, if it’s not good or safe, might I suggest adding replication or mirroring or some other solution for environments to a future release, guide or additional tool?
How many lines of code do you have for mirroring your prod environment? to dev? Feel free to place them here?
I am not sure why you would want a tool available to replace this. There are roughly 20 lines of event store code there. Perhaps you want to publish yours?
Oh no, I agree. But you were saying that things could go wrong. If there was a proven solution to avoid those pitfalls, it seems that would be a better option
just strip out the 30-40 lines of your specific code and put it up! link to the gist next time. The only needed thing is a catchupsubscription that ignores eventtypes that start with $
So you’re saying line 43 should be enough to prevent the problem? I guess it got ruined during the few minutes it was running without it then, I guess.
yes its just a check for starts with $. we prefix all system space with $
Ok. That was an assumption on my part; glad to know it was correct. That must have been what happened. Some snuck in there before I put that short-circuit in. I’ll just blow dev away and try again now that it’s in there. Thanks.