Server crash: unhandled exception crash in StorageWriterService

Hi guys,

I’m doing exploratory testing by sending abusive data from the C# client. I can now crash the server every time. It shows this error:

Exit reason: Unexpected error in StorageWriterService: LastEventNumber is less than FirstEventNumber
Parameter name: lastEventNumber

The throw point is in AlreadyCommitted():

            public AlreadyCommitted(Guid correlationId, string eventStreamId, int firstEventNumber, int lastEventNumber)
            {
                Ensure.NotEmptyGuid(correlationId, "correlationId");
                Ensure.NotNullOrEmpty(eventStreamId, "eventStreamId");
                Ensure.Nonnegative(firstEventNumber, "FirstEventNumber");
                if (lastEventNumber < firstEventNumber)
                    throw new ArgumentOutOfRangeException("lastEventNumber", "LastEventNumber is less than FirstEventNumber");

The error originates in **IndexWriter.**CheckCommit().

            if (expectedVersion == ExpectedVersion.Any)
            {
                // ...
                foreach (var eventId in eventIds)
                {
                    // ...
                }
                return first /* no data in transaction */
                    ? new CommitCheckResult(CommitDecision.Ok, streamId, curVersion, -1, -1, IsSoftDeleted(streamId))
                    : new CommitCheckResult(CommitDecision.Idempotent, streamId, curVersion, startEventNumber, endEventNumber, false);
            }

Here, the foreach loop ends gracefully, and sets the result to CommitDecision.Idempotent, but the last EventInfo retrieved does indeed have an endEventNumber less than startEventNumber. Later on, this causes an unhandled exception.

The client side abuse is to send the same 100000 events repeatedly, sliced into varying sizes.

      for (int batchSize = 10; batchSize <= 10000; batchSize *= 10)
      {
        int loops = 100000/batchSize;
        for (int x = 0; x < loops; x++)
        {
          List<EventData> slice = new List<EventData>(events.Skip(x*batchSize).Take(batchSize));

          connection.AppendToStreamAsync(
            _streamName,
            ExpectedVersion.Any,
            slice);
        }
      }

It always fails on the second pass through the top loop, which I guess makes sense because it is exercising the idempotency checks.

I’m running recent code from the dev branch. Apologies, I cant give an exact version, I wasn’t the one who fetched the source.

Thoughts?

-Sam Kanai

What happens when you await connection.AppendToStreamAsync or call .Wait() on the result? Just a WAG but I think that it’s blowing up because you are sending the same set of event ids multiple times without awaiting the result, causing something funny to happen in the idempotency check.

What is starteventnumber and endeventnumber that start end < start?

Originally I had a Wait in there, behavior was exactly the same. It’s a server side problem, anyway.

It varies from run to run, but here are 3 runs:

start 2850, end 1909

start 2350, end 1699

start 2860, end 1589

-Sam

You are writing in different batch sizes but expecting idempotency and using expected version.any … it shouldnt be throwing but it is not assured to be idempotent either.

There are also some funny edge conditions down the path of idempotency with changed data compared to what wrote the first time.

Greg

Sure, I understand that. Hence why I said “abusive testing.” I’m looking for corner cases, and testing robustness in the face of bad input.

-Sam