sequential event number

Hi, for use in an event sourcing app, I need to get the sequential event number to see if an event has been processed before. Currently I’m grabbing the “commitPosition” from the event and using that.
I have an error in my system where an event is at position x and the lastprocessdposition is at x - whatever but it has actually already been processed. So I’m up in the ES dashboard looking at events that have been received there, but I can’t find any reference to the commitPosition, so it is very difficult do determine where my error is.

a) am I still correct to be using the commitPosition as the sequential event number?

b) is there anyway to see what the commitPosition is via the dashboard ui?

I suspect that I could curl queries at the ES and find what I need but it would be a lot easier if I could do it in the ui.

Thanks,

R

Typically, the sort of thing you’re describing is addressed by doing a write with an expectedVersion based on a streamVersion you observed - if you get to write your event without a WrongVersionException, you ‘own’ the event number on a stream.

If you are talking about tailing a stream, then you can track the writes to the stream and only move forward your pull subscription as you process the events.

If neither of the above have anything to do with what you’re asking or attempting to accomplish, you might wish to restate your question in more concrete terms - your question is very abstract in nature.

–Ruben

OK, let me try to be more clear.
I keep track of the events processed so that should an event handler go down, when it comes back up it can read through all the events and only start processing when it has reached an event with a sequential number greater than the recorded last processed. Really shouldn’t read through all the events should just pick up starting with that last recorded one but hopefully this makes sense.
So that’s what’s happening. I"m testing that process and I’m getting an error, an event that has already been processed seems to have a number greater that the recorded last processed number.

So I’m trying to diagnose this but when I look at the streams in the ES dashboard I don’t see the sequential number so it’s making it difficult to figure out. To be clear, the error is in my code not ES :slight_smile: I’m just trying to diagnose and the sequential number would definitely help.

Thanks,

R

When reading events out of a ordinary stream, you will need to track the EventNumber property.

By “ordinary” stream, I mean one that you (not the system) created and wrote to directly.

Other streams exist being linked or emitted by projections. When you’re reading from those streams, you may want to track other properties instead.

Below I’ve pasted some code that I wrote that extracts event position data nicely out of what you get from EventStore, converting it to what I consider a more “plain-english”, “understandable” representation of event data and positions.

There’s quite a lot of it because I wrote my own event sourcing abstractions library and implemented an EventStore version of it as one of several possible dependency injection options. You’ll be able to take what you learn from reading this code to get the information you need.

My own abstraction library code (just the relevant bits)

[DebuggerDisplay("{DebuggerFormat}")]

public class EventData {

///

/// The name of the stream

///

public string StreamId;

///

/// A globally unique identifier for the event.

///

public Guid EventId;

///

/// The sequence number of the event within the stream.

/// When writing events to a store, you can set it to “EventNumbers.Any” or any other member of the EventNumbers class.

/// If you want the store to check for sequential integrity, set it to the sequence number that the store will be expecting.

///

public long EventNumber;

///

/// The name of the event type. A stream can contain multiple event types.

///

public string EventType;

public byte[] Data;

public byte[] Metadata;

///

/// True if the Data and MetaData bytes represent a json string.

///

public bool IsJson;

#region Debugger-view properties

string DebuggerFormat => $"{EventNumber}@{StreamId} <{DataAsString}><{MetadataAsString}>";

string DataAsString => null == Data ? “” : Encoding.UTF8.GetString(Data);

string MetadataAsString => null == Metadata ? “” : Encoding.UTF8.GetString(Metadata);

#endregion

public override bool Equals(object obj) {

return Equals(obj as EventData);

}

public bool Equals(EventData other) {

if (null == other) return false;

if (!(StreamId == other.StreamId)) return false;

if (!(EventId == other.EventId)) return false;

if (!(EventNumber == other.EventNumber)) return false;

if (!(EventType == other.EventType)) return false;

if (!(Data.SequenceEqual(other.Data))) return false;

if (!(Metadata.SequenceEqual(other.Metadata))) return false;

if (!(IsJson == other.IsJson)) return false;

return true;

}

}

///

/// An event that is retrieved from a data store

///

public class RecordedEventData : EventData {

///

/// The time that the event was stored in the event stream. (Not necessarily the time that the event was created).

///

public TimeStamp StoredAt;

}

///

/// An event that is retrieved from a data store. It is possibly a linked event, due to a projection.

///

public class ResolvedEventData {

///

/// The original event

///

public RecordedEventData Event;

///

/// Represents the link. Link.Metadata contains information about the link itself.

/// Null if this is not a linked event.

///

public EventLink Link;

}

public class EventPositionData {

///

/// Contains the position of the event in the stream in which it was first created.

/// If you’re reading events from a stream that contains linked events, and you’re interested in the

/// event’s position within the stream currently being read, you’ll need to use the “EventLinkPosition”

/// property instead.

///

public EventPosition EventPosition;

///

/// Null if the event was not created or linked by a projection.

/// Contains the position of the event in the stream it’s currently being read from.

/// NB: Not the stream in which the event was first created

///

public EventPosition EventLinkPosition;

///

/// Null if the event was not created by a custom user projection.

/// When created (or linked) by a custom user projection, it contains list of the positions of all the streams

/// used as inputs to the projection at the time the event was created or linked.

///

public List ProjectionPositions;

}

///

/// Use this class to describe the position of an event within a stream,

/// or the number of events written to a stream at a particular point in time.

///

public class EventPosition {

public string StreamId;

public long EventNumber;

public EventPosition Clone() {

return new EventPosition {

StreamId = StreamId,

EventNumber = EventNumber,

};

}

public override bool Equals(object obj) {

return Equals(obj as EventPosition);

}

public bool Equals(EventPosition other) {

if (null == other) return false;

return StreamId == other.StreamId && EventNumber == other.EventNumber;

}

}

Implementation Code (just the relevant bits)

/// Converts an EventStore event into my abstraction library object

static ResolvedEventData Convert(ESResolvedEvent e) {

return new ResolvedEventData {

Event = new RecordedEventData {

StreamId = e.Event.EventStreamId,

EventId = e.Event.EventId,

EventNumber = e.Event.EventNumber,

EventType = e.Event.EventType,

Data = e.Event.Data,

Metadata = e.Event.Metadata,

IsJson = e.Event.IsJson,

StoredAt = new TimeStamp(e.Event.Created.Ticks),

},

Link = null == e.Link ? null : new EventLink {

EventId = e.Link.EventId,

EventNumber = e.Link.EventNumber,

StreamId = e.Link.EventStreamId,

Metadata = e.Link.Metadata,

},

};

}

/// Gets the event position data from my abstraction library object

public EventPositionData GetPositionData(ResolvedEventData data) {

/// TODO: Not tested: Suspect this does not work properly with events from the $category-categoryname stream

/// which is created by the $stream_by_category projection which sends the first event of every category to the $category-categoryname stream.

/// Same issue probably exists with events in the $streams stream, created by the $streams projection.

var result = new EventPositionData();

result.EventPosition = new EventPosition {

StreamId = data.Event.StreamId,

EventNumber = data.Event.EventNumber,

};

if (null != data.Link) {

result.EventLinkPosition = new EventPosition {

StreamId = data.Link.StreamId,

EventNumber = data.Link.EventNumber,

};

}

// ----------------------------------------------------

// Extract projection positions

// ----------------------------------------------------

List FromMetaData(byte[] metadata) {

return ((metadata?.Length ?? 0) == 0) ? null : (JObject.Parse(Encoding.UTF8.GetString(metadata)).GetValue("$s") as JObject)

?.Values() // NB: null coalescing operator (?.) handles case when $s property does not exist by returning null

.Select(x => new EventPosition { StreamId = x.Name, EventNumber = (long)x })

.ToList();

}

/// Handle case when event is linked by a custom user projection

result.ProjectionPositions = FromMetaData(data.Link?.Metadata);

if (null == result.ProjectionPositions) {

/// Handle case when event is created by a custom user projection

result.ProjectionPositions = FromMetaData(data.Event.Metadata);

}

// ----------------------------------------------------

return result;

}

And one more class I forgot to include from my abstraction library, used in the code above:

[DebuggerDisplay("{DebuggerDisplay}")]

public class EventLink {

public Guid EventId;

public string StreamId;

public long EventNumber;

public byte[] Metadata;

#region Debugger-view properties

string DebuggerDisplay => $"{EventNumber}@{StreamId} <{MetaDataAsString}>";

string MetaDataAsString => null == Metadata ? “” : Encoding.UTF8.GetString(Metadata);

#endregion

}

HI Benjamin,
First I’m using node.js and as an aside, I never thought I"d say this but I miss C#, anyway, I think what is returned to me via the api is a bit different that what you get. For instance I don’t get an “eventNumber” but I do get an event position. Regardless, I think we are on the same page. I do record my eventPosition properly.

Again, my problem here is that using HTTP or preferably the dashboard, I can’t seem to get the eventPosition of a specific event. I’d have to create a test harness or something to retrieve via tcp in order to know what the eventPosition of a specific event is. Which makes debugging a bit of a PITA.

Thanks,

R

“, I can’t seem to get the eventPosition of a specific event.”

Are you referring to the overall position eg location like in $all or the stream position? The stream position is given and if not already there (I have to look through the varying types returned) it could be.

Right Greg thank you for giving me the proper phrase! I need the over all position in the $all stream! I know that there are actually two positions, the initial position and then the position that is the result of and scavenging activity. I need the original position of the event when it is added to the stream. I get it in the TCP result, but I don’t seem to get it in the dashboard or via http.
This is for use in reading the $all stream for creating (application based) projections for my read stores. It’s possible that the ES projections have advanced enough that I don’t need the application projections, but I got em and I can investigate that further later.

If you are reading the $all stream you definitely get them :slight_smile: that’s how the stream is read and how pagination works :P. I have to look at an individual stream read to see if we include them (doesn’t seem awful to).

Greg, are you saying that if I query the $all stream via http, the events in the payload will have the absolute positioning on them? If so this would certainly help. However, if that was surfaced in the Dashboard that would be super helpful.
Thanks,

R

Look at the URI there must be some way we can page no? :slight_smile: $all has absolute order (much as a fromAll projection or a subscribeToAll).

I am 99.9% sure sure in $all we also include the stream position (can’t be bothered to test). If we don’t open a ticket and it will be there.

Cool, i"m at work and can’t get into my system from here, but I’ll check when I get home.
Thanks,

R

Hi Raif,

Out of interest, are you calling the HTTP API directly from Node, or are you using one of the community-developed Node Clients?

I’m about to do some Node bashing and am keen to use a Client - but which one? Perhaps your experiences can save me trialing them all…

Thanks,

Raith

I’m using a node client “node-eventstore-client”: “^0.2.0” not sure what version they are on now. Works pretty well, I’m not currently using projections so I can’t speak to that. But bare in mind that there is still a shit ton of code you have to write to get read thruogh and write to streams. I can share mine with you if you like. It’s heavily non documented, I guess I could walk you through it if you wanted to as well.
R

Thanks Raif,

I have already had a quick look at that one - I used one of his samples for subscribing to events and was pleased with the result.

I shall continue with it then, somewhat reassured that I know where I can find someone else using it :slight_smile:

Yes, the proper clients do require quite a bit of plumbing. I have already used the .NET API so am looking for a close match in Node.

And thanks for the offer of code, but I shall decline for now. If either of us comes up with an elegant abstraction layer worth sharing then I’m up for that though.

I’m also consuming the HTTP API from PowerShell (I’m a recent convert now that it’s cross-platform) for some quick-hack tooling (e.g. duplicate a stream, copy streams to a different instance), etc.

Raith

I used to have some bash scripts automating some of this with curl. I will try to poke around for them.

HI Greg,
So finally got time to take a look and it seems that the way you do paging is via the event number specific to the stream, not the $all stream. For instance here are two sequential events returned from curling $all
{

“title”: “8@Trainer-e68533f8-ea60-4195-b9bf-f7a9e6c75a39”,

“id”: “http://127.0.0.1:2113/streams/Trainer-e68533f8-ea60-4195-b9bf-f7a9e6c75a39/8”,

“updated”: “2018-08-12T15:29:18.033161Z”,

“author”: {

“name”: “EventStore”

},

“summary”: “trainersClientsUpdated”,

“links”: [

{

“uri”: “http://127.0.0.1:2113/streams/Trainer-e68533f8-ea60-4195-b9bf-f7a9e6c75a39/8”,

“relation”: “edit”

},

{

“uri”: “http://127.0.0.1:2113/streams/Trainer-e68533f8-ea60-4195-b9bf-f7a9e6c75a39/8”,

“relation”: “alternate”

}

]

},

{

“title”: “55@$projections-$master”,

“id”: “http://127.0.0.1:2113/streams/%24projections-%24master/55”,

“updated”: “2018-08-12T15:29:16.338615Z”,

“author”: {

“name”: “EventStore”

},

“summary”: “$statistics-report”,

“links”: [

{

“uri”: “http://127.0.0.1:2113/streams/%24projections-%24master/55”,

“relation”: “edit”

},

{

“uri”: “http://127.0.0.1:2113/streams/%24projections-%24master/55”,

“relation”: “alternate”

}

]

},

``

In the TrainersClientsUpdated we have a title 8@Trainer and the uri is bla…/8 but the event from the projection emitted just before it is title 55@projections and the uri blah…/55

So what I was hoping for was given that $all has 100 events and the 8@trainer event was the 37th emitted I would see, somewhere 37. (obviously stream numbers are much higher than this, just an example).

I’m going to write a test that calls the tcp api for a specific event and this way I"ll be able to see the absolute position but it would be easier if it was in the ui or curl.

Thanks,

R

So again the reason I need this is that I may have a ( App side ) projection that is listening to 5 different streams. If it goes down, the streams keep growing. It comes back on line and says what was the last number event I processed? 1234? ok start processing and any event < 1234 do not process any event > 1234 do process then we catch up.

The http format is handled in the prev/next links for paging not per event unless I am misunderstanding you.

Yes, a bit, so I’m using tcp in my app, I have an error and I want to look at the absolute position relative to the $all stream of a couple of events. As it stands it seems I can not do that via the dashboard ui or via curling the EventStore. I have to write a test harness and read the events via tcp to get the absolute position of the event. Again by absolute position I mean in relation to the $all stream.
Thanks,

R