Setting maxAge per category stream

Hey!

We’re planning to remove old events from some of our streams and looking for a best approach to do that. The streams that we want to clear are in the form of Category-id.

Now the question is: can we set $maxAge parameter in a category stream metadata once, to make it respectable by all the streams in given category? Or do we have to set it for each individual stream?

My initial tests showed that setting $maxAge or $maxCount in category metadata streams doesn’t make any effect. We’re using v20.10.

If this is not possible, what would be the best approach to accomplish the same result with as little effort as possible?

Thanks!

jacek,

I think you are correct that adding the metadata on the category wouldn’t get rid of the actual events (since this stream is just linkTos)

One approach we are using here is having a single stream for such events (it’s enriched events).
Then we simply add the metadata to the single stream.
Alternatively, if you need to have different streams, perhaps when the stream is created, you simply add the metadata then?
Something like:

var aggregate = getLatest(aggregateId);
var initialVersion = aggregate.Version;

aggregate.Update();

saveChanges(aggregate);

if( initialVersion = -1) //a little crude{
//Update metadata on newly created stream
}

Steven, thanks!

Got it. Creating metadata for each individual stream is something that (1) I wanted to avoid because it’s a common logic, (2) would be hard/time-consuming to implement, given the number of streams that we already have.

Would there be any elegant and automated way of setting $maxAge for already existing streams?

I’m thinking about fetching all the individual streams with API and for each of them writing to it’s metadata stream. But that will take forever.

Could you simply update the metadata the next time you load events from the stream, or would you need to set the maxAge on all the streams now?

The dynamically-named-streams that we write to and we plan to clean, are not ever read directly anymore (actually, they never were, because of projections and persistent subscriptions that we use later in the process).

So to answer the question, I guess we’d need to set the maxAge on all the past streams now. Which is actually something that we wanted to avoid.

Hmm, that doesn’t sound like much fun.
Building a little harness to look round all the streams (from category perhaps) sounds like a sensible approach, or create a catchup / persistent subscription to the category, and then consume the event, which would simply update the metadata on the stream (so it’s not impacting any aggregate writes)

Yes, I was thinking about something similar, but I think I could make it as a projection. So it reads a category stream and writes to metadata stream of the individual event stream. I’m just afraid of the number of events that it’s going to process and number of streams that it’s going to write to… It’s a lot. I would really prefer to have it possible to do with single write to category stream.

Thank you for your input Steven!

interesting feature idea:
though a few things that would need to be cleared up :

  • what if maxAge / maxCount is not category specific but stream specific?
  • when you set it does it overwrite any preexisting values ?
  • does it merge with preexisitng metadata ?
  • does it set only the db reseved metadata values ?
  • is it an application level or operational level feature ?