Hey,
I’m most likely dumb but I don’t understand with the documentation how to change the $maxAge property.
Is this per stream or can I configure it globally?
In any case what document, event type, and stream shall I set?
I tried for example to create a stream foo and to push a new event in $foo with type=$metadata and data={"$maxAge": 10}
But nothing is scavenged.
Thanks for the clarifications
teviah,
Did you clear the browser cache to verify the events were soft deleted.
A “friend” of mine had the same problem and spent a while wondering what he did…
It is configured per stream.
But it looks like you wrote it to the wrong stream…
Did you push your event into “$$foo” - with TWO dollar signs?
“$$foo” is the metadata stream for event stream “foo”.
By the same convention (add two dollar signs) “$$$streams” is the metadata stream for event stream “$streams”.
If you apply your data to “$$foo” (with any event type) then it should be reflected in “foo” immediately.
As Steven says you will need to clear your browser cache.
Steven also refers to these events being “soft deleted”. The data won’t actually be scavenged until other criteria are met.
Oh it works! Thanks!
No way to configure it globally to all applicative streams?
However, how am I supposed to know that $$foo is the metadata stream of foo. The documentation is absolutely not clear!
What is unclear so I can create a ticket https://eventstore.org/docs/http-api/stream-metadata/index.html?tabs=tabid-1%2Ctabid-3%2Ctabid-5%2Ctabid-7%2Ctabid-9
I didn’t know it was possible. I just did it here: https://github.com/EventStore/docs.geteventstore.com/issues/270
Yet, I’m still interested to know whether it is possible to apply $maxAge for all user streams?
I’m creating streams of the fly following the same pattern: foo.001, foo.002 etc. Shall I configure the $maxAge for each stream?
Thanks
You will need to configure each stream with the relevant metadata. Other than categories being a baked-in convention (the Category-dash-ID stream names, $by_category projection, and fromCategory() selector method) ES doesn’t enforce any kind of ‘type’ on your streams - only you know the intention for each.
I agree it would be great to be able to configure default stream metadata per Category. But then ES would need to know in advance what Categories you were going to create, whereas they are currently dynamically derived during/after creating a stream.
Similarly, there are existing threads in this group where people have asked how to set the metadata for a stream emitted from a projection (you can’t), or how to use a projection to write to the metadata of another stream (you can’t) - so you are definitely not alone in wanting to do this.
I don’t know if the team welcome feature requests in the GitHub issues (can anyone confirm this please?). If nothing else then there is some old wiki documentation there which is not identical to the main site, and may still yield some valuable nuggets.
Create your streams with a dash rather than a dot (e.g. “foo-1”, “foo-2”) and you can then write projections which deal with these collectively as a Category. But as you said, this is “on the fly”; there is no preconception that the category will exist, and it’s only an optional convention anyway, so ES makes no assumptions that these streams will be for similar purpose or contain similar events. So it’s not unreasonable that it doesn’t offer a mechanism for you to configure them in a similar way.
I am just going to start a new post with a question relating to setting metadata immediately after creating a new stream. I expect the answer will be “it depends”, but you might be interested to follow the answer as it will likely be your next question too
Hi again teivah,
I’ve asked a new question and await confirmation of my observations, but it appears that you can write to your $$foo-1 metadata stream before the foo-1 stream exists. These streams seem to be independent, but work in harmony when they both exist.
I’m still learning about ES and one of the things I must remember to leave at the door is the baggage I’ve accumulated using other approaches and systems! Old assumptions don’t apply here
I know you would rather not have to set the $maxAge for each of your streams but, given that you DO have to do that, it might please you to learn that you can set it first, in readiness for your data stream. It’s not the answer you were looking for, but it might be a pacifying compromise.
(Unless Greg pops up and says not to do it, in which case listen to him.)
Raith
To be fair we looked at other ways of doing this long ago. One suggestion we came up with was interesting and we may still do it as a different selector (its a pretty powerful idea).
imagine you could write to a stream a stream link such as:
StreamLink : {
stream : “mycoolstream”
}
to “mystreamlinks”
Then you could do
fromStreamOfStreamLinks(“mycoolstream”).
foreach()
i other words just a different selector to control the selection of the streams but works the same as from category etc you could if we did a touch more work on it actually implement fromCategory in js at that point 
fromStreamOfStreamLinks()
We have also looked at a few other selectors like this that seem initially to be useful
We have talked in the past about supporting a global maxage (or perhaps by category/pattern/etc). Its not actually that hard to implement and could be a good feature to put in up for grabs.
Basically right now there is a bit of code that looks at the stream metadata for it. Instead it would just also look at the global one for the time or a pattern/time its definitely a small in terms of overall implementation complexity.
Would this feature help out the issue?
Thank you both for your help.
@Greg: Yes, I think it might help.
Greg,
Using a pattern for providing $maxCount and $maxAge sounds great.
I’d personally like to be able to set these for **projection-emitted partitioned streams. **I guess my pattern to match projection output would need to include a (partial) test for the partitionId so that it didn’t apply to the other partition-related streams, including the non-partitioned result…
e.g.
$projections-materialise_cart__-cart*__
$projections-materialise_cart-cart-1234-result
$projections-materialse_cart
$projections-materialise_cart-result
$projections-materialise_cart-partitions
$$$projections-materialise_cart
etc.
Furthermore, this has an advantage over writing to the metadata stream (e.g $$$projections-materialise_cart-cart1234-result) in that it won’t be wiped out by a projection reset.
(Rightly or otherwise, I am using projections to materialise my aggregates’ current states and generally only need the latest of each, hence wanting to set $maxCount=1 and let the old versions be scavenged)
From what I’ve gleaned from previous group threads, this would make a number of people very happy.
Thanks,
Raith
PS.
@Greg - your “StreamOfStreamLinks” post sounds very interesting, but I am struggling to fit it to this context. Did you post it in the right topic??