Hello Community!
I want to programmatically list all streams with names matching given pattern.
I’m using GRPC client API. Event Store version 20.6.
I can’t find any API method that returns all the existing streams - so I can iterate the result and display the matching ones.
How can I get names off all current existing streams?
The best thing to do here is create your own read model for this.
Thank you for the answer, but I’m afraid I don’t understand what you mean…
Maybe I’ll explain exactly what I want to achive.
I have few projections which produce temporary streams named TempStream-* and I want to delete that streams because they are not needed any more.
So I want to get all streams names which match TempStream-* pattern and invoke EventStoreClient.SoftDeleteAsync() for each name.
I want to do it programmatically in C# using GRPC EventStore API.
If you enable “$stream_by_category” system projection, it should create a stream called $category-TempStream which will have the first events of each stream. You can then read the stream name fro the resolved event.
Also, if you find you need to do this quite often, you could set the maxAge of the stream when you creating it, so it will automatically get soft deleted.
This would be the best solution, but the emit() function that we can use in projections does not allow to set the stream metadata. I’m creating that TempStream-* streams ad hoc in projections.
emit should allow metadata to be passed in
You can pass metadata in emit(), but this is event metadata, not stream metadata.
But I’ll try to set maxAge by writing directly to $$TempStream-* stream.
Ah yes, good point.
Didn’t think of writing directly to the metadata stream though!
Writing metadata directly fails - I have an error: The ‘$$ TempStream-616aa169-e12b-4288-88b7-3959e3871114’ stream managed by projection 7 has been written to from the outside.
I will probably give up using projections. I wanted to use them to calculate the interval between two specific events. It will be easier to do it directly in my application code.
And coming back to the beginning of the conversation. I find it shocking that there is no function in the EventStore API that will simply return a list of existing streams in the database.
Wouldn’t the projection $stream_by_category help with that, or $streams?
While you can enable projections for this, keep in mind that it will reduce your write throughput. This is why I suggest projecting this to your own read model.
Writing metadata directly fails - I have an error: The ‘$$ TempStream-616aa169-e12b-4288-88b7-3959e3871114’ stream managed by projection 7 has been written to from the outside.
If a projection needs to write to a stream nothing else can write to it. You definitely should not try to append to a metadata stream from a projection.
I will probably give up using projections. I wanted to use them to calculate the interval between two specific events. It will be easier to do it directly in my application code.
Based on my understanding of what you are doing, you shouldn’t need to create temporary streams. It would help if you included the source code of your projection.
Ok, here is source code for my two projections.
The goal is to compute time interval between event “Rest.Api.Events.RestHttpRequestBeganEvent” and event “Rest.Api.Events.RestHttpRequestFinishedEvent”. Each event has correlationId field (GUID) - so we want to compute interval between events which share the same value of correlationId.
First projection “waits” for events to appear in main-stream and links them to temporary streams.
Because CorrelationId is unique for each pair of Rest.Api.Events.RestHttpRequestBeganEvent and Rest.Api.Events.RestHttpRequestFinishedEvent - each temporary stream HttpRestRequest-GUID will contain only two events.
fromStream('main-stream').
when({
"Rest.Api.Events.RestHttpRequestBeganEvent" : function(s,e) {
var correlationId = e.body.CorrelationId;
var streamName = "HttpRestRequest-"+correlationId;
linkTo(streamName,e);
},
"Rest.Api.Events.RestHttpRequestFinishedEvent" : function(s,e) {
var correlationId = e.body.CorrelationId;
var streamName = "HttpRestRequest-"+correlationId;
linkTo(streamName,e);
},
});
Second projection computes time interval between events (event dispatch time is stored in PushTime property in event metadata). Computed interval is stored as new event in http-rest-request-interval-stream.
fromCategory("HttpRestRequest")
.foreachStream()
.when({
$init: function(state, event) { return { "startEvent": null } },
"Rest.Api.Events.RestHttpRequestBeganEvent": function(state, event){
state.startEvent=event;
},
"Rest.Api.Events.RestHttpRequestFinishedEvent": function(state, event){
t1 = Date.parse(JSON.parse(state.startEvent.metadataRaw).PushTime);
t2 = Date.parse(JSON.parse(event.metadataRaw).PushTime);
emit("http-rest-request-interval-stream","HttpRestRequestIntervalComputedEvent", {
"$type": "HttpRestRequestIntervalComputedEvent",
startEvent: {
data: state.startEvent.data,
metadata: JSON.parse(state.startEvent.metadataRaw)
},
endEvent: {
data: event.data,
metadata: JSON.parse(event.metadataRaw)
},
intervalMs: t2-t1
},
{
"SourceName" : "EventStoreProjection",
"AppName" : "EventStore",
"AppVersion" : "20.6",
"PushTime" : new Date().toISOString()
});
},
});
These projections work ok. The only problem is how to delete temporary streams HttpRestRequest-* created ad hoc by first projection after second projection has computed interval.
PS. CorrelationId and PushTime are our custom fields created by our application.
Slightly off what you are doing here, but could you use a partition to get the same result?
So rather than linking to “HttpRestRequest-”+correlationId, just create a partition?
And I am sure partition states get collected on the scavenge
Very good idea. I wrote a new version of projection using partitionBy - and it works. Thank you. 
However, the question takes a new form: how will EventStore know that a given partition is no longer needed? After a year of running the projection, many million partitions will be created. 99.99% percent of them are no longer needed because an event with a given CorrelationId will never come again.
fromStream("source-stream")
.partitionBy(function(event){ return event.data.CorrelationId; })
.when({
$init: function(state, event) { return { "startEvent": null } },
"Rest.Api.Events.RestHttpRequestBeganEvent": function(state, event){
state.startEvent=event;
},
"Rest.Api.Events.RestHttpRequestFinishedEvent": function(state, event){
t1 = Date.parse(JSON.parse(state.startEvent.metadataRaw).PushTime);
t2 = Date.parse(JSON.parse(event.metadataRaw).PushTime);
emit("http-rest-request-interval-stream","HttpRestRequestIntervalComputedEvent", {
"$type": "HttpRestRequestIntervalComputedEvent",
startEvent: {
data: state.startEvent.data,
metadata: JSON.parse(state.startEvent.metadataRaw)
},
endEvent: {
data: event.data,
metadata: JSON.parse(event.metadataRaw)
},
intervalMs: t2-t1
},
{
"SourceName" : "EventStoreProjection",
"AppName" : "EventStore",
"AppVersion" : "20.6",
"PushTime" : new Date().toISOString()
});
},
});Don’t take this as gospel, but my understanding was you could browse to:
$projections-(your projection name)-partitions
Then simply truncate this stream?
Probably would be safer getting someone else to verify this.
There is probably no such stream, I cannot browse it. But I saw that when I stop and start the projection, the “Partitions Cached” property is set to zero. Maybe the partitions are only stored in RAM? I will post a new topic on this forum, maybe one of the EventStore developers will answer me.
Be interested to see any replies.
I assumed partitions were cached in ram, but this was throttled to 4000, suggesting to me the other partitions not recently accessed are on disk.
Based on what you’ve posted, I don’t think you really need to use eventstoredb projections for this. You would be better off tracking this in a relational or document database off of a subscription.