GES 3.4.0 $stats streams doesn't run

EventStoreDB
#1

I have a cluster of ES nodes up and running.

However http://10.8.58.11:2113/web/index.html#/streams/$stats-10.8.58.11:2114 shows that it stopped one day ago. Actually all stats stream are dead.

I don’t need stats though, I just need reliable periodic source of events (and stats was working fine in development).

What could be the problem?

#2

What does gossip say? What is your config? Do you have errors in your logs about stats?

#3

Gossip is fine. All 3 nodes are visible with one elected master. And there is no stats errors for past 3 days.

#4

Restart a slave does stats start updating? What is the interval it’s set for? Is this windows or Linux? On windows you can get corrupt perf counters which can cause stats to stop running.

#5

Restarting slaves didn’t help. Default 1 minute interval. Linux (Debina 8).

#6

Could it be related to https://groups.google.com/forum/#!topic/event-store/iyyLRJMg-ww

#7

Generally the way to check if it’s a ui issue is to issue a curl request to the http API.
curl -i http://localhost:2113/streams/%24stats-127.0.0.1:2113 -H"accept: application/json" -u admin:changeit

``

That is what the UI uses. Replace 127.0.0.1:2113 with the appropriate variables for the node.

#8

I test with curl. I mean could it be a symptom of some common problem? hanging stats, missing streams in UI.

#9

So, any thoughts on why it doesn’t work?

#10

I would like to know what the following results in

  1. Start a fresh Event Store node

  2. Write an event to a stream

  3. The following is how the Recently Created Streams’ data is retrieved from ES
    curl -i “http://localhost:2113/streams/%24all/head/1?embed=rich” -u admin:changeit -H “accept:application/json”

``

  1. Recently Changed Streams are only available when the System Projection $streams is enabled and running.

If you are seeing results in curl and not in the UI. You can run ES with
–disable-http-caching

``

and let us know what you see.

#11

apologies /head/1 should be /head/100

#12

Point 4 clarifies everything a bit.
The content I see in UI reflects $streams stream. It’s way behind actual state of the system.

When I try to reenable $streams projection using HTTP
http -a admin:changeit POST http://eventstore-01.local:2113/projection/\$streams/command/enable’

I get
HTTP/1.1 404 Not Found
Access-Control-Allow-Headers: Content-Type, X-Requested-With, X-Forwarded-Host, X-PINGOTHER, Authorization, ES-LongPoll, ES-ExpectedVersion, ES-EventId, ES-EventType, ES-RequiresMaster, ES-HardDelete, ES-ResolveLinkTo, ES-ExpectedVersion
Access-Control-Allow-Methods:
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Location, ES-Position
Content-Length: 0
Content-Type:
Keep-Alive: timeout=15,max=100
Server: Mono-HTTPAPI/1.0

#13

When I try to reenable $streams projection using HTTP
http -a admin:changeit POST
http://eventstore-01.local:2113/projection/\$streams/command/enable’

\$ url encoding may be important here why not just do it from the
projections tab?

#14

I can’t see it there.

#15

Do you have --run-projections=all?

#16

Yep other system projections (e.g. $by_category) are running fine and visible in UI

#17

but $streams isn't there?

#18

Excactly!

#19

Apologies for the inaccuracy in my reply above.
**Recently Created Streams: **/streams/$streams/head/50?embed=rich (Requires Projections)

**Recently Changed Streams: **/streams/$all/head/100?embed=rich

#20

Recently Changed Streams are fine!
Recently Created Streams is not running