Hello,
We are running EventStore 5.0.2 on a AKS cluster with 3 Ubuntu nodes. I was playing around with the Scavenge and I noticed that if I start Scavenge from Admin UI it only executes on one of the pods (not really sure how he picked that one out of the three). I’ve checked the documentation and searched on the internet on how to run it on specific pod but without any luck. Is there a way to specify on which pod/node to run scavenge?
Scavenge operations are not coordinated between the nodes, so each node must be scavenged separately. It makes sense because each node has its own copy of the database. In addition, you don’t want to scavenge all the nodes at the same time because it will kill the IO. We recommend to run the scavenge operation on slave nodes only and change the master last, then run the final node.
If you notice that you get a random node, you probably have a load balancer for HTTP in front of the Admin UI. It actually doesn’t work that well because the UI is different and you really need to know which node you are talking to.
We described the scavenging better in the new docs, so you can check it out https://developers.eventstore.com/server/5.0.9/server/scavenge/
@alexey.zimarev thanks for your response. I’ve checked the new docs and it still not clear to me how to trigger scavenge on specific node. If I use this command curl -i -d {} -X POST http://localhost:2113/admin/scavenge -u "admin:changeit" where do I specify the node name? If the load balancer is the problem can you suggest some workaround? Thanks in advance.
Here the localhost is the node. If you have three nodes like 10.0.1.11, 10.0.1.12 and 10.0.1.13, you need to run it with three calls to each node:
curl -i -d {} -X POST http://10.0.1.11:2113/admin/scavenge -u "admin:changeit"
curl -i -d {} -X POST http://10.0.1.12:2113/admin/scavenge -u "admin:changeit"
curl -i -d {} -X POST http://10.0.1.13:2113/admin/scavenge -u "admin:changeit"
If you used the Helm chart (deprecated), you are probably accessing the cluster admin (and HTTP) via the ingress. The ingress points to a load balancer service, so each time you get a new node.
You need to send requests to each pod instead. Use the port forward to each pod and send the command.
