commit and prepare position are referring to logical places in the log
where an event was prepared and committed.
For what you describe above just the event numbers will probably be ok
I’m struggling/flailing with getting started. Here’s a list of things that could be documented a little more (do let me know if I’ve simply missed them)
-
Overall it’s such a new space some general guidance on partitioning - when to append to a generic stream, when to aggregate, when to produce temporal queries. I realise the difficulty here (each problem space is unique) but perhaps a forced/worked example would be useful (from generic streams to useful aggregates and then processing, maybe)
-
ACL/settings.js uploading. This was mentioned a little but we were borked for a few hours before we POSTed default ACL.
-
Debug how to / guide. If you’re not faulting and treat is carefully, all is well. But it’s not too simple first up
-
EventStore query language … we eventually simply looked at the prelude.js/blogs but a single page with syntax would be helpful
-
Projection maintanence and generally how to re-run a projection over existing streams would be useful.
-
When to use emit categories and when to emit streams. I think I know now (it depends on further usage, ie. if later we’re looking to combine)
Now it seems that the CEP-side of ES is a little under-documented, but again I could be missing things.
I realise there use-case for ES is rather broad, but I think the doco is (rightly) more focused towards SEP (or a more traditional Event Storage).
This is cool in and of itself, but (again I think) the CEP potential of ES is staggering. I think more doco could be provided as many such scenarios begin with streams (i.e. where the traditional/SEP/Event Storage ends) and produces ever more useful, complex streams.
Excuse an ignorance above, all. It’s an awesome technology. Trying to help both myself and others. Do let me know if I’m being a t*t.
"- EventStore query language ... we eventually simply looked at the
prelude.js/blogs but a single page with syntax would be helpful
- Projection maintanence and generally how to re-run a projection over
existing streams would be useful.
- When to use emit categories and when to emit streams. I think I know
now (it depends on further usage, ie. if later we're looking to
combine)
Now it seems that the CEP-side of ES is a little under-documented, but
again I could be missing things. "
These features are all in beta. One of the main things for them to
come out of beta is documentation.
Definitively projections with at least some API reference (maybe it exists but I haven’t been able to find it, being written in javascript it makes it even harder to comprehend).
Today my best reference about projections are 2013 blog posts from Rob Ashton.
Projections are in beta which is why the documentation is lacking on
them (also subject to change)
The two things that would have helped us most when starting out are:
-
Some meaningful documentation on what the configuration options do, what they mean, and what the conservative recommended values are for simple deployments (single-node, three-node open-source, three-node HA commercial). We were running an HA cluster for months with poor config, and had all manner of peculiar problems, because we had to guess at what the correct config might be. Eventually you guys spotted the issues in a long-running support request, but I was seriously ready to give up on running Eventstore in a Linux-AWS environment.
-
Some technical docs on how things work under the hood. I’m so desperate for information on the underlying mechanics that I’m considering writing it myself by reading the code. We see messages in the logs, but have no idea what they mean - is this normal behaviour, or is it an sign of the forthcoming apocalypse?
Other than that, I think we’re good - I think it’s quite hard to write good documentation on how to use EventStore effectively, because it’s a topic large enough to be a book in its own right; the missing documentation is about the technical aspects of configuring, troubleshooting, and diagnosing the system.
The two things that would have helped us most when starting out are:
1) Some meaningful documentation on what the configuration options do, what
they mean, and what the conservative recommended values are for simple
deployments (single-node, three-node open-source, three-node HA commercial).
We were running an HA cluster for months with poor config, and had all
manner of peculiar problems, because we had to guess at what the correct
config might be. Eventually you guys spotted the issues in a long-running
support request, but I was seriously ready to give up on running Eventstore
in a Linux-AWS environment.
Would a terraform script be a possibly better option here? for AWS
2) Some technical docs on how things work under the hood. I'm so desperate
for information on the underlying mechanics that I'm considering writing it
myself by reading the code. We see messages in the logs, but have no idea
what they mean - is this normal behaviour, or is it an sign of the
forthcoming apocalypse?
Can you give some specific examples of this?
It’s hard to give specific examples: Truthfully, we don’t know what any of the logs mean, because we don’t really know what the eventstore is doing under the hood.
For example, we’re building out our monitoring capabilities at the moment; with a bit of background research, I can find out what to look for in redis stats, or varnish stats so that we can raise alerts when things aren’t quite right; I have no idea what we should be monitoring in order to tell whether an eventstore cluster is healthy.
Terraform might have been useful to get started. You guys were good enough to offer to set the cluster up for us in the first place, but we said we’d rather do it ourselves, because we need to know how it’s all put together and what it’s doing, or we can’t maintain it in production.
Some information on how the election process works, how things actually replicate, what an epoch is, what the various .chk files do, what all the stats mean, would give us much more confidence in the system. Right now we often feel that we’re flying blind - the cluster would go read-only, we’d kill a node, it would work again - but why?
I don’t want to sound negative because, for the record, you and James have been great over the last couple of weeks, but when EventStore doesn’t work, there is an absolute paucity of information to help you troubleshoot.
Perhaps one simple way to move forward would be to set up a community effort, grep the codebase for log messages and create a wiki page for each one, just so that something is googleable when there’ a problem. Paired with a set of architectural overviews about how things fit together you’d have a good start. I’m guessing that a lot of the technical questions on this group are asking about the same core concepts, because they’re not written down anywhere.
– Bob.
ps. you still owe me some stickers for my laptop.
This seems like a reasonable idea.
Hi Bobg,
How are you monitoring the logs—is there a specific stream?
We are using the ELK stack to aggregate our logs and would like ot write a plugin for e sot bring logs in to the elk stack.
Cheers
Sean.
On Behalf Of Bob Gregory
Currently we’re not, except when something goes wonky, and then we monitor them with tail -f 
Most likely is that we’ll push the logs to the elk stack, and write something (a collectd plugin?) to push stats.csv to riemann; we could push stats through logstash -> riemann but it seems a little redundant.
I would love to write a dashboard for Eventstore similar to Elasticsearch’s kopf, but I need to properly understand the stats and the gossip first.
Hi,
Ok, I’m writing a LogStash Beats plugin for ES, but forst we need an ES Go client!
Are the logs written to the eventstore or just to files?
Cheers
Sean.
On Behalf Of Bob Gregory
I agree a go client would be very cool.
The statistics are currently available in a stream (atom/protobuf etc)
$statistics-{address:port} for general statistics. For cluster status
you can use any-node/gossip over http to get.
FYI you don’t need a Go client to build a FileBeats plugin - they just consume the log files from disk.
However, I would quite like a Go client. Perhaps that might be my OSS project for the week 
Hi,
That’s true, it just seemed an excuse to learn go!
On Behalf Of James Nugent
Not documentation per se, but I’d appreciate an announcement on this mailing list or the blog when a new version is released. (even better with release notes. Or maybe there’s a clever way to hook up githubs release notes to an rss reader?).
/Peter
If you subscribe to the GitHub repository the release should appear in the notifications whenever a new tag is added. We normally post to a mailing list for commercial customers as well.
Extremely late to this party, but I’ll second the several requests for overall general best practices for production environments as well as more understanding of the internals architecture and flow. I’m a visual learner, so even some more basic whiteboard type breakdowns of general architecture, streams, projections, etc., would be great. Maybe something more in the vein of the EventStore talks currently on SkillsMatter.com with quality visuals.
Also - absolutely love Bob Gregory’s community wiki idea. That would certainly be something I would both consume and contribute into.
-
More information on what GES is intended for and how GES is positioned. For example, it has been said GES can be seen similar to a database server in your infrastructure. But databases are shared by multiple applications, and have schemas, how to do this in GES.
-
Information on best practices with regards to running one or multiple instances on a machine (instance per application, one cluster shared between applications, …?)
-
More documentation on projections
-
Also documentation on projections in C# (Saw a skillsmatter video that said it is possible, although only for advanced scenarios)
Those are some of the things which come up right away.