Json log format - filebeat / logstash

[stefan.eriksson]

Hi there,

We are running EventstoreDB on linux and are shipping the logs using Elastic Filebeat.
The ingestion pipeline in Elastic is setup to accept logs as specified in the documentation:

Diagnostics | EventStoreDB Documentation

These have a EventProperties and Message properties that ar used in the pipeline.

However, the logs look like this:
{"@t":"2024-05-15T09:58:16.5258250+00:00","@mt":"Verifying hash for TFChunk '{chunk}'...","@l":"Debug","@i":2466452794,"chunk":"/datadisk/eventstore/db/chunk-001150.000000","SourceContext":"EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk","ProcessId":1172,"ThreadId":6}

The documentation says “This format is aligned with Serilog Compact JSON format”, just below a format that is clearly not serilog compact json. This seem to be the format used in the logs.

• Why does the documentation show a format that is not used in the logs?
• Is there a setting to change the format? (I only found plain/json)
• Does anyone have an example of server and elastic settings to make this work?

Thanks.

[stefan.eriksson]

We are now looking into ingesting the compact format to elastic.
However, it is not trivial to create the formatted message (@m in serilog compact format) in the ingestion pipeline. Does anyone have an example of this? As I understand I will have to run a javascript script in the filebeat of logstash setup.

It would be helpful if there was an option to include the formatted message in the log, as it would make the elastic ingestion a lot easier.