Access denied when mounting disks for EventStoreDB 20

EventStoreDB
1

Hello folks!

We have translated the Docker Compose example from the docs into a Kubernetes deployment so we can use run ES DB for our dev/test environments (production still uses ES Cloud).

We’ve created 3 pods, each one as node of the ES DB Cluster.

For each pod, we created 2 disks on Azure and mounted them on each individual pod so they have a dedicated pair of disks, one for logs and one for data mounted as /eventstore/logs and /eventstore/data respectively. We’ve also set the ENVVars accordingly:

image
image1640×752 81 KB

When the container starts, both volumes appear on it just fine with RW access but the application crashes at the startup:

==============================================================================================================
INSECURE MODE IS ON. THIS MODE IS *NOT* RECOMMENDED FOR PRODUCTION USE.
INSECURE MODE WILL DISABLE ALL AUTHENTICATION, AUTHORIZATION AND TRANSPORT SECURITY FOR ALL CLIENTS AND NODES.
==============================================================================================================

[    1, 1,20:10:00.423,INF] Quorum size set to 2
[    1, 1,20:10:00.424,WRN] Authentication and Authorization is disabled on all TCP/HTTP interfaces. It is recommended to run with Authentication and Authorization enabled in production
[    1, 1,20:10:00.424,WRN] TLS is disabled on all TCP/HTTP interfaces - no certificates are required to run EventStoreDB. It is recommended to run with TLS enabled in production.
[    1, 1,20:10:00.431,INF] Cannot find plugins path: "/usr/share/eventstore/plugins"
[    1, 1,20:10:00.598,DBG] MessageHierarchy initialization took 00:00:00.1376277.
[    1, 1,20:10:00.604,FTL] Host terminated unexpectedly.
**System.UnauthorizedAccessException: Access to the path '/eventstore/data/writer.chk' is denied.**
** ---> System.IO.IOException: Permission denied**
   --- End of inner exception stack trace ---
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
   at System.IO.FileStream.OpenHandle(FileMode mode, FileShare share, FileOptions options)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
   at EventStore.Core.TransactionLog.Checkpoint.MemoryMappedFileCheckpoint..ctor(String filename, String name, Boolean cached, Boolean mustExist, Int64 initValue) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.Core/TransactionLog/Checkpoint/MemoryMappedFileCheckpoint.cs:line 30
   at EventStore.Core.VNodeBuilder.CreateDbConfig(Int32 chunkSize, Int32 cachedChunks, String dbPath, Int64 chunksCacheSize, Boolean inMemDb, Boolean unbuffered, Boolean writethrough, Int32 chunkInitialReaderCount, Int32 chunkMaxReaderCount, Boolean optimizeReadSideCache, Boolean reduceFileCachePressure, Int64 maxTruncation, ILogger log) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.Core/VNodeBuilder.cs:line 1608
   at EventStore.Core.VNodeBuilder.Build(IOptions options, IPersistentSubscriptionConsumerStrategyFactory[] consumerStrategies) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.Core/VNodeBuilder.cs:line 1400
   at EventStore.ClusterNode.ClusterVNodeHostedService.Create(ClusterNodeOptions opts) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.ClusterNode/ClusterVNodeHostedService.cs:line 144
   at EventStore.Core.EventStoreHostedService`1..ctor(String[] args) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.Core/EventStoreHostedService.cs:line 45
   at EventStore.ClusterNode.ClusterVNodeHostedService..ctor(String[] args) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.ClusterNode/ClusterVNodeHostedService.cs:line 34
   at EventStore.ClusterNode.Program.Main(String[] args) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.ClusterNode/Program.cs:line 22

We’ve validated that the pod can indeed write to the mounted path but we still get access denied.

Do we need any special permissions on that directory? Any guidance on what may be wrong?

Thank you!

[]s

2

The user in the container image is not root, could it be that?

3

That may be the case. We are not enforcing/setting any user on our pod spec. Since the container crash very early in the process I’m not able to attach a terminal to see the exactly user account that is being used…

4

Ok, we figured out. Created another docker file based on the official ES DB one, and replaced the entry point. Inspected the UID and GID and then changed it on our volume mounts.

Now it can write to the disk.

Thanks!