Error starting eventstore for 21.10.0, ExtHttpPort is not a known option

I am upgrading from 20.10.4 to 21.10.0.

It seems there are no breaking changes for client gRCP and everything works fine when I spin up a local EventStoreDB container like this:

docker run --name esdb-node -it -p 2113:2113 -p 1113:1113 eventstore/eventstore:21.10.0-buster-slim --insecure --enable-atom-pub-over-http --mem-db=True

However, I am using GitLab CI/CD and spinning up a container for executing unit/integration tests. The tests fail when running in GitLab and the event store container does not initialize so I’m guessing there are some additional changes I should make here.

This is my .gitlab-ci.yml job that runs tests

test:
  stage: test
  services:
    - name: eventstore/eventstore:21.10.0-buster-slim
      alias: eventstore
  variables:
    EVENTSTORE_INSECURE: "true"
    EVENTSTORE_MEM_DB: "true"
    EVENTSTORE_CLUSTER_SIZE: "1"
    EVENTSTORE_EXT_TCP_PORT: "1113"
    EVENTSTORE_EXT_HTTP_PORT: "2113"
  before_script:
    - rounds=10;
      while [ $rounds -gt 0 ]; do
        curl http://eventstore:2113 && echo OK && break || echo FAIL
        rounds=`expr $rounds-1`;
        sleep 5;
      done;
      echo Continue with test execution
  script: dotnet test --blame --configuration Release
  rules:
    - exists:
      - test/**/*Tests.csproj

and I am seeing errors in gRPC connectivity such as

System.InvalidOperationException : Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: Name or service not known (eventstore:2113) SocketException: Name or service not known", DebugException="System.Net.Http.HttpRequestException: Name or service not known (eventstore:2113)
 ---> System.Net.Sockets.SocketException (0xFFFDFFFF): Name or service not known
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)

because the hostname eventstore is unreachable. It seems it fails to start as I can see traces such as

Could not resolve host: eventstore

and

ExtHttpPort is not a known option

As a side note I am using the connection string

esdb://eventstore:2113?tls=false

And locally it works fine also since I’ve got a 127.0.0.1 eventstore alias on my /etc/hosts

Where can I find details about configuration for 21.10.0 and how it differs from 20.10.4?

Apologies if I’ve missed any info on this at the docs.

Thanks!


PS: This is the full log

Service runner-z3wu8uu--project-24665749-concurrent-0-9ad0f0f99381ec75-eventstore__eventstore-0 probably didn't start properly.
Health check error:
start service container: Error response from daemon: Cannot link to a non running container: /runner-z3wu8uu--project-24665749-concurrent-0-9ad0f0f99381ec75-eventstore__eventstore-0 AS /runner-z3wu8uu--project-24665749-concurrent-0-9ad0f0f99381ec75-eventstore__eventstore-0-wait-for-service/service (docker.go:1143:0s)
Service container logs:
2021-11-04T15:16:50.978745067Z [    1, 1,15:16:50.913,FTL] Error while parsing options: The option ExtHttpPort is not a known option. (Parameter 'ExtHttpPort')
2021-11-04T15:16:50.978771130Z [    1, 1,15:16:50.960,INF] Options:
2021-11-04T15:16:50.978775474Z OPTION                                                    DESCRIPTION
2021-11-04T15:16:50.978779099Z 
2021-11-04T15:16:50.978782231Z Application Options
2021-11-04T15:16:50.978785453Z --help=false                                              Show help.
2021-11-04T15:16:50.978788760Z --version=false                                           Show version.
2021-11-04T15:16:50.978792131Z --config=/etc/eventstore/eventstore.conf                  Configuration files.
2021-11-04T15:16:50.978795412Z --what-if=false                                           Print effective configuration to console and then exit.
2021-11-04T15:16:50.978798773Z --start-standard-projections=false                        Start the built in system projections.
2021-11-04T15:16:50.978802103Z --disable-http-caching=false                              Disable HTTP caching.
2021-11-04T15:16:50.978805447Z --stats-period-sec=30                                     The number of seconds between statistics gathers.
2021-11-04T15:16:50.978808844Z --worker-threads=0                                        The number of threads to use for pool of worker services. Set to '0' to scale automatically (Default)
2021-11-04T15:16:50.978820548Z --enable-histograms=false                                 Enables the tracking of various histograms in the backend, typically only used for debugging, etc.
2021-11-04T15:16:50.978824063Z --log-http-requests=false                                 Log Http Requests and Responses before processing them.
2021-11-04T15:16:50.978827616Z --log-failed-authentication-attempts=false                Log the failed authentication attempts.
2021-11-04T15:16:50.978831012Z --skip-index-scan-on-reads=false                          Skip Index Scan on Reads. This skips the index scan which was used to stop reading duplicates.
2021-11-04T15:16:50.978837367Z --max-append-size=1048576                                 The maximum size of appends, in bytes. May not exceed 16MB.
2021-11-04T15:16:50.978840836Z --insecure=false                                          Disable Authentication, Authorization and TLS on all TCP/HTTP interfaces.
2021-11-04T15:16:50.978853846Z 
2021-11-04T15:16:50.978857231Z Logging Options
2021-11-04T15:16:50.978860489Z --log=/var/log/eventstore                                 Path where to keep log files.
2021-11-04T15:16:50.978863847Z --log-config=logconfig.json                               The name of the log configuration file.
2021-11-04T15:16:50.978868076Z --log-level=Default                                       Sets the minimum log level. For more granular settings, please edit logconfig.json. (Default, Verbose, Debug, Information, Warning, Error, Fatal)
2021-11-04T15:16:50.978872578Z --log-console-format=Plain                                Which format (plain, json) to use when writing to the console. (Plain, Json)
2021-11-04T15:16:50.978881168Z --log-file-size=1073741824                                Maximum size of each log file.
2021-11-04T15:16:50.978884651Z --log-file-interval=Day                                   How often to rotate logs. (Infinite, Year, Month, Day, Hour, Minute)
2021-11-04T15:16:50.978931015Z --log-file-retention-count=31                             How many log files to hold on to.
2021-11-04T15:16:50.978935348Z --disable-log-file=false                                  Disable log to disk.
2021-11-04T15:16:50.978938818Z 
2021-11-04T15:16:50.978942064Z Authentication/Authorization Options
2021-11-04T15:16:50.978945330Z --authorization-type=internal                             The type of authorization to use.
2021-11-04T15:16:50.978948774Z --authorization-config                                    Path to the configuration file for authorization configuration (if applicable).
2021-11-04T15:16:50.978952191Z --authentication-type=internal                            The type of Authentication to use.
2021-11-04T15:16:50.978955490Z --authentication-config                                   Path to the configuration file for Authentication configuration (if applicable).
2021-11-04T15:16:50.978958964Z --disable-first-level-http-authorization=false            Disables first level authorization checks on all HTTP endpoints. This option can be enabled for backwards compatibility with EventStore 5.0.1 or earlier.
2021-11-04T15:16:50.978962586Z 
2021-11-04T15:16:50.978965674Z Certificate Options
2021-11-04T15:16:50.978968871Z --trusted-root-certificates-path                          The path to a directory which contains trusted X.509 (.pem, .crt, .cer, .der) root certificate files.
2021-11-04T15:16:50.978972347Z --certificate-reserved-node-common-name=eventstoredb-node The reserved common name to authenticate EventStoreDB nodes/servers from certificates
2021-11-04T15:16:50.978975814Z 
2021-11-04T15:16:50.978978876Z Certificate Options (from file)
2021-11-04T15:16:50.978983258Z --certificate-file                                        The path to a PKCS #12 (.p12/.pfx) or an X.509 (.pem, .crt, .cer, .der) certificate file. If you have intermediate certificates, they should be bundled together in a PEM or PKCS #12 file containing the node's certificate followed by the intermediate certificates.
2021-11-04T15:16:50.978987241Z --certificate-private-key-file                            The path to the certificate private key file (.key) if an X.509 (.pem, .crt, .cer, .der) certificate file is provided.
2021-11-04T15:16:50.978990821Z --certificate-password                                    The password to the certificate if a PKCS #12 (.p12/.pfx) certificate file is provided.
2021-11-04T15:16:50.978994262Z 
2021-11-04T15:16:50.978997376Z Certificate Options (from store)
2021-11-04T15:16:50.979000538Z --certificate-store-location                              The certificate store location name.
2021-11-04T15:16:50.979003852Z --certificate-store-name                                  The certificate store location name.
2021-11-04T15:16:50.979007185Z --certificate-subject-name                                The certificate store subject name.
2021-11-04T15:16:50.979015165Z --certificate-thumbprint                                  The certificate fingerprint/thumbprint.
2021-11-04T15:16:50.979018672Z 
2021-11-04T15:16:50.979021781Z Cluster Options
2021-11-04T15:16:50.979025041Z --stream-info-cache-capacity=0                            The maximum number of entries to keep in the stream info cache. Set to '0' to scale automatically (Default)
2021-11-04T15:16:50.979028624Z --cluster-size=1                                          The number of nodes in the cluster.
2021-11-04T15:16:50.979031978Z --node-priority=0                                         The node priority used during leader election.
2021-11-04T15:16:50.979035329Z --commit-count=-1                                         The number of nodes which must acknowledge commits before acknowledging to a client.
2021-11-04T15:16:50.979038762Z --prepare-count=-1                                        The number of nodes which must acknowledge prepares.
2021-11-04T15:16:50.979042141Z --discover-via-dns=true                                   Whether to use DNS lookup to discover other cluster nodes.
2021-11-04T15:16:50.979045513Z --cluster-dns=fake.dns                                    DNS name from which other nodes can be discovered.
2021-11-04T15:16:50.979048862Z --cluster-gossip-port=2113                                The port on which cluster nodes' managers are running.
2021-11-04T15:16:50.979052191Z --gossip-seed                                             Endpoints for other cluster nodes from which to seed gossip.
2021-11-04T15:16:50.979055569Z --gossip-interval-ms=2000                                 The interval, in ms, nodes should try to gossip with each other.
2021-11-04T15:16:50.979059810Z --gossip-allowed-difference-ms=60000                      The amount of drift, in ms, between clocks on nodes allowed before gossip is rejected.
2021-11-04T15:16:50.979063346Z --gossip-timeout-ms=2500                                  The timeout, in ms, on gossip to another node.
2021-11-04T15:16:50.979066710Z --read-only-replica=false                                 Sets this node as a read only replica that is not allowed to participate in elections or accept writes from clients.
2021-11-04T15:16:50.979070229Z --unsafe-allow-surplus-nodes=false                        Allow more nodes than the cluster size to join the cluster as clones. (UNSAFE: can cause data loss if a clone is promoted as leader)
2021-11-04T15:16:50.979073847Z --dead-member-removal-period-sec=1800                     The number of seconds a dead node will remain in the gossip before being pruned.
2021-11-04T15:16:50.979077256Z --leader-election-timeout-ms=1000                         The timeout, in milliseconds, on election messages to other nodes.
2021-11-04T15:16:50.979080661Z --quorum-size=1                                           
2021-11-04T15:16:50.979083922Z --prepare-ack-count=1                                     
2021-11-04T15:16:50.979087140Z --commit-ack-count=1                                      
2021-11-04T15:16:50.979090380Z 
2021-11-04T15:16:50.979093486Z Database Options
2021-11-04T15:16:50.979099976Z --min-flush-delay-ms=2                                    The minimum flush delay in milliseconds.
2021-11-04T15:16:50.979103454Z --disable-scavenge-merging=false                          Disables the merging of chunks when scavenge is running.
2021-11-04T15:16:50.979106846Z --scavenge-history-max-age=30                             The number of days to keep scavenge history.
2021-11-04T15:16:50.979110164Z --cached-chunks=-1                                        The number of chunks to cache in unmanaged memory.
2021-11-04T15:16:50.979113481Z --chunks-cache-size=536871424                             The amount of unmanaged memory to use for caching chunks in bytes.
2021-11-04T15:16:50.979116824Z --max-mem-table-size=1000000                              Adjusts the maximum size of a mem table.
2021-11-04T15:16:50.979120308Z --hash-collision-read-limit=100                           The number of events to read per candidate in the case of a hash collision.
2021-11-04T15:16:50.979123761Z --db=/var/lib/eventstore                                  The path the db should be loaded/saved to.
2021-11-04T15:16:50.979127070Z --index                                                   The path the index should be loaded/saved to.
2021-11-04T15:16:50.979130709Z --mem-db=false                                            Keep everything in memory, no directories or files are created.
2021-11-04T15:16:50.979134125Z --use-index-bloom-filters=true                            Creates a Bloom filter file for each new index file to speed up index reads.
2021-11-04T15:16:50.979137537Z --index-cache-size=0                                      The maximum number of entries to keep in each index cache.
2021-11-04T15:16:50.979151195Z --skip-db-verify=false                                    Bypasses the checking of file hashes of database during startup (allows for faster startup).
2021-11-04T15:16:50.979154834Z --write-through=false                                     Enables Write Through when writing to the file system, this bypasses filesystem caches.
2021-11-04T15:16:50.979158306Z --unbuffered=false                                        Enables Unbuffered/DirectIO when writing to the file system, this bypasses filesystem caches.
2021-11-04T15:16:50.979161735Z --chunk-initial-reader-count=5                            The initial number of readers to start when opening a TFChunk.
2021-11-04T15:16:50.979165097Z --prepare-timeout-ms=2000                                 Prepare timeout (in milliseconds).
2021-11-04T15:16:50.979168390Z --commit-timeout-ms=2000                                  Commit timeout (in milliseconds).
2021-11-04T15:16:50.979171706Z --write-timeout-ms=2000                                   Write timeout (in milliseconds).
2021-11-04T15:16:50.979174964Z --unsafe-disable-flush-to-disk=false                      Disable flushing to disk. (UNSAFE: on power off)
2021-11-04T15:16:50.979178281Z --unsafe-ignore-hard-delete=false                         Disables Hard Deletes. (UNSAFE: use to remove hard deletes)
2021-11-04T15:16:50.979181716Z --skip-index-verify=false                                 Bypasses the checking of file hashes of indexes during startup and after index merges (allows for faster startup and less disk pressure after merges).
2021-11-04T15:16:50.979189417Z --index-cache-depth=16                                    Sets the depth to cache for the mid point cache in index.
2021-11-04T15:16:50.979192933Z --optimize-index-merge=false                              Makes index merges faster and reduces disk pressure during merges.
2021-11-04T15:16:50.979196330Z --always-keep-scavenged=false                             Always keeps the newer chunks from a scavenge operation.
2021-11-04T15:16:50.979200333Z --reduce-file-cache-pressure=false                        Change the way the DB files are opened to reduce their stickiness in the system file cache.
2021-11-04T15:16:50.979203928Z --initialization-threads=1                                Number of threads to be used to initialize the database. Will be capped at host processor count.
2021-11-04T15:16:50.979207429Z --reader-threads-count=0                                  The number of reader threads to use for processing reads. Set to '0' to scale automatically (Default)
2021-11-04T15:16:50.979211043Z --max-auto-merge-index-level=2147483647                   During large Index Merge operations, writes may be slowed down. Set this to the maximum index file level for which automatic merges should happen. Merging indexes above this level should be done manually.
2021-11-04T15:16:50.979214759Z --write-stats-to-db=false                                 Set this option to write statistics to the database.
2021-11-04T15:16:50.979218248Z --max-truncation=268435456                                When truncate.chk is set, the database will be truncated on startup. This is a safety check to ensure large amounts of data truncation does not happen accidentally. This value should be set in the low 10,000s for allow for standard cluster recovery operations. -1 is no max.
2021-11-04T15:16:50.979222171Z --chunk-size=268435456                                    
2021-11-04T15:16:50.979225430Z --stats-storage=File                                       (None, Stream, File, StreamAndFile)
2021-11-04T15:16:50.979228780Z --db-log-format=V2                                        The log format version to use for storing the event log. V3 is currently in development and should only be used for testing purposes. (V2, ExperimentalV3)
2021-11-04T15:16:50.979232470Z --stream-existence-filter-size=256000000                  The amount of memory & disk space, in bytes, to use for the stream existence filter. This should be set to roughly the maximum number of streams you expect to have in your database, i.e if you expect to have a max of 500 million streams, use a value of 500 megabytes. The value you select should also fit entirely in memory to avoid any performance degradation. Use 0 to disable the filter. Resizing the filter will cause a full rebuild.
2021-11-04T15:16:50.979238329Z 
2021-11-04T15:16:50.979241510Z gRPC Options
2021-11-04T15:16:50.979244932Z --keep-alive-interval=10000                               Controls the period (in milliseconds) after which a keepalive ping is sent on the transport.
2021-11-04T15:16:50.979248406Z --keep-alive-timeout=10000                                Controls the amount of time (in milliseconds) the sender of the keepalive ping waits for an acknowledgement. If it does not receive an acknowledgment within this time, it will close the connection.
2021-11-04T15:16:50.979255178Z 
2021-11-04T15:16:50.979258389Z Interface Options
2021-11-04T15:16:50.979261532Z --int-ip=127.0.0.1                                        Internal IP Address.
2021-11-04T15:16:50.979264844Z --ext-ip=127.0.0.1                                        External IP Address.
2021-11-04T15:16:50.979268110Z --http-port=2113                                          The port to run the HTTP server on.
2021-11-04T15:16:50.979271487Z --enable-external-tcp=false                               Whether to enable external TCP communication
2021-11-04T15:16:50.979274849Z --int-tcp-port=1112                                       Internal TCP Port.
2021-11-04T15:16:50.979278117Z --ext-tcp-port=1113                                       External TCP Port.
2021-11-04T15:16:50.979281336Z --ext-host-advertise-as                                   Advertise External Tcp Address As.
2021-11-04T15:16:50.979284665Z --int-host-advertise-as                                   Advertise Internal Tcp Address As.
2021-11-04T15:16:50.979288005Z --advertise-host-to-client-as                             Advertise Host in Gossip to Client As.
2021-11-04T15:16:50.979291336Z --advertise-http-port-to-client-as=0                      Advertise HTTP Port in Gossip to Client As.
2021-11-04T15:16:50.979294640Z --advertise-tcp-port-to-client-as=0                       Advertise TCP Port in Gossip to Client As.
2021-11-04T15:16:50.979297978Z --ext-tcp-port-advertise-as=0                             Advertise External Tcp Port As.
2021-11-04T15:16:50.979301252Z --http-port-advertise-as=0                                Advertise Http Port As.
2021-11-04T15:16:50.979304511Z --int-tcp-port-advertise-as=0                             Advertise Internal Tcp Port As.
2021-11-04T15:16:50.979307821Z --int-tcp-heartbeat-timeout=700                           Heartbeat timeout for internal TCP sockets.
2021-11-04T15:16:50.979311153Z --ext-tcp-heartbeat-timeout=1000                          Heartbeat timeout for external TCP sockets.
2021-11-04T15:16:50.979314458Z --int-tcp-heartbeat-interval=700                          Heartbeat interval for internal TCP sockets.
2021-11-04T15:16:50.979317819Z --ext-tcp-heartbeat-interval=2000                         Heartbeat interval for external TCP sockets.
2021-11-04T15:16:50.979321730Z --gossip-on-single-node                                   When enabled, tells a single node to run gossip as if it is a cluster.
2021-11-04T15:16:50.979325283Z --connection-pending-send-bytes-threshold=10485760        The maximum number of pending send bytes allowed before a connection is closed.
2021-11-04T15:16:50.979328767Z --connection-queue-size-threshold=50000                   The maximum number of pending connection operations allowed before a connection is closed.
2021-11-04T15:16:50.979332203Z --disable-admin-ui=false                                  Disables the admin ui on the HTTP endpoint.
2021-11-04T15:16:50.979335524Z --disable-stats-on-http=false                             Disables statistics requests on the HTTP endpoint.
2021-11-04T15:16:50.979338970Z --disable-gossip-on-http=false                            Disables gossip requests on the HTTP endpoint.
2021-11-04T15:16:50.979345492Z --enable-trusted-auth=false                               Enables trusted authentication by an intermediary in the HTTP.
2021-11-04T15:16:50.979348940Z --disable-internal-tcp-tls=false                          Whether to disable secure internal TCP communication.
2021-11-04T15:16:50.980229866Z --disable-external-tcp-tls=false                          Whether to disable secure external TCP communication.
2021-11-04T15:16:50.980289171Z --enable-atom-pub-over-http=false                         Enable AtomPub over HTTP Interface.
2021-11-04T15:16:50.980293970Z 
2021-11-04T15:16:50.980297857Z Projection Options
2021-11-04T15:16:50.980301346Z --run-projections=None                                    Enables the running of projections. System runs built-in projections, All runs user projections. (None, System, All)
2021-11-04T15:16:50.980305050Z --projection-threads=3                                    The number of threads to use for projections.
2021-11-04T15:16:50.980308426Z --projections-query-expiry=5                              The number of minutes a query can be idle before it expires.
2021-11-04T15:16:50.980311940Z --fault-out-of-order-projections=false                    Fault the projection if the Event number that was expected in the stream differs from what is received. This may happen if events have been deleted or expired.
2021-11-04T15:16:50.980319409Z --projection-compilation-timeout=500                      The time in milliseconds allowed for the compilation phase of user projections
2021-11-04T15:16:50.980325453Z --projection-execution-timeout=250                        The time in milliseconds allowed for the executing a handler in a user projection
2021-11-04T15:16:50.980329036Z
*********

I found the issue. Sorry, it was an obvious one, literally in front of my nose, but I skipped it :sweat_smile:

ExtHttpPort does not exist on this version. As per https://developers.eventstore.com/server/v21.6/networking/http.html#keepalive-pings (which seems to be also valid for 21.10.0)
the variable is HttpPort so the following contains the proper change on EVENTSTORE_HTTP_PORT

test:
  stage: test
  services:
    - name: eventstore/eventstore:21.10.0-buster-slim
      alias: eventstore
  variables:
    EVENTSTORE_INSECURE: "true"
    EVENTSTORE_MEM_DB: "true"
    EVENTSTORE_CLUSTER_SIZE: "1"
    EVENTSTORE_EXT_TCP_PORT: "1113"
    EVENTSTORE_HTTP_PORT: "2113"
  before_script:
    - rounds=10;
      while [ $rounds -gt 0 ]; do
        curl http://eventstore:2113 && echo OK && break || echo FAIL
        let rounds=`expr $rounds-1`;
        sleep 5;
      done;
      echo Continue with test execution
  script: dotnet test --blame --configuration Release
  rules:
    - exists:
      - test/**/*Tests.csproj

All good!

You don’t really need to look at KeepAlive docs to find out the right settings. It’s there in the server docs https://developers.eventstore.com/server/v21.6/networking/http.html

1 Like

For the benefit of those finding there way here when troubleshooting that error, there’s a gotcha with Kubernetes that I’d like to mention. If you deploy a service with a name starting with eventstore then Kubernetes’ service environment variables will interfere with Event Store’s configuration, likely causing the above error with some random option that you can’t find anywhere.

You can set enableServiceLinks: false on the Event Store’s pod spec to switch them off, and Event Store will start. You don’t want to know how long it took me to realise what was causing that :laughing:

1 Like

Yeah, this is a weird one. I got it too some time ago.

We have a warning in the docs, maybe it should mention Kubernetes service name explicitly as it happens way too often