DNS discovery and Certificate Management for EventStoreDB versions 21.10.5 and beyond

We've been working to simplify wildcard certificate management, so here is some new guidance on certificate management from EventStoreDB version 21.10.5 and beyond.

These instructions explain how to configure EventStoreDB with a wildcard certificate (from a public or private Certificate Authority (CA)) and cluster DNS discovery, how to rotate the certificates, and how to replace a node with this setup.

Be aware that these instructions will only work as of version 21.10.5.

There are also two assumptions made about your setup:

• Each node has a single IP address used for node-to-node and client-to-cluster communication
• Clients have the root Certificate Authority certificate installed
  
  

Note: see this post for more information.

Initial Setup

The initial setup of EventStoreDB will be a 3-node cluster. All nodes and the cluster DNS will use *.project.example.com as a wildcard certificate example.

Each node has 1 IP address used both for internal (node to node) communication and client communication.

DNS Records

The following table lists the necessary DNS records

Note that the records TTL are low (minutes, not hours) in order to allow more rapid propagation in case of a node being replaced.

Certificate

Public CA:

The certificate private key must be in RSA format. The certificate needs to have the following attributes:

• Subject Name - Common Name (CN): *.project.example.com
• Subject Alternative Name (SAN): *.project.example.com

The relevant configuration for the certificates for all nodes is:

# Certificates configuration
CertificateFile: /etc/eventstore/certs/node.crt
CertificatePrivateKeyFile: /etc/eventstore/certs/node.key
TrustedRootCertificatesPath: /etc/ssl/certs
CertificateReservedNodeCommonName: "*.project.example.com"

Note: The TrustedRootCertificatesPathis the standard location of trusted root certificates in Linux-based systems and is used in this case since the root certificate is a public Certificate Authority (CA).

Private CA:

This assumes you are using The Event Store gencert-cli.

Generate the root CA

./es-gencert-cli create-ca

Generate each node certificates

./es-gencert-cli.exe create-node --dns-names *.project.example.com -out ./certs/n1
./es-gencert-cli.exe create-node --dns-names *.project.example.com -out ./certs/n2
./es-gencert-cli.exe create-node --dns-names *.project.example.com -out ./certs/n3

Copy each node.crt and node.key files to /etc/eventstore/certs on each respective node.
Copy the public key of the root certificate authority, the ca.crt file, to /etc/eventstore/certs/ca on each node.

The relevant configuration for the certificates for each node is:

# Certificates configuration
CertificateFile: /etc/eventstore/certs/node.crt
CertificatePrivateKeyFile: /etc/eventstore/certs/node.key
TrustedRootCertificatesPath: /etc/eventstore/certs/ca

Note: The TrustedRootCertificatesPath in this case is not the standard location of trusted root certificates in Linux-based systems.

Node Configuration

Each node configuration is then, in the case of private CA: (for public CA issued certificates, replace the certificate configuration section)

Node1

---
# Paths
Db: / var/lib/eventstore
Index: / var/lib/eventstore/index
Log: / var/log/eventstore
 
# Certificates configuration
CertificateFile: /etc/eventstore/certs/node.crt
CertificatePrivateKeyFile: /etc/eventstore/certs/node.key
TrustedRootCertificatesPath: /etc/eventstore/certs/ca
 
# Network configuration
IntIp: 127.0.20.1
ExtIp: 127.0.20.1
AdvertiseHostToClientAs: node1.project.example.com
HttpPort: 2113
IntTcpPort: 1112
ExtTcpPort: 1113
EnableExternalTcp: true
EnableAtomPubOverHTTP: true
 
# Cluster gossip
ClusterSize: 3
DiscoverViaDns: true
ClusterDns: cluster.project.example.com
 
# Projections configuration
RunProjections: All

Node2

---
# Paths
Db: / var/lib/eventstore
Index: / var/lib/eventstore/index
Log: / var/log/eventstore
 
# Certificates configuration
CertificateFile: /etc/eventstore/certs/node.crt
CertificatePrivateKeyFile: /etc/eventstore/certs/node.key
TrustedRootCertificatesPath: /etc/eventstore/certs/ca
 
# Network configuration
IntIp: 127.0.20.2
ExtIp: 127.0.20.2
AdvertiseHostToClientAs: node2.project.example.com
HttpPort: 2113
IntTcpPort: 1112
ExtTcpPort: 1113
EnableExternalTcp: true
EnableAtomPubOverHTTP: true
 
# Cluster gossip
ClusterSize: 3
DiscoverViaDns: true
ClusterDns: cluster.project.example.com
 
# Projections configuration
RunProjections: All

Node3

---
# Paths
Db: / var/lib/eventstore
Index: / var/lib/eventstore/index
Log: / var/log/eventstore
 
# Certificates configuration
CertificateFile: /etc/eventstore/certs/node.crt
CertificatePrivateKeyFile: /etc/eventstore/certs/node.key
TrustedRootCertificatesPath: /etc/eventstore/certs/ca
 
# Network configuration
IntIp: 127.0.20.3
ExtIp: 127.0.20.3
AdvertiseHostToClientAs: node3.project.example.com
HttpPort: 2113
IntTcpPort: 1112
ExtTcpPort: 1113
EnableExternalTcp: true
EnableAtomPubOverHTTP: true
 
# Cluster gossip
ClusterSize: 3
DiscoverViaDns: true
ClusterDns: cluster.project.example.com
 
# Projections configuration
RunProjections: All

Client connection string

Note that when using certificates signed by a private CA, the root CA public key must be trusted by the client. Typically, it is placed in the trusted store of the user under which the application is running.

gRPC

esdb+discover://cluster.project.example.com:2113

TCP

ConnectTo=discover://cluster.project.example.com:2113;UseSslConnection=true

Renewing a node certificate

In both cases of certificates issued by a public and private CA the procedure is the same.

• Issue the new certificates
• Copy the public key to the same locations as the CertificateFile configuration setting e.g.: /etc/eventstore/certs/node.new.crt
• Copy the private key to the same location as the CertificatePrivateKeyFile configuration setting e.g.: /etc/eventstore/certs/node.new.key
• Change the configuration file:

  # Certificates configuration
  CertificateFile: /etc/eventstore/certs/node.new.crt
  CertificatePrivateKeyFile: /etc/eventstore/certs/node.new.key

• Force a reload of the certificate configuration
  
  Note: the -k forces curl to skip verification of the certificate revocation

  curl -k -X POST -u [user]:[password] --basic https://node1.project.examples.com:2113/admin/reloadconfig -d '' -v

Once the configuration has been reloaded, the server will contain log entries like:

[14476,74,19:15:23.591,INF] Reloading the node's configuration since a request has been received on /admin/reloadconfig.
[14476,23,19:15:23.593,INF] Loading the node's certificate(s) from file: "/etc/eventstore/certs/node.new.crt"
[14476,23,19:15:23.628,INF] Loading the node's certificate. Subject: "CN=eventstoredb-node", Previous thumbprint: "DA090ED6844C15F11031A1CE8A3841ECAEA5AA6C", New thumbprint: "DA090ED6844C15F11031A1CE8A3841ECAEA5AA6C"
[14476,23,19:15:23.628,INF] Loading trusted root certificates.
[14476,23,19:15:23.631,INF] Loading trusted root certificate file: "/etc/eventstore/certs/ca/ca.crt"
[14476,23,19:15:23.631,INF] Loading trusted root certificate. Subject: "CN=EventStoreDB CA 8a9b87f0cedb5230569473bc8bcca45c, O=Event Store Ltd, C=UK", Thumbprint: "24BEB8402D30DD7B62FC3DAF4F877D8911FD8CED"
[14476,23,19:15:23.635,INF] Certificate chain verification successful.
[14476,23,19:15:23.635,INF] All certificates successfully loaded.
[14476,23,19:15:23.635,INF] The node's configuration was successfully reloaded

Replacing a node

In order to replace a node in this configuration, follow these steps:

• Prepare a new DNS entry for the new node E.g: node4.project.example.com. 300 IN A 127.0.20.4
• Issue a new certificate for the new node
• On the new virtual machine:
  • Install EventStoreDB. The process may not be started yet.
  • Create a configuration file as described above
  • Eventually, restore a backup of the data and index files
• Kill the node to be replaced
  • At this point, the cluster will be in a degraded mode, but still functional
  • The killed node should be reported as “Dead” in EventStoreDB UI and the gossip endpoint
• Change the cluster DNS entries to replace the old node IP address with the new node IP address. E.g: replacing node3 with the IP address of node4

cluster.project.example.com. 300 IN A 127.0.20.1 
cluster.project.example.com. 300 IN A 127.0.20.2 
cluster.project.example.com. 300 IN A 127.0.20.4

• Start the EvenStoreDB process on the new node (node4)
• Remove the DNS entry of the old node (node3)

At this point, the old node will still show up as DEAD in the gossip endpoint and the UI for some time, depending on the DeadMemberRemovalPeriodSec configuration setting. The default setting is 30 minutes.

No changes are needed in the other nodes since they are using DiscoverViaDns and ClusterDns. The client applications do not need any changes as well if they use the cluster DNS discovery as described in the “client connection string” section.