The Oceanic Safety Validation Pipeline built with EventStoreDB

Disclaimer: Not every one of our customers can reveal the secrets behind their architecture. In this case, we’ve changed key details and names to protect their identity, whilst being able to share their unique use case and how EventStoreDB solved their problems.

Background

The software team behind this domain harness data of oceanic movements using instruments and data from other centres, to provide warnings for mariners all over the world. These warnings are designed to keep oceans safe and prevent human  tragedy: a task that needs a solution suitable for complexity.  

The Challenge

Mariners need accurate, real-time oceanic information. If there’s a shipwreck in an area, then our customer needs to ensure the section of the Electronic Nautical Chart (known as a cell) is updated, and co-ordinate this data with other centres - a critical part of the SOLAS responsibility (Safety Of Life At Sea).

Given its vital function, our customer needed to accurately validate all data that’s provided to them:

“Say, Country A owns a section of the ocean, and Country B owns another part, but they overlap. One reports a shipwreck is there and the other one doesn't. If we pick the wrong data, let's say we pick the latest report rather than picking the report of the actual country that manages that area, then a ship hits something - that comes back to us. That’s just one of the areas of validation.” - John Doe, Senior Software Engineer.

Previously, this was a manual process. Whenever data came in, there would be a labour and time intensive process to manually check everything. While some parts would always need to be manual, they needed a validation pipeline to reduce this workload and automate as much as possible.

“There's an awful term here, called a dead cat bucket. It's essentially where, if you were, let's say, in a production pipeline, building computers and your job was to put the CPU cooler on every computer. Then one day on a conveyor belt, there's a dead cat on one of the computers. Now you wouldn't have a specific automation element for that. You wouldn't be like, right okay, well we're gonna send it down and put it in the dead cat bucket.”

Rather than trying to assume every single outcome, John and his team set out to identify the most regular outcomes. Outside of those, they require manual checks. This resulted in automating the validation process for most of their electronic nautical chart validation, whilst keeping unpredictable parts of the process (dead cats!) manual.

Assessing The Options

With vital systems like the one this team had to build, they assessed multiple solutions including Event Streaming products, Event Driven Architecture and CQRS before finally choosing EventStoreDB.

 
“EventStoreDB is quite different from anything else that's available. Kafka, for example, has implemented features that may seem similar to Event Store, but is not a database. Kafka wasn’t designed to actually store events. I'm not only referencing Kafka, but the Azure options and RabbitMQ have similar limitations".

Ultimately, catch up subscriptions, projections, writing events in projections and multiple streams were must-haves for John’s team, leading them to EventStoreDB.

“EventStoreDB goes pretty much hand in hand with Event Driven Architecture and CQRS. We talked about CQRS as a solution; in our case, though the system is mainly driven by and reacts to events, the command side is almost non-existant. The Query side didn't need any additional stores because we could rely on EventStoreDB projections . Not only to feed and trigger microservices, but also using the projected events to display information and current state to our users.”

Getting Started with EventStoreDB

Our customer uses Azure as their main Cloud provider.

“We initially tried to discover Event Sourcing and CQRS through Microsoft - which proved to be difficult through their documentation. We found Event Store's website after deciding this was the architecture we were going to use and it is fantastic. It kind of just brought it alive a bit more. It's brilliant. And it goes into a lot of depth without getting confusing, which is really hard to do.”

Understanding the complexity of stakeholder buy-in when adopting new technology was important to John when looking into their options. Something that Event Store supported..

“Speaking as somebody who has to talk to different stakeholders about what we're doing. And quite a lot of what we do is very backend and doesn't really have any visuals or anything nice to show people. I've been like, yeah, go check this [Event Store’s website] out. Don't bother reading the Microsoft docs. Being able to do that is wicked.”

The Solution Built with EventStoreDB

Our customer built the validation pipeline project for their Electronic Nautical Charts with EventStoreDB as its backbone. Data comes in one end, and it goes through their microservices, completely decoupled.

Think of the world map being split into a grid. Each grid was a cell, each event for each cell was then split into different streams. An event would come in, attached to a certain cell name, and is then split into three streams.

Stream 1: The name of the event
Stream 2: The full cell name (an eight letter identifier, then the additional update).
Stream 3: The cell and that eight letter identifier without the additional update, so they can actually read the full stream for that particular cell.

Building their system in this way initially felt complicated but the pay-off was worth it. EventStoreDB gave flexibility and a real-source of truth for the unique and complex problem the team were facing.

The Support from Event Store

Being open-source, EventStoreDB is constantly updated based on the feedback from the community. Our support team is there to support our customers at all times, but especially when it’s time to upgrade .

“The feedback (on the upgrade support) has just been insane. We have been really, really pleased. They're (Event Store support team) so quick to get back to us and they've helped us loads, they ask for a load of detail, and they're really patient.”

The Future is Event Store Cloud

Short term goals for our customer is upgrading to the newest version of EventStoreDB and moving to the GRPC client. Larger goals for the team to move from self managed to Cloud to take advantage of the benefits that come with that change, such as multi-data center redundancy.

“We really want to move to Cloud. Either hosting the database in a container or actually using Event Store's cloud, which I think is probably the plan to be honest. We've had a couple of infrastructure related issues, so there wasn't really a push to Cloud but there certainly is now.”

It’s been a pleasure to work with John and his team on their fascinating project. A unique and complex use case, that is enabling the incredibly important work they do to keep oceanic travel safe for all.