In this article we cover a full end-to-end example, from Event Storming, to creating Event Sourcing and architecture diagrams to implementation in Python.
EventStoreDB is an event-native database built for applications using Event Sourcing, or Event-Driven Architectures. In this article, we run through a practical example of how you might implement EventStoreDB with Event Sourcing.
This article is based on our webinar, End-To-End Example with EventStoreDB, by Solutions Architect, Tony Young.
To demonstrate how to implement an event sourced application in EventStoreDB and to cover all the concepts surrounding Event Sourcing, we’re going to use a simple loan processing example.
Loans processing can be very complicated but we’re going to boil it down to some very simple business rules:
We're going to assume that our loan application is going to contain some data fields submitted by the requester.
There will be a credit check that's required and so we'll simulate actually calling out to an external system and getting back a credit score. Then based on that credit score and our loan application data, we'll work towards some automation.
The goal of this demo is to put together a system that will allow us to show:
We’ll also step through some of the terms we’re using such as Event Storming.
Let's get started.
Event Sourcing is a software development pattern where we aim to model the actions or commands that are taken within a particular system as a set of events that are produced by that system.
In Event Sourcing, the source of truth is always the collection of events that have taken place that capture the facts of what happened.
All that detailed information, that rich context and the state of your system can be derived at any time by collecting together and processing in order these events.
New to Event Sourcing? Read our Beginners Guide.
Event Sourcing is designed to support your business workflows as well as your technology partners. It’ll bring benefits to the entire organization as you work together to implement a new business process.
Let's take a look at four of the key benefits of Event Sourcing using our loan processing example.
In our loan example, Event Sourcing is going to make the solution fully auditable. We're going to have a durable trail of all of the steps taken in our loan application and our loan processing.
That’s because each event is stored in an immutable append-only log (your event store, in this case EventStoreDB), in sequential order. This gives you the ability to ‘time-travel’ through the application to see what has happened.
And as events contain the context of the change, at a very granular level, you’ll not only have the what, where and when, but the who and why of the change.
Because you can see what happened at each step of the loan application, rather than just the end result (loan approved/declined), you can do full auditing right down to the individual event level. This makes it extremely valuable for the business.
You’ll also have a lot of options when it comes to the downstream processing of those events, whether you need the full, rich context of what happened or just a summary. You can provide just the information that’s required.
Another benefit of Event Sourcing is adaptability. Over time our loan processing workflow will evolve and change. Event Sourcing allows easy integration between processes.
Changes that are made in one part of the workflow are easily absorbable by other parts of the workflow. Components can change in isolation without impacting each other and the systems are also fault tolerant.
It makes implementing changes a lot quicker because with Event Sourcing you have a set of loosely coupled processes that are all working together to solve a common problem. You can make a rapid change to one very easily.
Another benefit of Event Sourcing is that it future proofs your technology. If you want to send that data down to a new analytics, AI or machine learning tool, you have all that data, it's all available in its raw form, ready to go. You can transform it into whatever format you need for those new technologies.
As mentioned, Event Sourcing is a software development pattern. Event Storming on the other hand, is a workshop based method where you determine the commands, the events, the business workflow, how you integrate with external systems etc, that you want to model as a software system.
To put it simply, it's a free flowing discussion about the workflow.
Before you start your workshop, you need to identify your key stakeholders across the business.
Bring these different business function teams together in a workshop with the IT team and have a free flow discussion around requirements.
Ask open ended questions to get discussions going about the needs of the business.
The advantage of carrying out an event storming session with cross-functional teams is that it makes the process a partnership in which everyone is speaking the same language. It ensures that it brings together collective wisdom about business processes as well as the technical realities in order to arrive at the best solution.
These requirements can be captured on different coloured sticky notes:
These sticky notes can then be put up on a board for everyone to see. The team can move requirements around, expand on areas and look for optimizations.
At the end of the session you should arrive at a fully fleshed out business workflow that the IT team can go and implement.
Let’s now walk through what an Event Storming session looks like for our loan application example.
At the start of the Event Storming session the IT team asks the business:
"What are we here to collaborate on?"
The business team replies:
"We're talking about building a loan processing system."
Next the IT team asks:
"From a high level, what do we want our loan processing system to do?"
To which the business replies:
"Consumers will apply for loans. We’ll generate decisions to either approve or deny the loan, and notify the customer."
These questions give you the beginnings of your Event Sourcing diagram, of your workflow:
We have Tony, our customer, who's going to perform an action or a command to request a new loan, this is represented by the blue 'sticky note' LoanRequest.
On the right hand side of the diagram we have two further blue sticky notes that show whether Tony’s loan request will get approved or denied. Tony will then be notified of the outcome.
So already you can start to build out this diagram to document your workflow. Next you need to look at how you get from request to decision. Because obviously, there's a huge amount of 'meat' in the middle that needs to be filled in.
The next question the IT team would ask is:
"How do we determine if a loan should be approved or denied?"
This should prompt another free flowing discussion, to talk through the steps that lead to a loan approval or a loan decline. These steps might include an underwriter looking at a loan application as well as the credit score of the consumer, and then making the decision to either approve or deny that loan.
So here we have a little bit more information that we're pulling out. We know that there has to be some system in which an underwriter can look at a loan - we represent the Underwriting System with a pink 'sticky note', as shown below:
We also know that the underwriter needs to look at the loan application and the credit score which we call a read model. The read model is a presentation of information stored as events in some transformed way that a human (or machine) could look at and make a decision.
This read model is represented by a green 'sticky note' and that read model is connected to our Underwriting System, as shown below:
Based on the output of that read model and actions taken within the system, we’ll perform the commands to approve or deny the loan. So we’re beginning to build out this workflow.
Next, the IT team will want to know about the data the underwriter needs in order to make their decision.
The business collects only essential information on the customer, for example, name, address and the amount of the loan that they want.
The IT team will then want to understand how they keep applications separate from each other. They determine from the business that each loan has a unique ID number, so consumers can apply for more than one loan, and each loan is kept separate for privacy and approval reasons.
This then prompts the IT team to ask:
"How is the loan request ID generated?"
The business team confirms that they don’t mind how it’s generated as long as it’s unique. So the IT team can decide how it’s generated and in a way that best suits the chosen technology.
The diagram can now be updated to show the information submitted through the loan application as shown by the orange 'sticky notes' in the image below:
Our customer, applied for a loan that generated an event with all of the details that he supplied as part of his loan application.
We know that the detail is fed into the green read model so we connect those two. We also know that once the customer's loan is approved or denied, we’re approving or denying a specific loan. That loan request ID will be captured as part of the approval or denial event that takes place when the underwriter makes their decision.
Now we need to look at how we feed the credit scores into this process. Remember that part of the loan application looks at the applicants credit score, so how do we get these?
The business confirms that they get the national ID number and checks it against the credit clearing house to return a credit score back to our system. So we’ve got the missing piece of information which we now need to add to our process flow.
Click image to see larger version.
Now we’ve added a business rule to our process flow that tells us we need to take the national ID number and check the customer’s credit score - this is the purple 'sticky note'.
The business rule invokes a particular action, or a command, which is to check the credit score that involves calling another system that we have represented by the pink 'sticky note'.
The credit clearing house sends back a response that gives us the person's credit score. We want to hold that response in another event to say, "hey, the credit was checked and this is the credit score for this national ID number".
We also need to feed that into the read model presented to our underwriters. So now we have a line from this credit checked event and from this loan request event.
We now have all the pieces of information that the underwriter needs to look at our loan request and take an approval or denial action.
So, we're almost there!
However, one other business rule we need to take into account is the standard policies around when loans are approved or denied. Remember, a credit score >=7 loan is approved, a credit score of <=4 loan is denied.
The question then to the business is:
"Can we add some business logic based on these standard policies to automatically approve or deny a loan request?"
Of course the business is going to say yes to anything that reduces the workload of their underwriters!
Going back to our diagram, the business rules are defined so we can further expand our diagram here:
Click image to see larger version.
We've got our loan application, we’ve got our credit check and now we have this extra business rule that’s going to perform that automated decision of approval or denial or make the decision to send to the underwriter.
That expands our diagram significantly because now we want to make sure we document every possible path through the set of business rules.
We’ve now got other ‘actors’ in our diagram to add to Tony, our customer. We’ve got two new customers, Yves and Rob with different paths.
Tony’s path remains the same. He goes off for some manual underwriting, manual approval or denial because his credit score falls within that range that cannot be automatically approved or denied. So a ‘loan approval needed’ event is generated before any further action is taken.
Now, our new customer, Yves, is applying for a loan and he has a really good credit score. So Yves' loan request can be automatically approved. A command is taken which generates an automatic approval event
Finally, Rob's credit score is too poor so we automatically take the decision to decline Rob's loan.
Now we have accurately captured all of the simple business rules that we had set out in our problem domain.
Obviously as you know, if this was a large bank, this diagram would be much more complicated, but for the purposes of our example, we've captured what could be considered a fully functional loan processing system.
Now we have our Event Sourcing diagram following our Event Storming session, we now need to start transitioning it into more of an architecture diagram.
So, if we look at the workflow we’ve created, there's thousands of ways that we could break this up into components that we could implement with Event Sourcing. But as we look towards architecture, there's really four main parts to the solution:
On the left hand side of the diagram above, we have some system that handles submission of applications and reporting of status back to the person who requested the loan, so we can group these and call them Application Submission.
In the middle we have the part that’s performing the credit check and integrates with external systems to carry that out. So we can group these together and call it Credit Checking.
On the right hand side is where we have the decision making processes. We’ve got the business rules, automated rules that get applied along with the Underwriting System where manual decisions have to be made.
These are the four main components to our system.
We can now take these four components and create a processing architecture as shown below:
We’ve got our Loan Request where a customer applies for a loan by providing key information and submitting their application, creating a LoanRequest event that’s placed into EventStoreDB (the event-native database depicted in the middle of the diagram above).
From EventStoreDB we have a Credit Checker that 'listens' out for a new loan request event and performs a credit check, going off to integrate with that credit clearing house and returning the credit score. This will then generate a credit checked event which will go back into EventStoreDB.
The Loan Decider is going to pick this up and apply the business rule - either generate a LoanAutomaticallyApproved, LoanAutomaticallyDenied or a LoanApprovalNeeded event.
If a LoanApprovalNeeded event is created, the Underwriting System will populate a read model for a human to use and take a decision on the loan application. Depending on the outcome, this will either generate a LoanManuallyApproved event or a LoanManuallyDenied event which is appended to EventStoreDB..
Once those events are back in EventStoreDB the status of the loan application is updated and sent back to the customer.
Now let's look at how we implement this within EventStoreDB using Python.
Python as a language is super simple to write code, very readable, executes on pretty much any platform. So we chose Python to work with EventStoreDB, our Python client is fully supportable - however we do support other languages such as .Net, and Java.
You'll find all the code we're working with in our Event Store samples repo that you can find on GitHub. If you want a working example, you can clone that and get started.
Sample code: https://github.com/EventStore/samples/tree/main/LoanApplication/Python
Now, let’s get into the demo by watching the video below:
In this simple example, there's one EventStoreDB database allowing the events to flow through.
So what's happening from a technical level?
In this demo, we're able to walk through all four workflows very quickly and easily with this test injector. In a real running system, you would have lots of these applications coming in, the credit checks and automated decisions take place.
We have this build up of manual underwriting that's required, which can be processed as required and to distribute to multiple underwriters at the same time.
So, now we have all this data coming in as raw events. Let's take a look at some downstream use cases of this data.
Suppose you want to provide some kind of AI based chatbot where the end user would be able to ask for a status update of their loan, or request a change to their loan request.
To achieve this you can feed the raw events straight into something like an OpenAI model and get details back from ChatGPT. We've built an example of this in action, which you can view in the video below:
In today’s world, businesses want their AI models to work in real-time and that’s the power of EventStoreDB with an event sourced application.
Your downstream consumption of EventStoreDB's data can look like anything. The events within EventStoreDB can be sent in raw form, like our OpenAI example, or you could be 'listening' (also know as a subscription) for events that you can transform and summarize into a downstream database to enable analytics and reporting.
It's very common to build an Operational Data Store (ODS) for reporting. You can take those events and build them into a summary table and then allow other teams to run their Tableau, Power BI reports etc.
As new events come in, you can do the transformation and update your ODS so the business can get real-time access to that data.
Another use case is Machine Learning (ML). Suppose we want to have an ML model that is constantly listening and learning.
We can ask it for ways to process improvements by streaming those events into the ML models. We’re then able to get some recommendations to improve the workflow of our loan processing system, or some changes to our business workflow.
So, there you have it, our full end-to-end example of EventStoreDB with Event Sourcing. From Event Storming right through to downstream use cases with AI.
Finally, let's take a quick look at the next steps you can take to get started.
How do you get started with your own implementation using EventStoreDB?
We've got a wealth of resources to help get you started, including our Event Sourcing and EventStoreDB Training Series on our YouTube channel, as well as our training courses and consulting.
If you'd like to speak to a member of our team about your particular use case or other offerings such as Event Store Cloud, then please do reach out.