Event-Driven Architecture: Beyond the Schema Registry

Event-Driven Architecture is taking off.

All we have to do is look around us to see the amount of innovation that has been happening recently.

There is a huge investment of money and time into this space.

We have companies like Amazon and Upstash that are focusing on reducing the barrier to entry for Event-Driven Architectures (offering Serverless solutions) and it's totally understandable.

With this reduced barrier to entry I believe we are going to see an uptake in the amount of companies adopting Event Architectures. We already have 80% of all Fortune 100 companies trusting and using Kafka, this architecture is just going to continue to grow.

With the current rate of innovation and investment we are moving towards a world where Event-Driven Architecture is becoming more accessible than ever.

James Urquant suggests we are entering a world of data flow and real time integrations. A world where we can consume event streams, get value from real time data (streams) and start to see standards for event communication between technology start to emerge. We can already start seeing this with CloudEvents and AsyncAPI.

Source: https://eventmodeling.org

More recently we have seen tools like EventStorming and EventModelling gain popularity proving they are great ways for us to identify and capture Events within our systems.

It's proven that there is value capturing context and information for our events.

There is huge value to discuss why an event exists, the flow of events and collectively unfold the story in which our system is made up of events.

We understand that Events are not just for developers but the whole organisation to identify and own.

"It is not the domain expert's knowledge that goes into production. is is the assumption of the developers that goes into production" - Alberto Brandolini

Events are created for a reason. The context of existence is important.

Event context and information is the key, but unfortunately it's often the thing that is lost.

•••

Event Architectures evolve over time, more events get added and more services produce and consume them. The reasoning behind the event's existence is lost.

We have developed ways to help us as developers understand our Events, versions and schemas. We know this as the Schema Registry.

But I can't help but feel we can go beyond the Schema Registry and provide more value for our events...

Think for a minute...

What if we could document producers/consumers of events?
What if we could document the event context (why it exists)
What if we can visualise the relationships between events and services?
What if we could plugin to existing systems?
What if it was as easy as writing markdown files?

In this blog post I want to share with you my thoughts around Event Documentation, Schema Registries and why I believe there is much more we can do.

I have been working on an Open Source solution I believe can help our community and help us start to get more value from our Event Architectures.

If you are still with me, grab a drink and let's get started.

•••

Going beyond the Schema Registry

Schema Registry provides a shared repository of schemas that allows applications to flexibly interact with each other

Schema Registries are great.

They allow us to define the contract between consumers and producers within our Architecture. They can store versions of our events and enable tooling around our Architecture.

Schema Registry enables us to have a reliable development process, gives us better control over our data quality and helps us ensure our events are in the correct format.

But things tend to stop there....

Our events are much more than names and schemas.

We could document much more information with our events:

Event Creation (Why the event was created)
Evolution of the event (Why have things changed)
Examples of events (Schemas are good, but better with examples)
How to trigger the events (Code snippets on how to quickly trigger?)
Consumers/Producers
And more....

Which got me thinking....

I wonder if we can still have the benefits of our schema registry but also have the ability to document our events?

These thoughts lead me to research and ultimately starting to spike a solution to help us document our events called EventCatalog.

Let me explain how it works.

•••

Introducing EventCatalog

EventCatalog Landing Page

After speaking to many people and joining various communities I'm started to learn that...

Event observability and discovery is hard.

I believe part of the problem is lack of Event Documentation.

Now, one solution would be to keep an external document up to date with our events and ecosystem, but as time goes on this is hard to maintain and will soon get out of date....

So what else can we do?

I believe there might be a way in which we can document our events whilst integrating with existing solutions and have minimal effort to keep them up to date.

•••

Designing EventCatalog

I started to sketch some initial thoughts and ideas.

EventCatalog Initial Drawings

What if adding documents for our events was as easy as writing a blog post?

Could we have an enriched tool that allows us to visualise events, flows and document events with minimal effort?

But what does minimal effort mean?

For me it was simple:

EventCatlog needs to integrate with existing solutions (Schema Registries, AsyncAPI, Kafka, Amazon EventBridge etc)
EventCatlog needs to be easy to use
EventCatalog needs to provide instant value (little effort, maximum value)
Reduce overhead of documentation tooling as much as possible

With that in mind, I started to spike EventCatalog and I believe I have a solution that may help our community document Events, Visualise Flow and much more....

•••

How EventCatalog works

I'm a huge fan of the open source project docusaurus and the developer experience they offer. Docusaurus gives developers the ability to write documentation websites with minimal effort.

This is the experience I want for EventCatalog.

When using EventCatalog you will focus on three parts:

eventcatalog.config.js (configure your catalog)
Events (markdown files)
Services (markdown files)

Key Features

EventCatalog uses NextJS under the hood, which allows us to render static content and server side content if we wish. This let's us do some interesting things...

Render interactive documentation
Mermaid Diagrams for Events + Services
3D Node Graphs for your Architecture
Schema Renderers
Example (Code) Renderers
Trigger events within EventCatalog itself (coming later)
And much more....

I'm happy to share that most of this is now done, which you can see in the video below.

If you are into Event-Driven Architecture and Event Documentation checkout this tool I'm working on (WIP)

⚡️ Create docs in seconds
📄 Just add event markdown files
📊 Visual consumers, producers, and owners
💥 And more...

Hopefully, this open-source tool can help you!

Demo 👇 pic.twitter.com/NiiFUDCsSZ
— David Boyne 🚀 (@boyney123) November 18, 2021

Another key part to EventCatalog will be the ability to keep documentation up to date and integrate with your current systems.

EventCatalog supports a plugin architecture that will allow you to render custom elements, generate documentation and much more.

•••

Plugins

Plugin based projects are very common and great ways to expand projects into new requirements and user needs.

You will be able to write plugins using EventCatalog API and do things like the following:

Generate documentation
Write custom components
Trigger events into your architecture
and more....

For example, if you have a Schema Registry for Apache Kafka, you could write a plugin to consume your schemas and generate documentation for you. You can add custom metadata using EventCatalog (event context) and keep generating docs without overriding this information.

Here is a small video of me using the AsyncAPI plugin I built to generate documentation. AsyncAPI provides a specification that allows you to define services, messages, channels and events. This plugin reads the file and generates markdown files required for EventCatalog.

Our events and services within event architectures can be almost anything 🤔

EventCatalog will support plugins that will allow you auto-generate docs from any system ♥️🚀

Here is just one example of how you could use @AsyncAPISpec to generate your catalog👇📹

💯 Open Source pic.twitter.com/B0P0WHmTJV
— David Boyne 🚀 (@boyney123) December 3, 2021

With the plugin API you will be able to write your own plugins and integrate with your existing solutions.

•••

Want to learn more Amazon EventBridge?

Get the EventBridge Book!

A guide to walk you through Amazon EventBridge with practical examples, use-cases and advanced patterns.

•••

What's Next

I'm still working on getting the EventCatalog initial version released, (writing docs, tests, etc) and hopefully will have something released soon ready for feedback.

I'm a huge fan of Event Architectures, I have seen ways they can help companies scale and remain decoupled. We have some great tooling available like Schema Registries which allow us to document our schemas, but I believe we can do much more.

Hopefully EventCatalog will help you observe and discover events.

If you wish to contribute, provide feedback or join the community feel free to join the discord channel.

If you would like to follow my thought process or connect feel free to reach out or follow on Twitter.

Thanks for reading, and I hope this work helps!