- Published on
Event-Driven Architecture: Beyond the Schema Registry
Schema Registry is great, but can we do more? In this post I explore Event documentation and share with you EventCatalog.
- David Boyne
- Published on • 9 min read
Event-Driven Architecture is taking off.
All we have to do is look around us to see the amount of innovation that has been happening recently.
- More than 80% of all Fortune 100 companies trust, and use Kafka.
- Upstash Launches Serverless Kafka (Nov 2021)
- AWS Introduced MSK Serverless (Nov 2021)
- AWS Launches Amazon EventBridge (2019)
- Evolution of Event Modelling (2018)
- Ably raises $70 Million for realtime features (2021)
- Postman partner with AsyncAPI (2020)
There is a huge investment of money and time into this space.
We have companies like Amazon and Upstash that are focusing on reducing the barrier to entry for Event-Driven Architectures (offering Serverless solutions) and it's totally understandable.
With this reduced barrier to entry I believe we are going to see an uptake in the amount of companies adopting Event Architectures. We already have 80% of all Fortune 100 companies trusting and using Kafka, this architecture is just going to continue to grow.
With the current rate of innovation and investment we are moving towards a world where Event-Driven Architecture is becoming more accessible than ever.
James Urquant suggests we are entering a world of data flow and real time integrations. A world where we can consume event streams, get value from real time data (streams) and start to see standards for event communication between technology start to emerge. We can already start seeing this with CloudEvents and AsyncAPI.
More recently we have seen tools like EventStorming and EventModelling gain popularity proving they are great ways for us to identify and capture Events within our systems.
It's proven that there is value capturing context and information for our events.
There is huge value to discuss why an event exists, the flow of events and collectively unfold the story in which our system is made up of events.
We understand that Events are not just for developers but the whole organisation to identify and own.
"It is not the domain expert's knowledge that goes into production. is is the assumption of the developers that goes into production" - Alberto Brandolini
Events are created for a reason. The context of existence is important.
Event context and information is the key, but unfortunately it's often the thing that is lost.•••
Event Architectures evolve over time, more events get added and more services produce and consume them. The reasoning behind the event's existence is lost.
We have developed ways to help us as developers understand our Events, versions and schemas. We know this as the Schema Registry.
But I can't help but feel we can go beyond the Schema Registry and provide more value for our events...
Think for a minute...
- What if we could document producers/consumers of events?
- What if we could document the event context (why it exists)
- What if we can visualise the relationships between events and services?
- What if we could plugin to existing systems?
- What if it was as easy as writing markdown files?
In this blog post I want to share with you my thoughts around Event Documentation, Schema Registries and why I believe there is much more we can do.
I have been working on an Open Source solution I believe can help our community and help us start to get more value from our Event Architectures.
If you are still with me, grab a drink and let's get started.•••
Going beyond the Schema Registry
Schema Registry provides a shared repository of schemas that allows applications to flexibly interact with each other
Schema Registries are great.
They allow us to define the contract between consumers and producers within our Architecture. They can store versions of our events and enable tooling around our Architecture.
Schema Registry enables us to have a reliable development process, gives us better control over our data quality and helps us ensure our events are in the correct format.
But things tend to stop there....
Our events are much more than names and schemas.
We could document much more information with our events:
- Event Creation (Why the event was created)
- Evolution of the event (Why have things changed)
- Examples of events (Schemas are good, but better with examples)
- How to trigger the events (Code snippets on how to quickly trigger?)
- And more....
Which got me thinking....
I wonder if we can still have the benefits of our schema registry but also have the ability to document our events?
These thoughts lead me to research and ultimately starting to spike a solution to help us document our events called EventCatalog.
Let me explain how it works.•••
After speaking to many people and joining various communities I'm started to learn that...
Event observability and discovery is hard.
I believe part of the problem is lack of Event Documentation.
Now, one solution would be to keep an external document up to date with our events and ecosystem, but as time goes on this is hard to maintain and will soon get out of date....
So what else can we do?
I believe there might be a way in which we can document our events whilst integrating with existing solutions and have minimal effort to keep them up to date.•••
I started to sketch some initial thoughts and ideas.
What if adding documents for our events was as easy as writing a blog post?
Could we have an enriched tool that allows us to visualise events, flows and document events with minimal effort?
But what does minimal effort mean?
For me it was simple:
- EventCatlog needs to integrate with existing solutions (Schema Registries, AsyncAPI, Kafka, Amazon EventBridge etc)
- EventCatlog needs to be easy to use
- EventCatalog needs to provide instant value (little effort, maximum value)
- Reduce overhead of documentation tooling as much as possible
With that in mind, I started to spike EventCatalog and I believe I have a solution that may help our community document Events, Visualise Flow and much more....•••
How EventCatalog works
I'm a huge fan of the open source project docusaurus and the developer experience they offer. Docusaurus gives developers the ability to write documentation websites with minimal effort.
This is the experience I want for EventCatalog.
When using EventCatalog you will focus on three parts:
- eventcatalog.config.js (configure your catalog)
- Events (markdown files)
- Services (markdown files)
EventCatalog uses NextJS under the hood, which allows us to render static content and server side content if we wish. This let's us do some interesting things...
- Render interactive documentation
- Mermaid Diagrams for Events + Services
- 3D Node Graphs for your Architecture
- Schema Renderers
- Example (Code) Renderers
- Trigger events within EventCatalog itself (coming later)
- And much more....
I'm happy to share that most of this is now done, which you can see in the video below.
If you are into Event-Driven Architecture and Event Documentation checkout this tool I'm working on (WIP)— David Boyne 🚀 (@boyney123) November 18, 2021
⚡️ Create docs in seconds
📄 Just add event markdown files
📊 Visual consumers, producers, and owners
💥 And more...
Hopefully, this open-source tool can help you!
Demo 👇 pic.twitter.com/NiiFUDCsSZ
Another key part to EventCatalog will be the ability to keep documentation up to date and integrate with your current systems.
EventCatalog supports a plugin architecture that will allow you to render custom elements, generate documentation and much more.•••
Plugin based projects are very common and great ways to expand projects into new requirements and user needs.
You will be able to write plugins using EventCatalog API and do things like the following:
- Generate documentation
- Write custom components
- Trigger events into your architecture
- and more....
For example, if you have a Schema Registry for Apache Kafka, you could write a plugin to consume your schemas and generate documentation for you. You can add custom metadata using EventCatalog (event context) and keep generating docs without overriding this information.
Here is a small video of me using the AsyncAPI plugin I built to generate documentation. AsyncAPI provides a specification that allows you to define services, messages, channels and events. This plugin reads the file and generates markdown files required for EventCatalog.
Our events and services within event architectures can be almost anything 🤔— David Boyne 🚀 (@boyney123) December 3, 2021
EventCatalog will support plugins that will allow you auto-generate docs from any system ♥️🚀
Here is just one example of how you could use @AsyncAPISpec to generate your catalog👇📹
💯 Open Source pic.twitter.com/B0P0WHmTJV
With the plugin API you will be able to write your own plugins and integrate with your existing solutions.
Want to learn more Amazon EventBridge?Get the EventBridge Book!
A guide to walk you through Amazon EventBridge with practical examples, use-cases and advanced patterns.
I'm still working on getting the EventCatalog initial version released, (writing docs, tests, etc) and hopefully will have something released soon ready for feedback.
I'm a huge fan of Event Architectures, I have seen ways they can help companies scale and remain decoupled. We have some great tooling available like Schema Registries which allow us to document our schemas, but I believe we can do much more.
Hopefully EventCatalog will help you observe and discover events.
If you wish to contribute, provide feedback or join the community feel free to join the discord channel.
If you would like to follow my thought process or connect feel free to reach out or follow on Twitter.
Thanks for reading, and I hope this work helps!