Amazon EventBridge: Event Payload Standards

Amazon EventBridge is a great service that provides a Serverless event bus that allows us to write event driven architectures within AWS. With EventBridge we can consume events from AWS, SaaS providers and even create our own custom events.

When creating custom events using EventBridge we can specify any JSON payload that we like, this makes EventBridge extremely flexible but may lead us to defining our event payloads poorly.

A couple of years ago when I started to use EventBridge I didn’t really consider any “standards” for our events, we would just put in the data that made sense for that particular event/domain and that was it. Although this worked pretty well over time I learned that standardising your events can bring benefits and it looks like it's starting to become an industry standard.

In this short blog post I want to dive into event payload patterns that seem to be emerging and what I believe are becoming the standard for EventBridge events.

So, grab a drink and let’s get started.

Amazon EventBridge Events

Let’s take a quick look at a standard EventBridge event.

{
    "version": "0",
    "id": "0d079340-135a-c8c6-95c2-41fb8f496c53",
    "detail-type": "OrderCreated",
    "source": "myapp.orders",
    "account": "123451235123",
    "time": "2022-02-01T18:41:53Z",
    "region": "us-west-1",
    "detail": {...} // whatever we like
}

The version, account, time and region are all properties that AWS handles for us. That leaves core properties detail, detail-type and source to be defined by us.

We typically use the detail-type for the event name and the source to be the service that produced the event. (more info)

So if we look at the event above, we can understand that the event is a OrderCreated event which was created by a service called myapp.orders.

So far so good...

Let's now fill our event with custom data using the detail property.

{
  "version": "0",
  "id": "0d079340-135a-c8c6-95c2-41fb8f496c53",
  "detail-type": "OrderCreated",
  "source": "myapp.orders",
  "account": "123451235123",
  "time": "2022-02-01T18:41:53Z",
  "region": "us-west-1",
  "detail": {
    "amount": "19.99",
    "quantity": "2",
    "orderId": "0d079340-535a-c8c6-95c2-41fb8f496c53",
    "userId": "9e223efa-e318-4e06-84d3-8734db2f31ea"
  }
}

So here we have an OrderCreated event with some data containing the orderId, userId, amount and quantity.

If we leave the event like this, it would be fine. Downstream services could subscribe and everyone might be happy...

But can we do better?

Standardising Event payloads with EventBridge

With Amazon EventBridge it is entirely up to us to define the payloads for our custom events. AWS does not enforce any patterns on our payloads (detail) which have led the community to explore and define their own standards.

Back in 2020 I came across a great Article by Sheen Brisals where he talks about event standards and splitting event payloads into data and metadata sections.

Since then this pattern seems to be becoming more popular and recommended within the AWS community.

Sheen Brisals has shared with us a pattern of introducing metadata within our detail object.

"detail": {
      "metadata": {
        ...
      },
      "data": {
        ...
      }
   }

Implementing these kinds of standards within our events can provide us with some benefits:

• Better filtering of events (we can filter on metadata as well as the event payload)
• Easier downstream processing based on metadata
• Opens the doors to more observability patterns and debugging options

Let's take our OrderCreated event we created earlier and add the new metadata pattern to it.

{
    "version": "0",
    "id": "0d079340-135a-c8c6-95c2-41fb8f496c53",
    "detail-type": "OrderCreated",
    "source": "myapp.orders",
    "account": "123451235123",
    "time": "2022-02-01T18:41:53Z",
    "region": "us-west-1",
    "detail": {
      "metadata": {
        "correlation_id": "dddd9340-135a-c8c6-95c2-41fb8f492222"
        "service": "my-service",
        "domain": "SHOP",
      },
      "data": {
        "amount": "19.99",
        "quantity": "2",
        "orderId": "0d079340-535a-c8c6-95c2-41fb8f496c53",
        "userId": "9e223efa-e318-4e06-84d3-8734db2f31ea"
      }
   }
}

So what has changed here? What benefits can we see?

• trace events using the correlation_id set in the metadata.
• publisher information using the service property.
• know which domain the event was triggered from using the domain field.
• use all these properties for more complex rule filtering downstream.

Now imagine if all of our events within our architecture followed the same pattern and included metadata then we could have a collection of information we can use for filtering, debugging and much more.

Adding metadata in our events is a great pattern introduced by LEGO, and it’s a pattern we can start to see being used widely and recommended within the community...

Who is using EventBridge Event Standards?

Event standardisation is becoming very popular with specifications like cloudevents emerging.

Amazon EventBridge does not provide any patterns or recommendations on how you should structure your events, but as we can see the metadata pattern seems to be getting widely adopted.

Luc van Donkersgoed in his talk Event-Driven Architecture at PostNL Scale shares with us that they follow a similar patterns at PostNL.

Another interesting part is the Event Structure.@donkersgood shows us an example payload for EventBridge, and you will notice it's split between metadata and data.

This is also a pattern I have seen from @sheenbrisals and although I don't think it's documented in AWS... pic.twitter.com/LZ4vJJY6BC

— David Boyne 🚀 (@boyney123) February 1, 2022

Lars Jacobsson shares with us that they are using the metadata pattern.

We also split data/metadata and that idea was stolen straight from the mouth of @sheenbrisals' on the AWS Community Day Nordics 2020. Great talk and great convention 🙌

— Lars (@lajacobsson) February 1, 2022

Matt Lewis and the team at DVLAgovuk are also using the metadata pattern with their Events. (more info coming soon)

In Danio Poccia talk I Build Applications - Event-driven Architecture (Level 300) he references the metadata pattern for EventBridge users.

It looks like the metadata pattern is growing and becoming very popular within the AWS community, and many people are seeing the beneifts of standards within their payloads.

If you are using any event standards with EventBridge, feel free to contact me, I would love to learn about it and add your use case to this blogpost.

Summary

Recently I have spent time diving into EventBridge, Event Architectures and Standardisations (AsyncAPI, Serverless Workflow, CloudEvents).

Across the industry we can start to see a need for event payload standards forming such as CloudEvents and custom event standards within our applications.

A few years ago when I started to use EventBridge my team added any information into the event we felt was necessary and logical without considering any standards. We learnt that simple event standards such as adding metadata into our events can give us many benefits and help us filter, observe and manage our events.

So next time you are defining your events, I recommend considering if you need any standards for your event payloads.

If you want to know more, watch and read the resources linked at the bottom of this blog post and see if any of these patterns can help you and your team.

If you have any questions and would like to connect with me on Twitter feel free, I would love to listen to your experiences.

Thanks for reading, and I hope it helps.

More Reading & Learning

• Sheen gives us a great talk at AWS Community Day Nordics in 2020 and talks about event patterns and payload structure
• Danilo Poccia gives us a great overview of EventBridge and also mentions customer usecases
• Luc van Donkersgoed gives a great talk about EventBridge and Patterns they use at PostNL
• Check out awesome-eventbridge where I collect more awesome talks, articles and videos around EventBridge

In my next blog post I will be writing about Fat and Thin events, the pros and cons of them and what it all means? If you want to be notifie, simply follow on Twitter.