Building an event-driven library is one thing. Putting it inside a realistic application is another.
While integrating NERV Event into a full payment showcase, several edge cases surfaced that were easy to overlook when looking at the Outbox and Inbox patterns individually.
None of them changed the fundamental architecture. The Outbox still provides durable publication. The Inbox still provides durable consumption, retries, and idempotency.
But they exposed something more important: production reliability lives in the boundaries between those mechanisms.
Those findings became the focus of NERV Event 2.1.
This release strengthens four areas:
- atomic Inbox processing;
- fail-fast Outbox configuration;
- aggregate-aware event ordering;
- runtime portability for polling.
Let's look at why each one matters — and what these changes taught us about building reliable event-driven systems.
1. The Inbox Transaction Boundary Matters
Consider a consumer processing an event:
Inbox Event
|
v
Handler
|
+--> Update local database
|
v
Mark Inbox PROCESSED
At first glance, this looks correct.
But there is a dangerous failure window if the handler's database transaction commits independently from the Inbox state transition.
Imagine the following sequence:
1. Claim Inbox event 2. Execute handler 3. Commit handler database changes 4. Application crashes 5. Inbox record is still not PROCESSED 6. Event is retried 7. Handler executes again
The Inbox correctly believes the event still requires processing.
The business database, however, has already been changed.
The result can be duplicate local effects.
The NERV Event 2.1 Transaction Model
NERV Event 2.1 closes this failure window.
Local handler database effects and the Inbox transition to PROCESSED now execute within the same transaction.
Claim event
|
v
BEGIN TRANSACTION
|
+--> Execute handler
|
+--> Update business data
|
+--> Mark Inbox PROCESSED
|
COMMIT
If anything fails:
BEGIN TRANSACTION
|
+--> Execute handler
|
+--> FAILURE
|
ROLLBACK
Both the business changes and Inbox completion are rolled back.
Only after that rollback does NERV Event record the retry or failure metadata through a separate transactional boundary.
This gives us an important invariant:
Local handler effects and successful Inbox completion either commit together or do not commit at all.
What This Does Not Solve
This guarantee applies to transactional work performed against the local database.
It cannot magically make an external system part of that database transaction.
If a handler performs:
Inbox Handler
|
+--> Local PostgreSQL update
|
+--> External payment API
the HTTP request cannot simply be rolled back because the database transaction failed afterward.
External side effects therefore still require idempotency.
This distinction is important:
Atomic transactions protect local effects. Idempotency protects effects outside the transaction boundary.
Reliable consumer processing needs both.
2. Silent Outbox Misconfiguration Is a Reliability Problem
Another issue appeared while configuring the showcase.
Imagine an application that can successfully write events into its Outbox, but has no functional dispatcher capable of delivering them.
The application starts normally.
Business transactions succeed.
Outbox records accumulate.
And nothing gets published.
This is one of the more dangerous infrastructure failures because the system can appear healthy while event delivery is effectively disabled.
Fail Fast Instead
NERV Event 2.1 validates active Outbox publication during application startup.
Publication is considered active when the application indicates publishing intent through mechanisms such as:
nerv.event.outbox.enabled=true RetryPolicy custom OutboxDispatcher configured outbound destinations
If publication is active but there is no functional dispatcher, startup fails with an actionable configuration error.
That changes the failure model from:
Production traffic
|
v
Outbox grows
|
v
Events are never delivered
|
v
Someone eventually notices
into:
Application startup
|
v
Configuration validation
|
v
FAIL FAST
For infrastructure configuration, finding the problem before production traffic reaches the application is considerably safer than silently accepting a non-functional configuration.
What About Consumer-Only Applications?
Fail-fast validation introduced another design question.
Some applications use NERV Event only for Inbox consumption.
Requiring every consumer-only application to add:
nerv.event.outbox.enabled=false
would make an otherwise backward-compatible upgrade unnecessarily disruptive.
NERV Event 2.1 therefore distinguishes between having the library installed and actually expressing publication intent.
Consumer-only applications continue to start without additional configuration.
Explicitly disabling the Outbox remains available, but isn't required merely because NERV Event is present.
The result is both:
backward compatibility and fail-fast publication safety.
3. Kafka Keys Alone Don't Guarantee Application-Level Ordering
The largest new capability in NERV Event 2.1 is aggregate-aware ordering.
Consider an Order aggregate producing:
OrderCreated PaymentAuthorized OrderConfirmed
These events have a natural sequence.
But with multiple Outbox workers, retrieving records in database order alone does not guarantee that they will actually reach the broker in that same order.
Worker A might claim:
OrderCreated
while Worker B claims:
PaymentAuthorized
If Worker B publishes first, downstream consumers could observe:
PaymentAuthorized OrderCreated
The database rows were ordered correctly.
The actual dispatch was not.
"Just Use a Kafka Key"
Kafka developers will immediately recognize one part of the solution:
Events belonging to the same aggregate should use the same Kafka record key.
That's necessary.
But it isn't sufficient.
Kafka can preserve ordering within a partition, but the Outbox dispatcher still controls the order in which records are handed to Kafka.
If two concurrent workers effectively perform:
send(event2) send(event1)
using the same Kafka key doesn't magically reverse those calls.
The ordering guarantee therefore has to begin before the broker.
4. Introducing orderingKey
NERV Event 2.1 introduces an optional orderingKey throughout the publication pipeline.
Conceptually:
EventPublication.builder()
.type("ORDER_UPDATED")
.payload(payload)
.orderingKey(orderId)
.build();
The ordering identity travels through the full pipeline:
Application
|
v
EventPublication
|
v
Outbox Persistence
|
v
Claiming
|
v
Dispatcher
|
v
Transport
|
+--> Kafka
|
+--> SQS
The ordering key is persisted in the Outbox so retries continue using the same ordering identity.
It is deliberately optional.
Existing applications do not need to provide one, and unkeyed events retain their existing concurrent dispatch behavior.
5. Ordering Starts in the Outbox
The interesting part isn't adding another property to the publication API.
The important change is how Outbox records are claimed.
NERV Event now ensures that events sharing the same ordering key are claimed and dispatched sequentially.
Conceptually:
Order A / Event 1 --┐ Order A / Event 2 --+-- sequential Order A / Event 3 --┘ Order B / Event 1 --┐ Order B / Event 2 --+-- sequential Order B / Event 3 --┘
But Order A and Order B remain independent:
+-- Order A events -- sequential
Workers -----+
+-- Order B events -- sequential
This distinction matters.
We could preserve ordering by forcing everything through a single global worker:
ALL EVENTS
|
v
single queue
|
v
single worker
But that would sacrifice much of the scalability gained through concurrent Outbox dispatch.
Instead, serialization occurs at the ordering-key level.
Events belonging to the same aggregate remain ordered while unrelated aggregates can continue progressing concurrently.
This protection also applies across batches and application replicas.
Adding more NERV Event instances should increase concurrency between independent ordering keys without allowing two instances to reorder events belonging to the same key.
6. Transport-Specific Ordering
Once the Outbox establishes the ordering boundary, each transport can map that identity onto the guarantees offered by the underlying broker.
Kafka
For Kafka, orderingKey becomes the Kafka record key.
orderingKey
|
v
Kafka record key
|
v
partition selection
Events sharing an ordering key therefore follow Kafka's key and partition ordering semantics.
Amazon SQS FIFO
For SQS FIFO queues, the ordering key becomes:
MessageGroupId = orderingKey
NERV Event also uses:
MessageDeduplicationId = eventId
This maps aggregate ordering naturally onto SQS FIFO message groups while retaining event-level deduplication identity.
Amazon SQS Standard
SQS Standard queues do not provide FIFO ordering guarantees.
NERV Event therefore does not pretend otherwise.
An event may still carry an ordering key at the publication level, but choosing SQS Standard does not transform the underlying transport into an ordered queue.
A useful abstraction should provide stronger guarantees where possible without hiding the limitations of the infrastructure underneath it.
7. Minimal Runtime Images Expose Hidden Dependencies
The showcase also uncovered a smaller issue with an important lesson.
Polling previously relied on a random-generator implementation whose provider was not guaranteed to be available in minimal Java runtime images.
On a normal development JDK, this is easy to miss.
Inside a deliberately reduced production runtime, it becomes a deployment failure.
For NERV Event 2.1, that dependency has been replaced with:
ThreadLocalRandom
from java.base.
For polling jitter and backoff behavior, requiring an additional random provider was adding deployment complexity without providing a meaningful architectural benefit.
Using a standard Java runtime facility removes that unnecessary dependency and makes NERV Event friendlier to minimal runtime images.
There is a broader lesson here:
The deployment environment is part of the architecture.
A library that works perfectly inside a developer's full JDK can still contain assumptions that surface only once the application is packaged for production.
8. Database Compatibility
Supporting durable aggregate ordering requires preserving the ordering key with the event itself.
NERV Event 2.1 therefore adds the PostgreSQL migration:
006-add-outbox-ordering-key.sql
The new Outbox column is nullable.
That is intentional because ordering remains opt-in.
Existing Outbox records remain valid, and applications that do not require aggregate ordering do not need to change how they publish events.
Existing two-argument EventPublication construction also remains supported.
This allows the new capabilities to ship as a backward-compatible 2.1.0 minor release rather than requiring a new major version.
9. The Showcase Was the Important Test
The most interesting part of these changes isn't any individual feature.
It's where the problems were discovered.
The basic Outbox pattern was already working.
The basic Inbox pattern was already working.
Retries were working.
Kafka was working.
SQS was working.
The edge cases appeared when those components were combined inside a realistic application with:
- transactional business logic;
- multiple services;
- concurrent processing;
- multiple application replicas;
- Kafka and SQS transports;
- real failure scenarios;
- and production-style deployment constraints.
This is exactly why I wanted a full NERV showcase rather than another isolated example application.
Examples demonstrate APIs.
Realistic showcases challenge architectural guarantees.
10. The Bigger Lesson
An Outbox isn't reliable merely because events are stored before publication.
An Inbox isn't reliable merely because consumed event IDs are persisted.
Kafka ordering isn't guaranteed merely because a message has a key.
And an application isn't correctly configured merely because Spring Boot managed to start.
The guarantees have to survive the boundaries between those components.
That is what NERV Event 2.1 focuses on:
Durability
+
Atomicity
+
Idempotency
+
Ordering
+
Fail-fast configuration
=
Predictable event processing
The goal isn't to eliminate failure.
Distributed systems will fail.
The goal is to make those failures:
- explicit instead of silent;
- recoverable instead of destructive;
- observable instead of hidden;
- deterministic instead of surprising.
That's the direction NERV Event continues to take: providing Spring applications with event-delivery infrastructure whose behavior remains understandable not only when everything works, but especially when it doesn't.
Try NERV Event
NERV Event is an open-source Spring Boot library for building reliable event-driven applications using transactional Outbox and Inbox patterns.
It provides infrastructure for:
- transactional Outbox publishing;
- durable Inbox consumption;
- idempotent event processing;
- automatic retries and failure tracking;
- multi-instance-safe event claiming;
- aggregate-aware event ordering;
- Kafka integration;
- Amazon SQS integration;
- scheduler resilience;
- and operational inspection APIs.
GitHub: https://github.com/czetsuyatech/nerv-event
If you're building event-driven Spring Boot systems, I'd especially like to hear how you're handling ordering, consumer transaction boundaries, and multi-instance Outbox dispatch in production.
If NERV Event is useful to you, consider starring the project on GitHub. Feedback, issues, and contributions are always welcome.
NERV — Next-Generation Engineering for Runtime Velocity
Production-ready infrastructure for Java engineers building reliable enterprise systems.
Post a Comment