Reliable Event-Driven Architecture in Spring Boot: Outbox, Inbox, Retries, and Idempotency
Introduction
Event-driven architecture looks simple at first.
Your application performs a business operation, publishes an event to Kafka or SQS, and another service consumes it.
For example:
@Transactional
public void createOrder(CreateOrderCommand command) {
Order order = orderRepository.save(...);
kafkaTemplate.send(
"orders",
new OrderCreatedEvent(order.getId())
);
}
The order is saved, an OrderCreatedEvent is published, and other services can react to it.
But there is a problem hiding in those few lines.
What happens if the database transaction succeeds, but publishing the event fails?
And on the other side:
What happens if the consumer receives the same event twice?
These two questions lead to some of the most important patterns in reliable event-driven systems: the Transactional Outbox, Inbox, Idempotent Consumer, and Durable Retry patterns.
Let's build the architecture step by step and see what changes when these patterns have to work in a real Spring Boot application running across multiple instances
The Dual-Write Problem
Imagine an order service that needs to do two things:
-
Save an order to PostgreSQL.
-
Publish
OrderCreatedto Kafka.
Conceptually:
Database ──────> COMMIT ✓
|
Kafka ─────────> PUBLISH ✗
The database and Kafka are two independent systems. A successful database commit does not guarantee a successful Kafka publish. Consider this sequence:
1. INSERT order 2. COMMIT 3. Publish OrderCreated 4. Application crashes
If the application crashes between steps 2 and 3, the order exists but the event doesn't.
Other services may never know that the order was created.
Reversing the operations doesn't solve the problem either:
1. Publish OrderCreated 2. INSERT order 3. Database transaction fails
Now consumers may receive an event for an order that doesn't exist.
This is the classic dual-write problem.
The Transactional Outbox Pattern
Instead of trying to atomically update the database and message broker, we make the event part of the same database transaction as the business operation.
Database Transaction
┌─────────────────────────────┐
│ │
Request ──> Business Data │
│ + │
│ Outbox Event │
│ │
└────────── COMMIT ───────────┘
|
v
Outbox Dispatcher
|
v
Kafka / SQS
When an order is created, we persist both the business data and the event:
ORDER + OUTBOX EVENT
inside the same database transaction.
Either both are committed or neither is committed.
A separate dispatcher then finds pending outbox records and publishes them to the broker.
A simplified lifecycle might look like this:
PENDING | v PROCESSING | +──── success ────> PUBLISHED | └──── failure ────> RETRY / FAILED
This removes the dangerous database-and-broker dual write from the business transaction.
But it also gives us something extremely valuable in production:
persistent delivery state.
Your Outbox Should Be Debuggable
Reliability isn't only about retrying failed operations.
When something goes wrong in production, someone eventually needs to answer:
What happened to this event?
A useful outbox record should contain enough information to answer that question:
eventId eventType source correlationId payload status attempts createdAt availableAt publishedAt lastError
Instead of searching through distributed logs hoping to reconstruct what happened, an engineer can inspect the actual delivery state:
SELECT * FROM event_outbox WHERE status = 'FAILED';
This leads to an important principle:
Reliability mechanisms should also improve debuggability.
If an event cannot be delivered, that failure should be visible and inspectable.
Publishing Reliably Is Only Half the Problem
Suppose our outbox works perfectly.
Every event eventually reaches Kafka.
We still aren't finished.
Most event-driven architectures use at-least-once delivery. That means the same event may be delivered more than once.
Producer | v Kafka | +──── OrderCreated #123 ────> Consumer | +──── OrderCreated #123 ────> Consumer
A consumer might successfully process an event but crash before acknowledging it. The broker can then deliver it again.
If processing means sending an email, the customer might receive two emails.
If processing means performing a financial operation, the consequences can be much worse.
A reliable consumer therefore needs to assume:
Every event can arrive more than once.
The Inbox Pattern
The Inbox Pattern provides a durable record of received events.
Before processing an event, the consumer registers its unique event ID.
Broker
|
v
Inbox Registration
|
+── event already exists ──> DUPLICATE
|
└── new event
|
v
RECEIVED
|
v
PROCESSING
/ \
v v
PROCESSED FAILED
If the same eventId arrives again, the consumer knows that it has already seen the event.
The event ID becomes an idempotency boundary.
Instead of depending on exactly-once delivery, we design the consumer so duplicate delivery does not result in duplicate business effects.
The Inbox Is More Than a Deduplication Table
A minimal inbox could contain nothing more than processed event IDs.
For production systems, however, it can provide something much more useful:
a durable history of event processing.
Consider storing:
eventId eventType source correlationId payload status attempts receivedAt processedAt availableAt lastError
Now imagine someone reports:
Order 123 was created, but the downstream action never happened.
You can inspect the inbox.
Was the event received?
Was processing started?
Did processing fail?
How many times was it attempted?
When is the next retry?
What was the last error?
Those questions become much easier to answer when processing state is explicit.
Retries Should Survive Application Restarts
Spring provides excellent retry mechanisms.
For example:
@Retryable
public void handle(OrderCreatedEvent event) {
...
}
This can be perfectly appropriate for short-lived transient failures.
But event processing introduces another question:
What happens if the JVM dies?
Event processing fails
|
v
Retry scheduled in memory
|
v
Application restarts
If retry state exists only in memory, it disappears with the process.
For durable event processing, retry state can instead be persisted:
FAILED
|
| availableAt <= now
v
PROCESSING
|
+──── success ────> PROCESSED
|
└──── failure ────> FAILED
|
+ attempts++
+ availableAt = next retry
A scheduler periodically finds events whose retry time has arrived and attempts them again.
Because the state lives in the database, restarting the application doesn't destroy the retry information.
Use Backoff Instead of Hammering a Failing Dependency
Retrying immediately and continuously can make an outage worse.
Suppose a downstream service is unavailable.
Thousands of failed events retrying as quickly as possible simply add more pressure to an already failing system.
A better strategy is exponential backoff:
Attempt 1 → immediate Attempt 2 → +1 second Attempt 3 → +2 seconds Attempt 4 → +4 seconds Attempt 5 → +8 seconds
Eventually, the configured retry limit is exhausted.
At that point, the event can remain explicitly marked as failed:
status = FAILED availableAt = null
Automatic processing stops, but the event doesn't disappear.
It remains available for investigation and operational recovery.
What About Dead-Letter Queues?
Dead-letter queues are valuable, particularly for broker-level failures.
But a broker DLQ doesn't necessarily have to become the application's primary record of processing failure.
There is a useful distinction:
Broker concern Application concern
Delivery failure Processing failure
Malformed message Business handler failure
Transport problem Retry exhaustion
| |
v v
DLQ INBOX
The two mechanisms can coexist.
A database-backed inbox gives the application direct visibility into its own processing state, while a DLQ remains available for appropriate broker and transport-level failures.
Then You Deploy Multiple Pods
Everything becomes more interesting once the application runs more than one instance.
OUTBOX
|
pending event
|
┌──────────┴──────────┐
v v
Pod A Pod B
Dispatcher Dispatcher
Both instances may discover the same pending event.
Without concurrency control, both may attempt to process it.
Production implementations therefore need a concept of claiming or locking.
For example:
status lockOwner lockedAt
An instance claims records transactionally before processing them.
Other instances can then determine that those records are already being processed.
But this creates another question:
What happens if a pod claims an event and then dies?
The system needs a deterministic mechanism for recovering stale claims after an appropriate timeout.
At this point, the outbox is no longer just a database table plus a scheduled query.
It has become infrastructure.
Even the Scheduler Can Fail
There is another failure mode that is surprisingly easy to overlook.
Suppose the dispatcher completes successfully and schedules its next execution:
Dispatcher completes
|
v
schedule(nextRun)
|
X
TaskScheduler rejects the task
If the scheduler continues reporting itself as running, the application has entered a dangerous state.
Everything appears healthy.
But no future dispatch will happen.
Events can quietly accumulate in the outbox.
A reliable scheduler therefore benefits from an explicit lifecycle:
STOPPED | v STARTING | v RUNNING | +──── scheduling failure ────> FAILED
A useful invariant is:
A scheduler must not report itself as running if no task is scheduled and no work is currently executing.
Scheduling infrastructure itself needs observable failure semantics.
Observability Is Part of Reliability
Imagine receiving a production incident at 2 AM:
We created the order, but the downstream system didn't process it.
Ideally, you should be able to follow the event:
Order | v Outbox Event | +── created +── claimed +── publish attempts +── published | v Broker | v Inbox Event | +── received +── processing attempts +── failure reason +── retry schedule +── processed
Correlation metadata should connect the pieces.
Payloads should be readable.
State transitions should be explicit.
Failures should remain inspectable.
Logs should explain what the infrastructure is doing without becoming the only source of truth.
A reliable system isn't only one that recovers from failures. It is one that helps engineers understand those failures.
From Architecture to Implementation: NERV Event
These are the problems I wanted to solve consistently across Spring Boot applications.
None of the individual patterns are new.
Transactional outbox is well understood. Idempotent consumers are well understood. Retries, locking, and message brokers are well understood.
The difficulty is making all of them work together consistently in a production application.
That led me to build NERV Event: an open-source event infrastructure library for Spring Boot.
At a high level, the architecture looks like this:
Spring Boot Application
|
┌──────────┴──────────┐
| |
v v
OUTBOX INBOX
| ^
v |
Outbox Dispatcher |
| |
v |
Kafka / AWS SQS ───────────┘
NERV Event brings together:
-
Transactional outbox persistence
-
Durable inbox processing
-
Idempotent event consumption
-
Persistent retries
-
Exponential retry policies
-
Multi-instance-safe processing
-
Kafka integration
-
AWS SQS integration
-
Scheduler lifecycle and failure visibility
-
Event retention
-
Operational inspection
-
Correlation metadata
-
Human-readable persisted payloads
The goal isn't to hide event-driven architecture behind magic.
The goal is to make its behavior predictable, observable, and easy to debug.
Explore NERV Event
NERV Event is open source and available on GitHub.
Source code, documentation, configuration, and examples:
NERV Event on GitHub:
https://github.com/czetsuyatech/nerv-event
If you're building event-driven Spring Boot services, you can also use the project as a reference architecture even if you don't adopt the library itself.
I'll be writing more about the individual pieces behind NERV Event—including transactional publishing, inbox processing, Kafka and SQS integration, retries, multi-pod deployment, and operational tooling—in future articles.
Final Thoughts
Adding Kafka or SQS to a Spring Boot application doesn't automatically make the application reliably event-driven.
The difficult parts exist around the broker:
Business Transaction
|
v
Transactional Outbox
|
v
Reliable Delivery
|
v
At-Least-Once Messaging
|
v
Inbox + Idempotency
|
v
Durable Processing
|
v
Retries + Recovery
|
v
Observability
Each layer addresses a different failure mode.
And in distributed systems, those failure modes aren't theoretical. Processes restart. Networks fail. Messages are redelivered. Dependencies become unavailable. Schedulers stop. Multiple instances compete for the same work.
The goal isn't to pretend those failures won't happen.
The goal is to design the system so that when they do happen:
state is preserved, recovery is predictable, and engineers can understand exactly what happened.
That's the philosophy behind NERV Event.
NERV Event
Open-source event infrastructure for reliable Spring Boot applications.
Post a Comment