Designing Retry-Safe Event-Driven Systems with Kafka

In event-driven architectures, executing business mutations alongside event publication is fraught with silent failure modes. A prevalent anti-pattern is executing a database update and immediately calling kafkaTemplate.send() in the same service method. If the network blips before the broker acknowledges the record, the downstream systems miss the event entirely. Conversely, if the database transaction fails to commit after the event is dispatched, downstream consumers act on phantom state.

In my Portfolio Management System (PMS) project, I structured trade ingestion around the Transactional Outbox Pattern. Instead of attempting a fragile distributed two-phase commit across PostgreSQL and Apache Kafka, the application writes both the trade mutation and an event record into an outbox table within the same local relational ACID transaction.

@Transactional
public TradeExecutionResult executeTrade(TradeOrderCommand cmd) {
  Trade trade = tradeRepository.save(new Trade(cmd));
  
  OutboxEvent event = OutboxEvent.builder()
      .aggregateType("TRADE")
      .aggregateId(trade.getId())
      .eventType("TRADE_EXECUTED")
      .payload(objectMapper.writeValueAsString(new TradeExecutedEvent(trade)))
      .status(EventStatus.PENDING)
      .createdAt(Instant.now())
      .build();
      
  outboxRepository.save(event);
  return new TradeExecutionResult(trade.getId(), ExecutionStatus.SUBMITTED);
}

A decoupled publisher process reads committed rows from the outbox table and delivers them to Kafka with publisher acknowledges enabled (acks=all). Because the event is persisted inside the relational database transaction, zero events are lost even during immediate node crashes or broker outages.

Kafka guarantees at-least-once delivery out of the box when using acknowledged commits. Consumers must therefore be strictly idempotent. Relying on Kafka offset commits alone is insufficient because application restarts between message processing and offset acknowledgment cause previous messages to be redelivered.

• Unique Event Identifiers: Every payload carries a globally unique eventId and deterministic aggregate version.
• Redis Deduplication Window: A fast pre-flight check in Redis (SETNX with a 24-hour TTL) discards immediate duplicate deliveries before hitting database layers.
• Optimistic Concurrency Constraints: Final state persistence enforces version checks (WHERE version = :expectedVersion) and relational unique constraints on (aggregate_id, event_id).

When a downstream dependency (such as an external market clearing API or risk evaluation service) experiences transient latency, naive thread sleeping in Kafka consumers is catastrophic. Pausing the consumer thread blocks partition consumption, stalling all subsequent unrelated messages and eventually triggering consumer group partition rebalances.

To maintain partition throughput, failed messages are dispatched to tiered retry topics with exponential backoff delays:

• Primary Topic (trades.execution): Ingests real-time events at peak throughput.
• Tier 1 Retry (trades.execution.retry-15s): Messages failing transient checks are forwarded here and consumed with a 15-second delay.
• Tier 2 Retry (trades.execution.retry-60s): Messages failing a second time back off to a 60-second cycle.
• Dead Letter Queue (trades.execution.dlq): Messages that fail 3 consecutive attempts are routed to the DLQ for manual inspection and alerting.

A Dead Letter Queue without active alerting is merely a graveyard for lost business transactions. Every message routed to the DLQ must emit Prometheus metrics tagged with the error category, aggregate ID, and original payload offset. Furthermore, an administrative replay CLI allows operators to re-inject resolved DLQ messages back into the primary topic once downstream dependencies recover.