Implementing The Outbox-Pattern With Kafka – Part 1: The Outbox-Table

Previously in This Blog-Series…

Based on a very simple example-project we will implemnt the Outbox-Pattern with Kafka.

In this part, we will implement the outbox (aka: the queueing of the messages in a database-table).

The Outbox

The outbox is represented by an additionall table in the database. This table acts as a queue for messages, that should be send as part of the transaction. Instead of sending the messages, the application stores them in the outbox-table. The actual sending of the messages occures outside of the transaction.

Because the messages are read from the table outside of the transaction context, only entries related to sucessfully commited transactions are visible. Hence, the sending of the message effectively becomes a part of the transaction. It happens only, if the transaction was successfully completed. Messages associated to an aborted transaction will not be send.

The Implementation

No special measures need to be taken when writing the messages to the table. The only thing to be sure of is that the writing takes part in the transaction.

In our implementation, we simply store the serialized message, together with a key, that is needed for the partitioning of your data in Kafka, in case the order of the messages is important. We also store a timestamp, that we plan to record as Event Time later.

One more thing that is worth noting is that we utilize the database to create an unique record-ID. The generated unique and monotonically increasing id is required later, for the implementation of Exactly-Once semantics.

The SQL for the table looks like this:

CREATE TABLE outbox (
  id BIGINT PRIMARY KEY AUTO_INCREMENT,
  key VARCHAR(127),
  value varchar(1023),
  issued timestamp
);

Decoupling The Business Logic

In order to decouple the business logic from the implementation of the messaging mechanism, I have implemented a thin layer, that uses Spring Application Events to publish the messages.

Messages are send as a subclass of ApplicationEvent:

publisher.publishEvent(
  new UserEvent(
    this,
    username,
    CREATED,
    ZonedDateTime.now(clock)));

The event takes a key (username) and an object as value (an instance of an enum in our case). An EventListener receives the events and writes them in the outbox table:

@TransactionalEventListener(phase = TransactionPhase.BEFORE_COMMIT)
public void onUserEvent(OutboxEvent event)
{
  try
  {
    repository.save(
        event.getKey(),
        mapper.writeValueAsString(event.getValue()),
        event.getTime());
  }
  catch (JsonProcessingException e)
  {
    throw new RuntimeException(e);
  }
}

I chose a @TransactionalEventListener with the phase set to BEFORE_COMMIT that the listener has to be executed in the same transaction context in which the event was published. Otherwise, the writing of the messages would not be coupled to the success or abortion of the transaction, thus violating the idea of the pattern.

A normal EventListener would also suffice here, because spring immediately executes all registered normal event listeners. Therefore, the registered listeners run in the same thread that published the event and participate in the existing transaction.

May The Source Be With You!

Since this part of the implementation only stores the messages in a normal database, it can be published as an independent component that does not require any dependencies on Kafka. To highlight this, the implementation of this step does not use Kafka at all. In a later step, we will separate the layer, that decouples the business code from our messaging logic in a separate package.

The complete source code of the example-project can be cloned here:

This version only includes the logic, that is needed to fill the outbox-tabel. Reading the messages from this table and sending them through Kafka will be the topic of the next part of this blog-series.

The sources include a Setup for Docker Compose, that can be run without compiling the project. And a runnable README.sh, that compiles and run the application and illustrates the example.

Leave a Reply

Your email address will not be published. Required fields are marked *