Implementing The Outbox-Pattern With Kafka – Part 0: The example

The Plan

In this mini-series I will implement the Outbox-Pattern as described on Chris Richardson’s fabolous website

The pattern enables you, to send a message as part of a database transaction in a reliable way, effectively turining the writing of the data to the database and the sending of the message into an atomic operation: either both operations are successful or neither.

The pattern is well known and implementing it with Kafka looks like an easy straight forward job at first glance. However, there are many obstacles that easily lead to an incomplete or incorrect implementation. In this blog-series, we will circumnavigate these obstacles together step by step.

The Example Project

To illustrate our implementation, we will use a simple example-project. It mimics a part of the registration process for an web application: a (very!) simplistic service takes registration orders for new users.

  • Successfull registration requests will return a 201 (Created), that carries the URI, under which the data of the newly registered user can be accessed in the Location-header:
    echo peter | http :8080/users
    HTTP/1.1 201 
    Content-Length: 0
    Date: Fri, 05 Feb 2021 14:44:51 GMT
    Location: http://localhost:8080/users/peter
  • Requests to registrate an already existing user will result in a 400 (Bad Request):
    echo peter | http :8080/users
    HTTP/1.1 400 
    Connection: close
    Content-Length: 0
    Date: Fri, 05 Feb 2021 14:44:53 GMT
  • Successfully registrated users can be listed:
    http :8080/users
    HTTP/1.1 200 
    Content-Type: application/json;charset=UTF-8
    Date: Fri, 05 Feb 2021 14:53:59 GMT
    Transfer-Encoding: chunked
            "created": "2021-02-05T10:38:32.301",
            "loggedIn": false,
            "username": "peter"

The Messaging Use-Case

As our messaging use-case imagine, that there has to happen several processes after a successful registration of a new user. This may be the generation of an invoice, some business analytics or any other lengthy process that is best carried out asynchronously. Hence, we have to generate an event, that informs the responsible services about new registrations.

Obviously, these events should only be generated, if the registration is completed successfully. The event must not be fired, if the registration is rejected, because a duplicate username.

On the other hand, the publication of the event must happen reliably, because otherwise, the new might not be charged for the services, we offer…

The Transaction

The users are stored in a database and the creation of a new user happens in a transaction. A “brilliant” colleague came up with the idea, to trigger an IncorrectResultSizeDataAccessException to detect duplicate usernames:

User user = new User(username);;
// Triggers an Exception, if more than one entry is found

The query for the user by its names triggers an IncorrectResultSizeDataAccessException, if more than one entry is found. The uncaught exception will mark the transaction for rollback, hence, canceling the requested registration. The 400-response is then generated by a corresponding ExceptionHandler:

public ResponseEntity incorrectResultSizeDataAccessException(
    IncorrectResultSizeDataAccessException e)
{"User already exists!");
  return ResponseEntity.badRequest().build();

Please do not code this at home…

But his weired implementation perfectly illustrates the requirements for our messaging use-case: The user is written into the database. But the registration is not successfully completed until the transaction is commited. If the transaction is rolled back, no message must be send, because no new user was registered.

Decoupling with Springs EventPublisher

In the example implementation I am using an EventPublisher to decouple the business logic from the implementation of the messaging. The controller publishes an event, when a new user is registered:

publisher.publishEvent(new UserEvent(this, usernam));

A listener annotated with @TransactionalEventListener receives the events and handles the messaging:

public void onUserEvent(UserEvent event)
    // Sending the message happens here...

In non-critical use-cases, it might be sufficient to actually send the message to Kafka right here. Spring ensures, that the message of the listener is only called, if the transaction completes successfully. But in the case of a failure this naive implementation can loose messages. If the application crashes, after the transaction has completed, but before the message could be send, the event would be lost.

In the following blog posts, we will step by step implement a solution based on the Outbox-Pattern, that can guarantee Exactly-Once semantics for the send messages.

May The Source Be With You!

The complete source code of the example-project can be cloned here:

It includes a Setup for Docker Compose, that can be run without compiling the project. And a runnable, that compiles and run the application and illustrates the example.

Leave a Reply

Your email address will not be published. Required fields are marked *