[website] /

---
_edit_last: "2"
author: kai
categories:
  - demos
  - explained
  - java
  - kafka
  - spring
  - spring-boot
classic-editor-remember: classic-editor
date: "2021-02-05T17:59:38+00:00"
guid: http://juplo.de/?p=1201
parent_post_id: null
post_id: "1201"
title: 'Implementing The Outbox-Pattern With Kafka - Part 0: The example'
url: /implementing-the-outbox-pattern-with-kafka-part-0-the-example/

---
_This article is part of a Blog-Series_

Based on a [very simple example-project](/implementing-the-outbox-pattern-with-kafka-part-0-the-example/)
we will implemnt the [Outbox-Pattern](https://microservices.io/patterns/data/transactional-outbox.html) with [Kafka](https://kafka.apache.org/quickstart).

- Part 0: The Example-Project
- [Part 1: Writing In The Outbox-Table](/implementing-the-outbox-pattern-with-kafka-part-1-the-outbox-table/ "Jump to the explanation what has to be added, to enqueue messages in an outbox for successfully written transactions")

## TL;DR

In this part, a small example-project is introduced, that features a component, which has to inform another component upon every succsessfully completed operation.

## The Plan

In this mini-series I will implement the [Outbox-Pattern](https://microservices.io/patterns/data/transactional-outbox.html)
as described on Chris Richardson's fabolous website [microservices.io](https://microservices.io/).

The pattern enables you, to send a message as part of a database transaction in a reliable way, effectively turining the writing of the data
to the database and the sending of the message into an **[atomic operation](https://en.wikipedia.org/wiki/Atomicity_(database_systems))**:
either both operations are successful or neither.

The pattern is well known and implementing it with [Kafka](https://kafka.apache.org/quickstart) looks like an easy straight forward job at first glance.
However, there are many obstacles that easily lead to an incomplete or incorrect implementation.
In this blog-series, we will circumnavigate these obstacles together step by step.

## The Example Project

To illustrate our implementation, we will use a simple example-project.
It mimics a part of the registration process for an web application:
a (very!) simplistic service takes registration orders for new users.

- Successfull registration requests will return a 201 (Created), that carries the URI, under which the data of the newly registered user can be accessed in the `Location`-header:

`echo peter | http :8080/users
  HTTP/1.1 201
  Content-Length: 0
  Date: Fri, 05 Feb 2021 14:44:51 GMT
  Location: http://localhost:8080/users/peter
  `
- Requests to registrate an already existing user will result in a 400 (Bad Request):

`echo peter | http :8080/users
  HTTP/1.1 400
  Connection: close
  Content-Length: 0
  Date: Fri, 05 Feb 2021 14:44:53 GMT
  `
- Successfully registrated users can be listed:
  `http :8080/users
  HTTP/1.1 200
  Content-Type: application/json;charset=UTF-8
  Date: Fri, 05 Feb 2021 14:53:59 GMT
  Transfer-Encoding: chunked
  [
      {
          "created": "2021-02-05T10:38:32.301",
          "loggedIn": false,
          "username": "peter"
      },
      ...
  ]
  `

## The Messaging Use-Case

As our messaging use-case imagine, that there has to happen several processes after a successful registration of a new user.
This may be the generation of an invoice, some business analytics or any other lengthy process that is best carried out asynchronously.
Hence, we have to generate an event, that informs the responsible services about new registrations.

Obviously, these events should only be generated, if the registration is completed successfully.
The event must not be fired, if the registration is rejected, because a duplicate username.

On the other hand, the publication of the event must happen reliably, because otherwise, the new might not be charged for the services, we offer...

## The Transaction

The users are stored in a database and the creation of a new user happens in a transaction.
A "brilliant" colleague came up with the idea, to trigger an `IncorrectResultSizeDataAccessException` to detect duplicate usernames:

`User user = new User(username);
repository.save(user);
// Triggers an Exception, if more than one entry is found
repository.findByUsername(username);
`

The query for the user by its names triggers an `IncorrectResultSizeDataAccessException`, if more than one entry is found.
The uncaught exception will mark the transaction for rollback, hence, canceling the requested registration.
The 400-response is then generated by a corresponding `ExceptionHandler`:

`@ExceptionHandler
public ResponseEntity incorrectResultSizeDataAccessException(
    IncorrectResultSizeDataAccessException e)
{
  LOG.info("User already exists!");
  return ResponseEntity.badRequest().build();
}
`

Please do not code this at home...

But his weired implementation perfectly illustrates the requirements for our messaging use-case:
The user is written into the database.
But the registration is not successfully completed until the transaction is commited.
If the transaction is rolled back, no message must be send, because no new user was registered.

## Decoupling with Springs EventPublisher

In the example implementation I am using an `EventPublisher` to decouple the business logic from the implementation of the messaging.
The controller publishes an event, when a new user is registered:

`publisher.publishEvent(new UserEvent(this, usernam));
`

A listener annotated with `@TransactionalEventListener` receives the events and handles the messaging:

`@TransactionalEventListener
public void onUserEvent(UserEvent event)
{
    // Sending the message happens here...
}
`

In non-critical use-cases, it might be sufficient to actually send the message to Kafka right here.
Spring ensures, that the message of the listener is only called, if the transaction completes successfully.
But in the case of a failure this naive implementation can loose messages.
If the application crashes, after the transaction has completed, but before the message could be send, the event would be lost.

In the following blog posts, we will step by step implement a solution based on the Outbox-Pattern, that can guarantee Exactly-Once semantics for the send messages.

## May The Source Be With You!

The complete source code of the example-project can be cloned here:

- `git clone /git/demos/spring/data-jdbc`
- `git clone https://github.com/juplo/demos-spring-data-jdbc.git`

It includes a [Setup for Docker Compose](https://github.com/juplo/demos-spring-data-jdbc/blob/master/docker-compose.yml), that can be run without compiling
the project. And a runnable [README.sh](https://github.com/juplo/demos-spring-data-jdbc/blob/master/README.sh), that compiles and run the application and illustrates the example.