From: Kai Moritz Date: Tue, 16 Dec 2025 19:19:05 +0000 (+0100) Subject: WIP:?MEhr Beispiel-Content X-Git-Url: https://juplo.de/gitweb/?a=commitdiff_plain;h=de6c8a50f7431999335354f5082cc02aeb115e5d;p=website WIP:?MEhr Beispiel-Content --- diff --git a/content/blog/wp2hugo.html b/content/blog/wp2hugo.html new file mode 100644 index 00000000..a614c261 --- /dev/null +++ b/content/blog/wp2hugo.html @@ -0,0 +1,157 @@ +--- +_edit_last: "2" +author: kai +categories: + - demos + - explained + - java + - kafka + - spring + - spring-boot +classic-editor-remember: classic-editor +date: "2021-02-05T17:59:38+00:00" +guid: http://juplo.de/?p=1201 +parent_post_id: null +post_id: "1201" +title: 'Implementing The Outbox-Pattern With Kafka - Part 0: The example' +url: /implementing-the-outbox-pattern-with-kafka-part-0-the-example/ + +--- +_This article is part of a Blog-Series_ + +Based on a [very simple example-project](/implementing-the-outbox-pattern-with-kafka-part-0-the-example/) +we will implemnt the [Outbox-Pattern](https://microservices.io/patterns/data/transactional-outbox.html) with [Kafka](https://kafka.apache.org/quickstart). + +- Part 0: The Example-Project +- [Part 1: Writing In The Outbox-Table](/implementing-the-outbox-pattern-with-kafka-part-1-the-outbox-table/ "Jump to the explanation what has to be added, to enqueue messages in an outbox for successfully written transactions") + +## TL;DR + +In this part, a small example-project is introduced, that features a component, which has to inform another component upon every succsessfully completed operation. + +## The Plan + +In this mini-series I will implement the [Outbox-Pattern](https://microservices.io/patterns/data/transactional-outbox.html) +as described on Chris Richardson's fabolous website [microservices.io](https://microservices.io/). + +The pattern enables you, to send a message as part of a database transaction in a reliable way, effectively turining the writing of the data +to the database and the sending of the message into an **[atomic operation](https://en.wikipedia.org/wiki/Atomicity_(database_systems))**: +either both operations are successful or neither. + +The pattern is well known and implementing it with [Kafka](https://kafka.apache.org/quickstart) looks like an easy straight forward job at first glance. +However, there are many obstacles that easily lead to an incomplete or incorrect implementation. +In this blog-series, we will circumnavigate these obstacles together step by step. + +## The Example Project + +To illustrate our implementation, we will use a simple example-project. +It mimics a part of the registration process for an web application: +a (very!) simplistic service takes registration orders for new users. + +- Successfull registration requests will return a 201 (Created), that carries the URI, under which the data of the newly registered user can be accessed in the `Location`-header: + +`echo peter | http :8080/users + HTTP/1.1 201 + Content-Length: 0 + Date: Fri, 05 Feb 2021 14:44:51 GMT + Location: http://localhost:8080/users/peter + ` +- Requests to registrate an already existing user will result in a 400 (Bad Request): + +`echo peter | http :8080/users + HTTP/1.1 400 + Connection: close + Content-Length: 0 + Date: Fri, 05 Feb 2021 14:44:53 GMT + ` +- Successfully registrated users can be listed: + `http :8080/users + HTTP/1.1 200 + Content-Type: application/json;charset=UTF-8 + Date: Fri, 05 Feb 2021 14:53:59 GMT + Transfer-Encoding: chunked + [ + { + "created": "2021-02-05T10:38:32.301", + "loggedIn": false, + "username": "peter" + }, + ... + ] + ` + +## The Messaging Use-Case + +As our messaging use-case imagine, that there has to happen several processes after a successful registration of a new user. +This may be the generation of an invoice, some business analytics or any other lengthy process that is best carried out asynchronously. +Hence, we have to generate an event, that informs the responsible services about new registrations. + +Obviously, these events should only be generated, if the registration is completed successfully. +The event must not be fired, if the registration is rejected, because a duplicate username. + +On the other hand, the publication of the event must happen reliably, because otherwise, the new might not be charged for the services, we offer... + +## The Transaction + +The users are stored in a database and the creation of a new user happens in a transaction. +A "brilliant" colleague came up with the idea, to trigger an `IncorrectResultSizeDataAccessException` to detect duplicate usernames: + +`User user = new User(username); +repository.save(user); +// Triggers an Exception, if more than one entry is found +repository.findByUsername(username); +` + +The query for the user by its names triggers an `IncorrectResultSizeDataAccessException`, if more than one entry is found. +The uncaught exception will mark the transaction for rollback, hence, canceling the requested registration. +The 400-response is then generated by a corresponding `ExceptionHandler`: + +`@ExceptionHandler +public ResponseEntity incorrectResultSizeDataAccessException( + IncorrectResultSizeDataAccessException e) +{ + LOG.info("User already exists!"); + return ResponseEntity.badRequest().build(); +} +` + +Please do not code this at home... + +But his weired implementation perfectly illustrates the requirements for our messaging use-case: +The user is written into the database. +But the registration is not successfully completed until the transaction is commited. +If the transaction is rolled back, no message must be send, because no new user was registered. + +## Decoupling with Springs EventPublisher + +In the example implementation I am using an `EventPublisher` to decouple the business logic from the implementation of the messaging. +The controller publishes an event, when a new user is registered: + +`publisher.publishEvent(new UserEvent(this, usernam)); +` + +A listener annotated with `@TransactionalEventListener` receives the events and handles the messaging: + +`@TransactionalEventListener +public void onUserEvent(UserEvent event) +{ + // Sending the message happens here... +} +` + +In non-critical use-cases, it might be sufficient to actually send the message to Kafka right here. +Spring ensures, that the message of the listener is only called, if the transaction completes successfully. +But in the case of a failure this naive implementation can loose messages. +If the application crashes, after the transaction has completed, but before the message could be send, the event would be lost. + +In the following blog posts, we will step by step implement a solution based on the Outbox-Pattern, that can guarantee Exactly-Once semantics for the send messages. + +## May The Source Be With You! + +The complete source code of the example-project can be cloned here: + +- `git clone /git/demos/spring/data-jdbc` +- `git clone https://github.com/juplo/demos-spring-data-jdbc.git` + +It includes a [Setup for Docker Compose](https://github.com/juplo/demos-spring-data-jdbc/blob/master/docker-compose.yml), that can be run without compiling +the project. And a runnable [README.sh](https://github.com/juplo/demos-spring-data-jdbc/blob/master/README.sh), that compiles and run the application and illustrates the example.