demos/kafka/outbox
3 years agoSent messages are deleted individually after a crash
Kai Moritz [Sun, 31 Jan 2021 21:24:16 +0000 (22:24 +0100)]
Sent messages are deleted individually after a crash

* If all messages with a sequence-number lower than or equal than the
  sequence number of a message already send are deleted, this can only
  be done for the lowest sequence-number seen accross all partitions
  without violating the at-least-once semantics.
* Deleting seen messages individually by specifying their id (aka
  sequence-number) explicitly in the delete-command allows to delete
  _all_ messages from the outbox, that have been seen on the topic,
  hence, achieving exactly-once-semantics even in the case of an
  unclean shutdown (aka crash) of the OutboxProducer

3 years agoWIP: Using assign instead of subscribe
Kai Moritz [Tue, 20 Apr 2021 20:50:26 +0000 (22:50 +0200)]
WIP: Using assign instead of subscribe

This clearifies, that the current implementation does not scale.
Subscribe suggests, that one can run multiple instances of the service
concurrently. But doing so would break this version of the implementation.

TODO: Partitionen bestimmen...

3 years agoThe OutboxProducer restores the sequence-number from the wirtten topic
Kai Moritz [Sat, 30 Jan 2021 21:39:41 +0000 (22:39 +0100)]
The OutboxProducer restores the sequence-number from the wirtten topic

* The OutboxProducer consumes the written messages on start-up, extracts the
  send sequence-numbers and recalculates the seen watermarks and the maximal
  sequence-number, for at-least-once semantics.
* This implementation will most probably resend some messages after a crash,
  but it will never skip any unsend messages.
* Messages may be resend despite the recalculated watermarks, because only
  messages with a sequence number lower than the lowest watermark can be
  discarded safely from the outbox, if message-loss is not acceptable.
* Deactivated the integration-test for the loading of the context, because
  it cannot work without an available Kafka-Cluster.

3 years agoOld entries are removed from the outbox-table in batches
Kai Moritz [Sun, 7 Feb 2021 13:18:13 +0000 (14:18 +0100)]
Old entries are removed from the outbox-table in batches

3 years agoPreparing for the next version polling-outbox-3
Kai Moritz [Sun, 16 May 2021 21:51:18 +0000 (23:51 +0200)]
Preparing for the next version polling-outbox-3

Bumped only the version numbers for outbox-delivery and
polling-outbox-parent, because the other components are expected to stay
unchanged.

3 years agoReleased the version of the implementation for part 2 of the blog series part-2
Kai Moritz [Sun, 16 May 2021 21:31:38 +0000 (23:31 +0200)]
Released the version of the implementation for part 2 of the blog series

3 years agoEnabled logging-level DEBUG for the "outbox"-service
Kai Moritz [Sun, 31 Jan 2021 15:55:58 +0000 (16:55 +0100)]
Enabled logging-level DEBUG for the "outbox"-service

3 years agoMoved the name of the header for the sequence-number into a static field
Kai Moritz [Sun, 31 Jan 2021 15:51:59 +0000 (16:51 +0100)]
Moved the name of the header for the sequence-number into a static field

3 years agoEnabled idempotence for the producer
Kai Moritz [Sat, 30 Jan 2021 15:45:34 +0000 (16:45 +0100)]
Enabled idempotence for the producer

3 years agoReplaced handwritten property-names by references
Kai Moritz [Sat, 30 Jan 2021 15:37:31 +0000 (16:37 +0100)]
Replaced handwritten property-names by references

3 years agoMade the polling-interval configurable
Kai Moritz [Sun, 1 Nov 2020 20:58:07 +0000 (21:58 +0100)]
Made the polling-interval configurable

3 years agoMoved postage of messages into a reusable standalone implementation
Kai Moritz [Sun, 1 Nov 2020 11:58:20 +0000 (12:58 +0100)]
Moved postage of messages into a reusable standalone implementation

* Renamed maven-module outbox into delivery
* Renamed app polling-outbox into outbox-polling-delivery
* Added new artifact outbox-postage as maven-module postage
* Added fully qualified names for the docker-images
* Adapted docker-compose.yml and README.sh
* TODO: Move flyway-scriptes for outbox into module postage

3 years agoAdded Google Guava for byte-conversions
Kai Moritz [Fri, 30 Oct 2020 10:57:01 +0000 (11:57 +0100)]
Added Google Guava for byte-conversions

3 years agoFIX: Added missing annotation @EnableScheduling
Kai Moritz [Wed, 28 Oct 2020 17:31:56 +0000 (18:31 +0100)]
FIX: Added missing annotation @EnableScheduling

* The application did not do anything, because scheduling was not activated
* Adding @EnableScheduling also keeps the application alive
* Therefore, waiting on the CountDownLatch is not needed any more

3 years agoAdded a simple implementation of the polling outbox pattern
Kai Moritz [Sun, 16 May 2021 15:20:27 +0000 (17:20 +0200)]
Added a simple implementation of the polling outbox pattern

3 years agoAdded data-jdbc as git submodule jdbc
Kai Moritz [Sat, 31 Oct 2020 15:59:17 +0000 (16:59 +0100)]
Added data-jdbc as git submodule jdbc