demos/kafka/outbox
6 months agoFixed bug on first start-up (no current offsets) master
Kai Moritz [Mon, 1 Feb 2021 14:45:54 +0000 (15:45 +0100)]
Fixed bug on first start-up (no current offsets)

6 months agoSent messages are deleted individually after a crash
Kai Moritz [Sun, 31 Jan 2021 21:24:16 +0000 (22:24 +0100)]
Sent messages are deleted individually after a crash

* If all messages with a sequence-number lower than or equal than the
  sequence number of a message already send are deleted, this can only
  be done for the lowest sequence-number seen accross all partitions
  without violating the at-least-once semantics.
* Deleting seen messages individually by specifying their id (aka
  sequence-number) explicitly in the delete-command allows to delete
  _all_ messages from the outbox, that have been seen on the topic,
  hence, achieving exactly-once-semantics even in the case of an
  unclean shutdown (aka crash) of the OutboxProducer

6 months agoWIP: Using assign instead of subscribe
Kai Moritz [Tue, 20 Apr 2021 20:50:26 +0000 (22:50 +0200)]
WIP: Using assign instead of subscribe

This clearifies, that the current implementation does not scale.
Subscribe suggests, that one can run multiple instances of the service
concurrently. But doing so would break this version of the implementation.

TODO: Partitionen bestimmen...

6 months agoThe OutboxProducer restores the sequence-number from the wirtten topic
Kai Moritz [Sat, 30 Jan 2021 21:39:41 +0000 (22:39 +0100)]
The OutboxProducer restores the sequence-number from the wirtten topic

* The OutboxProducer consumes the written messages on start-up, extracts the
  send sequence-numbers and recalculates the seen watermarks and the maximal
  sequence-number, for at-least-once semantics.
* This implementation will most probably resend some messages after a crash,
  but it will never skip any unsend messages.
* Messages may be resend despite the recalculated watermarks, because only
  messages with a sequence number lower than the lowest watermark can be
  discarded safely from the outbox, if message-loss is not acceptable.
* Deactivated the integration-test for the loading of the context, because
  it cannot work without an available Kafka-Cluster.

6 months agoOld entries are removed from the outbox-table in batches
Kai Moritz [Sun, 7 Feb 2021 13:18:13 +0000 (14:18 +0100)]
Old entries are removed from the outbox-table in batches

6 months agoPreparing for the next version polling-outbox-3
Kai Moritz [Sun, 16 May 2021 21:51:18 +0000 (23:51 +0200)]
Preparing for the next version polling-outbox-3

Bumped only the version numbers for outbox-delivery and
polling-outbox-parent, because the other components are expected to stay
unchanged.

6 months agoReleased the version of the implementation for part 2 of the blog series part-2
Kai Moritz [Sun, 16 May 2021 21:31:38 +0000 (23:31 +0200)]
Released the version of the implementation for part 2 of the blog series

6 months agoEnabled logging-level DEBUG for the "outbox"-service
Kai Moritz [Sun, 31 Jan 2021 15:55:58 +0000 (16:55 +0100)]
Enabled logging-level DEBUG for the "outbox"-service

6 months agoMoved the name of the header for the sequence-number into a static field
Kai Moritz [Sun, 31 Jan 2021 15:51:59 +0000 (16:51 +0100)]
Moved the name of the header for the sequence-number into a static field

6 months agoEnabled idempotence for the producer
Kai Moritz [Sat, 30 Jan 2021 15:45:34 +0000 (16:45 +0100)]
Enabled idempotence for the producer

6 months agoReplaced handwritten property-names by references
Kai Moritz [Sat, 30 Jan 2021 15:37:31 +0000 (16:37 +0100)]
Replaced handwritten property-names by references

6 months agoMade the polling-interval configurable
Kai Moritz [Sun, 1 Nov 2020 20:58:07 +0000 (21:58 +0100)]
Made the polling-interval configurable

6 months agoMoved postage of messages into a reusable standalone implementation
Kai Moritz [Sun, 1 Nov 2020 11:58:20 +0000 (12:58 +0100)]
Moved postage of messages into a reusable standalone implementation

* Renamed maven-module outbox into delivery
* Renamed app polling-outbox into outbox-polling-delivery
* Added new artifact outbox-postage as maven-module postage
* Added fully qualified names for the docker-images
* Adapted docker-compose.yml and README.sh
* TODO: Move flyway-scriptes for outbox into module postage

6 months agoAdded Google Guava for byte-conversions
Kai Moritz [Fri, 30 Oct 2020 10:57:01 +0000 (11:57 +0100)]
Added Google Guava for byte-conversions

6 months agoFIX: Added missing annotation @EnableScheduling
Kai Moritz [Wed, 28 Oct 2020 17:31:56 +0000 (18:31 +0100)]
FIX: Added missing annotation @EnableScheduling

* The application did not do anything, because scheduling was not activated
* Adding @EnableScheduling also keeps the application alive
* Therefore, waiting on the CountDownLatch is not needed any more

6 months agoAdded a simple implementation of the polling outbox pattern
Kai Moritz [Sun, 16 May 2021 15:20:27 +0000 (17:20 +0200)]
Added a simple implementation of the polling outbox pattern

6 months agoAdded data-jdbc as git submodule jdbc
Kai Moritz [Sat, 31 Oct 2020 15:59:17 +0000 (16:59 +0100)]
Added data-jdbc as git submodule jdbc