demos/kafka/outbox
3 years agoFixed bug on first start-up (no current offsets) eod-outbox-alt
Kai Moritz [Mon, 1 Feb 2021 14:45:54 +0000 (15:45 +0100)]
Fixed bug on first start-up (no current offsets)

3 years agoSend messages are deleted individually after a crash
Kai Moritz [Sun, 31 Jan 2021 21:24:16 +0000 (22:24 +0100)]
Send messages are deleted individually after a crash

* If all messages with a sequence-number lower than or equal than the
  sequence number of a message already send are deleted, this can only
  be done for the lowest sequence-number seen accross all partitions
  without violating the at-least-once semantics.
* Deleting seen messages individually by specifying their id (aka
  sequence-number) explicitly in the delete-command allows to delete
  _all_ messages from the outbox, that have been seen on the topic,
  hence, achieving exactly-once-semantics even in the case of an
  unclean shutdown (aka crash) of the OutboxProducer

3 years agoThe OutboxProducer restores the sequence-number from the wirtten topic
Kai Moritz [Sat, 30 Jan 2021 21:39:41 +0000 (22:39 +0100)]
The OutboxProducer restores the sequence-number from the wirtten topic

* The OutboxProducer consumes the written messages on start-up, extracts the
  send sequence-numbers and recalculates the seen watermarks and the maximal
  sequence-number, for at-least-once semantics.
* This implementation will most probably resend some messages after a crash,
  but it will never skip any unsend messages.
* Messages may be resend despite the recalculated watermarks, because only
  messages with a sequence number lower than the lowest watermark can be
  discarded safely from the outbox, if message-loss is not acceptable.
* Deactivated the integration-test for the loading of the context, because
  it cannot work without an available Kafka-Cluster.

3 years agoSwitched submodule-path to publicly available sources part-1
Kai Moritz [Sat, 6 Feb 2021 10:14:15 +0000 (11:14 +0100)]
Switched submodule-path to publicly available sources

3 years agoEnabled logging-level DEBUG for the "outbox"-service
Kai Moritz [Sun, 31 Jan 2021 15:55:58 +0000 (16:55 +0100)]
Enabled logging-level DEBUG for the "outbox"-service

3 years agoMoved the name of the header for the sequence-number into a static field
Kai Moritz [Sun, 31 Jan 2021 15:51:59 +0000 (16:51 +0100)]
Moved the name of the header for the sequence-number into a static field

3 years agoEnabled idempotence for the producer
Kai Moritz [Sat, 30 Jan 2021 15:45:34 +0000 (16:45 +0100)]
Enabled idempotence for the producer

3 years agoReplaced handwritten property-names by references
Kai Moritz [Sat, 30 Jan 2021 15:37:31 +0000 (16:37 +0100)]
Replaced handwritten property-names by references

3 years agoOld entries are removed from the outbox-table in batches
Kai Moritz [Sat, 30 Jan 2021 14:06:54 +0000 (15:06 +0100)]
Old entries are removed from the outbox-table in batches

3 years agoImproved example execution in README.sh
Kai Moritz [Sat, 30 Jan 2021 13:43:04 +0000 (14:43 +0100)]
Improved example execution in README.sh

* The setup for Docker Compose uses a specific version of Kafka
* README.sh uses Docker to execute kafka-commands
* No error-message is issued, if the topic already exists

3 years agoMade the polling-interval configurable
Kai Moritz [Sun, 1 Nov 2020 20:58:07 +0000 (21:58 +0100)]
Made the polling-interval configurable

3 years agoAdded an executable README.sh, that demonstrates the example
Kai Moritz [Sun, 1 Nov 2020 16:41:06 +0000 (17:41 +0100)]
Added an executable README.sh, that demonstrates the example

3 years agoMoved postage of messages into a reusable standalone implementation
Kai Moritz [Sun, 1 Nov 2020 11:58:20 +0000 (12:58 +0100)]
Moved postage of messages into a reusable standalone implementation

* Renamed maven-module outbox into delivery
* Renamed app polling-outbox into outbox-polling-delivery
* Added new artifact outbox-postage as maven-module postage
* Added fully qualified names for the docker-images
* TODO: Move flyway-scriptes for outbox into module postage

3 years agoAdded Google Guava for byte-conversions
Kai Moritz [Fri, 30 Oct 2020 10:57:01 +0000 (11:57 +0100)]
Added Google Guava for byte-conversions

3 years agoAdded missing annotation @EnableScheduling
Kai Moritz [Wed, 28 Oct 2020 17:31:56 +0000 (18:31 +0100)]
Added missing annotation @EnableScheduling

* The application did not do anything, because scheduling was not activated
* Adding @EnableScheduling also keeps the application alive
* Therefore, waiting on the CountDownLatch is not needed any more

3 years agoAdded a simple implementation of the polling outbox pattern
Kai Moritz [Sat, 31 Oct 2020 16:03:24 +0000 (17:03 +0100)]
Added a simple implementation of the polling outbox pattern

3 years agoAdded data-jdbc as git submodule jdbc
Kai Moritz [Sat, 31 Oct 2020 15:59:17 +0000 (16:59 +0100)]
Added data-jdbc as git submodule jdbc