demos/kafka/chat
2 months agoWIP:poc-setup rebase--2024-02-26--19-46
Kai Moritz [Mon, 26 Feb 2024 14:14:22 +0000 (15:14 +0100)]
WIP:poc-setup

2 months agoWIP:poc-setup
Kai Moritz [Mon, 26 Feb 2024 14:13:51 +0000 (15:13 +0100)]
WIP:poc-setup

2 months agoWIP:fix-sse
Kai Moritz [Sun, 25 Feb 2024 20:44:37 +0000 (21:44 +0100)]
WIP:fix-sse

2 months agofix: Sensible value for sink-buffer
Kai Moritz [Sun, 25 Feb 2024 20:44:16 +0000 (21:44 +0100)]
fix: Sensible value for sink-buffer

2 months agofeat: Made polling-interval for Kafka configurable
Kai Moritz [Sun, 25 Feb 2024 19:50:21 +0000 (20:50 +0100)]
feat: Made polling-interval for Kafka configurable

2 months agoWIP:wait
Kai Moritz [Sun, 25 Feb 2024 19:43:43 +0000 (20:43 +0100)]
WIP:wait

2 months agoWIP:FIX
Kai Moritz [Sun, 25 Feb 2024 19:42:59 +0000 (20:42 +0100)]
WIP:FIX

2 months agoWIP:loading
Kai Moritz [Sun, 25 Feb 2024 09:32:20 +0000 (10:32 +0100)]
WIP:loading

2 months agoWIP:wait--FIX
Kai Moritz [Sun, 25 Feb 2024 09:00:51 +0000 (10:00 +0100)]
WIP:wait--FIX

2 months agoWIP:wait--FIX
Kai Moritz [Sun, 25 Feb 2024 08:50:44 +0000 (09:50 +0100)]
WIP:wait--FIX

2 months agoWIP:wait--FIX
Kai Moritz [Sun, 25 Feb 2024 08:46:57 +0000 (09:46 +0100)]
WIP:wait--FIX

2 months agoWIP:wait
Kai Moritz [Sat, 24 Feb 2024 19:15:26 +0000 (20:15 +0100)]
WIP:wait

2 months agoWIP:wait
Kai Moritz [Sat, 24 Feb 2024 18:13:42 +0000 (19:13 +0100)]
WIP:wait

2 months agoWIP:wait
Kai Moritz [Sat, 24 Feb 2024 17:59:05 +0000 (18:59 +0100)]
WIP:wait

2 months agoWIP:wait
Kai Moritz [Sat, 24 Feb 2024 17:57:12 +0000 (18:57 +0100)]
WIP:wait

2 months agoWIP:setup
Kai Moritz [Sat, 24 Feb 2024 17:41:44 +0000 (18:41 +0100)]
WIP:setup

2 months agoWIP:setup
Kai Moritz [Sat, 24 Feb 2024 12:28:29 +0000 (13:28 +0100)]
WIP:setup

2 months agoWIP:setup
Kai Moritz [Sat, 24 Feb 2024 12:18:21 +0000 (13:18 +0100)]
WIP:setup

2 months agofix: The number of chards is the number of partitions of the `data_channel`
Kai Moritz [Sat, 24 Feb 2024 11:56:24 +0000 (12:56 +0100)]
fix: The number of chards is the number of partitions of the `data_channel`

2 months agoWIP:setup
Kai Moritz [Sat, 24 Feb 2024 11:50:34 +0000 (12:50 +0100)]
WIP:setup

2 months agoWIP:setup
Kai Moritz [Sat, 24 Feb 2024 11:39:04 +0000 (12:39 +0100)]
WIP:setup

2 months agoWIP:setup
Kai Moritz [Sat, 24 Feb 2024 11:35:14 +0000 (12:35 +0100)]
WIP:setup

2 months agoWIP:startup
Kai Moritz [Fri, 23 Feb 2024 16:04:51 +0000 (17:04 +0100)]
WIP:startup

2 months agoWIP:startup
Kai Moritz [Fri, 23 Feb 2024 15:52:36 +0000 (16:52 +0100)]
WIP:startup

2 months agoWIP:startup
Kai Moritz [Fri, 23 Feb 2024 10:05:27 +0000 (11:05 +0100)]
WIP:startup

2 months agoWIP:neu
Kai Moritz [Fri, 23 Feb 2024 09:37:17 +0000 (10:37 +0100)]
WIP:neu

2 months agoWIP:neu
Kai Moritz [Fri, 23 Feb 2024 09:37:11 +0000 (10:37 +0100)]
WIP:neu

2 months agoWIP:neu
Kai Moritz [Thu, 22 Feb 2024 22:00:07 +0000 (23:00 +0100)]
WIP:neu

2 months agoWIP:neu
Kai Moritz [Thu, 22 Feb 2024 21:59:00 +0000 (22:59 +0100)]
WIP:neu

2 months agoWIP:neu
Kai Moritz [Thu, 22 Feb 2024 21:10:43 +0000 (22:10 +0100)]
WIP:neu

2 months agoWIP:neu
Kai Moritz [Thu, 22 Feb 2024 20:55:33 +0000 (21:55 +0100)]
WIP:neu

2 months agoWIP:neu
Kai Moritz [Thu, 22 Feb 2024 20:49:27 +0000 (21:49 +0100)]
WIP:neu

2 months agoWIP:neu
Kai Moritz [Thu, 22 Feb 2024 20:46:10 +0000 (21:46 +0100)]
WIP:neu

2 months agoWIP:neu
Kai Moritz [Thu, 22 Feb 2024 20:40:17 +0000 (21:40 +0100)]
WIP:neu

2 months agoWIP:neu
Kai Moritz [Thu, 22 Feb 2024 20:06:43 +0000 (21:06 +0100)]
WIP:neu

2 months agoWIP:neu
Kai Moritz [Thu, 22 Feb 2024 19:51:30 +0000 (20:51 +0100)]
WIP:neu

2 months agoWIP:neu
Kai Moritz [Thu, 22 Feb 2024 19:42:46 +0000 (20:42 +0100)]
WIP:neu

2 months agoWIP
Kai Moritz [Sat, 3 Feb 2024 20:38:24 +0000 (21:38 +0100)]
WIP

2 months agowIP
Kai Moritz [Sat, 3 Feb 2024 20:09:51 +0000 (21:09 +0100)]
wIP

2 months agoWIP
Kai Moritz [Sat, 3 Feb 2024 20:05:35 +0000 (21:05 +0100)]
WIP

2 months agoWIP
Kai Moritz [Sat, 3 Feb 2024 11:12:22 +0000 (12:12 +0100)]
WIP

2 months agoWIP
Kai Moritz [Sat, 3 Feb 2024 08:02:29 +0000 (09:02 +0100)]
WIP

2 months agoWIP
Kai Moritz [Sat, 3 Feb 2024 07:29:06 +0000 (08:29 +0100)]
WIP

2 months agofix: Disabled mongodb-autoconfig, if mongodb is not used
Kai Moritz [Fri, 23 Feb 2024 10:39:26 +0000 (11:39 +0100)]
fix: Disabled mongodb-autoconfig, if mongodb is not used

2 months agofeat: Switched to `spring-boot-starter-data-mongodb-reactive`
Kai Moritz [Sat, 3 Feb 2024 23:24:07 +0000 (00:24 +0100)]
feat: Switched to `spring-boot-starter-data-mongodb-reactive`

2 months agofeat: Added logging for io.projectreactor to `MongoDbStorageStrategy`
Kai Moritz [Tue, 20 Feb 2024 16:35:14 +0000 (17:35 +0100)]
feat: Added logging for io.projectreactor to `MongoDbStorageStrategy`

2 months agorefactor: DRY for logging-category from io.projectreactor
Kai Moritz [Tue, 20 Feb 2024 16:32:54 +0000 (17:32 +0100)]
refactor: DRY for logging-category from io.projectreactor

2 months agofeat: Added counting of restored instances
Kai Moritz [Tue, 20 Feb 2024 15:14:22 +0000 (16:14 +0100)]
feat: Added counting of restored instances

2 months agotest: `StorageStrategy`-IT are restoring instead of recreating
Kai Moritz [Thu, 22 Feb 2024 15:46:27 +0000 (16:46 +0100)]
test: `StorageStrategy`-IT are restoring instead of recreating

2 months agofix: GREEN - Fixed the restore-mechanism
Kai Moritz [Thu, 22 Feb 2024 15:03:08 +0000 (16:03 +0100)]
fix: GREEN - Fixed the restore-mechanism

* The code of a reactive flow _must not_ call blocking functions.
* In order to solve this, the restore-process is triggered explicitly
  after the creation of the classes.

2 months agorefactor: RED - Refined success/error-handling for restore-operations
Kai Moritz [Tue, 20 Feb 2024 15:12:00 +0000 (16:12 +0100)]
refactor: RED - Refined success/error-handling for restore-operations

* This innocent little change discloses a severe missconception in the
  implementation of the storage strategies.
* The call to `Mono.block()`, though not really changing the behaviour
  during the restore-process, triggers a sanity-check from
  io.projectractor.

2 months agotest: Simplified the integration-tests for `StorageStrategy`
Kai Moritz [Thu, 22 Feb 2024 14:35:39 +0000 (15:35 +0100)]
test: Simplified the integration-tests for `StorageStrategy`

2 months agotest: Simplified `InMemoryWithMongoDbStorageIT`
Kai Moritz [Thu, 22 Feb 2024 13:49:46 +0000 (14:49 +0100)]
test: Simplified `InMemoryWithMongoDbStorageIT`

* Switched to `@ServiceConnection` instead of hand-coded initializer.

2 months agotest: Added IT for `ChatRoomRepository` and `MessageRepository`
Kai Moritz [Wed, 21 Feb 2024 17:30:01 +0000 (18:30 +0100)]
test: Added IT for `ChatRoomRepository` and `MessageRepository`

2 months agotest: Simplified & Unified the Kafka-tests
Kai Moritz [Thu, 22 Feb 2024 13:39:30 +0000 (14:39 +0100)]
test: Simplified & Unified the Kafka-tests

2 months agotest: Simplified `ChatHomeServiceTest`
Kai Moritz [Thu, 22 Feb 2024 13:24:28 +0000 (14:24 +0100)]
test: Simplified `ChatHomeServiceTest`

* Removed the unnecessary explicit instantiation of the `ObjectMapper`.

2 months agotest: Simplified & unified test-setup for ``ChatHomeServiceTest``s
Kai Moritz [Thu, 22 Feb 2024 12:10:09 +0000 (13:10 +0100)]
test: Simplified & unified test-setup for ``ChatHomeServiceTest``s

* Using existing configuration-classes instead of hand-coded configuartion.
* Aligned the `KafkaChatHomeServiceTest` with the `in-memory`-tests.

2 months agorefactor: Cleaned up code
Kai Moritz [Tue, 20 Feb 2024 12:05:21 +0000 (13:05 +0100)]
refactor: Cleaned up code

2 months agofeat: Introduced counting of stored instances in `StorageStrategy`
Kai Moritz [Tue, 20 Feb 2024 12:02:09 +0000 (13:02 +0100)]
feat: Introduced counting of stored instances in `StorageStrategy`

2 months agofeat: Introduced config-parameters for the `io.projectreactor`-logging
Kai Moritz [Tue, 20 Feb 2024 11:48:40 +0000 (12:48 +0100)]
feat: Introduced config-parameters for the `io.projectreactor`-logging

2 months agorefactor: Set logging-level for io.projectreactor to `DEBUG`
Kai Moritz [Tue, 20 Feb 2024 10:43:00 +0000 (11:43 +0100)]
refactor: Set logging-level for io.projectreactor to `DEBUG`

2 months agorefactor: Refined stream-definition in `StorageStrategy#write`
Kai Moritz [Tue, 20 Feb 2024 10:28:22 +0000 (11:28 +0100)]
refactor: Refined stream-definition in `StorageStrategy#write`

* Changed the stream-definition to a more natural order.
* As a result, the stored `ChatRoomInfo`-instances do not have to be
  handed clumsily to the following stream.

2 months agorefactor: Refined return-type of `StorageStrategy#write`
Kai Moritz [Tue, 20 Feb 2024 10:14:43 +0000 (11:14 +0100)]
refactor: Refined return-type of `StorageStrategy#write`

2 months agorefactor: One stream -> using `flatMap` instead of an inner `subscribe`
Kai Moritz [Tue, 20 Feb 2024 07:50:56 +0000 (08:50 +0100)]
refactor: One stream -> using `flatMap` instead of an inner `subscribe`

2 months agorefactor: Moved succes/error-logging to outer stream-definitions
Kai Moritz [Tue, 20 Feb 2024 07:23:13 +0000 (08:23 +0100)]
refactor: Moved succes/error-logging to outer stream-definitions

2 months agorefactor: Simplified `StorageStrategy`
Kai Moritz [Tue, 20 Feb 2024 06:47:22 +0000 (07:47 +0100)]
refactor: Simplified `StorageStrategy`

* Reconfigurable success/error-logging was introduced for
  `NoStorageStorageStrategy`.
* But as it turns out, this strategy can simply apply its logging in the
  overwritten method, that disables the whole storing-logic.
* Hence, the interface was greatly simplified again, by removing this
  ununsed mechanism.

2 months agorefactor: Moved extracted the `subscribe()`-call from `StorageStrategy`
Kai Moritz [Mon, 19 Feb 2024 14:01:58 +0000 (15:01 +0100)]
refactor: Moved extracted the `subscribe()`-call from `StorageStrategy`

* The subscription does no more happen inside the implementations of the
  interface `StorageStrategy`.
* Instead, the methods, that are defined in `StorageStrategy` return the
  created `Flux`.
* The call to `subscribe()` happens in the code, that uses this methods.
* This faciliates feature refinements concerning the asynchronous handling
  of success- and error-cases and so forth.

2 months agorefactor: Extracted subscription into a separate method-call
Kai Moritz [Sat, 3 Feb 2024 23:37:23 +0000 (00:37 +0100)]
refactor: Extracted subscription into a separate method-call

* This is in preparation to a planed refinement.
* The refinment will move the `subscribe()`-call out of the implementation
  of the `StorageStrategy`.

2 months agotest: Added missing JSR-310 module to not Spring-Boot-based tests
Kai Moritz [Mon, 19 Feb 2024 13:50:34 +0000 (14:50 +0100)]
test: Added missing JSR-310 module to not Spring-Boot-based tests

2 months agofeat: Added log message for disabled storage
Kai Moritz [Mon, 19 Feb 2024 13:31:21 +0000 (14:31 +0100)]
feat: Added log message for disabled storage

2 months agofix: Without `@DirtiesContext` the app is not teared down correctly
Kai Moritz [Sun, 18 Feb 2024 19:12:19 +0000 (20:12 +0100)]
fix: Without `@DirtiesContext` the app is not teared down correctly

* `StorageStrategy` depends on Spring to call the method
  `ChatBackendApplication#onExit()`, that is annotated with `@PreDestroy`.
* If this method is not called, the strategy is not applied, which leads
  to errors in the integration-tests.
* This happens, if all tests are run through Maven, because Spring
  recycles the application-context and only tears down the context after
  all tests have run.
* The addition of `@DirtiesContext` on those tests forces Spring to tear
  down the application completely after each so annotated test.
* Hence, the method is called as expected, which fixes the described
  errors.

2 months agotest: Introduced integration-tests for `NoStorageStorageStrategy`
Kai Moritz [Mon, 19 Feb 2024 10:05:17 +0000 (11:05 +0100)]
test: Introduced integration-tests for `NoStorageStorageStrategy`
--
ALIGN

2 months agotest: Introduced integration-tests for `NoStorageStorageStrategy`
Kai Moritz [Mon, 19 Feb 2024 10:01:13 +0000 (11:01 +0100)]
test: Introduced integration-tests for `NoStorageStorageStrategy`
--
COPY

2 months agorefactor: Extracted annonymous class into `NoStorageStorageStrategy`
Kai Moritz [Mon, 19 Feb 2024 12:07:35 +0000 (13:07 +0100)]
refactor: Extracted annonymous class into `NoStorageStorageStrategy`
--
ALIGN

2 months agorefactor: Extracted annonymous class into `NoStorageStorageStrategy`
Kai Moritz [Mon, 19 Feb 2024 12:05:51 +0000 (13:05 +0100)]
refactor: Extracted annonymous class into `NoStorageStorageStrategy`
--
COPY

2 months agorefactor: Added success- and failure-callbacks for `ChatHomeService`
Kai Moritz [Sun, 18 Feb 2024 20:46:56 +0000 (21:46 +0100)]
refactor: Added success- and failure-callbacks for `ChatHomeService`

2 months agorefactor: Added success- and failure-callbacks to `StorageStrategy`
Kai Moritz [Sat, 3 Feb 2024 21:44:02 +0000 (22:44 +0100)]
refactor: Added success- and failure-callbacks to `StorageStrategy`

2 months agorefactor: `KafkaServicesApplicationRunner` aufgeräumt
Kai Moritz [Sat, 3 Feb 2024 19:43:29 +0000 (20:43 +0100)]
refactor: `KafkaServicesApplicationRunner` aufgeräumt

2 months agofix: Config-ITs do not fail, if run multiple times
Kai Moritz [Sat, 3 Feb 2024 11:08:15 +0000 (12:08 +0100)]
fix: Config-ITs do not fail, if run multiple times

2 months agofix: Removed manual version-management for testcontainers.org
Kai Moritz [Sat, 3 Feb 2024 07:35:38 +0000 (08:35 +0100)]
fix: Removed manual version-management for testcontainers.org

2 months agofeat: Upgraded Spring Boot 3.1.3 -> 3.2.2
Kai Moritz [Sat, 3 Feb 2024 07:32:40 +0000 (08:32 +0100)]
feat: Upgraded Spring Boot 3.1.3 -> 3.2.2

2 months agofix: `getChatRoomInfo()` thrwos `LoadInProgressException` when loading
Kai Moritz [Sat, 3 Feb 2024 16:25:45 +0000 (17:25 +0100)]
fix: `getChatRoomInfo()` thrwos `LoadInProgressException` when loading

- The method `InfoChannel.getChatRoomInfo(UUID)` has to check, if loading
  is in process.
- Otherwise, an existing chat-room might erronously not be found, if it is
  requested, while `InfoChannel` is loading, because it is not yet loaded.

2 months agorefactor: compute `loadInProgress` on offset-change
Kai Moritz [Sat, 3 Feb 2024 14:32:25 +0000 (15:32 +0100)]
refactor: compute `loadInProgress` on offset-change

2 months agorefactor: separated message- and record- (aka offset-) handling
Kai Moritz [Sat, 3 Feb 2024 14:20:25 +0000 (15:20 +0100)]
refactor: separated message- and record- (aka offset-) handling

2 months agorefactor: Renamed method in `InfoChannel` according to conventions
Kai Moritz [Sat, 3 Feb 2024 14:13:09 +0000 (15:13 +0100)]
refactor: Renamed method in `InfoChannel` according to conventions

2 months agorefactor: Made only locally used method in `KafkaChatHomeService` private
Kai Moritz [Sat, 3 Feb 2024 14:11:50 +0000 (15:11 +0100)]
refactor: Made only locally used method in `KafkaChatHomeService` private

2 months agofeat: The position of the last seen messages is stored on a revoke
Kai Moritz [Sun, 24 Sep 2023 19:42:22 +0000 (21:42 +0200)]
feat: The position of the last seen messages is stored on a revoke

2 months agofix: The actual position has to be requested from the consumer
Kai Moritz [Sun, 24 Sep 2023 19:39:01 +0000 (21:39 +0200)]
fix: The actual position has to be requested from the consumer

* If the last seen offset and the current offset differ, although the
  partition did not contain any messages between this offsets, the loading
  process got stuck, because the position never advanced.
* Therefore, the actual position, that is compared against the read
  end-offset, has to be requested from the consumer.

2 months agorefactor: Added logging of loaded messages
Kai Moritz [Sun, 24 Sep 2023 19:38:46 +0000 (21:38 +0200)]
refactor: Added logging of loaded messages

2 months agorefactor: Refined success/error-handling when publishing shard-ownership
Kai Moritz [Tue, 20 Feb 2024 09:26:25 +0000 (10:26 +0100)]
refactor: Refined success/error-handling when publishing shard-ownership

2 months agofeat: Implemented `HaproxyShardingPublisherStrategy`
Kai Moritz [Sun, 17 Sep 2023 09:31:22 +0000 (11:31 +0200)]
feat: Implemented `HaproxyShardingPublisherStrategy`

* Implemented a first simple `ShardingPublisherStrategy`, that uses the
 https://www.haproxy.com/documentation/haproxy-runtime-api/[HAProxy Runntime API]
 to publish changed ownerships.
* Added configuration-properties `kafka.haproxyRuntimeApi` and
  `kafka.haproxyMap` to configure the strategy.

2 months agofeat: Introduced interface `ShardingPublisherStrategy`
Kai Moritz [Tue, 20 Feb 2024 08:53:10 +0000 (09:53 +0100)]
feat: Introduced interface `ShardingPublisherStrategy`

* The interface is used by `DataChannel` to publish the changed ownership
  each time, a new partition is assigned to the consumer-group.
* Added a dummy-implementation in `KafkaServicesConfiguration`.

2 months agofeat: Introduced a configurable instance-id
Kai Moritz [Tue, 20 Feb 2024 08:48:03 +0000 (09:48 +0100)]
feat: Introduced a configurable instance-id

2 months agofix: `ConsumerTaskRunner` waits until the data-loading is finished
Kai Moritz [Fri, 22 Sep 2023 16:20:31 +0000 (18:20 +0200)]
fix: `ConsumerTaskRunner` waits until the data-loading is finished

2 months agofeat: Introduced events that are send, if a shard is assigned/revoked
Kai Moritz [Sat, 16 Sep 2023 19:40:45 +0000 (21:40 +0200)]
feat: Introduced events that are send, if a shard is assigned/revoked

* In order to redirect requests to the appropriate instances, each
  instance must know the mapping from  shard-IDs to instance-URIs.
** For the static `in-memory`-implementation `ShardedChatHomeService`,
   this is a static mapping, that can be configured and read on start-up.
** For the dynamic `KafkaChatHomeService`, this mapping has to be
   propageted to all instances each time the partition-assignment of the
   consumer-group changes.
* Changes for `ShardedChatHomeService`
** Introduced `ChatBackendProperties.shardOwners` (of type `URI[]`).
** Each instance reads the static mapping from shard-ID to instance-URI
   on start-up.
* Changes for `KafkaChatHomeService`
** Introduced `KafkaServicesProperties.instanceUri` (of type `URI`).
** Each instance reads its URI on start-up.
** `DataChannel` sends an event for each assigned/removed partition,
   when the partition-assignment of the consumer-group changes.
** These events propagete the changed mapping from partition-ID to
   instance-URI.
** `InfoChannel` receives these events and updates the dynamic mapping
   from partition-ID (aka:shard-ID) to instance-URI.
* The shards, that are owned by an instance, can be queried via `/shards`.

2 months agofix: `metadata` is never `null`
Kai Moritz [Sun, 28 Jan 2024 17:50:33 +0000 (18:50 +0100)]
fix: `metadata` is never `null`

2 months agorefactor: DRY for computation of new `ChatRoomData` instances
Kai Moritz [Fri, 15 Sep 2023 19:35:17 +0000 (21:35 +0200)]
refactor: DRY for computation of new `ChatRoomData` instances

2 months agofix: GREEN - `DataChannel` creates entries for existent chat-rooms
Kai Moritz [Fri, 15 Sep 2023 18:56:14 +0000 (20:56 +0200)]
fix: GREEN - `DataChannel` creates entries for existent chat-rooms

2 months agotest: RED - Putting a message in a newly created chat-room
Kai Moritz [Fri, 15 Sep 2023 18:04:01 +0000 (20:04 +0200)]
test: RED - Putting a message in a newly created chat-room

2 months agorefactor: Splitted test in `AbstractConfigurationIT` into smaler pieces
Kai Moritz [Fri, 15 Sep 2023 15:29:05 +0000 (17:29 +0200)]
refactor: Splitted test in `AbstractConfigurationIT` into smaler pieces