Yet another blog post of the kafEEne series – this time, its Java EE Concurrency Utilities in action along with Kafka. As usual, the code is on Github and you can refer to the README on how to get this up and running using Docker
Setup
As mentioned, the application is Docker based
- It uses Zookeeper, Kafka and TomEE (airhacks) Docker images
- For ease of development and demo, the entire application can be run as a single unit using Docker Compose
Kafka
We have the following setup for Kafka
- One topic with three partitions (configured to be auto-created)
- Automatic offset commit is disabled, and
- Links to Zookeeper (container) of course
Consumer
- The Consumer is the meat of the application
- It’s a
Runnable
whoserun
methodpoll
s Kafka cluster (with a configurable time out – 20 secs by default) - It also doubles as a
ManagedTask
(from the Java EE Concurrency Utilities) - Provides implementation for a
ManagedTaskListener
whose life cycle events are used to commit offsets to Kafka (automatic offset commit is disabled) - The Consumer is triggered using a
@Singleton
bean which schedules it within theManagedExecutorService
thread pool
Producer
- It’s a
Runnable
- Triggered using a
@Singleton
bean within theManagedScheduledExecutorService
thread pool - It waits for a configurable (3 seconds by default) before producing the next record
Checking the results
The results you’ll see might be similar to this. Here is the gist
The above snapshot shows data from three (consumer) poll loops
- Producer pushed one record
- Each loop fetched 1 record (that’s just co-incidence, not by design) and its completely independent of the producer
- After each loop (task), the offset commit process got triggered – in this example we have 3 partitions and the offset for each partition is printed to the console
- If you notice carefully, the committed offset (for a particular loop) will be in line with the data which was consumed (note: committed offset points to the next offset i.e. one more than the offset that offset which was last consumed)
Further reading
- kafEEne blog series
- Other Kafka blogs on simplydistributed
- Kafka based apps on Github
Cheers!