kafEEne #2: Kafka & Concurrency Utilities

Yet another blog post of the kafEEne series – this time, its Java EE Concurrency Utilities in action along with Kafka. As usual, the code is on Github and you can refer to the README on how to get this up and running using Docker

Setup

As mentioned, the application is Docker based

• It uses Zookeeper, Kafka and TomEE (airhacks) Docker images
• For ease of development and demo, the entire application can be run as a single unit using Docker Compose

Kafka

We have the following setup for Kafka

• One topic with three partitions (configured to be auto-created)
• Automatic offset commit is disabled, and
• Links to Zookeeper (container) of course

Consumer

• The Consumer is the meat of the application
• It’s a Runnable whose run method polls Kafka cluster (with a configurable time out – 20 secs by default)
• It also doubles as a ManagedTask (from the Java EE Concurrency Utilities)
• Provides implementation for a ManagedTaskListener whose life cycle events are used to commit offsets to Kafka (automatic offset commit is disabled)
• The Consumer is triggered using a @Singleton bean which schedules it within the ManagedExecutorService thread pool

Producer

• It’s a Runnable
• Triggered using a @Singleton bean within the ManagedScheduledExecutorService thread pool
• It waits for a configurable (3 seconds by default) before producing the next record

Checking the results

The results you’ll see might be similar to this. Here is the gist

The above snapshot shows data from three (consumer) poll loops

• Producer pushed one record
• Each loop fetched 1 record (that’s just co-incidence, not by design) and its completely independent of the producer
• After each loop (task), the offset commit process got triggered – in this example we have 3 partitions and the offset for each partition is printed to the console
• If you notice carefully, the committed offset (for a particular loop) will be in line with the data which was consumed (note: committed offset points to the next offset i.e. one more than the offset that offset which was last consumed)

Further reading

• kafEEne blog series
• Other Kafka blogs on simplydistributed
• Kafka based apps on Github

Cheers!

Advertisements