This repo contains the source code for the "Building Interactive Query Service" presentation at Current 2022 in Austin, Texas on October 5th 2022.

About the project

The goal of this GitHub repo is to server as a basic reference implementation for building an interactive query service in Kafka Streams. For building your own IQ application you should be able to fork this project and follow the pattern here for adding your own controllers and web page for displaying the results. This project also uses Standby tasks for demonstrating high-availability for Kafka Streams as well as interactive queries.

Dependencies

The project uses the following dependencies

• For the web serving layer the application uses Spring Boot Note that we're only using Spring Boot for providing the dependency injection required to wire up the application - no specific spring-kafka or spring-kafka-streams functionality is leveraged like @EnableKafkaStreams
• The application is using JQuery for making REST API calls and rendering the results in a dynamic web page found in
• The integration tests use Testcontainers and Junit 5. The integration tests don't use any spring specific testing utilities

Running the application

To run the application you have two options

1. COMING SOON: Build an uber jar from the project by running ./gradlew shadowJar then run java -jar kafka-streams-iq-app.jar from the command line. Note to run more than one instance on the same host, you'll have to set a unique port for each instance using a -Dserver.port=NNNN setting for each one.
2. Run from IntelliJ. Again you'll have to set a unique port for running more than one instance using -Dserver.port=NNNN in the Spring-Boot section of the run/debug configuration settings
3. To push records through the Kafka Streams application run the TestDataProducer class again from either IntelliJ or a separate command from the command line java -jar jarFile TestDataProducer
4. The application.properties file contains the Kafka Streams properties. The configuration class KafkaStreamsAppConfiguration class ingests these properties. For connecting to Confluent Cloud there's the confluent.properties.orig you can also use this file for secure connections to any Kafka cluster. The KafkaStreamsAppConfiguration class is expecting a confluent.properties file to exist to use for secure connections. To make sure you don't check in any sensitive configs save the confluent.properties.orig file as confluent.properties which is set to ignore in the .gitignore file.
5. There is a configuration secure.configs in the application.properties file which is set to true by default - to disable loading the confluent.properties file (testing, or on-prem unsecure brokers) set this configuration to false

Testing

There is an integration test for both key and range query: InteractiveQueriesIntegrationTest For both tests, they start two Kafka Streams applications, produces some records then asserts both active hosts return results then shuts down one application and asserts that the standby returns query results. The integration tests serve to show an example of testing an IQ application and will also help validate any changes you may make in the application.

NOTE: that the test uses testcontainers and the Kafka image is for M1 macs so if you're on x86 you'll have to adjust the docker image pulled for the test.