Kafka divides communication into topics. Producer and Consumer send and receive from Topics. The topics are defined in their configuration. Before topics can be used, they usually have to be created. (Only on test systems this is sometimes done automatically.) Producers and consumers should check if their topics exist before using them. Unfortunately, Kafka behaves unfavorably here: depending on the situation, the logs are flooded with warnings without informing the application in time. Furthermore, the error message is of limited help.

Improvement

Kafka Producer and Consumer should check if their Topics exist. If they do not exist, an attempt is made to create them. If this does not succeed, the start of the application is aborted.

Application

The setting is automatically active. There are the following configuration options:

• create-topics="false": do not create topics if they do not exist.
• disabled="true": Do not check if the topics exist. Reactivates the old behavior. This is useful for developers, for example, to be able to launch applications without having to launch Kafka.
• wait-timeout="30s": When checking and creating topics, wait at most this long for the answer from the server. Default is 10 seconds.
  • For format see: com.top_logic.basic.config.format.MillisFormat

Example: {{#!xml <application xmlns:config="http://www.top-logic.com/ns/config/6.0"><configs> <config config:interface="com.top_logic.kafka.services.common.TopicChecker$Config" disabled="true" /> </configs> </application> }}}

Code Migration

• It should be clarified with the clientele if the application is allowed to create the topics itself if they do not exist.
• For development, it is recommended to disable the check in the personal configuration if the application should be able to start locally even without Kafka.

Test

No automatic test, because here different Kafka server settings are necessary, and the tests do not support this.

• The Kafka tests do not find any error.
• Start a Kafka application with Kafka: This must work.
• Launching a Kafka application without Kafka: This must fail.

• In the Kafka server configuration file, specify that topics are not created automatically.
• The Kafka tests do not find an error.
• Launch a Kafka application with Kafka: This must work.
• Start a Kafka application without Kafka: This must fail.

• Set the configuration TopicChecker.Config.create-topics to false.
• The Kafka tests fail.
• Launch a Kafka application with Kafka: This must fail.
• Launch a Kafka application without Kafka: This must fail.

• Set the TopicChecker.Config.disabled configuration to true.
• The Kafka tests fail.
• Launch a Kafka application with Kafka: This must work.
• Launch a Kafka application without Kafka: This must work.