Major
Detail
Major
Detail
Challenge
DiFa's systems within a network architecture make heavy use of Kafka interfaces, be it with TL-Sync or self-implemented protocols (e.g. JSON).
For HTTP servers, it is common that all communication is logged on the server side, e.g. request-logs that log the meta information of any request. These logs make it possible to trace the communication across systems and different network components. In the event of an error, it is also possible to see at which point communication is interrupted or which components refuse further processing.
Communication via Kafka, on the other hand, is not traceable in the current situation, for several reasons:
- Producers do not log the sending of messages adequately
- Recipients do not log the receipt of messages
- Recipients do not sufficiently log the processing of messages
- Log files are flooded with message content
- Log information is retained for only about 20 minutes, after which log entries are overwritten due to sheer volume, file size and rolling file numbers are insufficient
- Message contents are encrypted and therefore not detectable and inspectable by Daimler tools like Lenses.
Solution
Our Kafka implementation generates a communication protocol. The sending and receiving of each message is logged before further functional processing and independent of the processing result (exception save/finally). The logs contain the necessary meta-information to trace the path of a message from the producer to the consumer.
The log format is standardized and largely symmetrical between the send log and the receive log for easier analysis.
The protocol recognizes two log levels:
- INFO: meta information about a message per line
- DEBUG: Metainformation + content of a message, if necessary several lines
There is a send log for all producers and a receive log for all consumers (unclear if this is enough or if each producer and comsumer creates an own log (an own file)).
The meta information must include (obda):
- Timestamp of sending / receiving
- Topic
- Message ID
- Chunk information (if available)
- Offset (if possible)
- Application server (IP, hostname, cluster node ID, or similar)
- Kafka server (IP, hostname or similar)
- name/ID/class of the producer or consumer
- tbd...
The log settings are configured sufficiently to keep track of the communication in the INFO level for at least 1 week.
The solution approach is not limited to EPP, but applies equally to all ASCon systems in the DiFa landscape.
Implementation
I would just ask that we look at the implementation together before finalizing, especially with the log information.
Code Migration
- Kafka sender and receiver have a new mandatory configuration option: CommonClientConfig.getLogWriter().
- The tl:KafkaLogWriter is used to log meta data and content from Kafka messages in a usable form.
- As templates for own implementations the tl:TLSyncRecordLogWriter and the tl:KafkaStringLogWriter can be used.
- To find all occurrences where a tl:KafkaLogWriter must be entered, search for KafkaProducerService and KafkaConsumerService in all .config.xml files.
- As an example configuration, the file com.top_logic.kafka/src/main/webapp/WEB-INF/conf/kafkaConf.config.xml can be used, in which both tl:KafkaProducerService and tl:KafkaConsumerService occur.
- The constructor of the tl:ProducerProxy class now takes a tl:KafkaLogWriter as an argument.
- The classes tl:CommonClientConfig and tl:KafkaCommonClient now have as additional type parameter the type of the value that is sent or received.
Test
Start the Kafka demo once with log level DEBUG and once with INFO. Create at least one object each time. Then check if the expected amount of data is in the log for the Kafka messages.
- INFO: Only the meta data.
- DEBUG: Additionally also the whole content.
The data must be logged by both the sender and the receiver.