#26626 Systematic logging for Kafka

Enhancement

Major

#26626

Systematic logging for Kafka

#26864

Force password change when administrator assigns password

#26893

Optional dependencies between services (ManagedClass)

#26905

Extinguishing callback at the subject object

#26920

Suggested values for number and string attributes

#26959

Drag&Drop: Generalization of the DropByExpression API

Detail

#26549

Form tables: Customization standard columns

#26755

Update CKEditor to version 4.19.1

#26858

Reduce calls to ModelBuilder.supportsModel()

#26871

Simplified mail configuration for IMAPS and SMTPS

#26872

Selectable ID column on type

#26887

Specify database passwords encrypted

#26889

Encrypt Pepper in Argon Hashing

#26913

Improve performance when deleting multiple objects

#26917

Default selection for value suggestions: Options via printout

#26918

Prioritize implementations for in-app annotations

#26938

Allow configuring control tag and CSS classes for all configurable control renderers

#26946

Show resource path of the application in the environment variables

#26953

Icon library: Bootstrap Icons

Nice to have

#26914

Alias and System Properties for ZooKeeper Port and Kafka Port

Bugfix

Detail

#23801

Label calculation in Excelexport does not consider table configuration

#24788

Missing deployment of top-logic.js and CSS for production mode

#25322

Form field for mandatory DateTime attribute incorrectly marked as changed

#25612

Model Editor displays link ends in the attribute table when references are changed

#25613

BPE: Creation of new process instances with mandatory fields not possible

#25652

In-app modeling: backreferences of compositions are created as multiple=true

#25732

Model-based search: Input validation: Filter query leads to NullpointerException

#25858

Model editor reloads page on script execution

#26396

Drag and Drop: Icon from table header to drop zone: NPE

#26406

WYSIWYG YouTube plugin has problems with embedding

#26441

No session invalidation during login and password change process

#26447

Reference checks: Are only checked in the form, not when committing

#26461

Overridden attribute inherits mandatory "Default value" customization

#26484

In-app template for grid and tables: Function "Verifier for use as list item" does not get component model

#26529

Model transfer to development environment: ResKeys remain in dynamic properties

#26554

Calculated web folder references delete folder when deleting the base object

#26569

Missing constraint violations when deleting objects referenced by mandatory fields

#26583

Encoding problems in system overload hint page

#26584

Wiki formatting in multiline text attributes / text fields broken

#26587

Option providers do not update calculated suggested values

#26691

Webfolder: Erroneous behavior of multiple upload file selection

#26730

Column definitions are not applied to composition tables

#26750

Prohibit drop from non-TL object to tables and trees

#26836

StackOverflow due to configuration error: Calculated string attribute returns integer

#26852

ApplicationTesting: No two sessions possible for the same user in JUnit test

#26853

EditComponent loses edit mode after deletion

#26856

Lack of documentation adaptation to new + context

#26857

Inconsistent "source" properties in documentation pages

#26859

Correct various Eclipse settings

#26865

Missing selection marker in composition tables

#26866

Instance Browser: New transient object does not show type

#26867

Configuration uses unencrypted SMTP and IMAP password

#26868

End animation when table is cleared from GUI

#26870

Inactive scheduler crashes the application

#26873

Application monitor "Environment variables" may display safety-critical values

#26874

Service Editor displays "secret" of the OpenID service

#26880

Selection of icons in the icon chooser is not scriptable

#26883

Drag preview image in script recorder does not use label

#26884

Declarative forms: Missing GUI update for programmatic changes of a list-valued property

#26885

Constraints on declarative forms with arguments from a container reference lead to errors for new elements

#26890

Overridden properties are not initialized correctly when booting from model definition

#26891

Jerky table contents in FrozenTables

#26894

Missing dependency ModelBasedSearch -> SearchBuilder

#26895

Missing dependency MailServer -> MailReceiverService

#26896

DnD: Drop operation with drop type "child" on table gets the wrong row object

#26903

Under unfavorable circumstances data from future revisions visible in AssociationCache's

#26904

CSS editor does not work with Top-Logic theme variables

#26906

Default annotations cannot be inherited from the attribute type

#26911

OutOfMemoryError on App-WAR generation

#26922

With generated subject classes, a default provider of an attribute in a non-structure class does not get a create context

#26923

ModelNamingSchemes do not get "value context" in some cases

#26924

Horizontally displayed radio button selection extends beyond form boundaries

#26925

BreadCrumbs do not update when inner tabbars are not visible

#26926

Rare ArrayIndexOutOfBoundsException on tab change

#26928

Hidden component does not appear anymore, although it gets a supported model

#26929

Scripting recorder: No templates are loaded in project demo

#26934

Update yFiles to version 2.4.0.6

#26939

Changed order of standard columns in new table

#26942

Frozen table: drag selection contains duplicates

#26945

Inconsistent lock timeout at application startup

#26957

No update of node properties after model change

#26960

Schedule: Automatic collision avoidance does not work

#26965

Theme icon with value "none" is not resolved correctly

#26972

Scripting recorder: stops execution after reloading the page

#26988

In-app documentation generator does not extract documentation for overwritten config properties

#26996

ChartJS Update to 3.9.1

#26997

JSoup Update to 1.15.3

#26998

Update Batik Graphics to 1.15

Enhancement

Major

#26626

Systematic logging for Kafka

Code migration Logging TLSync

Challenge

DiFa's systems within a network architecture make heavy use of Kafka interfaces, be it with TL-Sync or self-implemented protocols (e.g. JSON).

For HTTP servers, it is common that all communication is logged on the server side, e.g. request-logs that log the meta information of any request. These logs make it possible to trace the communication across systems and different network components. In the event of an error, it is also possible to see at which point communication is interrupted or which components refuse further processing.

Communication via Kafka, on the other hand, is not traceable in the current situation, for several reasons:

Producers do not log the sending of messages adequately
Recipients do not log the receipt of messages
Recipients do not sufficiently log the processing of messages
Log files are flooded with message content
Log information is retained for only about 20 minutes, after which log entries are overwritten due to sheer volume, file size and rolling file numbers are insufficient
Message contents are encrypted and therefore not detectable and inspectable by Daimler tools like Lenses.

Solution

Our Kafka implementation generates a communication protocol. The sending and receiving of each message is logged before further functional processing and independent of the processing result (exception save/finally). The logs contain the necessary meta-information to trace the path of a message from the producer to the consumer.

The log format is standardized and largely symmetrical between the send log and the receive log for easier analysis.

The protocol recognizes two log levels:

INFO: meta information about a message per line
DEBUG: Metainformation + content of a message, if necessary several lines

There is a send log for all producers and a receive log for all consumers (unclear if this is enough or if each producer and comsumer creates an own log (an own file)).

The meta information must include (obda):

Timestamp of sending / receiving
Topic
Message ID
Chunk information (if available)
Offset (if possible)
Application server (IP, hostname, cluster node ID, or similar)
Kafka server (IP, hostname or similar)
name/ID/class of the producer or consumer
tbd...

The log settings are configured sufficiently to keep track of the communication in the INFO level for at least 1 week.

The solution approach is not limited to EPP, but applies equally to all ASCon systems in the DiFa landscape.

Implementation

I would just ask that we look at the implementation together before finalizing, especially with the log information.

Code Migration

Kafka sender and receiver have a new mandatory configuration option: CommonClientConfig.getLogWriter().
- The tl:KafkaLogWriter is used to log meta data and content from Kafka messages in a usable form.
- As templates for own implementations the tl:TLSyncRecordLogWriter and the tl:KafkaStringLogWriter can be used.
- To find all occurrences where a tl:KafkaLogWriter must be entered, search for KafkaProducerService and KafkaConsumerService in all .config.xml files.
- As an example configuration, the file com.top_logic.kafka/src/main/webapp/WEB-INF/conf/kafkaConf.config.xml can be used, in which both tl:KafkaProducerService and tl:KafkaConsumerService occur.
The constructor of the tl:ProducerProxy class now takes a tl:KafkaLogWriter as an argument.
The classes tl:CommonClientConfig and tl:KafkaCommonClient now have as additional type parameter the type of the value that is sent or received.

Test

Start the Kafka demo once with log level DEBUG and once with INFO. Create at least one object each time. Then check if the expected amount of data is in the log for the Kafka messages.

INFO: Only the meta data.
DEBUG: Additionally also the whole content.

The data must be logged by both the sender and the receiver.