major
minor
major
minor
TL-Sync detects duplicate messages and automatically skips them. Missing messages are not currently detected and can lead to complex technical issues.
Improvement
TL-Sync should detect when a message is missing. When this happens, it should pause reception, log an error, and display this in the monitor view.
Implementation
TL-Sync transmits the revision number of the sending system in every message. However, it transmits only the relevant revisions. These numbers are therefore not consecutive. However, if either the number of the most recently sent revision or the first unsent revision is also transmitted, the recipient will have enough data to detect this situation.
Technical Background
TL-Sync uses Kafka. And that essentially guarantees that no messages are lost. Therefore, this feature would actually be unnecessary. Unfortunately, however, it happens time and again with customers that Kafka is configured or administered incorrectly. This leads to data loss and, consequently, to technical problems that can sometimes be very difficult to resolve—yet the customer simultaneously exerts significant pressure to fix them within the shortest possible time. Therefore, this feature is, unfortunately, still useful.
Recognizing This Situation
On the application’s Monitor page, there is a line for the TL-Sync receiver with the following message:
Failed.
Cause: Processor class com.top_logic.kafka.knowledge.service.importer.KBDataProcessor failed.
Cause: LOG MARK: 'in-tl-sync-context' = 'true'.
Cause: Detected that messages are missing when receiving changeset 12. The last processed changeset was 7. However, the new message indicates that the last processed changeset should have been 9.
Resolving this issue
The sending application must resend the missing data. It can also simply resend everything. TL-Sync will recognize and skip the revisions that have already been processed.
Before the sending application resends its data, all “too new” revisions must be removed from Kafka. Typically, all remaining data in Kafka can be deleted. Applications that use “chunking” must also delete their stored chunks.
Code Migration
Background
- The TL-Sync message format changes as a result of this transition.
- The updated receiver can also process old messages. However, old receivers cannot process the new messages and will log error messages.
- The sender can be configured to continue sending messages of the old type. However, if this is done, the detection of missing messages will not work; it will be disabled.
- If messages of the new type are sent, all receiving applications must also be updated to support them. Otherwise, reception will fail and error messages will be logged.
Required Migration
- Either all receiving applications must be updated to the new version at the same time,
- Or all sending, updated applications must be configured to send messages of the old type. See: ChangeSetSerializer.Config.getMessageVersion()
Test
Preparation
- Start the ZooKeeper, Kafka, and TL Kafka demo.
- Create an object to send a message.
- This initializes the transmission. As a result, the receiver stores which revision it last received.
Ensure that a changeset is lost
- Set a breakpoint in the consumer so that no more messages can be received. For example, at the beginning of: ConsumerDispatcher.poll(int, int)
- Create another object to send another message.
- Terminate the application without allowing the consumer to consume this message.
- Terminate Kafka and ZooKeeper.
- Delete the data from ZooKeeper and Kafka in the "tmp" folder.
- The location of this folder depends on the operating system.
- Restart ZooKeeper, Kafka, and the TL Kafka Demo.
Verify that the transmission is paused
- Create another object to send another message.
- This object must not be delivered. The monitor page must indicate to the TL-Sync recipient that there is a problem.
Restore transmission
- Pause the tl:KBDataProducerTask in the SchedulerGUI.
- Stop the TL Kafka Demo, Kafka, and ZooKeeper.
- Delete the data from ZooKeeper and Kafka in the "tmp" folder.
- Restart ZooKeeper, Kafka, and TL Kafka Demo.
- Delete the entries from the sender in the TL_PROPERTIES database table:
{{{#!sql delete from TL_PROPERTIES where "propKey" like 'TLSync.lastSentRevisionAtDate%' }}}
- Let the tl:KBDataProducerTask continue running in the SchedulerGUI.
Verify that the transmission is working again
- Verify that the two missing objects have arrived within 15 seconds at the latest.
- If the application was not terminated while the Kafka data was being deleted, it may take up to 11 minutes. This is because the exponential backoff continues to run, slowing down subsequent reception attempts to prevent the log from becoming overloaded.