#26910 Optimize TL-Sync for larger loads

enhancement

critical

#26910

Optimize TL-Sync for larger loads

major

#26365

Enable explicit option ordering in selector.template.xml

#26936

WrapperGenerator: lookup methods for enums/classifiers, better typing for references

#26986

Do not use theme variables directly but only via CSS variables

#26999

Typed configuration: short-cut format for list-value properties

#27001

ProgressDialog: Better reaction possibility to errors in the background process

#27053

TLScript for resolving a model part

#27057

BulkIdLoad should also load flex attributes

#27060

New type: tl.util:JSON

#27067

Better tooltips for model elements, icons for TLClassifier

#27068

ResKey with fixed number of values

#27109

Show the process cockpit by default in new app

#27118

Install TopLogic via Debian packages

#27135

Delimit layout editor functions from specialist functions

#27159

Model editor: translation of all module names plus display of technical name

#27186

Context-local environment variables: Assigning application properties via JNDI

minor

#26784

Migrate TLDoclet to jdk.javadoc.doclet

#26877

Better context information for deleted objects

#26952

Customizability of breadcrumb display

#26982

Control width of selection and detail column via theme variable

#26983

Adjust width of filter dialogs via theme variable(s)

#26985

Introduce styleable DIV container around demo license display

#27000

Rebuild form on invalidation

#27015

Logging in KBDataProducerTask should cover error cases better

#27055

Service editor commands should not always be active

#27063

Java class cache for TLObject implementation

#27075

Navigation of calculated config properties via collections

#27095

Bulk processing of object deletions during layout update

#27106

Delete command should (can) close open dialog

#27108

Risk analysis: Move coordinate origin down/left

#27111

Get log output even without log viewer

#27116

Support for MariaDB in combination with JNDI configuration

#27133

Adjust width of minimized sidebar via theme variable

#27151

ApplicationAction, which calls other action scripts

#27158

Shortening of button labels in the change check dialog

#27169

Declarative forms: Set property editors via configuration instead of annotation

#27173

Automatic language determination during wrapper generation

#27175

Auto-update of Debian deployments

#27180

Default label for components in the Inspector

defect

major

#27123

Meaningless selection at "Configure Tabs

#27137

In-app configured tables not exportable

#27138

Error when dragging the selected row of a non-initial slice of a table

#27193

Configuration export via context menu in model editor defective

minor

#26663

TreeGridComponent: Selection is not reset after discarding a transient row

#26807

Possible NPE in MonitorEventAccessor

#26962

TL-Sync: Tabs are lost with string attributes

#26984

Not all tab bars take into account the theme variable setting 'LEVEL_ONE_BAR_SIZE'.

#26992

Asynchronous loading of layouts for themes crashes short-running test cases

#27018

SchedulerGui: Detail view "Selected run" always empty

#27019

Inconsistent logging: scheduler uses its own log file, but its tasks do not

#27027

Declarative forms: SelectField disappears after upload if option list depends on mandatory property

#27028

Data migration: Delay migration processors

#27029

Failure to use PropertyInitializer for list-valued properties.

#27035

Missing constraint evaluation for list-valued properties

#27042

MaintenanceJspBase should write to the log first, then to the client, instead of the other way around.

#27047

Avoiding data migration in #25732: Saved search queries may no longer be readable

#27049

Transaction monitor renders collapse buttons twice

#27065

Missing CDATA quoting when serializing ConfigurationItem

#27069

Minimize an EditComponent discards changes

#27070

Duplicate error icons for TL script fields

#27073

LDAPAccessService: connection establishment using the host name instead of the IP address

#27078

Prevent unnecessary, incorrect conversion of simple attribute values

#27085

Create in tree grid on misconfiguration leads to NullpointerException

#27087

Map-valued properties in configurations of form editor plugins do not work

#27088

Multiline text in layout template parameters is reformatted

#27090

TreeTableComponent shall initially select the first visible row, if initially one row shall be selected

#27094

Reconfiguration of a component in the tile environment removes toolbar commands

#27098

Default-for definition in in-app tile layouts does not work

#27107

Tab "Process Cockpit" is available twice for selection at "Configure Tabs

#27110

Launch new app without test code

#27113

In the Process Cockpit, the number of tasks is not displayed directly after login

#27115

Error in deployed app without autoconf directory

#27119

Mail dispatch: Startup notification leads to NullpointerException

#27121

In-app reconfiguration and re-export of a modular layout crashes the application

#27122

Reconfiguring and exporting an inherited (modular) layout is not applied

#27129

Declarative forms: Values of @MapBinding properties are not taken over

#27130

TLScript fields not visible in BPE

#27132

Chart.js JavaScript is included in the application CSS

#27140

Personalization of forms for local types crashes the application

#27142

Missing error with wrong command clique

#27146

Memory leak due to multiple registration of selection listeners in the GridComponent

#27147

Update Dependency Check to version 7.4.4 or above

#27148

Scriptrecorder: Input of date values via date popup control is not recorded

#27153

Error when running tests in a build reactor via Maven

#27164

Missing sender address in mails

#27165

Platform-Specific Folder Separator for Mail Folders

#27170

ClassCastException on representation of a tl.core:DateTime attribute in a composition table

#27172

LayoutEditor: StackOverflowError with wrong model declaration

#27176

Unfolding in the model element tree causes selection of another unfolded element

#27178

Infinite tree in table selection dialogs leads to errors

#27179

Service configurations cannot be changed in a deployed system

#27182

Memory overflow during execution of scripted tests

#27191

Time selection logs errors

#27197

Missing component model for grid object initialization with multiple input channels

nice-to-have

#25295

Refresh empty search returns NPE

#27128

Double pop-up in view "External selection in the tree

enhancement

critical

#26910

Optimize TL-Sync for larger loads

Performance TLSync

When using TL-Sync with large amounts of data, we've noticed that it can be very slow at times and could likely be significantly optimized.

By “large amounts of data,” we mean changesets with about one million changes.

Improvement

The sending process has been optimized. Previously, sending a changeset with 1,800,000 changes took 6 hours. After the optimization, it now takes only 2.5 minutes. That is perfectly sufficient. Specifically, the following optimizations were implemented:

A quadratic complexity issue in filtering out changes that should not be sent was resolved.
Ticket #27057: A database query is no longer made for each object.
Ticket #27063: ClassLoader.loadClass(String) is no longer called for each object.
The tl:KBDataProducerTask now logs whether progress was made in the event of errors.

Test

No automated test. It can be tested manually in the Kafka demo with some effort:

Reconfigure the Kafka demo to use MySQL.
- H2 appears to cause extreme performance issues starting at around one million objects.
Start the Kafka Demo.
Disable the tl:KBDataProducerTask in the SchedulerGUI.
Use the "Create Many Objects" button to generate a large changeset.
Restart the application to clear all tl:KnowledgeBase caches.
Reactivate the task above.
Measure how long the task takes to transfer the objects.