Problem

A PR build in Jenkins Build_Git (clean install spotbugs:spotbugs, tests on H2) takes 98–108 min during the day and 77–79 min when running alone. Per-module timings (builds #16353, #16350):

The build is strictly sequential: the job is a Jenkins "Maven project" without -T; the module times add up to 94 min. The longest dependency chain in the reactor is only 58 min, 36 min of which is tl-demo plus the test-app-7-4-0 module built after it.

Of the 259 non-renovate PRs merged since 2026-04-01, 179 had tl-demo downstream of a changed module; 82 touched only the React UI modules, which tl-demo does not depend on.

Lösung

PR builds of all TopLogic projects run in shared Jenkins pipelines kept in the repository tl-ci (https://git.top-logic.com/TopLogic/tl-ci): job tl-ci-pr (pr/Jenkinsfile) for pull requests, job tl-ci-deploy (deploy/Jenkinsfile) for the release jobs that call Build_Git today. The engine provides the test-harness and build features the PR pipeline relies on; a project switches to tl-ci-pr once it uses an engine version with this change.

1. Sharded scripted tests

• The test harness (AbstractBasicTestAll, test.com.top_logic.basic.util.ShardSelection) takes the system property TestAll.scripted: all (default, all tests), none (all tests except the scripted ones), or <i>/<n> (only the scripted tests of shard i of n). none and all shards together run exactly the tests of all.
• Scripted tests are distributed in units: the scripts directly contained in one script directory, or all scripted tests of one test class. A unit is never split, because its scripts depend on their order and on shared application state. Each module assigns its units by decreasing size to the shard with the smallest load; on a tie, the shards are preferred starting at the module's index in the list TestAll.shardModules (the modules sharing the shards), so that the units of small modules spread over the shards. Every shard JVM computes the same assignment.
• The system property TestAll.scratchDir (default tmp, test.com.top_logic.basic.ScratchDirectory) sets the directory for temporary test files. With a non-default scratch directory, the application under test gets its storage path (tl_storage_path) and a writable web application overlay in front of its resource path inside the scratch directory, so that several test runs in the same module directory neither interfere nor modify the module's sources (generated style sheets and scripts, exported layouts and models). IDEResources resolves a relative missing-keys-file/all-keys-file against the top-level web application's module directory.
• TestModelReporting.script.xml no longer depends on objects created by scripts of other directories.
• Command of a shard: mvn surefire:test -pl <modules> -DskipTests=false -DTestAll.scripted=<i>/<n> -DTestAll.shardModules=<modules> -DTestAll.scratchDir=tmp/shard-<i> -Dsurefire.reportNameSuffix=shard-<i>.

2. Parallel reactor build

• -Dtl.javadoc.aggregate=false activates the profile javadoc-module-local (in tl-parent-build and tl-parent-engine): the TLDoclet of the framework modules writes into the module's build directory instead of the shared tree tl-doc/javadoc, and the JavaDoc index is skipped. The doclet still runs in every module (message resources, warnings). Without the switch, the build is unchanged (the job Create_JavaDoc relies on the shared tree).
• gwt.localWorkers (default 1) bounds the workers of each GWT compilation.
• The goals translate, check-resources, touch and normalize of the tl-maven-plugin are marked thread-safe.
• A full parallel reactor build (mvn -T 1C clean install -Dtl.javadoc.aggregate=false, no tests) takes 2:39 min on a developer machine instead of ~8 min sequentially.
• In the reactor, a module starts only after its dependencies have run their tests. The PR pipeline therefore builds without running tests and runs the tests afterwards, so that e.g. the 7 min of tl-core tests no longer delay all modules depending on tl-core.

3. PR pipeline (tl-ci: pr/Jenkinsfile, job tl-ci-pr)

• Stages: Checkout (fresh workspace, tl-ci plus the branch BRANCH of the repository REPO), Select (scripts/affected-modules.sh), Build (mvn -T <MAVEN_THREADS> clean install without running tests), Test (concurrently: the module tests with scripted tests excluded, spotbugs:spotbugs, and SHARDS shards over the affected modules with scripted tests), Check sources (fails if the build or the tests modified versioned sources), then test results, SpotBugs issues (without Git blame) and the log rules (build warnings → UNSTABLE, test JVM crashes → FAILED).
• scripts/affected-modules.sh determines from git diff against the merge base with TARGET_BRANCH (default master): MODE (full if the reactor's root POM or .mvn/ changed or a full build is forced; none if no reactor module changed; partial otherwise), the changed modules, the affected modules (changed modules and their dependents, including modules inheriting from a changed parent POM), the affected modules whose packaging runs tests, and those with scripted tests. A reactor in a subdirectory (BUILD_MODULE) and single-module projects are supported. Requires Maven >= 3.9.
• partial: (1) compile-only install of the changed modules and their dependencies, (2) clean install -pl <changed> -amd without running tests, (3) tests and spotbugs:spotbugs over the modules of (2). SpotBugs thereby covers only the affected modules.
• Replayed over the last 30 merged engine PRs: 17 needed 2–27 modules, 13 of them no scripted tests at all; one needed a full build.

4. Deploy pipeline (tl-ci: deploy/Jenkinsfile, job tl-ci-deploy)

• Takes the parameters of Build_Git (BUILD_NAME, REPO, BRANCH, BUILD_MODULE, GOAL, SKIP_TESTS, SKIP_JAVADOC, TAG, additionalOptions, ...), runs one sequential Maven invocation of GOAL, sets and pushes TAG after a successful build, and mails the result like Build_Git. No requirements on the engine version. Unlike Build_Git, a test failure fails the build instead of making it unstable.

5. Site configuration

• No repository contains site-specific settings. Everything specific to the build server (database and mail servers of the tests, credentials, Maven settings) is kept in the Jenkins credential tl-ci-env (kind "Secret file", a shell environment file). The pipelines source it before each Maven call without echoing it; TopLogic resolves its ${env:...} aliases from these environment variables, so no value appears on the command line, in the build parameters or in the build log. Maven options go into MAVEN_ARGS. Project-specific settings go into the parameter additionalOptions.
• The build time depends strongly on the load of the build node: a full engine build with the first version of the pipeline (still running the tests within the reactor build) took 67 min while five Build_Git builds ran on the same 8-CPU node, which themselves took 1 h 41 min – 2 h 04 min instead of 77–108 min; with the build without tests, 30 min.

6. Full-database test runs (one worker per database)

A PR build tests only with the default database (H2). A full test run (ONLY_DEFAULT_DB off, as in the nightly build) runs every multi-database test once per database (MySQL, MSSQL, H2, Oracle, Oracle 12, Oracle 19, PostgreSQL). All modules share one set of external schemas per database, so the tests of one database must never run concurrently, while different databases can.

• The test harness (AbstractBasicTestAll, test.com.top_logic.basic.util.DBSelection) takes the system property TestAll.db: all (default, unchanged behavior), none (all tests except those bound to a worker database), or the name of one database (only the tests bound to it: no module-independent, unbound or scripted tests). TestAll.dbWorkers lists the databases that run in workers of their own (default: all databases of the multi-database tests except the default database). A value other than all implies the full multi-database enumeration (tl_test_onlyDefaultDB=false). none and one run per worker database together run every test of a full run exactly once; the selection composes with TestAll.scripted.
• A test is bound to a database if it lies below a DatabaseTestSetup or KBSetup for it (interface DBBoundTest); a nested binding to another database is reported as an error. The databases of the multi-database tests are listed once (DatabaseTestSetup.MULTI_DB). The in-place pruning of the test tree (TestPruner) is shared with ShardSelection.
• A test suite that cannot be built (e.g. an invalid TestAll.scripted / TestAll.db value) now fails the test run instead of yielding an empty, successful one.
• tl-ci pr/Jenkinsfile: with ONLY_DEFAULT_DB off, the module tests and the shards run with TestAll.db=none, and the Test stage gets one branch per database of the parameter TEST_DBS (default mysql,mssql,oracle,oracle12,oracle19,postgresql): surefire:test of test.TestAll sequentially over the modules with tests, with its own scratch directory, test ports and report suffix.
• Verified with tl-ci-pr #3 (full build, all databases, branch CWS/CWS_29707_db_workers of tl-ci): 73 min instead of 3 h 44 min – 4 h 37 min of the nightly build TL7 (#789). In tl-core the runs selected 4194 (none) + 1207 (MySQL) + 1063 (MSSQL) + 1067 + 1066 + 1066 (Oracle) + 919 (PostgreSQL) = 10582 test cases, i.e. exactly the full suite. Load-sensitive tests found on the way: #29710, #29711.

Out of scope

• Running the per-module conformance tests once for the whole workspace: only the pure file-scan checks could be moved, and under -T their cost is mostly off the critical path.
• Slow individual tests: #29706.
• test.com.top_logic.kafka.demo keeps its application and Kafka data in the fixed directory tmp/ (storage path from the variable storage_path, Kafka/ZooKeeper data directories in the Kafka configuration), not in the scratch directory. With its single scripted unit, only one shard runs it.
• Splitting the scripted unit scripted.TestDemo (76 ordered scripts, ~15 min), which bounds the duration of the scripted-test shards from below.
• Moving the nightly build TL7 itself to the tl-ci pipelines: besides the tests it tags the build, builds the aggregated JavaDoc, deploys the snapshot and triggers the dependent builds and demo deployments; it also uses its own database schemas (tltrunktest*), while tl-ci-env holds those of the CWS builds (cwstest*).