The XML import (com.top_logic.xio) resolves IDs only among the objects created in the current import (AbstractModelBinding keeps an in-memory map). Importing a second file that shares data with an earlier one therefore creates that data again. Typical case: exchange formats that carry shared master data with every document. Such formats also state settings like the language of their texts in the data itself, which the import definition cannot pick up yet.

This ticket consolidates #29794, #29795, #29796 and the original request of this ticket.

Use case

KNX ETS project files (E-Home, TL 8.0.0-alpha10) each contain the datapoint types of the KNX standard, the manufacturer list and the catalogs, hardware and application programs of the devices used. Every import of a project duplicated all of it (hundreds of manufacturers per import, the same application program once per project). These objects have stable IDs (M-0008_A-10D7-11-A0E5-O000A, DPST-9-1) stored in an attribute (extId). The texts of an element are in the language named by an attribute (<ApplicationProgram DefaultLanguage="de-DE">), translations are grouped by language (<Language Identifier="de-DE"><TranslationUnit><TranslationElement RefId="..."><Translation AttributeName="Text" Text="..."/>).

Requested

1. Resolve IDs against existing objects: a way to declare that IDs not created in this import are looked up among existing objects. Together with join-duplicates, existing objects are then reused and updated, missing ones created.
2. Skip a known subtree: a handler that imports an element only if the object its ID names does not exist yet, otherwise skips the element with all its content. For data that never changes under the same ID (an ETS application program, whose ID includes its version) this avoids re-importing thousands of nested objects that have no IDs of their own and could not be joined.
3. Optional object references: an object-ref (and reference) that ignores an ID that resolves to nothing, instead of reporting ERROR_UNRESOLVED_ID when the import context is closed. Needed e.g. for translation sections that also list objects the import does not model.
4. Bind an import variable to an XML attribute (#29796): make the value of an attribute of the current element available to the handlers of the nested elements.
5. set-translation with the language taken from the input (#29795): from an XML attribute of the current element or from an import variable, instead of only a fixed language. Optionally fill the application's fallback language with the same text, if it has none yet.

Defect: joined object linked again (#29794)

<object join-duplicates="true"> (ObjectCreate) reuses an object whose ID was already imported, but then runs the handler's <linking> again, so the object is added to its container reference a second time:

java.lang.IllegalArgumentException: 'AttributedWrapper(type:ehome.catalog:Manufacturer, ..., name: Siemens)' is already used and must not be added again to 'com.top_logic.element.meta.kbbased.storage.SetStorage@...'
	at com.top_logic.element.meta.kbbased.storage.SetStorage.checkAddValue(SetStorage.java:156)
	at com.top_logic.xio.importer.binding.AbstractModelBinding.addValue(AbstractModelBinding.java:133)
	at com.top_logic.xio.importer.linkings.ListLinking.lambda$link$0(ListLinking.java:50)

For an ordered (list) reference the object ended up in the list twice instead.

Workarounds in E-Home

• Importer.registerSharedObjects() registers all objects of the shared modules under their extId with ModelBinding.assignId() before reading (1.).
• App handler de.haumacher.ehome.importer.UnlessKnown (<unless-known id-attribute="Id">) (2.).
• TranslationsImport, which looks objects up with resolveObjectBackwards() and ignores unknown ones (3.).
• DefaultLanguage (<default-language>) binding the Locale of DefaultLanguage to the variable language (4.).
• I18NPropertyImport / I18NTexts (5.).
• JoiningModelBinding, whose addValue() does nothing if the reference already contains the element (defect).

Lösung

• Linking is idempotent: AbstractModelBinding.addValue() does nothing if the reference already contains the element. This fixes join-duplicates and also covers objects found among existing data, which are usually linked already.
• <resolve-existing objects="..." key="..." argument-var="this"> (scoped handler): within its content, an ID that is not known to the import is looked up among existing objects. The index is built once per import, on the first miss, from the objects computed by objects (which receives the value of argument-var), keyed by the result of the function key (converted to a string). A hit is registered as an import ID, so references resolve to it and <object join-duplicates="true"> updates it instead of creating a new object. Nested handlers are asked innermost first; a duplicate key is reported as warning, the first object wins. Example: <resolve-existing objects="all(`ehome.catalog:Manufacturer)" key="m -> $m.get(ehome.catalog:Manufacturer#extId)">`. Java code can install its own lookups through ImportContext.withIdLookup(IdLookup, ...).
• <unless-known id-attribute="Id">: skips the element with its content if its ID is known (created earlier in the import or found via resolve-existing), otherwise runs the content.
• ignore-unresolved="true" on <object-ref>, <reference> and <multi-reference>: an ID that is still unresolved at the end of the import is ignored instead of being reported (the reference stays unset, the content of an <object-ref> is dropped). It is still reported if another reference to the same ID requires it. (Named ignore-unresolved, since <object-ref> already has mandatory for the presence of the ID attribute.)
• <multi-reference> with forward references (defect found on the way): the list was stored immediately, so IDs defined later in the file ended up as internal placeholder objects in the reference. The list is now stored once all its IDs are resolved, in ID order; an ID still unresolved at the end of the import is left out.
• <with-var var="language" xml-attribute="DefaultLanguage">: runs its content with the variable bound to the attribute value; without the attribute the content runs with the variable unchanged.
• <set-translation> takes exactly one of language, language-attribute (attribute of the current element) and language-var (import variable with a language tag or a Locale). The language tag is reduced to its language (de-DE -> de), also for a fixed language. fill-fallback="true" also stores the text under the fallback locale of the application, if no text is stored for it yet.

Test

• test.com.top_logic.xio.importer.TestXmlImporter