#29234 OpenAPI server cannot return binary responses

enhancement

major

#29084

JSON Schema Support for Typed Configuration

minor

#29334

Status code as argument for responses in REST requests

defect

major

#28993

Filters of dynamic columns ignore the attribute type

#29042

Error in column filter for double values

#29234

OpenAPI server cannot return binary responses

#29325

Generated JSON schema marks a non-nullable layout-template property (dialog title) as nullable/optional

#29336

7.10: Bookmark links do not work for logged-in users

#29338

CreateTLSingletonProcessor creates duplicate annotations

minor

#28456

Reference to missing files in subsession.jsp

#28485

Immutable/Disbled Boolean fields accessible/focusable via tab

#28651

Unnecessary space for inline tables

#28725

Pointless scroll bar in the help editor

#29053

TL-Script: format() with null input returns IllegalArgumentException

#29066

"SafeHTML not started" error with scripted tests

#29069

A calculated attribute of the type "General search expression" cannot return a closure

#29074

The form data of a component cannot be accessed in a value transformation of a component channel

#29075

After changing the model, calculated attributes that have become invalid can no longer be edited

#29076

API key in REST interface configuration is not hidden

#29091

Values of a composition attribute cannot be edited if previously used in the same form

#29240

Update possible with incorrect attributes

#29284

Flow diagram: Font metric does not fit - texts flow beyond their frame

#29304

Replacement value attribute is not applied correctly in transient objects

#29339

Failed migration in #28305

#29340

TL Script: Optimizing database queries with transient objects fails

defect

major

#29234

OpenAPI server cannot return binary responses

OpenAPI

Problem

The current server-side OpenAPI implementation does not allow defining an endpoint that returns binary data. Any operation result is either JSON-serialized or coerced to a String before being written to the HTTP response - there is no path for streaming raw bytes (e.g. application/octet-stream, image/*, application/pdf) to the caller.

Root Cause

In com.top_logic.service.openapi.server.impl.ServiceMethodByExpression#handleRequest, the response body is always written via the character Writer:

{{{#!java resp.getWriter().write(content); // content is always a String }}}

The value of content is built in exactly two ways:

Default (no Response wrapper): content = JSON.toString(result) - any return value is JSON-serialized.
With Response wrapper (HTTPResponse TL-Script function): if the MIME type is application/json, then content = JSON.toString(response.getResult()); otherwise content = String.valueOf(response.getResult()).

HttpServletResponse#getOutputStream() is never used for user endpoint responses. There is no special-casing for BinaryData, byte[], or InputStream return values - they fall into the else branch and are either JSON-serialized or coerced via String.valueOf(...), which destroys the binary content.

The Javadoc of com.top_logic.service.openapi.server.impl.Response already documents this limitation:

> "written to HttpServletResponse#getWriter()" - i.e., character output only.

Schema vs. runtime mismatch

com.top_logic.service.openapi.server.parameter.ParameterFormat defines a BINARY entry, which is reused by OperationResponse#getFormat(). BINARY is emitted into the generated OpenAPI spec document as a schema annotation, but there is no corresponding runtime handling in the response writer. The published contract therefore lies about what the endpoint can actually deliver.

Asymmetry with the Client Side

The client side (consuming a remote API) does support binary responses: com.top_logic.service.openapi.client.registry.impl.response.ResponseHandlerByExpression calls EntityUtils.toByteArray(entity) for BINARY format and returns a byte[] to TL-Script. Only the server side (producing a response) is broken.

Proposed Fix

In ServiceMethodByExpression#handleRequest, branch on the result type:

If the result is a BinaryData, byte[], or InputStream (or if the declared/response ParameterFormat is BINARY), write raw bytes via resp.getOutputStream() instead of resp.getWriter().
Preserve the current JSON/string path for all other cases.

Additionally, Response (the HTTPResponse TL-Script wrapper) should accept binary payloads so that a script can explicitly return application/octet-stream, application/pdf, image/*, etc., with the correct Content-Type and Content-Length set.

Affected Files

com.top_logic.service.openapi.server/src/main/java/com/top_logic/service/openapi/server/impl/ServiceMethodByExpression.java (lines 79-112)
com.top_logic.service.openapi.server/src/main/java/com/top_logic/service/openapi/server/impl/Response.java (Javadoc + possibly new binary payload field)
com.top_logic.service.openapi.server/src/main/java/com/top_logic/service/openapi/server/parameter/ParameterFormat.java (runtime hook for BINARY on the response side)