Known Limitations

The honest gaps - what is proven against a real running app and what is not.

Known Limitations

BX Agents is under active development. This page tracks the honest gaps - what's tested against a real running app, what still only runs against bx-ai's "mock" provider, and real upstream quirks this project ran into.

Testing runs against the mock provider only

Every fast-lane spec (build pipeline, generators, CLI verbs) and the ColdBox integration suite exercise bx-ai's built-in "mock" provider - never a real network call to an LLM. This is deliberate (fast, free, deterministic CI), but it means no automated test currently proves a real provider (OpenAI, Anthropic, etc.) actually round-trips correctly end-to-end. Do at least one manual chat/serve run against a real provider and a throwaway API key before depending on this in production.

Real ColdBox integration testing

./gradlew testColdBoxIntegration boots a real boxlang-miniserver process against a generated app's own Application.bx/Bootstrap, and makes a genuine HTTP request through a toAi()-registered route. This is the strongest proof point in the suite - it caught three real bugs in ColdBoxAppGenerator.bx during development (a missing WireBox Binder extends, wrong Bootstrap constructor argument order, and a bare getInstance() call that doesn't exist on Binder).

The toAi() first-request race

The very first HTTP request to a freshly booted app's toAi() route can transiently fail with "Function [getInstance] not found" - a genuine ColdBox/WireBox lazy-injection race on the Router's own getInstance delegate, not a BX Agents bug. It succeeds reliably on every request after something else has already forced WireBox to build the GeneratedAgent singleton once (a health check, a chat session, another route). Send a warm-up request before relying on a freshly deployed toAi() route under load.

Three integration checks - closed via a different route than originally planned

tests/specs/integration/RuntimeStartupSmokeSpec.bx still has three xit()s, since this file runs via the CLI runner (runTests.bxs/testBx), and BoxLang's CLI mode never gives a cgi scope - which ColdBox's RoutingService needs even to load the router at startup. That's a structural fact about CLI mode, not a gap: all three checks are now proven for real elsewhere, in tests/specs/integration/coldbox/ColdBoxRuntimeSpec.bx (which runs inside a real HTTP request served by a real boxlang-miniserver process):

• A real ColdBox-routed HTTP request reaching the generated agent end-to-end - proven by ColdBoxRuntimeSpec.bx plus runColdBoxIntegrationTests.bxs's own POST /api/chat/invoke assertion.
• schedules/* actually registering with a live ColdBox Scheduler - proven via SchedulerService.getSchedulers()["appScheduler@coldbox"].hasTask(...) against a real boot, without waiting out an actual cron fire (the coarsest supported granularity is 1 minute, which would tax every CI run for marginal extra proof once the task is confirmed live and registered).
• chat and a served HTTP route never diverging - corrected from its original wording (chat deliberately never boots WireBox at all, so it can never share WireBox's singleton object by design) to what actually matters: instantiating GeneratedAgentFactory outside of WireBox produces a behaviorally-equivalent agent to WireBox's own singleton.

Real OS-process CLI testing - and a real bug it found

ModuleCliProcessTest.java spawns genuine java -jar <boxlang-jar> module:bxagents <verb> ... child processes against a copy of the actual installable module structure (build/modules/bxagents), pointed at a real modulesDirectory - exactly how a real BoxLang installation loads this module. Every other CLI spec calls ModuleConfig.main()/each verb's run() in-process instead, which is faster but never proves the module resolves correctly once genuinely installed.

That gap was real: this test caught every CLI verb failing with "class not located" the moment it ran through an actual installed-module process, because internal cross-references used a bare bxagents.models.... dotted path that only ever resolved thanks to this repo's own dev/test boxlang.json (a hand-declared /bxagents mapping) - a genuinely installed module never gets that mapping. Fixed by switching every nested class's internal references to the Class@bxagents module-relative suffix form (ModuleConfig.bx itself, sitting at the module root, resolves plain relative paths fine and needed no change) - see BuildPipeline.bx's init() docblock for the full explanation. build.gradle's module-structure output moved from build/module to build/modules/bxagents (folder name must equal the module name for modulesDirectory discovery to find it at all) and the dev/test boxlang.json now also loads it as a real module, so the whole existing suite exercises the same resolution path production does, not just the convenience mapping.

A real, foundational bug found building the testing framework: agents never received their own tools

Building BaseAgentSpec's toHaveCalledTool matcher (M15) surfaced a serious, previously-undiscovered bug: ColdBoxAppGenerator's generated aiAgent() call never passed a tools: argument at all - confirmed against real bx-ai source (AiAgent.bx never references aiToolRegistry() internally). A project's tools/ were copied into the build and made name-resolvable for MCP wiring, but no agent BX Agents ever built - in any context: a real served app, chat, or a test spec - actually received its own declared tools. This had gone undetected because no existing test ever asserted on a real tool invocation, only on a non-empty response.

Fixed in ColdBoxAppGenerator.renderAgentFactory(): every generated GeneratedAgentFactory.bx now loads its own tools/ directory via a new ToolRegistryLoader.bx (using an ABSOLUTE path embedded at generation time, not a relative one depending on whatever "/" mapping happens to be in effect for the loading context - confirmed that aiToolRegistry().scan("tools")'s own relative-path resolution silently fails when called from a DynamicClassLoader-loaded context like chat), then passes tools: aiToolRegistry().getAll() to every aiAgent() call.

A third top-level-script pitfall, found the hard way in CI: var cannot be used at the top level of a .bxs script. var declares into the local scope, which only exists inside a function, so a top-level var x = ... throws Scope [local] is not available in this context at runtime - not at parse time, so it survives review and only fires on the code path that reaches it. It cost a CI cycle in runColdBoxIntegrationTests.bxs, where the offending line sat inside a failure-diagnostics branch and therefore crashed exactly when something else had already gone wrong, replacing the real failure with its own. Two rules follow: never write var outside a function in a .bxs, and wrap diagnostic output in its own try/catch so it can never mask the failure it is describing.

The Application setting for a default datasource is this.datasource, NOT this.defaultDatasource. The plural registration key really is this.datasources[ "name" ] = { ... }, so this.defaultDatasource is the name everyone reaches for - and BoxLang accepts it silently and does nothing with it. Verified directly against the runtime rather than inferred: with this.defaultDatasource = "testds" set, an unqualified queryExecute() still fails with No default datasource defined in the application or globally or in the query options. Registered datasources are: [testds]; changing the single line to this.datasource = "testds" resolves the datasource and the call proceeds to a real driver concern. There is no warning and no hint in the error that the setting you wrote was ignored - the message names the registered datasource you are trying to select, which reads as though the selection mechanism is broken rather than misspelled.

qb's moduleSettings.qb.defaultOptions did not reach QueryBuilder in a real ColdBox boot. The generated config/ColdBox.bx sets moduleSettings.qb.defaultOptions = { datasource : "<name>" }, and qb's own ModuleConfig.cfc maps QueryBuilder@qb with .initArg( name = "defaultOptions", value = settings.defaultOptions ) inside onLoad() - which reads as correct, and is why the generated ChatDb.query() originally named no datasource. It did not work: every query failed with No default datasource defined in the application or globally or in the query options. Registered datasources are: [<name>], i.e. the datasource was registered and the builder still had empty options. Why the module setting did not arrive was not run to ground - it stopped mattering, because naming the datasource explicitly is better than depending on that plumbing regardless. ChatDb.query() now calls .mergeDefaultOptions( { datasource : static.DATASOURCE } ) on every builder, mirroring what schemaOptions() already had to do for SchemaBuilder (which qb never passes defaultOptions to at all). If you add a new qb call path to generated code, name the datasource on it; do not assume the module setting covers you.

A generated app now declares this.defaultDatasource, not just this.datasources. With one datasource, anything that runs a query without naming one - ColdBox itself, another module, a project's own code - should reach it instead of failing. Nothing asserted on the emitted datasource block before, which is how it shipped without a default; ColdBoxAppGeneratorSpec now covers both the web-UI case and the no-web-UI case (where neither line should appear).

The generated models/ChatDb.bx did not compile, and no unit spec noticed. WebUiGenerator builds BoxLang source out of BoxLang string literals, where a literal double quote is written as a doubled quote - so an empty-string literal needs FOUR quote characters, and writing the natural-looking two emits a single stray quote that opens a string and swallows the rest of the line. Two template lines carrying an elvis-to-empty-string (?: "") emitted ?: ", so every generated web-UI project shipped a ChatDb.bx that failed to parse. Every WebUiGeneratorSpec case asserted on substrings of the generated text, which all still matched; only a real ColdBox boot ever tried to compile it, and that path was itself broken for several cycles by the harness bugs described below, so the real bug stayed hidden behind them. There is now a WebUiGeneratorSpec case asserting that every line of generated source has an even number of double quotes - an invariant that catches the whole class rather than the two lines that happened to be wrong. The wider lesson: a generator spec that only greps its own output does not test that the output is valid code.

GET /chat/api/health proves far less than it looks like it does. The generated health() action renders a literal { status: "ok", success: true } and touches nothing else - so a 200 proves ColdBox booted and routing reaches the generated ChatUi handler, and says nothing about WireBox, ChatDb or SQLite. The comment in runColdBoxIntegrationTests.bxs used to claim it exercised "ColdBox routing -> ChatUi -> WireBox -> ChatDb", which is false and gave real false confidence: one CI run returned a green probe while models/ChatDb.bx could not compile at all. Corrected in place; treat the integration specs, not the probe, as the store's coverage.

TestBox's JSONReporter cannot report on specs that touch live framework singletons. The ColdBox integration specs resolve real controller/WireBox/qb objects, and those object references end up in the TestBox result memento. JSONReporter serializes the memento wholesale, so BoxLang reflects over the live objects (StructUtil.objectToStruct -> DynamicInteropService.getMethodNames) and follows their cyclic references until the JVM throws StackOverflowError. The specs themselves had already run green when this fired, which is exactly what made it confusing: a passing suite reported as a failed run. tests/runner-coldbox.bxm therefore assembles its own report instead, forcing every value through a toScalar() helper so no object can reach the serializer regardless of what a spec leaves behind, and recording the pass/fail counts in the progress markers before the report is built - so a serialization failure can never again be mistaken for a test failure.

In a BoxLang time mask, nn is NANOSECONDS, not minutes - use mm. dateTimeFormat( now(), "HH:nn:ss" ) silently produced timestamps like 02:777491298:32, which read as corrupt output rather than a wrong mask. (Note the inverse trap in a date mask, where mm is the month: yyyy-mm-dd yields the minute, not the day.)

Deriving an identity server-side is not the same as authorizing against it. The generated handlers/ChatUi.bx took care to derive userId from the session on every action and never from the request body - and two actions still had an authorization hole, because they were addressed by threadId rather than by the caller. /pending and /resume loaded a checkpoint by id and acted on it; /resume even derived decidedBy from the session, which reads like a scoping check but only governs the label on the decision, not whose run is being decided. A visitor with someone else's threadId could read their pending tool calls and answer their approvals. Both now compare the caller against the userId the agent checkpointed in the run options. The general rule this is worth remembering for: when a route is keyed by an opaque id rather than by the caller, deriving identity buys you attribution, not access control - the two need separate thought, and a server-derived value sitting in the same function can easily be mistaken for both.

ProjectValidator's path checks must consider .., not just a leading separator. database.path is spliced into an expandPath() call in the generated Application.bx, so it was checked for being absolute - and ../../var/lib/chat.db escapes the app directory exactly as effectively while passing that check cleanly. Now rejected on both separators, matching whole path segments so legitimate names like ..hidden or a..b still pass. Any future validator that guards a generated path needs both halves of this check.

Running the test suite locally needs bx-ai overlaid from source, exactly as CI does. ./gradlew downloadModules fetches the published bx-ai snapshot, which lags its own development branch - it overwrites the vendored copy under src/test/resources/modules/bxai and the suite then fails ~40 specs with Method 'isRunning' not found and The method aiGatewayRegistry does not exist, both APIs that exist upstream but not in the published build. That is not a regression, and it is easy to misread as one. Rebuild and overlay before running testBx: ( cd <bx-ai checkout> && ./gradlew createModuleStructure ) then cp -R <bx-ai>/build/module/. src/test/resources/modules/bxai/. tests/coldbox/, tests/testbox/ and tests/qb/ come from box install (run it in tests/), and all three are gitignored.

chr() does not exist in BoxLang - the BIF is char(). This is not context-dependent, despite a comment in ModuleConfig.bx that used to claim it was unavailable only "in this module-CLI execution context" (now corrected). Verified directly against the runtime: char( 10 ) returns the newline, chr( 10 ) throws Function [chr] not found, in a plain CLI script and in a served template alike. It is worth knowing because the failure is a runtime one - chr() parses fine and survives review, then throws on whatever code path first reaches it. It cost several CI cycles here: tests/runner-coldbox.bxm called chr( 10 ) in its very first progress marker, so every request 500'd on entry, and the retry loop described below turned that plain, one-line error into an opaque "HTTP 408, never responded".

A fourth lesson from the same CI harness, and the one that cost the most cycles: instrumentation that a retry can overwrite records nothing. tests/runner-coldbox.bxm opened by truncating its own progress file (fileWrite( progressFile, "" )), and runColdBoxIntegrationTests.bxs retried the runner request once a second until a deadline. So every retry wiped the markers the hung first attempt had written, and the failure diagnostics faithfully printed an empty file - which read as "the page never got anywhere" when the truth was "the evidence was deleted". Worse, the retries were actively harmful independent of the diagnostics: the runner page is neither cheap nor idempotent (it boots TestBox and runs every integration spec), so retrying it stacked concurrent full test runs onto a MiniServer worker pool of only a handful of threads - a way to cause a hang, not to recover from one. Both are fixed: the orchestrator clears the marker file once and makes exactly one request (the health probe ahead of it has already proven the server is up, so there is nothing for a retry to wait for), and the page only ever appends, tagging each line with a per-request UUID so overlapping attempts stay distinguishable. The general rule: a progress log must be append-only and must outlive the thing it is instrumenting, and anything that is expensive or stateful should not sit behind a retry loop at all.

A related, separate BoxLang pitfall found while diagnosing this: request is a reserved built-in scope name. A local/loop variable named request can silently shadow it - for ( var request in someArray ) { request.someKey } iterated the correct number of times, but every request.someKey access inside it silently read as the empty built-in scope instead of the loop variable, with no error at all. Fixed by renaming to recordedRequest in BaseAgentSpec.bx's matchers - worth remembering for any future BoxLang code in this project that loops over anything sensibly named request.

serve's miniserver lookup is PATH-only

serve looks for boxlang-miniserver only on PATH (MiniServerLauncher.findExecutable()). There's no fallback to a configured path or a bundled binary - if it isn't installed and on PATH, serve fails with a clear, actionable error, but there's no alternate way to point it at one. invoke --server reuses serve internally, so it inherits the same PATH-only lookup - its own InvokeSpec.bx test for the real HTTP round-trip checks MiniServerLauncher.findExecutable() first and skips (rather than fails) when no real binary is on PATH, the same idiom RuntimeStartupSmokeSpec.bx already uses for its own jar-presence check. Do at least one manual bxAgents invoke --message=... --server run on a machine with boxlang-miniserver installed before depending on the real HTTP path in production - the same honest framing already used elsewhere in this page for gaps this suite can't close itself in every environment.

The scoped BoxLang runtime home reaches serve/invoke --server unconditionally; the in-process verbs only if your BoxLang install loads .env

serve scopes the miniserver's own BoxLang runtime home to .build/runtime via serverHome (a real, confirmed MiniServerConfig field - boxlang-web's own MiniServer CLI help text: -s, --serverHome <PATH> BoxLang server home directory (default: ~/.boxlang)), so each project's compiled-class cache and any config overrides are isolated per-project rather than shared globally at ~/.boxlang. invoke --server inherits this since it reuses serve internally. This part is unconditional - we write the launch config for that process ourselves.

chat, build, test, and default (in-process) invoke are different: they run inside the already-started bxAgents process, whose own BoxLang runtime - and therefore its home - was resolved before any of our BoxLang code, including ModuleConfig.bx's own main(), ever got a chance to run (the engine has to exist to interpret that code at all). BoxRuntime is a JVM-wide singleton; its home is fixed at first initialization, so nothing a verb class does from inside that already-running process can change it retroactively.

There IS a real lever for this, though: BoxRunner (BoxLang's own core CLI entry point, not just the miniserver) reads a genuine BOXLANG_HOME environment variable before BoxRuntime initializes - confirmed directly against the real runtime jar (not just documentation): running it with BOXLANG_HOME=<path> set as a real OS environment variable (relative paths resolve against the CWD, same as an absolute path) populates the full home structure at that path instead of ~/.boxlang, every time. new scaffolds a .env declaring BOXLANG_HOME=.build/runtime - the same path serve uses - for exactly this reason (mirroring ortus-boxlang/bx-ai-intro's own .env/BOXLANG_HOME convention), at the project root - i.e. the directory a user actually runs bxAgents <verb> from for the in-process verbs, which is the right place for a CWD-based .env loader to find it.

What we confirmed, and what we couldn't. boxlang-miniserver (what serve launches) does have real, built-in .env auto-loading - confirmed by decompiling and then actually running ortus.boxlang.web.MiniServer directly: with no envFile configured, it resolves .env relative to the server's webRoot (not the project root) and, if found, loads it as Java Properties, applying each key via System.setProperty() - real values loaded this way ARE visible to BoxLang code via getSystemSetting(). The raw core runtime jar (BoxRunner, used by chat/build/test/default invoke) has no equivalent logic anywhere in it (confirmed by grepping every class in the jar for .env) - so those verbs only pick up .env if the actual boxlang CLI you have installed (a BVM-provided native launcher, not this raw jar) does its own .env loading before the JVM starts, the way ortus-boxlang/bx-ai-intro relies on. That's plausible and consistent with real-world usage of that project, but isn't something this sandbox has the real binary to verify directly.

One specific, confirmed gotcha even where .env loading does happen: BOXLANG_HOME itself does not take effect unless it's a real OS environment variable - not a JVM system property, and not a -D flag. Verified twice directly against the real jars: (1) running boxlang-miniserver against a webRoot whose .env declared BOXLANG_HOME=customhome loaded the file (confirmed via its own "Loaded environment variables from:" log line and via getSystemSetting() correctly returning other .env values), yet the server still logged Logs Directory: /root/.boxlang/logs - the default home, not customhome; (2) launching BoxRunner directly with -DBOXLANG_HOME=<path> (a JVM system property, no .env involved) also had no effect on the resolved home, even though getSystemSetting( "BOXLANG_HOME" ) happily returned the flag's value from BoxLang code. So BOXLANG_HOME resolution specifically reads only the true OS environment variable - unlike most settings, getSystemSetting() returning a value for it doesn't mean the runtime home actually moved. If your boxlang CLI's .env loader works the same way MiniServer's does internally (System.setProperty after the JVM has already started) rather than exporting a real env var before launching the JVM, BOXLANG_HOME in .env won't reach the runtime home even though .env itself loaded successfully - everything else in it will still work. If your install doesn't give you this at all, source it yourself before running commands (e.g. set -a; source .env; set +a) to get the same isolation serve already gets unconditionally.

new's box install convenience step isn't exercised by an automated test

new runs box install inside the scaffolded tests/ folder by default, so bxAgents test works immediately (see CLI Reference). This is a real network operation (CommandBox resolves testbox against ForgeBox) - confirmed to take ~25 seconds and fail with a certificate error in this development sandbox specifically, the same ForgeBox-unreachable constraint already noted elsewhere in this project's own tooling. NewSpec.bx only exercises the fast, deterministic --skipInstall path (and the OS-process ModuleCliProcessTest.java/in-process ModuleConfigCliSpec.bx invocations of new also pass --skipInstall for the same reason) - the default install-attempt path itself is verified by manual testing only, not CI.

chat needs a real TTY

chat uses BoxLang's own MiniConsole, which shells out to stty to set up raw terminal mode - it can only run against a genuine interactive terminal. It doesn't work piped, redirected, or from a non-interactive process (a CI job, a script). There's no non-interactive fallback mode.

Fixed: chat and default (non---server) invoke used to fail for a class-based Agent.bx

Previously, both chat and default invoke threw The requested class [agent.classes.agentClass] has not been located in any class resolver. before ever reaching the agent. Root cause: both verbs load the generated GeneratedAgentFactory.bx in-process via DynamicClassLoader.instantiate() (a raw RunnableLoader call against an absolute path, no ColdBox container involved), and the generated factory used to instantiate a class-based Agent.bx via a relative dotted-path lookup, new "agent.classes.agentClass"() - which only resolves once something has registered a mapping making the app root resolvable, and nothing did outside a real ColdBox boot.

Registering one mid-script (Configuration.registerMapping( "/", appDir ) right before DynamicClassLoader.instantiate()) does not fix it either - confirmed empirically by hand-rolling the exact same sequence in a standalone .bxs script. Same class of limitation already documented above for TestRunnerLauncher's TestBox discovery: a mapping registered via Configuration.registerMapping() mid-script does not reliably propagate to a class's own relative-path lookups made within that same process.

Fix: ColdBoxAppGenerator.copyAgentClass() now returns the copied class's own absolute file path (instead of a dotted component path), and renderClassBasedAgentStatement() instantiates it via DynamicClassLoader.instantiate( absolutePath, context ) - the exact same primitive chat/invoke already use to load GeneratedAgentFactory.bx itself - rather than a relative new "..."(). This sidesteps mapping resolution entirely, so it now works identically whether or not a real ColdBox container is booted. Confirmed against examples/class-based-agent/: chat, default invoke, invoke --server, and serve all now build and run the same agent correctly.

No box.json executable install smoke test

box.json declares "boxlang": { "executable": "bxAgents" } so a real module install produces a native bxAgents command (see Installation). This wiring itself isn't exercised by an automated test - it relies on the BoxLang module installer's own documented behavior for generating executable wrappers, verified by reading its source, not by an install-and-run test in this repo's own CI.

schedules/Scheduler.bx isn't validated at build time

Since it's real, hand-written ColdBox code passed through untouched (see schedules/), build can't meaningfully check it the way { cron, action } config used to be checked - a syntax error, a typo in a getInstance( "..." ) call, or referencing an agent name that doesn't exist all pass build cleanly and only surface when the generated app actually boots (serve), same as any other real BoxLang class this project doesn't own the contents of. build does catch one narrower, adjacent mistake: two agents (root or subagent, any depth) declaring the same name - that's a config/WireBox.bx binding collision this project generates itself, so it's checked at validation time same as everything else it generates.

A real bug found building the test verb: resolving "the current BoxLang jar" is ambiguous once boxlang-miniserver is on the classpath

TestRunnerService.bx spawns a fresh child process to run a project's tests/specs, and needs to know which jar to launch it with. The first implementation used the standard "which jar was this class loaded from" trick (BoxRuntime.class.getProtectionDomain().getCodeSource().getLocation()) - this worked in isolated manual testing, but failed unpredictably once run as part of this project's own full testBx suite, which also needs boxlang-miniserver-*.jar on the classpath (for serve/MiniServerLauncher-related specs). Confirmed via direct inspection: boxlang-miniserver-*.jar is a fat jar that bundles its own copy of ortus.boxlang.runtime.BoxRuntime - with both jars on the classpath, the classloader can resolve BoxRuntime.class to the miniserver jar instead of the real runtime jar, silently launching MiniServer.main (which rejects --bx-config and exits 1 immediately, before any BoxLang script or test report is ever produced) instead of BoxRunner.main. This surfaced as tests/specs/cli/TestSpec.bx's real-process cases failing with exitCode=1 and an empty report - reproducible only when run through the full suite, not in isolation, which is what made it easy to miss.

Fixed by resolving the jar from java.class.path instead - scanning for a boxlang-*.jar entry that does not contain miniserver, falling back to the old codeSource trick only if no such entry is found.

deploy's ssh/docker/digitalocean targets aren't exercised by an automated real-process test

SshTargetSpec.bx/DockerTargetSpec.bx assert on the exact scp/ssh/docker command each target builds (a real ProcessBuilder argument array) without ever invoking the real binary - the same "capture, don't execute" approach used elsewhere in this suite for anything that needs a binary that might not be on PATH in CI (see MiniServerLauncherTest's assumeTrue skips). local's copy logic IS exercised for real (LocalTargetSpec.bx), including a regression test for a real latent bug this refactor fixed: the original Deploy.bx picked the "newest" .bxa via a lexical filename sort, which silently mis-picks once a project reaches double-digit versions (v9.0.0 sorts after v10.0.0) - fixed by sorting on actual file modification time (DistArtifactLocator).

DigitalOceanTargetSpec.bx similarly only unit-tests the pure buildAppSpec()/findExistingAppId() logic - the real GET/POST /v2/apps calls are never exercised in CI, since that needs a live DigitalOcean account and API token. Do at least one manual deploy --name=<ssh-entry> against a real disposable VM, and one deploy --name=<digitalocean-entry> against a real DO account with a throwaway app, before depending on either in production - the same honest framing already used above for the mock-provider-only testing gap.

deploy's ftp/sftp targets: real connection handling proven, a real successful upload isn't

Unlike ssh/docker (which shell out to external binaries and so can only have their command construction tested without a real server), ftp/sftp call the real bx-ftp module's bx:ftp component in-process - there's no "capture the command, don't execute" option. BaseFtpTargetSpec.bx instead makes a genuine connection attempt against 127.0.0.1 on a port nothing listens on, and asserts that the real refused-connection error is caught and re-thrown as a clear BxAgents.DeployFailed (confirmed via bx-ftp's own source that every action throws on failure rather than returning a soft succeeded: false). This proves the real connect/error-wrap/cleanup path end-to-end - what it can't prove is a real successful upload.

That gap is specifically because this development sandbox has no outbound raw TCP egress at all - only HTTPS through the environment's proxy (confirmed directly: curl ftp://test.rebex.net and a raw /dev/tcp connect to a public FTP host both hung and timed out, and docker info shows no daemon running, so bx-ftp's own bundled Docker FTP/SFTP test servers couldn't be started here either). This is a sandbox constraint, not a code limitation - do at least one manual deploy --name=<ftp-entry> and one deploy --name=<sftp-entry> against a real reachable server (or bx-ftp's own docker-compose up test servers, from a machine that has Docker/network access) before depending on either in production, the same honest framing already used above for ssh/digitalocean.

build doesn't roll back a partially-written .build/app on a mid-generation crash

BuildPipeline.build() deletes and recreates .build/app up front, then runs Phase 5's generators in sequence. Every input that Phase 3 (ProjectValidator) can check is checked before any of that happens, so a genuine mid-generation crash should be rare in practice - but if one still occurs (e.g. a models//schedules//mcp/ entry that fails to load, or an environment/filesystem problem), .build/app is left on disk in a partially-written state rather than restored to its prior contents or cleaned up. A subsequent successful build overwrites it cleanly, so this isn't sticky, but anything that inspects .build/app between a failed build and the next one (a CI step, a manual package retry) can see a broken half-generated app. There is no rollback/temp-directory-then-swap step yet.

An intermittent StackOverflowError was observed in testBx, unrelated to this project's own code

While investigating this milestone, ./gradlew testBx occasionally (not every run) crashed the whole JVM with a StackOverflowError inside the BoxLang engine's own generic-object JSON serialization (DynamicObjectSerializer/BoxStructSerializer calling each other, alternating, until the stack is exhausted - confirmed to still happen with -Xss16m, so it's a genuine cycle, not merely a deep-but-finite structure). Isolated via git stash (the crash reproduced identically with none of this session's changes applied, against the exact same commit already on development) and via bisecting tests/specs/** into every subdirectory both individually and combined (every subset, and even every subset-but-one, ran clean) - only the single full run occasionally reproduces it, and re-running the identical full suite immediately after a crash sometimes passes clean. This points to a timing-dependent race (plausibly interacting with ExampleScheduler's real background everySecond() task, which prints from its own thread pool concurrently with whichever spec happens to be running at that moment) rather than a bug in any one spec or in this project's own generated code. If testBx fails with a StackOverflowError and no other explanation, retry it before assuming a real regression - it is not currently reproducible on demand, so no automated regression test exists for it, and running it down further to a single root cause was out of scope for this round.

Push-style gateways (Telegram) are tested against a mocked API/scheduler seam only - no live platform integration runs in CI

TelegramGatewaySpec.bx exercises TelegramGateway's own logic (inbound normalization, outbound chunking at the 4096-char limit, HITL inline-keyboard building, scheduler task registration/removal) entirely against an injectable apiCaller/setScheduler() test seam - never a real Telegram Bot API call, and never a real ColdBox scheduler boot. This proves the gateway's own code is correct, not that it actually works end-to-end against Telegram's real API or a real running scheduler. Do at least one manual bxAgents serve against a project with a real botTokenEnvVar-backed Telegram bot before depending on it in production - the same honest framing already used above for the mock-provider-only/no-live-connection testing gaps elsewhere in this file. The same caveat will apply to every future push-style gateway (Slack, Discord, Email, WhatsApp) built the same way.

SlackGatewaySpec.bx carries the same gap, one level deeper: SlackGateway's persistent websocket connection is tested via an injectable setSocketOpener() seam (a fake object standing in for the real java.net.http.WebSocket), so every frame-handling/reconnect-logic assertion in the spec runs with zero real network I/O. What genuinely was verified directly (not mocked): a standalone smoke test instantiated the real SlackSocketListener(gateway) (which implements="java:java.net.http.WebSocket$Listener" directly - BoxLang compiles it as a genuine JVM implementer, no proxy needed) and called the real HttpClient.newWebSocketBuilder().buildAsync(...) against an unreachable address, confirming the BoxLang-to-Java interop itself works correctly all the way to the network boundary (it failed with a plain java.net.ConnectException, not a casting/interop error) - but no test here has ever completed a real Socket Mode handshake against Slack's actual servers. Do at least one manual bxAgents serve against a project with real botTokenEnvVar/appTokenEnvVar-backed Slack app credentials before depending on it in production.

DiscordGatewaySpec.bx carries the identical gap, for the identical reason: DiscordGateway's frame-handling/heartbeat/reconnect logic is exercised entirely against injectable setApiCaller()/setSocketOpener() seams, zero real network I/O. The same standalone smoke-test discipline was applied here too - gateway.onConnect() driven against a real HttpClient.newWebSocketBuilder().buildAsync(...) call to an unreachable address failed with a plain java.net.ConnectException, not a casting/interop error, confirming the interop chain works. What was NOT verified: a real Gateway handshake (Hello → Identify → READY) against Discord's actual servers, real heartbeat timing under Discord's own tolerance, or that the default intents value (GUILDS+GUILD_MESSAGES+DIRECT_MESSAGES+MESSAGE_CONTENT = 37377) is actually enough to receive message content once MESSAGE_CONTENT is enabled/approved for a real bot in the Discord Developer Portal. Do at least one manual bxAgents serve against a project with a real botTokenEnvVar-backed Discord bot (with MESSAGE_CONTENT enabled) before depending on it in production.

EmailGatewaySpec.bx carries a larger version of the same gap. Inbound IMAP is tested entirely via an injectable setImapPoller() seam (canned normalized message structs, no real mailbox), and outbound is tested entirely via an injectable setMailService() seam (FakeMailService/FakeMail, standing in for MailService@cbmailservices - EmailGateway never goes through real WireBox at all in these specs, since there's no real ColdBox boot). What genuinely was verified directly this session (not mocked, not assumed): the real jakarta.mail API surface fetchInboundMessages() depends on (Session.getDefaultInstance(), Flags/Flags.Flag/FlagTerm, Store.getStore("imaps"), Folder.READ_WRITE, MimeMultipart, InternetAddress) against the real jakarta.mail-api/Angus Mail jars (downloaded standalone for this - they aren't vendored in this repo's own test classpath), confirming every class/method name used actually exists and resolves; a real Store.connect() against an unreachable address, driven through EmailGateway.pollInbox() itself (not a bypassed helper), failed with a plain connection-timeout error, not an interop/casting error, confirming the interop chain reaches the real network boundary correctly, same discipline as Slack/Discord's websocket smoke tests. What was explicitly NOT verified, and is a strictly bigger gap than the chat-platform gateways: no real IMAP handshake against a real mailbox, no real cbmailservices/bx-mail module installed anywhere in this repo or its test harness (neither is vendored the way bx-ai/TestBox are - see the snapshot-lag entry below for that same workaround applied to a different module), so the WireBox resolution path (MailService@cbmailservices really existing, BXMail's bx:mail call really sending) has never been exercised at all, mocked or real, in this codebase. Do at least one manual bxAgents serve against a project with real IMAP credentials AND a real cbmailservices/bx-mail install (confirm box install succeeded and moduleSettings.cbmailservices resolves) before depending on this gateway in production - this is the least-verified of the four push-style gateways shipped so far.

WhatsAppCloudGatewaySpec.bx covers the gateway's own logic thoroughly and for real, not mocked: the signature-verification path is exercised with genuinely computed HMAC-SHA256 signatures (javax.crypto.Mac/SecretKeySpec, cross-checked independently this session against both openssl dgst -hmac and Python's own hmac module before trusting the BoxLang computation - a real reference-vector mismatch was caught and turned out to be a typo in the hand-copied expected value, not a bug, but only cross-verification caught that), the verify handshake, webhook dispatch/dedup, outbound send, and interactive button/list rendering are all driven through the gateway's real public methods with only the outbound Graph API HTTP call itself stubbed (setApiCaller()). What was NOT verified: the generated handlers/WhatsAppCloud.bx's own ColdBox request-context calls (event.getHTTPContent()/event.getHTTPHeader()/event.renderData(), rc's URL-scope-merged dotted-key query param access for the GET handshake) against a real ColdBox boot - these ARE the documented, standard ColdBox REST-handler idioms (confirmed against ColdBox's own "Building REST APIs" recipe docs, not guessed), a meaningfully more trustworthy starting point than the undocumented aiGatewayRegistry() key assumption that turned out to be wrong elsewhere in this file - but "documented" isn't the same as "proven to work in this generated context," and extending the project's own real runColdBoxIntegrationTests.bxs/miniserver harness to cover it would have required either passing fake env vars into a separately-spawned miniserver subprocess (no existing mechanism for that) or risking real config-related boot failures in the shared e2e-coldbox-route fixture other passing tests depend on - deferred rather than risking that blast radius for what's a lower-confidence-of-actually-being-wrong gap than the registry-key bug was. Do at least one real bxAgents serve + a real Meta webhook test (or curl) against /webhooks/whatsapp-cloud before depending on this route in production. No real Graph API call has ever been made either - deliver()/requestHumanInteraction()'s HTTP layer is exercised only via the apiCaller test seam.

TeamsGatewaySpec.bx covers the gateway's own logic thoroughly and for real, not mocked: JWT verification is exercised against a genuinely generated 2048-bit RSA keypair (java.security.KeyPairGenerator) and hand-signed test JWTs built entirely inside the spec (no pre-computed fixtures, no external openssl dependency at test time) - a valid signature is accepted and dispatches, while a tampered signature, a wrong aud, a wrong iss, and an expired exp are each independently confirmed to reject with 401. The invoke-activity (Adaptive Card button click) path, message dispatch, personal-scope-only filtering, threading via replyToId, chunking, and Adaptive Card rendering are all driven through the gateway's real public methods with only the outbound Connector REST call itself stubbed (setApiCaller()) and the JWKS/OAuth2 token fetches stubbed (setJwksFetcher()/setTokenFetcher()). What was NOT verified: the generated handlers/Teams.bx's own ColdBox request-context calls against a real ColdBox boot (same category of gap as WhatsApp Cloud's own handler, and deferred for the identical reason - see that entry above); no real OAuth2 token fetch or Connector REST call has ever been made against Microsoft's actual endpoints; and the JWKS-cached-for-instance-lifetime tradeoff (see docs/conventions/gateways.md's Teams section) means a real key-rotation scenario has never been exercised either. Do at least one real bxAgents serve + a real Teams app registration (App ID/password from the Azure/Bot Framework portal) + a real Teams client sending a DM before depending on this gateway in production.

TwilioGatewaySpec.bx covers the gateway's own logic thoroughly and for real, not mocked: the X-Twilio-Signature HMAC-SHA1/base64 verification path is exercised with a genuinely computed signature built inline in the spec, cross-checked independently this session against Python's own hmac/hashlib modules before trusting the BoxLang implementation (same discipline as WhatsApp Cloud's own HMAC-SHA256 cross-check) - a real reference signature was computed in Python for a known auth token/URL/params combination and confirmed to match BoxLang's output exactly. Form-body parsing (including a literal + correctly round-tripping through %2B percent-encoding), the publicUrl override for proxy/tunnel deployments, the dual-path TwiML-ack-then-async-REST-reply model, outbound chunking, and the phone-number-keyed HITL reply correlation are all driven through the gateway's real public methods with only the outbound Messages API HTTP call itself stubbed (setApiCaller()). What was NOT verified: the generated handlers/Twilio.bx's own event.getUrl() call against a real ColdBox boot (same category of gap as WhatsApp Cloud/Teams's own handlers, deferred for the identical reason) - event.getUrl() is a documented ColdBox Routable/Request Context method (confirmed via the ColdBox docs MCP), not a guess, but "documented" isn't "proven in this generated context." No real Twilio Messages API call has ever been made either. Do at least one real bxAgents serve + a real Twilio phone number webhook test before depending on this route in production - and note the phone-number-keyed HITL correlation (see docs/conventions/gateways.md's Twilio section) has a real, documented limitation: a second HITL request to the same phone number before the first is answered would overwrite the first's pendingApprovals entry, silently orphaning it. No allowlist/rate-limiting is built for inbound SMS either - unlike Eve, which documents (but doesn't enforce in code) that its own allowFrom config is "mandatory," this port has no equivalent gate at all; any phone number can message a deployed Twilio number and reach the agent.

GitHubGatewaySpec.bx covers the gateway's own logic thoroughly and for real, not mocked, and this gateway's core logic was additionally driven through a standalone real-BoxLang smoke test during development (not just the permanent spec) - which is how a genuine bug was caught before it ever reached the test suite: the mention-extraction helper's substring logic called left( body, 0 ) whenever the @mention occurred at the very start of a comment (a very common case), and BoxLang's left() throws "Count cannot be zero" rather than returning an empty string for a zero count - confirmed by triggering it with a real comment body, then fixed by explicitly branching around the zero-length case rather than assuming left()/mid() tolerate it. The X-Hub-Signature-256 verification, @mention regex-lookahead gating (confirmed via a dedicated smoke test that a bot named mybot does NOT fire on @mybot2), bot-loop guards, delivery-id dedup, issue-vs-review-thread conversation identity, deliver(), and the @mention-to-reply HITL correlation are all driven through the gateway's real public methods with only the outbound GitHub REST call itself stubbed (setApiCaller()). What was NOT verified: the generated handlers/GitHub.bx against a real ColdBox boot (same category of gap as every other webhook gateway's own handler in this project, deferred for the identical reason). No real GitHub API call has ever been made, and no real GitHub App/PAT has ever been used against a real repository - do at least one real bxAgents serve + a real GitHub webhook configured against a test repository before depending on this gateway in production.

SignalGatewaySpec.bx covers the gateway's own logic thoroughly and for real, not mocked: handleSseEvent()'s JSON-RPC/SSE parsing (blank-line/invalid-JSON handling, group-message filtering, quote-threading, sourceUuid fallback when a display name is absent), deliver()'s send shape and chunking, and HITL decision correlation/matching are all driven through the gateway's real public methods with only the outbound rpcCaller/connector I/O calls stubbed, the same seam-testing discipline as every other gateway in this project. Two BoxLang-level findings surfaced during development, both resolved and worth recording as general landmines rather than gateway-specific bugs: (1) a throwaway smoke-test script that named a local variable request was silently interacting with BoxLang's own reserved request scope instead of creating a plain variable, producing misleading "method not found"/"argument type mismatch" errors from HttpClient.send() that looked exactly like a genuine Java-interop limitation but disappeared entirely once the variable was renamed - SignalGateway.bx itself never had a bug; (2) a try/catch placed directly at the top level of a standalone .bxs smoke-test script (not inside a function) triggered a java.lang.VerifyError: Inconsistent stackmap frames, a real bytecode-verification limitation of BoxLang's top-level-script compiler, traced via the stack trace to the test script's own generated class, not SignalGateway.bx - fixed by wrapping the try/catch inside a named function instead. What was NOT verified, and is the largest gap among all the push-style gateways shipped so far: no real signal-cli daemon has ever been available in this environment, so the entire async SSE connection lifecycle - opening the stream via HttpClient.sendAsync()+BodyHandlers.ofLines(), the exponential reconnect-backoff loop against a genuinely flaky connection, the 30s/120s idle watchdog forcing a reconnect, and a live JSON-RPC round trip - has never been exercised end-to-end, only smoke-tested at the interop-plumbing level (a standalone test reached a real java.net.ConnectException against an unreachable address, proving the chain is sound, not that it works against a live daemon). Do at least one manual bxAgents serve against a project with a real, running signal-cli daemon and a real linked Signal account before depending on this gateway in production - this is a genuinely new transport architecture in this codebase (the only SSE-based gateway among Telegram/Slack/Discord/Email/Signal), not just a new platform on an already-proven transport shape.

WhatsApp Personal (unofficial personal-account bridge) - researched, not built

The original plan (matching Hermes Agent's own architecture) called for a WhatsAppPersonalGateway built by spawning a Node.js subprocess running @whiskeysockets/baileys (the multi-device WhatsApp Web protocol client Hermes itself uses - MIT licensed, its full bridge.js was read directly from Hermes's real source, not summarized). That approach was set aside mid-session on direct instruction to prefer a native BoxLang/JVM integration over a subprocess bridge, and to use a native Java library only if it's open source and neither GPL nor LGPL.

That search found Cobalt (com.github.auties00:cobalt, formerly WhatsappWeb4j) - a real, MIT-licensed, actively maintained (900+ stars) Java implementation of WhatsApp's multi-device "Linked client" protocol, with a documented fluent API (WhatsAppClient.builder().linkedApi().webClient()..., addNewMessageListener(), sendMessage()) that would coerce cleanly from a BoxLang closure via BoxLang's own documented Java-SAM coercion (confirmed via the BoxLang docs MCP - no createDynamicProxy() needed for a single-abstract-method listener). Two real blockers surfaced during verification, not guessed:

1. The first pom.xml read (Cobalt's master branch, an in-progress multi-module rewrite) requires Java 25 - two majors ahead of BoxLang's own documented baseline (Java 21+, confirmed via the BoxLang docs MCP and matching this project's own 21.0.10 JDK). Re-checking against the artifact actually published to Maven Central (cobalt:0.0.10, the real thing a <dependency> would resolve today, not the unreleased rewrite) showed <java.version>21</java.version> - so the Java-25 finding was a false alarm caused by reading the wrong branch, not a real blocker. Worth recording as a caution: a GitHub repo's default-branch pom.xml is not necessarily what's on Maven Central.
2. The real, published cobalt:0.0.10 pulls in com.aspose:aspose-words as a hard compile-time dependency (used internally for generating link-preview thumbnails from Word documents) - Aspose.Words for Java is commercial/proprietary-licensed, not open source, so bundling it would violate the same license constraint Cobalt itself was chosen to satisfy. The full dependency graph (~15 jars: zxing, qr-terminal, curve25519, protobuf-base, jackson or fastjson2 depending on version, libphonenumber, dd-plist, apk-parser, link-preview, jaffree, ez-vcard, slf4j, plus Aspose) would all need to be manually downloaded and bundled under this module's libs/ folder - BoxLang modules have no Maven-style dependency resolution of their own (confirmed via the BoxLang docs MCP: third-party jars are bundled directly into a module's libs/ folder, loaded by a per-module classloader - there's no javaLibraries key in box.json that would resolve a dependency tree automatically).

Given the license contamination via Aspose and the manual-fat-jar-assembly effort with no dependency-resolution tooling available to verify the result actually loads, WhatsApp Personal was descoped rather than shipped as either a real gateway or a stub. ProjectValidator's validGatewayTypes and GatewayGenerator's TYPE_CLASS_MAP do not include a whatsapp-personal entry - a project that tries to declare one gets the existing "unknown gateway type" validation error, same as any other unsupported type, rather than a misleading half-built stub. Revisiting this is reasonable if Cobalt ever drops the Aspose dependency (it's a narrow feature - link-preview thumbnailing of Word docs - not core to messaging), or if a future session decides the Node/Baileys subprocess-bridge approach (rejected this round on architecture preference, not on a technical blocker) is preferable after all.

GatewaySession is project-wide and root-agent-only (v1)

A project with at least one push-style gateway entry gets exactly one generated GatewaySession, bundling every push-style gateway and always bound to the project's root agent - matching the existing precedent that exposes: "agent" HTTP exposure is also always root-agent-only (see gateways/). A project with subagents cannot yet route different gateways to different subagents (e.g. "Telegram talks to SupportBot, Slack talks to ResearchBot"). A future per-gateway targetAgent: "SubagentName" key, consumed by a per-agent-node GatewaySession instead of one project-wide session, is the natural extension point - not built yet.

Fixed: GatewaySessionBootstrap.bx looked gateways up by the wrong aiGatewayRegistry() key (shipped broken across all four push-style gateways, caught during the WhatsApp research pass)

A real, previously-shipped bug: the generated interceptor's aiGatewayRegistry().get(...) call used the discovered gateways/* entry's own filename (e.g. "telegramChannel", from gateways/telegramChannel.bx), but bx-ai's real GatewayRegistry.register() always keys by the gateway CLASS's own fixed getName() (e.g. "telegram", set once in TelegramGateway.init()) - never anything caller-supplied. Confirmed both by reading bx-ai source directly and empirically (registering a real gateway, then calling .get() with its discovered entry name, threw "No item found in registry"). This meant GatewaySession construction would throw at ColdBox boot (afterConfigurationLoad) for every generated project with a push-style gateway - Telegram, Slack, Discord, and Email all shipped with this bug; it went undetected because the only prior test coverage asserted on the generated file's raw STRING CONTENT, never against a live registry.

Fixed in GatewayGenerator.generate(): the interceptor now looks gateways up by their TYPE string (which is always identical to the registered name, for every push-style gateway built so far), deduplicated. GatewayGeneratorSpec.bx gained a permanent regression test that registers a REAL gateway instance and proves the EXACT key the generator just emitted resolves it via a live aiGatewayRegistry() - closing the exact gap that let this ship undetected the first time.

A real, permanent consequence this fix surfaces (not new behavior, just now correctly reachable): because the registry is keyed by type, not by entry, two gateways/* entries of the same push-style type collide on the same registry slot project-wide - e.g. two type: "telegram" entries (two different bot tokens) would silently have the second registration overwrite the first, and GatewaySession would only ever see one of them. There's no per-entry alias/registration-name override today. One instance per push-style type, per project, is the actual v1 ceiling - not documented as such before this fix made it visible.

./gradlew downloadModules may fetch a bx-ai snapshot that's temporarily behind GatewayGenerator's aiGatewayRegistry() codegen

bx-ai renamed gatewayRegistry() to aiGatewayRegistry() on its development branch (confirmed directly - bifs/gatewayRegistry.bx was deleted outright, no back-compat alias) - GatewayGenerator was updated to match, since bx-ai hasn't cut a release yet and this project's own instruction was to follow it straight, not shim around it. The catch: downloadModules fetches a fixed, continuously-republished snapshot artifact (bx-ai@3.4.0-snapshot) from downloads.ortussolutions.com, and that published zip can lag behind bx-ai's own git development HEAD by some amount of time (confirmed directly this session: right after this rename landed upstream, the downloadable snapshot still had the OLD gatewayRegistry.bx). If a fresh downloadModules pulls a snapshot that predates this rename, any project with a channel-adapter gateways/* entry will fail to boot with Function 'aiGatewayRegistry' not found, since the generated code now calls the new name but the fetched module still only has the old one. This isn't fixable from BX Agents' side - it resolves itself on its own once ForgeBox republishes the snapshot from current bx-ai development. Verified this project's own generated code is correct against bx-ai's true HEAD by building a local module structure directly from its git source (ortus-boxlang/bx-ai) rather than relying on the possibly-stale downloaded zip.

Reconfirmed the same lag independently while building the push-style gateway/GatewaySession work: the downloaded bx-ai@3.4.0-snapshot at the time still had no GatewaySession.bx, no aiGatewaySession()/aiGatewayRegistry() BIFs, and a BaseGateway.bx/IGateway.bx with no onMessage()/onError() at all - a snapshot from well before this same rename. testBx was run in this state by replacing src/test/resources/modules/bxai's bifs//models//public//ModuleConfig.bx with a fresh copy from bx-ai's own git source (its libs//box.json left untouched), the same workaround as above - not something build/serve end users need to do themselves once ForgeBox catches up, but necessary for this session's own CI-less verification.

CI now works around this automatically, because it had to. The first real GitHub Actions run of this suite proved the lag is not cosmetic: the published bx-ai@3.4.0-snapshot still ships bifs/gatewayRegistry.bx (renamed to aiGatewayRegistry upstream long ago) and contains no models/gateway/BaseGateway.bx whatsoever - the class all nine push-style gateways in this module extend. Tested against it, 41 specs fail for reasons unrelated to this module's own code (The method aiGatewayRegistry does not exist, then every gateway spec cascading off a base class that is not there). .github/workflows/tests.yml therefore clones bx-ai's development branch, runs its createModuleStructure, and overlays the result over whatever downloadModules fetched - the same manual workaround described above, automated. bx-ftp and bx-sqlite are stable releases and still come from downloadModules untouched. Delete the overlay step once a published snapshot catches up; until then, note that CI is testing against bx-ai's branch HEAD rather than a pinned artifact, so an upstream breakage surfaces here as a bx-agents failure.

Hit a third time while wiring up /compact: bx-ai scoped IAiMemory.summarize() by userId/conversationId on development (commit f9ac7bd), but a fresh downloadModules still fetched a snapshot carrying the old single-argument summarize( struct config = {} ). Same workaround - the module was rebuilt from bx-ai's own git source (./gradlew createModuleStructure in that repo, copied over src/test/resources/modules/bxai) and the scoped behaviour verified against that build directly. Unlike the earlier two, this one has a runtime consequence an end user can hit: /compact calls mem.summarize( config, userId, conversationId ), and on a bx-ai predating that commit the extra arguments are simply ignored, so compaction would summarize the memory instance's default scope rather than the caller's conversation. A generated app therefore needs bx-ai at or past f9ac7bd for /compact specifically; every other route is unaffected. Nothing in this repo's own test suite regresses on an older snapshot either, since the web UI specs assert on generated source text rather than executing it.

The v1 web chat UI (exposes: "webui") - what's real, and what could only be checked against docs, not a live server

WebUiGeneratorSpec.bx and a BuildPipelineSpec.bx end-to-end test both drive the real WebUiGenerator/BuildPipeline classes against real fixtures: the static <path>/index.html shell is confirmed to be written and correctly templated (__API_BASE__/__APP_TITLE__ placeholders substituted, never left in the output), the optional interceptors/WebUiAuthGate.bx is confirmed to be generated only when apiKeyEnvVar is configured, to gate exactly <path>/api/* and never the bare <path> shell itself, and to be correctly registered into config/ColdBox.bx's interceptors:[...] list end-to-end through the real BuildPipeline. The generated interceptor was also confirmed this session to compile and instantiate cleanly via a standalone smoke test (loaded through the same DynamicClassLoader primitive the build pipeline itself uses).

What was NOT verified - and couldn't be, in this dev environment: a real bxAgents serve + real browser test of the page actually loading, streaming a reply, and the X-API-Key gate actually rejecting/accepting requests over real HTTP. This project's own runColdBoxIntegrationTests.bxs/tests/coldbox harness (the same one that proved toAi()'s /invoke route works end-to-end for http-gateway-agent) requires tests/coldbox to be present via a real box install inside tests/ - CommandBox and ForgeBox network access were both unavailable in this session's sandbox, so that harness could not be exercised for the web UI (or re-confirmed for anything else) this session. Two direct consequences:

• The exact /invoke JSON response shape used by this project's own generated pages/docs ({"input": "..."} in, a response containing "success": true) is empirically confirmed only via runColdBoxIntegrationTests.bxs's own prior, already-passing assertion (a substring check, not a full shape assertion) - not re-verified this session.
• The /stream SSE wire format the web UI's own JS parses (data: {"token":"..."} lines, terminated by data: [DONE]) is taken directly from ColdBox's own official "AI Routing" documentation, not independently re-confirmed against a live server this session - unlike almost every other wire-format claim in this project's docs, which were cross-checked against real running code wherever possible (see e.g. the HMAC-SHA256/SHA1 signature schemes' independent Python/openssl cross-checks elsewhere in this file).

Do at least one real bxAgents serve + a real browser test (message sent, reply streams in, the X-API-Key gate actually 401s a request missing the key) before depending on this feature in production - the same standing advice already given for every generated webhook handler's own untested-against-a-real-boot gap elsewhere in this file.

The web UI's SQLite store - verified at the library level, not through a real ColdBox boot

The qb + bx-sqlite stack underneath models/ChatDb.bx was verified directly this session against the real jars, not inferred from docs: qb 13.1.0's .cfc sources compile and run natively on BoxLang 1.16 with no bx-compat-cfml; SQLiteGrammar + SchemaBuilder really do create the v1 tables and indexes against a real SQLite file; a second migration pass is a clean no-op; QueryBuilder round-trips inserts, filtered/ordered reads and deletes; and the preferences composite primary key really does reject a duplicate (userId, prefKey). Two constraints were found this way rather than assumed - qb requires a named datasource (its own appendSqlComments() types that argument as string, so an inline struct throws before any SQL runs), and SchemaBuilder@qb is mapped with its grammar argument only, never receiving moduleSettings.qb.defaultOptions - and the generated code is shaped around both.

What was not verified, for the same reason as the rest of the web UI: none of this has run inside a real ColdBox boot. tests/coldbox requires a real box install, and CommandBox/ForgeBox network access was unavailable in this session's sandbox. So the migration logic is proven, but three wiring assumptions are not exercised end to end: that getInstance( "ChatDb" ) resolves through the generated app's own WireBox, that SchemaBuilder@qb/QueryBuilder@qb resolve once qb is installed as a real ColdBox module (qb is a box.json dependency, not vendored here - the same honest gap cbmailservices already carries), and that this.datasources in the generated Application.bx is picked up as expected. Do one bxAgents serve against a webui project with qb and bx-sqlite genuinely installed before relying on the store in production; the failure mode if any of those is missing is loud at boot (an unresolvable WireBox mapping or an unknown JDBC driver), not silent.

The web UI over real HTTP: proven by the integration runner's probe

runColdBoxIntegrationTests.bxs fetches GET /chat/api/health from outside the server, against the generated fixture app booted by a real boxlang-miniserver, and fails the build if it is not a 200. On CI it returns:

+ App probe GET /chat/api/health -> status=200
  body: {"success":true,"status":"ok"}

That single request is the end-to-end proof for the web UI's server side: ColdBox routing reaching handlers/ChatUi.bx, WireBox resolving the handler and ChatDb, this.datasources giving bx-sqlite a usable SQLite file, and qb being a genuinely activated ColdBox module - none of which any source-text assertion can establish. WebUiRuntimeSpec.bx then covers the store's behaviour (migrations, scoped conversation CRUD, the cross-user guard, preference upserts) by reading real objects out of WireBox inside that same boot.

What is still not covered: the other webui routes over HTTP. Only /health is fetched from outside the process. Driving the rest from WebUiRuntimeSpec is not possible as structured, because that spec runs inside a request served by the same single miniserver, so a loopback call would compete for the worker pool the runner itself occupies. Covering them needs either a second server process or a larger miniserver thread pool, and is a natural follow-up rather than something to fake.

Note that this paragraph previously cited runner-coldbox.bxm never responded: HTTP 408 as evidence of that starvation. That attribution was wrong, and is worth recording because it misled this project for several CI cycles: the 408 had nothing to do with loopback calls. The runner page called chr( 10 ) - a BIF that does not exist - so it returned 500 on entry to every request, and the orchestrator's retry loop ground through 90 seconds of those 500s before reporting the final timeout. Both are fixed. The loopback-starvation concern above is a genuine structural reason not to drive more routes from inside that spec, but it remains a well-founded expectation rather than something that has actually been observed failing here.

The web UI's own front-end: driven in a real browser, but against a mocked API

The shipped page's JavaScript was exercised for real this session, not just asserted against as source text: the generated index.html was loaded in headless Chromium and driven end to end with every <path>/api/* route intercepted and answered with realistic payloads. Confirmed working that way - the conversation sidebar rendering from GET /conversations and switching/renaming/deleting through its own routes; GET /info shaping the toolbar (Compact appearing only when capabilities.compact is true, the model name landing in the header); the theme arriving from server-side preferences and applying; /history rehydrating the transcript; a real SSE turn streaming content, reasoning and tool-call chips out of the bx-ai envelope; markdown rendering; and New chat creating a conversation server-side and opening it. Zero JavaScript errors, zero unreplaced __TOKEN__ placeholders, and the narrow-screen layout was screenshotted at 390px.

What that does not prove: the API was a Playwright mock, not the generated handlers/ChatUi.bx running under a real ColdBox boot against a real SQLite store. The request and response shapes were taken from the generated handler's own source, so a drift between the two would not be caught by this. Everything in the known-limitations entry above about the store applies here too - one manual bxAgents serve against a webui project with qb and bx-sqlite genuinely installed remains the honest gate before production use.