An agent that calls tools through text makes two format choices: the encoding of the calls it writes, and the encoding of the tool results and state it reads back. JSON is the default in most function-calling interfaces; XML-style tags are common in prompts and agent frameworks. We wanted to know whether the choice matters in multi-step workflows where every call changes state, and whether the two directions interact.
Across three models, no format came out ahead everywhere. What a model writes and what it reads interacted more strongly than either choice alone, in opposite directions for different models. For one model, most failures traced to how our client read a reply that arrived as two messages, not to the format.
The test
We built 48 workflow templates, 12 in each of four families: allocating stock to orders, changing dependent settings, multi-hop lookups, and provisional transactions with rejection and recovery. Each template has an ordinary variant and a stress variant whose protected strings carry ampersands, angle brackets, quotes, backslashes, line breaks and non-ASCII text.[1] Each of the 96 cases ran once in five conditions:
jj: the model writes JSON, the environment speaks JSONxj: the model writes XML, the environment speaks JSONjx: the model writes JSON, the environment speaks XMLxx: XML both wayssj:jjplus a requested JSON-schema response format
That is 480 workflows per model; a denominator of 95 below marks one workflow with an
unknown outcome. A workflow allows up to 8 model turns and 64 tool calls. A batch that
fails to parse runs nothing and gets one format-error retry; a second failure ends the
workflow. Both formats use one strict codec: no repair, no Markdown fences, no extra
text, exact string preservation. A workflow succeeds only if the final state matches
the gold answer exactly, every unrelated value survives, every transition was legal,
and the model called finish. Scoring is deterministic, with no model acting as
judge.[2]
The three models were GPT-6 Luna, GLM-5.3 and Muse Spark 1.3 Contributor, each at high reasoning effort, queried through hosted commercial API endpoints from 28 October 23:02 to 29 October 08:59 UTC.[3] A fourth model rejected requests in the pilot and was held before main collection, so none of its main workflows were sent.
The format effect
Each contrast below is a difference in task success, in percentage points, with an unadjusted 95% interval from resampling whole templates:
XML minus JSON averages the XML-output effect over both environments.[4] The estimates ranged from 0.0 to +3.4 points, and only GLM-5.3’s interval excluded zero. The interaction measures how much more XML output helps when the environment is also XML, and it was larger. GLM-5.3 did best writing XML into an XML environment (92 of 96, against 84 of 95 writing JSON into it). For GPT-6 Luna the pattern reversed: XML output did better than JSON in a JSON environment (94 of 96 against 92 of 96) and worse in an XML one (92 of 95 against 95 of 96). Muse Spark 1.3’s estimate leaned the same way as GLM-5.3’s (+9.4, interval −1.0 to +19.8), but see the next section. Requesting a JSON schema did not reliably help any model.
With nine intervals, some could exclude zero by chance. After a Bonferroni adjustment, only GLM-5.3’s +9.0 interaction still does.
Where the errors came from
Format errors varied far more between models than between formats. Each column counts the workflows, of 96, with at least one reply that failed to parse:
GPT-6 Luna’s 9 failed replies were spread across all five conditions. GLM-5.3’s 30 were
mostly XML: 26 came from the two XML-output conditions, half of them malformed
(mismatched or unclosed tags) and half well-formed but off contract, mostly array
entries in elements other than item. The other 4 were JSON missing a delimiter. The
single retry repaired nearly all of them: every GPT-6 Luna workflow and all but 3
GLM-5.3 workflows reached finish.
Muse Spark 1.3’s errors had one dominant cause, behind 680 of its 682 failed replies. Its replies often arrived as two
assistant messages: a short progress note labeled with the phase commentary
(“Round-robin allocation totals 8 units — reserving the consolidated batch next.”), then
a message holding the calls. Our client joined the text of every assistant message
without checking the phase, so the strict parser saw a sentence followed by JSON or XML
and rejected it. Every reply
with a commentary message failed; 822 of the 823 without one parsed. It finished 230
workflows, all correctly, and the other 250 stopped at the retry limit.
Reading the reply’s labels
We re-parsed every stored reply from all three models with the same frozen codec,
keeping only messages not labeled commentary. No model was called.
- GPT-6 Luna: 2 failed replies become valid and 2 valid ones break, because in those the calls sat in the commentary message. It nets to zero.
- GLM-5.3: every reply is a single unlabeled message, so nothing changes. Its 30 errors are format errors in the ordinary sense.
- Muse Spark 1.3: 678 of its 682 failed replies parse, and no valid reply breaks. Workflows with a format error fall from 432 to 4.
The prompt asked for the calls and nothing else, so Muse Spark 1.3’s commentary is an instruction-following lapse; but it arrived in its own labeled channel, and a client that honored the label would have parsed almost every reply. Its contrasts above largely reflect how often each condition’s workflows hit that parsing failure; its failed replies were spread almost evenly, 126 to 143 per condition.[5]
What we take from it
On this one synthetic benchmark, with one run per cell and AI-written gold, neither JSON nor XML was better in general. The interaction between what a model writes and what it reads mattered more than either choice and changed sign between models, so the two encodings are worth testing together, with the model in question. Format errors had different causes in different models: mostly XML syntax and contract errors that one retry nearly always repaired in one, notes in a separate channel that our client ignored in another. The right parsing rule depends on each service’s response-channel contract: honoring the label made almost every Muse Spark 1.3 reply parse and broke two GPT-6 Luna replies.[6]
Footnotes
-
The 48 templates are synthetic, authored by an AI agent, and cover four families. Each cell ran once, so the intervals describe variation across these templates. Tool argument schemas are JSON Schema in every condition. The schema arm has no matched XML constraint. ↩
-
An AI coding agent wrote the tasks and the 140 gold answers from the public task files only, without seeing the runtime or any model output, and a separate agent reviewed all 140 while blinded to outcomes, finding no concrete defect; its recorded serving model was GPT-6 Astra for 48 of its 50 messages and Muse Spark 1.3 Contributor for the other 2. A blinded human check of the gold has not been completed. The task and gold authors were recorded as GPT-6.1 Sol, and GPT-6 Luna appears by name to share a model family with them and with the main reviewer; we cannot verify this. Shared conventions could make tasks easier for that family, which affects absolute levels and cross-model comparisons more than within-model contrasts, since every condition is scored against the same gold. ↩
-
GPT-6 Luna and GLM-5.3 main collection ran first, from 28 October 23:52 to 29 October 04:15 UTC, in three segments split by a transport interruption and a local storage interruption; no attempted workflow was retried. Muse Spark 1.3 was added after the fourth model was held, before any of its requests; its main collection ran afterward from 04:51 to 08:24. Backend identity and version were not attested, GPT-6 Luna’s requested output cap was not forwarded, and sampling settings differed by interface: the chat-style requests to GLM-5.3 set temperature 0 and top_p 1, while the Responses-style requests to GPT-6 Luna and Muse Spark 1.3 used endpoint defaults. Retries or fallbacks inside the serving stack were not observable, and echoed request parameters did not always match those sent. Absolute differences between models may include serving differences; within-model contrasts do not. Three workflows (1 GPT-6 Luna, 2 GLM-5.3) have unknown outcomes and are not counted as failures. ↩
-
XML minus JSON is (
xj−jj+xx−jx) / 2; the interaction is (xx−jx) − (xj−jj); schema minus raw JSON issj−jj. Each is averaged over the two variants of a template, then over templates within a family, then equally over the four families. Intervals come from 10,000 resamples of 12 templates per family, with the Bonferroni adjustment taken over all nine. We computed no p-values. ↩ -
Muse Spark 1.3 also produced far more output per request (requests with known usage): 692.4 tokens against 89.2 for GPT-6 Luna and 116.3 for GLM-5.3, with reasoning making up 88.2%, 41.0% and 50.0% respectively. The excess is reasoning, not commentary. ↩
-
A code and data release is forthcoming. ↩