Structured Outputs: The Cost of Masking "Illegal Characters" Per Token

Status: Draft (2026-10-27), empirically measured. Measured on 2026-10-27 using a single 8GB-class laptop GPU (Ada generation), vLLM 0.31.0 (xgrammar 0.2.7, llguidance 1.7.6), and Qwen3-Reranker-0.6B. Official documentation was checked for 0.32.0, and the structured output documentation was identical to 0.31.0. Added 2026-10-27. Results from measuring an extraction task with correct answers using conversational Qwen3-0.6B and Qwen3-4B (AWQ) with vLLM 0.32.0 (xgrammar 0.2.8) have been added to Section 3-1. The 0.31.0 figures remain unchanged. Writing conventions: Empirical data is the default; unmeasured items are marked as "Unmeasured", and unverified items are marked as "Unverified".

Table of Contents

• 1. Principle: Filtering Out-of-Grammar Candidates from Next Token Predictions
• 2. How to Use and Evolution of Names
• 3. Direct Measurement
  • Method
  • Result 1. When Enforced, the Only Failures Were Truncations
  • Result 2. Costs: The First Request and a Small Per-Token Overhead
  • Incident: A Few Seconds on the First Sampling Batch
• 3-1. Re-measurement on 2026-10-27: Extraction Task with Correct Answers
• 4. Production Takeaways
• 5. Unverified Items
• References

1. Principle: Filtering Out-of-Grammar Candidates from Next Token Predictions

At each step, the model computes scores for the entire vocabulary (151,669 tokens for this model) and selects one. Structured outputs insert a step just before selection: the text generated so far is fed into a grammar machine (a finite state machine or pushdown automaton) to obtain the set of "tokens allowed to appear now," and the scores of all other tokens are overwritten with negative infinity. This set is converted into a bitmask and applied to the GPU score tensor all at once.

지금까지: {"priority": "
문법상 다음에 올 수 있는 것: l(ow), m(edium), h(igh) 로 시작하는 토큰만
나머지 151,000여 토큰 → 점수 -inf

Consequently, no matter how erratic the model is, the output adheres to the grammar. However, this introduces two types of costs.

1. Grammar compilation cost: The process of converting a JSON schema or regular expression into a token-level machine. Because the vocabulary has 150,000 tokens, pre-calculating whether "this token is allowed in this state" for every token is heavy. Identical schemas are cached.
2. Per-step mask generation cost: For each request, the state advances by one step, a mask is generated, and it is sent to the GPU.

The XGrammar paper (Dong et al., MLSys 2025) reports that by pre-computing tokens determined independently of the context and overlapping grammar computation with GPU execution, it is up to 100x faster than traditional methods, and when used with a serving engine, the overhead is nearly negligible (based on the abstract; conditions are not specified in the abstract). vLLM uses both xgrammar and llguidance (guidance backend), and the default value auto chooses based on the request.

2. How to Use and Evolution of Names

There are two formats: the OpenAI-style response_format and the vLLM-exclusive structured_outputs .

The naming changed as the API evolved. The old guided_json, guided_regex, guided_choice, guided_grammar and per-request guided_decoding_backend were removed in 0.12.0, and they are now consolidated into a single structured_outputs . The backend is determined server-side via --structured-outputs-config.backend , and the ability to choose a backend per request was removed in V1. Regular expression grammars differ by backend: xgrammar and guidance use Rust-style syntax, while lm-format-enforcer uses Python re syntax. Starting from 0.32.0, JSON schemas nested deeper than 64 levels are rejected with a 400 error (based on release notes, unmeasured).

3. Direct Measurement

Method

• Environment: A single 8GB-class laptop GPU (Ada generation), vLLM 0.31.0, Qwen3-Reranker-0.6B (bf16). When asked conversationally, this model almost always answers "No." Therefore, the difference between unforced and forced outputsis very clearly visible.
• Request: Using the chat API, asked to "Answer only with JSON containing the keys name, priority (low/medium/high), tokens (integer), and tags (list of strings) for a single task," and temperature 0.7, max_tokens 160, providing a different seed for each request.
• Variants: No enforcement, json_schema(four required keys, no additional keys, string length upper bound), json_object, choice, regex.
• Server: Launched three instances with the backends set to auto(default), and guidance, xgrammar .
• Iteration: Concurrency 1 ran five batches (first batch as warmup), concurrency 16 ran four batches (first batch as warmup). "Correctness" was judged by parsing the response as JSON to verify that the four keys were precisely present and had correct value types.

Result 1. When Enforced, the Only Failures Were Truncations

Totals for concurrency 16 across three measurement batches (48 requests).

• All incorrect responses in json_schema were due to finish_reason being length . In guidance and xgrammar, responses were cut off before the JSON could close due to hitting the 160 token limit twice each (the auto server ran before this item was logged, so the cause was not recorded). Grammar restricts "next tokens" but does not guarantee "timely completion." You must set max_tokens generously relative to the schema's maximum length, and clients must always verify finish_reason .
• json_object only enforces JSON grammar. The sampled output was syntactically valid JSON, but lacked any of the requested keys. If keys matter, use json_schema.
• Correctness and usefulness are different things. Most responses adhering to the schema contained empty names, 0, or empty lists. Grammar cannot make a model do something it does not know how to do.
• The output token counts for json_schema responses on the auto server matched those of the fixed xgrammar server batch by batch (28, 35, 32, 27 at concurrency 1). We infer that auto selected xgrammar for these requests (unconfirmed as selection logs were absent).

Result 2. Costs: The First Request and a Small Per-Token Overhead

• The first request for a new schema is the most expensive. From the second request onwards for the same schema, TTFT became identical to the unforced case (approx. 14ms). The cost of the first request included kernel preparation time on the server alongside grammar creation, making it impossible to separate the two.
• Token intervals increased by about 7% (5.9 → 6.3ms) at single concurrency and about 17% (6.3 → 7.4ms) at concurrency 16. However, unforced responses consisted of only three tokens ("No."), resulting in a short baseline for comparison. The larger increase at concurrency 16 is likely due to generating masks for 16 requests at every step.
• Because regex and choice have small grammars, their first-request costs were nearly negligible.

Incident: A Few Seconds on the First Sampling Batch

During measurements, the following warning appeared in the server logs.

WARNING [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_topp_kernel.
This causes a latency spike; consider extending warmup to cover this shape/config.

When a batch of temperature sampling (top-k, top-p) requests arrived for the first time on a new server, the kernel was compiled on the fly, increasing the TTFT for that batch from 3.9 seconds (guidance server, concurrency 1) to up to 11 seconds (guidance server, concurrency 16). This warning did not appear while running greedy decoding exclusively. Furthermore, compilation did not happen just once. When sending requests with temperature 1.0 sequentially at concurrency levels 1, 8, and 32 to another server, that batch slowed down again every time the concurrency level changed for the first time (generating 256 tokens took 5.4s then 1.5s for concurrency 1; 7.4s then 1.9s for concurrency 8; 8.4s then 3.2s for concurrency 32). It appears new kernels are generated for each batch size. A warning was also logged stating that the model's generation_config.json overwrote the default sampling values (temperature 0.6, top_k 20, top_p 0.95). In production, servers should be warmed up with requests matching actual sampling configurations after startup before accepting traffic. The 0.32.0 release notes mention a reduction in sampler warmup compilation overhead (unmeasured).

3-1. Re-measurement on 2026-10-27: Extraction Task with Correct Answers

The previous test only checked for "valid JSON." This time, we also checked whether the values were correct.

• Environment: vLLM 0.32.0 official container. Models include conversational Qwen3-0.6B (bf16), its FP8 online quantized version, the GPTQ 8-bit edition, and the Qwen3-4B AWQ 4-bit edition. Thought processes were disabled.
• Task: Sent six sentences simultaneously, such as "The job called backup-db has high priority, needs 1200 tokens and should be tagged nightly and storage. Reply with JSON only." (each with differing job names, priorities, token counts, and tags), measuring across three batches (18 requests). The first batch was discarded as warmup.
• Metrics: "Immediate parseability" means the response parses as json.loads directly as received, while "Correct values" means all four values are correct after stripping code fences.

• The small conversational model got the values right but the format wrong. Without enforcement, Qwen3-0.6B always wrapped the output in markdown fences like json … . While readable for humans, programs cannot parse it directly. json_object solved this issue completely.
• Even the same model changed its habits depending on quantization. The FP8 version answered without fences 6 out of 18 times. This means parsers relying on model habits can break when changing models or quantization.
• The large model answered without fences. 4B was parsed correctly all 18 times even without enforcement.
• Structured outputs did not change the values. All three variants achieved 18/18 for values, partly because this task was easy.
• The cost trend was the same as in 0.31.0. For 0.6B, json_schema had a response about 19ms (7%) longer than no enforcement, and the first request for a new schema showed a TTFT of 195ms in the warmup batch.
• Re-running the previous test on 0.32.0 with the reranker model (concurrency 16, json_schema 47/48, one failure due to length limit) yielded identical results to 0.31.0. Kernel compilation delays in the first sampling batch were also present in 0.32.0 (5.3s for the reranker model, 6.2s for Qwen3-0.6B).

4. Key Takeaways for Operations

• Warm up schemas in advance. If the number of schemas used by your service is fixed, sending each one once right after server startup prevents the first user from waiting an extra few hundred milliseconds. Services where schemas vary per request (APIs where users submit schemas) will continue to incur this cost.
• User-submitted schemas are untrusted input. Since the cost of building grammars grows with the size and complexity of the schema, place upper limits on size and depth if you accept external schemas. This is likely why 0.32.0 started rejecting schemas exceeding a depth of 64.
• finish_reason Check . Structured output's "always valid JSON" applies only when the generation completes.
• If keys are required, use json_schema rather than json_object .
• Keep the backend set to default auto, but if first-request latency is a concern, test both backends with the same schema as in this measurement and choose. In this schema test, xgrammar's first request latency was half that of guidance, while subsequent values were similar.

5. Unverified Items

• The costs of large schemas (deeply nested objects, long enums), context-free grammars (EBNF), and structural tags were not measured.
• We could not isolate the cost of grammar construction versus kernel preparation in the first-request cost. The reason why the first json_schema batch at concurrency 16 took over 1.5 seconds on the auto and guidance servers, compared to 30ms on the xgrammar server, also remains unverified.
• Behavior when used with reasoning models (models that output a chain of thought first) (--structured-outputs-config.enable_in_reasoning) was not measured.

References

• vLLM Documentation, Structured Outputs (identical for 0.31.0 and 0.32.0): https://docs.vllm.ai/en/latest/features/structured_outputs.html
• Dong et al., "XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models", MLSys 2025. https://arxiv.org/abs/2411.15100
• llguidance: https://github.com/guidance-ai/llguidance
• vLLM 0.32.0 Release Notes (schema depth limits, sampler warmup): https://github.com/vllm-project/vllm/releases/tag/v0.32.0
• Part 5 of this series: Tool Calling