Tool Calling: The parser turns text into function calls, and strict binds the arguments

Status: Draft (2026-10-27), Empirical. Measurements were taken using a single 8GB-class laptop GPU (Ada generation), vLLM 0.31.0, and Qwen3-Reranker-0.6B (a Qwen3 0.6B causal language model fine-tuned for reranking). Official documentation was checked on 2026-10-27 based on version 0.32.0 (released 2026-10-18), and differences from the 0.31.0 documentation are noted separately. Added 2026-10-27. Results re-measured with conversational Qwen3-0.6B, Qwen3-0.6B-GPTQ-Int8, Qwen3-4B-AWQ, and vLLM 0.32.0 have been added to Section 3-1. The previous figures measured with 0.31.0 remain unchanged. Writing rules: Empirical data is the default. Unmeasured items are marked as [Unmeasured] and unverified items as [Unverified].

Table of Contents

• 1. Tool calling is text in a predefined format
• 2. Four types of tool_choice
  • Shape and arguments are bound separately
• 3. Direct Measurement
  • Environment and Method
  • Results
  • How to Read
• 3-1. Re-measurement on 2026-10-27: Tool-aware models and 0.32.0
• 4. Takeaways for Operations
• 5. Unverified Items
• References

1. Tool calling is text in a predefined format

Imagine a scenario where an agent calls a weather API. There is no special output device called a "function call" inside the model. The model still generates tokens one by one, and it simply learned during training that "when calling a tool, write it in this shape." The Qwen series uses the Hermes format.

<tool_call>
{"name": "get_weather", "arguments": {"city": "Seoul", "unit": "c"}}
</tool_call>

The server performs three tasks:

1. It inserts the list of tools into the prompt. The model's chat template writes the request's tools into the system prompt as text.
2. It selects the text written by the model. The message.tool_calls tool parser is responsible for finding the above shape in the generated text and translating it into an OpenAI-format. Since shapes vary by model, there are multiple parsers such as hermes, llama3_json, mistral, qwen3_xml .
3. It enforces the shape if necessary. tool_choice If required is

or a specific function name, the structured output from Part 4 enforces the shape of the call at the token level. --enable-auto-tool-choice The options to enable this are as follows. If tool_choice="auto" is absent,

vllm serve <모델> --enable-auto-tool-choice --tool-call-parser hermes

requests are not accepted.

The official documentation states regarding named calls: "Guarantees a parsable call, but does not guarantee a high-quality call." This sentence neatly summarizes the empirical results of this article.

Shape and arguments are bound separately

Starting from version 0.21.0, xgrammar introduced structural tags to enforce tool calls, and current documentation divides enforcement into two layers.

• Envelope: <tool_call> : Markers like and function names. Always enforced for required and named specifications.
• Argument Schema: Enforced only when "strict": true is attached to the tool or the server is launched with --tool-strict-level parameter . Otherwise, arguments are bound only as something "resembling JSON".

The server-side baseline --tool-strict-level has three options: auto(default, follows request), function(enforces envelope on all requests), and parameter(treats all tools like strict). Since most agent frameworks do not attach strict , this allows operators to raise the baseline. The environment variable VLLM_ENFORCE_STRICT_TOOL_CALLING=false completely disables structural tags.

3. Direct Measurement

Environment and Method

• GPU: Single 8GB-class laptop GPU (Ada generation), Driver 595.99, CUDA 13.0 build
• vLLM 0.31.0 official container, PyTorch 2.13.0, xgrammar 0.2.7, llguidance 1.7.6
• Model: Qwen3-Reranker-0.6B (bf16). Since it is a model fine-tuned for reranking, it has almost no conversational capability. When asked via chat, it usually replies "No." Please read this test not as evaluating "a model that is good at using tools," but as evaluating " how far the server can support a small model that doesn't know how to use tools".
• Server: --enable-auto-tool-choice --tool-call-parser hermes, others default
• Request: "What is the weather in Seoul right now?", one tool get_weather(city: string, unit: "c"|"f"), both fields required, temperature 0.7, max_tokens 200, five times with seeds 0-4. Sent the same request without strict, and with strict: true and additionalProperties: false attached.
• "Schema followed" is the result of the client reading arguments as JSON and checking whether city is a string and unit is either c or f.

Results

Transfers several raw responses as-is.

• auto: All five times, the body was "No." and no tool was called. auto , the model simply chose not to call it.
• required, no strict: The first request produced <tool_call> inside "location": "1801, 401, 181, 182, …" , writing out-of-schema fields and number sequences all the way to the 200-token limit until it was cut off, preventing the parser from moving it to tool_calls (1,180ms). The remaining four were {} three times and {"city": "Hawaii"} once. The function names were correct, but the required field unit was never present.
• required, with strict: {"city": "Baltic", "unit": "c"}, {"city": "Berlin", "unit": "c"}, {"city": "Hawaii", "unit": "f"}, {"city": "none", "unit": "f"}, {"city": "New York", "unit": "f"}.

With the same seed, required and the named output were identical down to the character. This is because having only one tool causes both approaches to generate the same grammar.

The server counts this as well. After the test, /metrics of vllm:tool_call_parser_invocations_total was outcome="tool_call" 18, outcome="no_tool_call" 12 in non-streaming chat requests. This matches the values counted by the client (18 responses with tool_calls, 12 without).

How to Read

1. required without strict only guaranteed that "a call exists." While the function name was correct, the arguments were either an empty object or missing required fields. Both the 0.31.0 and 0.32.0 documentation state that "tools without strict accept only a loose argument grammar even with required and naming," and actual measurements matched this statement.
2. Enabling strict followed the schema, but the semantics were incorrect. All arguments were parsed as JSON and adhered to the required fields and enum values, but Seoul never appeared in the questions, and values like "none" also appeared. Grammar only restricts the remaining tokens to "allowed tokens"; the model still decides which token to pick among them.
3. The time differences are mostly due to token counts. The strict side took about 50ms longer, but {} also included the increased token count from using both fields, so it should not be interpreted purely as the cost of grammar validation. The values isolating just the grammar cost are in Part 4.

3-1. Re-measurement on 2026-10-27: Tool-aware models and 0.32.0

The previous test examined the "scope enforced by the server" using a model unaware of tools. This time, we look at the content using a model trained for tool calling.

• The environment is the same except vLLM was updated to the official 0.32.0 container. The models are Qwen3-0.6B (bf16) for conversation, the GPTQ 8-bit version of the same model, and the AWQ 4-bit version of Qwen3-4B. The thought process was disabled using chat_template_kwargs: {"enable_thinking": false} .
• Request: "What is the weather in {city} right now? Answer in celsius." The six cities are Seoul, Paris, Tokyo, Cairo, Lima, and Oslo. "Pass" means the argument city contains that city and unit is c.

• Tool-aware models called the tool every time even with auto, and arguments were correct even without strict. This contrasts with the reranker model, which barely followed the schema when strict was enabled and still got the city wrong.
• For 0.6B, enabling strict increased response time for required and named by about 7ms and auto by 22ms. Since auto does not use grammar at all unless strict is enabled, the difference is larger. Because the generated argument text was identical regardless of strict, this difference can be attributed to the cost of the grammar mask.
• Re-running the previous test with the reranker model in the same 0.32.0 container yielded identical results down to the response characters as 0.31.0 (schema compliance for required without strict: 0/5, with strict: 5/5; cities: Baltic, Berlin, Hawaii, none, New York).

This test is relatively simple with a single tool and a short prompt. Accuracy with multiple tools or complex arguments remains unmeasured.

4. Takeaways for Production

• For agent servers, raise the baseline. Rather than relying on clients to attach strict, running with --tool-strict-level parameter ensures that all tool arguments adhere to the schema. Conversely, if there are many tools with deep or large schemas, a grammar generation cost (initial request latency in Part 4) is incurred, so you should evaluate the latency of incoming requests accordingly.
• Arguments are always untrusted input. Even arguments that follow the schema "none" containing nonsensical values. The side executing the tool must recheck the range and permissions of values, and dangerous arguments like file paths, SQL, and shell commands must be accepted only via allowlists. You should also design assuming that prompt injection—where external text (web pages, documents) entering the prompt encourages the model to invoke tools—cannot be prevented for the same reason.
• We treat auto failing silently as a metric. auto If the model messes up the format in , it does not return an error but rather plain body text. If the above vllm:tool_call_parser_invocations_total in no_tool_call 's ratio is placed on a dashboard, it becomes immediately visible when the model, parser, or chat template changes. However, since this metric cannot distinguish between "a tool was needed but not called" and "no tool was needed," it should be viewed alongside application-side expectations.
• Parsers are paired with models. Qwen2.5 and Qwen3 use hermes, Qwen3-Coder uses qwen3_xml, and Llama 3.1 uses llama3_json , using the parser specified in the model documentation. Version 0.32.0 includes a response_template parser that reads the checkpoint's hf to interpret it without model-specific parsers, but the documentation notes that this parser does not support strict, required, or naming specifications (untested).

5. Unverified Items

• The accuracy of models trained on tool calling was measured only with a single tool and short questions in Section 3-1. Multiple tools or long conversations are untested.
• Parallel calling to invoke multiple tools at once, tool_calls fragments in streaming, and --tool-strict-level server options are untested.
• The hf parser and parallel_tool_calls=false enforcement in 0.32.0 are untested.

References

• vLLM Documentation, Tool Calling (0.32.0, 0.31.0): https://docs.vllm.ai/en/latest/features/tool_calling.html
• vLLM 0.21.0 Release Notes (strict tool calling with structural tags): https://github.com/vllm-project/vllm/releases/tag/v0.21.0
• Qwen Documentation, Function Calling and vLLM: https://qwen.readthedocs.io/en/latest/framework/function_call.html
• Part 4 of this series: Structured Output
• Raw Data: The "Raw Data" section in the README of this series' folder