Status: Draft (2026-11-15), Measured. The figures in this post were measured directly by the author on 2026-11-15 using Kubernetes v1.34, Dynamo v1.6.2 vllm-runtime:1.6.2 image (vLLM 0.31.0), the Qwen3-Embedding-0.6B model, and a single 8GB laptop-class GPU. Since the comparison target is a custom server built by the author rather than a general-purpose product, these numbers should not be applied directly to other servers. Writing rule: Measured data is the default; unmeasured items are marked as "Unmeasured", and unverified items are marked as "Unverified".

Migrating an Embedding Server to Dynamo

Table of Contents

• 1. Why Embeddings?
• 2. Configuration
• 3. The Vectors Were Identical
• 4. Drop-in Replacement Was Not Possible
• 5. Latency and Throughput
• 6. What Cannot Be Migrated
• 7. Conclusion
• References

1. Why Embeddings?

Part 3and Part 4evaluated Dynamo using a test chat model. This time, we tested Dynamo against a service currently in production. However, upon inspecting the service in use, it turned out not to be a chat model. It was a custom FastAPI server running on a single GPU that handled embeddings for search (RAG), reranking, and speech recognition all within a single process.

Dynamo's flagship features are useless for this server. Because embeddings do not generate tokens, prefix caching, KV routing, and disaggregated serving do not apply. Dynamo's code says the same thing: the embedding worker's help documentation notes that it skips KV event and KV router registration (v1.6.2, components/src/dynamo/vllm/backend_args.py).

Thus, the question changes: Can we remove the custom server and swap it out for a Dynamo embedding worker, and what changes if we do?.

2. Configuration

The vLLM backend in Dynamo becomes an embedding-only worker using the --embedding-worker argument. On the vLLM side, we also provide --runner pooling .

command: ["python3", "-m", "dynamo.vllm"]
args:
  - --model
  - Qwen/Qwen3-Embedding-0.6B
  - --embedding-worker
  - --runner
  - pooling
  - --max-model-len
  - "1024"
  - --gpu-memory-utilization
  - "0.4"

The graph consists of a single frontend and a single worker. The model used the same revision pre-downloaded by the existing server, and the worker was launched with HF_HUB_OFFLINE=1 . --max-model-len 1024 is a value matched to the input limit of the existing server.

The comparison procedure was as follows.

1. Send requests to the existing server to collect vectors and latencies (3 iterations).
2. Shut down the existing server and deploy the Dynamo graph on the same GPU.
3. Send the same requests to the Dynamo frontend (3 iterations).
4. Tear down the graph and restart the existing server.

The requests consisted of 40 documents (Korean and English, averaging 84 tokens per document) grouped into batches of 8. This is because clients of the existing server actually send them in batches of 8. Latency was measured with 30 sequential requests and 4 concurrent requests for a total of 40 requests, using the median of 3 iterations.

3. The Vectors Were Identical

Token counts were exactly identical, and vectors diverged at the fourth decimal place. Because the existing server performs direct inference using transformers while the Dynamo side uses vLLM, execution order and precision handling differ. We did not separately measure whether this difference is enough to alter search rankings (Unmeasured). We cannot conclude solely from these numbers whether it is safe to mix them without rebuilding indexes.

4. Drop-in Replacement Was Not Possible

Both servers expose /v1/embeddings . Because their names are identical, it might seem like simply changing the endpoint address would suffice, but four things responded differently to the same request.

dimensions. The reason was left in the worker logs. vLLM views this model as not supporting Matryoshka embeddings and rejects requests containing dimensions . Existing clients send dimensions included in every request. Because even values matching the model's original dimension trigger the "must not be present" check, all requests will fail unless clients are updated. We understand there is an override option in vLLM to treat models as Matryoshka, but it was not tested in this post (Unmeasured).

Query instructions. This is more dangerous because it does not produce errors. Qwen3-Embedding is designed to have instructions prepended to queries, and the existing server handles this automatically by accepting input_type or instruction fields. The Dynamo side discards unknown fields and embeds the text verbatim. The cosine similarity of the vectors generated by both sides for the same query was 0.82 . Comparing them without instructions yields 0.9995. Simply changing the address causes all requests to return 200 while search quality silently degrades. Instructions must be explicitly prepended as text by the client.

Long inputs. The existing server truncates at the limit, whereas vLLM rejects inputs exceeding the limit. It is hard to say which side is correct. However, if clients have relied on the server to handle truncation, that responsibility must be brought over.

Error codes. dimensions and long inputs both represent malformed client requests, but the frontend returned 500 along with Internal server error . The worker logs contained vLLM's validation errors along with the reasons. Since many clients treat 500 as retryable, this would cause unfixable requests to be continuously retried. This behavior was observed in v1.6.2; whether it has changed in the latest version remains unverified.

5. Latency and Throughput

Values measured on the same GPU (8GB class). These represent the latency for a single batch of 8 requests, taking the median of 3 iterations. Error counts were 0 for both sides.

Sequential latency is 0.68x, and concurrent 4 throughput is 2.8x. Per-iteration throughputs were 139.4, 141.0, and 141.2 docs/s for the existing server, and 409.1, 395.8, and 351.9 docs/s for Dynamo.

The reason behind the performance under 4 concurrent requests appears to lie in the architecture of the existing server. It wraps inference in a global lock, processing only one request at a time. This is evidenced by the concurrent 4 latency (226ms) being roughly four times the sequential latency (54ms). vLLM groups concurrently arriving requests into a single batch. However, since locks were not released nor batches disabled for separate measurements, this explanation is an assumption fitting the numbers rather than a verified fact.

There are three things to keep in mind when reading this table.

• Conditions are not entirely identical. The existing server also hosts a reranker model and a speech recognition model on the same card (card utilization 3.7GB). The Dynamo worker hosted only a single embedding model (3.1GB).
• Real traffic may have been mixed into the existing server. Other requests were not blocked during measurements.
• The comparison target is a custom-built server. This should not be read as "Dynamo is 2.8x faster," but rather as "this is how much things change when you replace a server processing requests one-by-one with a batching engine." The same difference would likely occur even if deploying vanilla vLLM without Dynamo. Comparisons against standalone vLLM were not performed (Unmeasured).

6. What Cannot Be Migrated

The existing server performs three tasks, and this post only migrated one of them.

Migrating only embeddings increases the number of servers from one to two. The existing server hosts three models in a single process sharing one 8GB card, but separating embeddings results in two processes on that card plus an additional frontend pod. If there is only a single worker, the role of the Dynamo frontend is merely to forward requests to that single worker.

7. Conclusion

• The Dynamo embedding worker launched with two arguments (--embedding-worker, --runner pooling) and produced identical vectors to the existing server using the same model (minimum cosine similarity 0.9995).
• On the same GPU, sequential latency changed from 54ms to 37ms, and concurrent 4 throughput changed from 141 docs/s to 396 docs/s. Most of this difference appears to stem from the contrast between single-request processing and batching architectures.
• A drop-in replacement by merely changing endpoints fails. dimensions results in a 500 error, query instructions are silently ignored altering the vectors (cosine 0.82), and long inputs are not truncated but result in 500 errors.
• Preparatory work must be done on the client side before migration. dimensions must be removed, instructions must be explicitly appended as text, lengths must be pre-truncated, and 500 errors must be configured not to trigger retries.
• For an embedding service with a single worker, Dynamo adds little value. The gains achieved likely originate from switching the engine to vLLM, which can be accomplished even without Dynamo.

Connecting this to the earlier parts of the series: KV routing proves its worth with chat requests sharing overlapping prefixes (Part 3). Disaggregated serving requires a fast network and large models (Part 4). Neither applies to workloads without generation like embeddings, leaving the remaining question: do the API contracts match? (this part).

References

• NVIDIA Dynamo v1.6.2, components/src/dynamo/vllm/backend_args.py 's --embedding-worker help documentation and embedding_worker_processes.py.
• vLLM documentation, Pooling Models, and OpenAI-compatible Embeddings API.
• Qwen3-Embedding model card instruction usage.
• Manifests, comparison scripts (embed_probe.py), and median summaries are located in the repository's docs/experiments/2026-11-15-dynamo-embedding/ .