Parallelization, Multimodal, and Pooling: What You Can and Cannot Benchmark with a Single GPU

Status: Draft (2026-10-27), pooling and multimodal benchmarked, parallelization unbenchmarked. Pooling measurements were conducted on 2026-10-27 using a single 8GB laptop GPU (Ada generation) and vLLM 0.31.0. Parallelization and multimodal features were unbenchmarked, and details are based solely on official documentation (0.32.0) and papers. Measurements for LoRA multi-serving can be found in Toss ML/AI Infra Preparation, Part 15. Added on 2026-10-27. Following approval, we downloaded vLLM 0.32.0 images and Qwen3-VL-2B-Instruct (Apache-2.0, safetensors 4.26GB) to add the multimodal measurements in Section 2-1. This content was marked as "Unbenchmarked" in the initial edition. Writing Rules: Measured data is the default. Unmeasured items are marked as "Unbenchmarked," and unverified items are marked as "Unverified."

Table of Contents

• 1. Four Types of Parallelism: What is Divided?
• 2. Multimodal Inputs
• 2-1. 2026-10-27 Benchmark: Running a 2B Vision-Language Model on an 8GB GPU
  • First, Getting It to Start
  • Results (Size-Hinted Server)
  • SSRF: Does the Server Download on Behalf of the User?
• 3. Pooling: Embeddings and Reranking on the Same Engine
  • Method
  • Results
• 4. LoRA Multi-Serving
• 5. Unverified Items
• References

1. Four Types of Parallelism: What is Divided?

• TP involves frequent communication. vLLM splits matrices using the Megatron-LM method (Shoeybi et al., 2019), which performs all-reduce twice per transformer layer (after attention and after MLP). Therefore, the benefits are significant when GPU interconnects are fast, such as NVLink. The official documentation also recommends PP instead of TP for GPUs without NVLink (e.g., L40S).
• PP has less communication but introduces bubbles. Because later stages must wait for preceding stages to finish, GPUs alternate idleness when viewing a single request. Multiple requests are pipelined to fill these bubbles. The documentation recommends using TP within a server and PP across servers (GPUs per server = TP, number of servers = PP).
• DP maintains separate KV caches. Because prefix caches are isolated per engine, identical prefixes must be routed to the same engine to utilize the cache. Routing requests based on KV state was the subject of the Dynamo post. The documentation also notes that while --max-num-seqs applies per engine, queue limits such as --max-num-queued-reqs apply system-wide.
• MoE models mix and match strategies. The documentation introduces configurations for models like DeepSeek, where attention uses DP and expert layers use EP.

To check whether a model fits, look for two lines in the startup logs: GPU KV cache size: N tokens and Maximum concurrency for M tokens per request: Kx . The experimental server in this series achieved "9.82x based on 40,208-token and 4,096-token requests." The documentation advises adding GPUs if this concurrency falls below the target.

Because we only have a single GPU, all four methods are unbenchmarked. Toss ML/AI Infra Preparation Part 24 includes measurements of transferring KV caches by linking two GPUs of the same family over 1GbE (where split serving was over 20x slower), so that article serves as a proxy for the cost of connecting GPUs over slow links.

2. Multimodal Inputs

vLLM serves models that accept images, video, and audio using the same OpenAI-compatible API. If you insert content blocks into chat message image_url, video_url, input_audio , the API server's processor downloads and preprocesses the media, sending it to the model's encoder. A single image is converted into multiple placeholder tokens within the prompt, and the prefix cache mixes the image hash into the placeholder block's hash so that identical images share caches (refer to Part 2's prefix cache design document).

In operations, you should prioritize this over performance.

• SSRF: Because the server downloads URLs provided in requests on behalf of the user, clients can probe internal addresses accessible to the server. The documentation recommends specifying allowed domains with --allowed-media-domains and blocking redirects with VLLM_MEDIA_URL_ALLOW_REDIRECTS=0 . It notes that this is even more critical when running inside a container.
• Memory Reservation: vLLM pre-allocates encoder memory based on the maximum media size the model can accept. If unused inputs are set to 0 like --limit-mm-per-prompt '{"image":0,"video":0}' , that share can be reallocated to KV cache. You can also reduce reservations using size hints (count, width, height, num_frames).
• CPU Memory: The processor cache, which prevents reprocessing identical media, defaults to 4GiB (mm_processor_cache_gb). This share must be accounted for when setting container memory limits, and can be disabled by setting it to 0 if there are no benefits.
• Version 0.32.0 changed behavior to open untrusted images only against an allowed list of raster formats (rejecting EPS, PostScript, and WMF, according to release notes).

In the initial edition, vision-language models were unbenchmarked because they required downloading new models. Following approval, we measured them in Section 2-1 below.

2-1. 2026-10-27 Benchmark: Running a 2B Vision-Language Model on an 8GB GPU

• Environment: A single 8GB laptop GPU (Ada generation), vLLM 0.32.0, Qwen3-VL-2B-Instruct (bf16, 4.08GiB weights), --max-model-len 4096.
• Image: A PNG drawn via PIL passed as a data URL (no external addresses fetched). A white background with a single colored rectangle and a four-digit number. Five images each at 448x448 and 896x896 resolutions.
• Prompt: Three times per image: ① "What is the number" (initial), ② asking the same question again, ③ "What is the color of the rectangle." Greedy decoding.

First, Getting It to Start

When starting up, vLLM measures memory by running the encoder once with the "largest image the model can accept." Logs indicated that the encoder cache budget was set to 16,384 tokens. With 4GB of weights plus that budget, no KV space remained on the 8GB GPU. Reducing just the number of image counts did not help, but providing the actual expected image size increased KV space by 4.5x. This is the primary knob to tweak when running vision-language models on small GPUs.

Results (Size-Hinted Server)

• Image tokens were proportional to area. Doubling side length resulted in 3.7x more tokens and 3.5x higher time to first token.
• Resending the same image skips the encoder and prefill stages. The prefix cache mixed the image hash into block hashes so identical images shared blocks (even with different questions, the prefix up to the image hit), and the processor cache metric vllm:mm_cache_hits_total recorded 20 hits out of 30 tries (second and third usages).
• Accuracy: Color 10/10, Four-digit number 0/10. Numbers were read by substituting or dropping digits, such as reading 6082 as "6002," 5435 as "543," and 9532 as "932." Whether this is a limitation of the 2B model or an artifact of preprocessing is unverified.

SSRF: Does the Server Download on Behalf of the User?

The default server's message is "not an image." This means the server actually sent a request to that internal address and received a response. Although the response body was not returned, a requester could probe the internal network via this server and discover what is open based on response times and errors. Specifying allowed domains rejected requests before downloading. Local files were blocked by default. For any service accepting image URLs, --allowed-media-domains is not optional but mandatory.

3. Pooling: Embeddings and Reranking on the Same Engine

Models that extract the final hidden states instead of generating text are called pooling modelsby vLLM. Running with --runner pooling provides /v1/embeddings, /score, /rerank, /classify, /pooling instead of generation. Causal models default to using the last token (LAST), while BERT-family models default to the first token (CLS). The official documentation notes upfront that "pooling model support is for convenience and does not guarantee being faster than other approaches."

Method

• Embedding: Running Qwen3-Embedding-0.6B with --runner pooling , sending 64 sentences composed of 120 random words (~127 tokens) in a single request, with the first of six runs used for warmup.
• Reranking: Qwen3-Reranker-0.6B is originally a generative model that calculates relevance using scores for "yes" and "no" tokens. Following the method in the vLLM example, --hf-overrides '{"architectures":["Qwen3ForSequenceClassification"],"classifier_from_token":["no","yes"],"is_original_qwen3_reranker":true}' and using the chat template from the example, I deployed it like a classification model. I sent one question and 32 short documents (6 types repeated) /rerank .

Result

• The token processing speed of embedding (approx. 21,900 tokens/sec) was similar to the prefill speed of the generative model on the same GPU (Part 2, approx. 18,000 tokens/sec). This is because pooling consists only of prefill operations.
• For reranking, in response to the question "How does the serving engine rewrite common prefixes?", it placed the prefix cache sentence at the very top with 0.993. Since it is a model trained for reranking, it performs well at this task.
• The pooling server also allocated 4.3GiB (40,272 tokens) for KV cache. The feature table states that chunked prefill and prefix caching work in causal attention pooling, so it is used for that purpose. If it is an embedding-only server, you can lower --gpu-memory-utilization to share the GPU with other tasks (unmeasured).
• According to the security documentation, paths without /score, /rerank, /pooling, /classify like /v1 will not be authenticated even if --api-key is provided. In Part 8, I actually tested paths with similar properties.

4. Multi-LoRA Serving

This is a feature that mounts multiple LoRA adapters on a single base model and uses a different adapter for each request (--enable-lora --max-loras --max-lora-rank). The values measured with a 1.7B model on the same 8GB-class GPU are covered in LoRA adapter serving. Using adapters was 8.5% to 17.9% slower, and throughput dropped significantly when the number of adapters exceeded the number of slots. There are two additional points to add in this edition.

• On-the-fly adapter loading is a risky feature. VLLM_ALLOW_RUNTIME_LORA_UPDATING=True Enabled via /v1/load_lora_adapter , the documentation warns "Do not use in production unless the environment is isolated and fully trusted." This is because the requester determines the path that the server will read.
• It cannot be used together with speculative decoding. In the feature compatibility table of the 0.32.0 documentation, the cell for LoRA and speculative decoding is marked with ❌ (Part 3).

5. Unverified Items

• The actual gains and communication costs of TP, PP, DP, and EP were unmeasured because there was only one GPU.
• Video/audio inputs, multiple images, and encoder latency under concurrent requests are unmeasured. The causes of incorrect numbers were also not verified.
• Changes in embedding throughput by batch size, reranking of long documents, and accuracy evaluations are unmeasured.

References

• vLLM Documentation, Parallelism and Scaling, Data Parallel Deployment, Multimodal Inputs, Conserving Memory, Pooling Models, Security (0.32.0): https://docs.vllm.ai/en/latest/
• Shoeybi et al., "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism", 2019. https://arxiv.org/abs/1909.08053
• vLLM Example, Deploying Qwen3 Reranker as a Classification Model: https://github.com/vllm-project/vllm/blob/v0.31.0/examples/pooling/score/qwen3_reranker_online.py
• vLLM 0.32.0 Release Notes (Image Format Whitelist): https://github.com/vllm-project/vllm/releases/tag/v0.32.0
• Previous Post: LoRA Adapter Serving, NVIDIA Dynamo