Measuring LoRA Adapter Serving Directly: Throughput Cost, Adapter Slots, and Dynamic Loading

Status: Draft (2026-10-07), based on actual measurements. Because the adapters are random weightscreated for performance testing, their output quality is meaningless, and only speed and resource behavior were measured. Calculated items are marked as "Calculated," and unmeasured items are marked as "Unmeasured." Writing Rule: Measured data is the baseline, and unmeasured items are marked as "Unmeasured."

Table of Contents

• 1. Why Serve LoRA
• 2. Environment and Methodology
• 3. The Cost of Throughput
• 4. Adapter Slots: What Happens When the Number of Adapters Exceeds --max-loras
• 4-1. Running the Same Test on SGLang
• 5. Time Taken to Load Adapters at Runtime
• 6. Takeaways for Operations
• 7. Potential Interview Questions
• External Evidence: Confirmed through papers and official documentation
• Unconfirmed Items
• References
• Verified Version and Date

1. Why Serve LoRA

Fine-tuning a model for each use case requires loading the entire model into memory for each. LoRA keeps the base weights W as they are, trains two low-rank matrices (B, A) to change to W + (α/r)·B·A , so you can attach multiple small adapters to a single base model and use a different adapter for each request. This is why people try to serve customer-specific and task-specific models on a single GPU. vLLM documentation explains that adapters can be served per request with low overhead. This post measures what percentage that "low" overhead actually is in this environment.

2. Environment and Methodology

Pay attention to the number of concurrent requests. The tables in this post list the "number of requests sent at once" (3, 18, 48). Due to the argument names of the testing tool, we initially called it "concurrency 1, 16," but we corrected this after confirming that it was actually sending 3x the requests.

Because the adapters are random, their outputs are meaningless. Since the size and computational load of adapter matrices are independent of whether they are trained, speed and memory behavior are assumed to be identical to trained adapters, but this was not verified with trained adapters (unmeasured).

3. The Cost of Throughput

Identical requests were sent to the baseline server (LoRA disabled) and the LoRA-enabled server.

• Leaving the LoRA feature enabled itself was free. The throughput for requests not using adapters was identical to the server with LoRA disabled (200 vs 199, 2,664 vs 2,667).
• Requests using adapters showed reduced throughput. The single-measurement values above were 8.5% for 3 requests and 17.9% for 48 requests, but the "Repeated Test" below revealed that the 48-request value was inflated due to a low initial measurement (about 10% when repeated). The document's "low overhead" was a not-so-small valuein this environment. We interpret this to mean that the additional computation of the adapter is not negligible compared to the weight-loading cost (Part 21) in the 1.7B model, though we did not break down which operations take up how much (estimated).
• Initially, we wrote that "mixing adapters did not make things worse," but repeated tests proved otherwise. In the single-measurement values, mixing a and b (2,323) was higher than adapter a alone (2,190), but in 5 repetitions, adapter a alone had a median of 2,390, while mixing them yielded 2,330. The single measurement of 2,190 was slow because it was the first adapter request called after starting the server (see below).
• The median ITL increased from 15 to 20ms. The magnitude of the latency increase aligns with the throughput decrease.
• The ratio did not decrease even when the model grew larger. We expected that because this model is small at about 15ms per token, the fixed cost of the adapter would be relatively large. Therefore, we spun up a server twice with Qwen3-4B-FP8 (about 19ms per token, randomly created adapters made the same way, max_loras 2) to measure, and the throughput change for adapter a was −11% (from 155 to 138 tok/s) in both runs at concurrency 1, and −18.3% and −10.7% at concurrency 16. The −18.3% case was a server with 6,320 tokens of KV cache, and the −10.7% case was a server with 10,896 tokens (Part 14). Concurrency 6 mixing adapters a, b, and c gave 397 and 396 tok/s, showing the slot shortage cliff remained as well. Since the precision (fp8) is different, this is not a strict comparison with 1.7B.

Repeated test (5 times each, same server). 48 requests (3 times concurrency 16) were sent 5 times in shuffled order across three conditions: no adapter, adapter a, and alternating adapters a and b (--max-loras 2).

• Only the first measurement was low. The first value for adapter a, 2,183, was the measurement from calling the adapter for the first time after launching the server, while the subsequent four values (2,390~2,398) were stable. It appears that loading and initialization costs were mixed in when using the adapter for the first time (not broken down). This is why the previous single-measurement value appeared inflated to 17.9%. This is an example showing that first values in benchmarks should be treated as warmupand tests should be repeated to use the median.
• The adapter cost is about 10–12% . Having a single adapter was slightly cheaper than mixing two (-10.3% vs -12.5%). The variation across repetitions was small: ±0.4% for no adapter, ±0.2% for adapter a (excluding the first value), and ±0.1% for a and b, making this difference meaningful.
• The −8.5% (182 vs 199) from the 3-request test above was also a single measurement, so it may have included warmup effects (not repeated).

4. Adapter slots: When the number of adapters exceeds --max-loras ,

--max-loras is the maximum number of different LoRAs that can fit in a single batch(default 1). Initially, we wrote it as "the number of adapters that can be loaded on the GPU simultaneously," but after directly reading the source code (vllm/config/lora.py) of the vLLM 0.31.0 image used in this post, the description was "Max number of LoRAs in a single batch," and the number of adapters to place on the CPU side was a separate --max-cpu-loras(default is the same value as max_loras ). We sent 18 requests with 3 adapters all at once (2 repetitions per setting. While the values for both runs were often nearly identical, adapters a and b with 2 slots differed at 886 and 984).

• When slots are insufficient, throughput drops by nearly half. When alternating 3 adapters with 2 slots, the throughput dropped to 529 tok/s, which was 56% of 950 tok/s with 4 slots and 47% of the adapter-free baseline (1,120). The cause was verified in the source code. The scheduler source code (vllm/v1/core/sched/scheduler.py) of the same image shows that when selecting pending requests, if the number of different LoRAs already in the batch reaches max_loras and the request's LoRA is not in the batch, the scheduler skips the request and leaves it in the queue (the comment says "Scheduling would exceed max_loras, skip"). In other words, if 3 adapters alternate in 2 slots, requests for the third adapter are continually postponed, causing the batch size to shrink and consequently reducing total throughput. Our initial speculation that the time spent switching adapters was the cause was corrected based on this source code fact; the waiting time distribution and actual batch sizes of postponed requests were not read via server metrics (unconfirmed).
• If slots are sufficient, the cost remains almost identical even if the number of adapters increases. The difference between 2 adapters (987) and 3 adapters (950) with 4 slots was 4%.
• Therefore, --max-loras should be aligned with "the number of adapters used in the same batch at any given time." Even if there are hundreds of adapters, as long as the ones appearing in requests simultaneously fall within the slot count, it is fine; otherwise, performance can drop to about half as shown above. Since slots pre-allocate adapter weights and computation space, they consume GPU memory. On this server, the total memory usage of the server with slots enabled was 6,144MiB, which was similar to the server without them enabled (6,168MiB) (likely absorbed by a corresponding reduction in the KV cache, though KV capacity was not verified).

4-1. Running the same test in SGLang

We measured LoRA performance in SGLang 0.5.21 (--enable-lora --max-loras-per-batch N) using the same model and three identical random adapters (rank 16, 0.19% of parameters).

• The cost per adapter was lower in SGLang.(-1.1% vs -10.3%). Initially, I wrote that since the SGLang value was measured only once, its initial value might be low like vLLM's, but when I restarted the server three times and measured again, the decrease for adapter a was almost identical at -0.9%, -1.3%, and -1.1%, and there was no phenomenon of a low initial value. At concurrency 1, it was 199 vs 194 tok/s (-2.0 to -2.5%). This is a clue that the LoRA kernels and batching methods of the two engines are different, but I did not break down the cause (conjectural). Since these results are values for this model and a rank 16 adapter, they should not be generalized.
• The cliff where slots were insufficient was the same for both engines(these values were also measured once each). When there were more different adapters in a batch than the slots, the throughput dropped by nearly half (SGLang 558 vs 1,045, vLLM 529 vs 950), and it recovered when the slots were increased beyond the number of adapters. Section 4's conclusion that the number of slots must not be smaller than the number of adapter types held true regardless of the engine.
• Limitations: Since it is a random adapter, it may differ from actually trained adapters. SGLang's single adapter and slot 4 values were repeated across three server runs, but the slot 2 cliffs (558, 529) were measured only once each.

5. Time Taken to Load Adapters During Runtime

VLLM_ALLOW_RUNTIME_LORA_UPDATING=True after launching the server, I added adapter c with /v1/load_lora_adapter .

• Loading a single adapter (6.4MB) finished within 0.2 seconds without restarting the server. This contrasts with server restarts taking 35 to 55 seconds (Part 14). The operational advantage of LoRA serving is that deploying adapters is much lighter than deploying models.
• However, there is a warning in the documentation not to use this feature in production. The vLLM documentation states not to use runtime adapter changes in production unless the environment is isolated and fully trusted. I understand this is a security risk because requests could cause adapters from arbitrary paths to be loaded (an interpretation I did not verify directly). In production, it is safer to specify them at startup with --lora-modules or to control adapter deployment via a pipeline.
• Server preparation time was 70 seconds when LoRA was enabled, which was longer than a server without LoRA enabled (35 to 55 seconds restart time). The server with LoRA enabled captured CUDA graphs in more forms (38 vs 19 in the logs).

6. Takeaways for Production

1. Measure the throughput cost of requests using adapters in advance. It was 8 to 18% in this environment and varies depending on the model and concurrency.
2. --max-loras Match to the number of concurrently appearing adapters. If slots are insufficient, throughput drops to about half.
3. Adapter deployment is lightweight, but uncontrolled loading is dangerous. Control it via pipelines and load only verified adapters.
4. There was no cost to keeping LoRA enabled. Requests not using adapters were unaffected.
5. Measurements must be retaken with trained adapters and larger models. The figures in this post were obtained from a small model (1.7B) and a 4B-FP8 instance with random adapters.

7. Potential Interview Questions

• Why serve multiple LoRA adapters? Because a single base model is shared and only the adapters (around 0.19% of the base model) are swapped and used, requiring much less GPU memory than loading full models for each use case.
• How much slower does it get when using adapters? In this environment, for the 1.7B model, it decreased by about 10 to 12% based on 48 requests (median of 5 iterations; the 17.9% measured once initially included warm-up effects). Merely keeping LoRA enabled incurred no cost.
• --max-loras What does determine, and what happens if it is insufficient? It is the number of different LoRAs that can fit in a single batch (default 1). If adapters exceed this number, the scheduler skips requests for those adapters, making the batch smaller. Serving 3 adapters with 2 slots dropped the throughput to 529 tok/s, which is 56% of 950 with 4 slots.
• Why is adapter deployment easier than model deployment? A 6.4MB adapter was loaded in 0.2 seconds without restarting the server. Models require a restart of 35 to 55 seconds plus weight loading time.
• Why should runtime LoRA loading not be used in production? The documentation warns not to use it unless it is an isolated, trusted environment.

External Evidence: Verified with Papers and Official Materials

External figures had different model, GPU, and adapter configurations, so direct comparisons were not made; instead, only the consistency of direction and order of magnitude was judged.

Things Not Verified

• Speed and output quality with trained adapters. Measured only speed with random adapters.
• Cost of larger models, higher ranks, and when adapters target all linear layers..
• Iteration error. Only the slot test was repeated twice, and Section 3 was done once. The discrepancy where mixing a and b yielded higher values than a alone was not interpreted.
• Latency and batch size of deferred requests. Did not read server metrics (Prometheus).
• --max-cpu-loras Effects of CPU-side cache settings such asand scale of hundreds of adapters.
• Interaction with prefix cache when adapters are present(whether the cache is separated per adapter).

References

• vLLM documentation, LoRA Adapters: https://docs.vllm.ai/en/latest/features/lora/
• Hugging Face PEFT, LoRA: https://huggingface.co/docs/peft/package_reference/lora
• Parts 14 (vLLM load and server prep time) and 21 (token speed and memory bandwidth) of this series

Verified Version and Date

As of 2026-10-07. vLLM 0.31.0, PEFT 0.21.2, accelerate 1.15.0, Qwen3-1.7B (bf16).