Running 2,000 fine-tunes on one graphics card costs just 5.5% of peak throughput, because every adapter shares the same heavy base weights [1]. That tiny cost covers a leap from five models to 2,000, which moves the buying decision off the graphics card and onto system memory. The old habit priced local servers model by model as the fleet grew. The new math buys one graphics processing unit (GPU) and stacks host memory instead.

The big picture:

A fine-tuned model looks like a whole new product, so that label quietly sets the hardware budget. What strikes me here is that a low-rank adaptation (LoRA) adapter is not a full standalone model. It is just a pair of small matrices bolted onto specific weights inside the base model, so its size tracks rank and layer width rather than the total parameter count [6]. The base weights load once into video memory (VRAM), and every task shares them. That setup turns a giant fleet of models into a single running engine plus a file catalog.

The math behind that shift is plain. A rank-16 adapter over a 4096-wide matrix in a 7-billion-parameter model trains under 1% of the base parameter count [6]. Measured on public files, one adapter is 13.6 megabytes (MB) against a 14.48 gigabyte (GB) base [3][4]. Thirty of them add roughly 3% to the memory the base already holds. On an 80 GB card, a thousandth of a model per task is not a rounding problem, but a completely different buying category [5].

What breaks first is the serving path. The study tested an old 2023 system that copied adapters per batch, which held just 2.04 requests a second with five adapters and ran out of memory at 100 [2]. The card was not full, but the packing rules failed. Once a server schedules adapters per request, the limit moves to system memory (RAM). The authors state plainly that adapter count scales with the host’s main memory [1].

That flip belongs on every purchase order. The VRAM holds the base model, the key-value cache for live requests, and a small set of active adapters. The broader catalog sits in RAM. A team quoting a second GPU for their twentieth task model is pricing the wrong part entirely, because the real choke point is a memory and disk bill.

By the numbers

  • 7.61 requests a second — throughput at 2,000 adapters: One 80 GB card ran 2,000 adapters at 7.61 requests a second against 8.05 with five, which cost 5.5% of the served rate [1].
  • 2.04 requests a second — the naive baseline: Older rules that copied adapters per batch managed 2.04 requests a second with five adapters and crashed at 100 [2].
  • 13.6 MB — one adapter’s size: A public adapter measures 13.6 MB against a 14.48 GB base model, adding barely a thousandth of its size [3][4].
  • 27.2 GB — the catalog’s real home: Two thousand adapters eat 27.2 GB of host memory, whereas 2,000 full copies of the 14.48 GB base would need nearly 29 terabytes [3].

What I’d watch:

The standard payback math treats a single model as the unit to compare. Everything above breaks that rule. What I keep coming back to is that the second through twentieth models are purely a memory and scheduler problem. The physical limits show up long before a new hardware order goes out.

  • The live set against the catalog: The card holds a working set of adapters while the fleet waits in RAM [1]. A hundred task models come to roughly 1.36 GB, which is 1.7% of an 80 GB card [3][5], so the real pressure lands on server memory and how fast those adapters page in.
  • The packing setting: The old 100-adapter crash came from rebuilding adapters per batch, which caused speed to sink as the catalog grew [2]. Current servers take a maximum adapter count and a maximum rank as startup rules, which means the default setting is often a hard ceiling.
  • The rank bill: The rank you set is memory reserved for every slot, so a server set for rank 128 pays for 128 even when each loaded adapter is only rank 16 [1][6]. Raising that rank moves both memory use and output quality.
  • The quality ledger: Low-rank adapters did worse than full fine-tuning on code and math tests in the same study, came out level with it on a general instruction mix, and forgot less of the source domain [6]. Serving twenty models on one card is cheaper only if each one actually clears the bar its specific task needs.

The catch

My read is that the 2,000-adapter headline is a test of a serving design, not a hard promise for your live traffic. The authors ran 7-billion and 13-billion-parameter models on A100 cards against fake traffic where a few adapters got most of the calls [1][2]. Real traffic spreading evenly across a big catalog keeps more adapters live at once, which changes how much swapping the box does.

The breaking point has also moved since the study. The memory crash belongs to a 2023 setup that handled adapters poorly [2], while current servers batch mixed adapters naturally. A modern setup is far more likely to lose speed to a bad adapter limit or a bloated rank setting than to die at a hundred models. That is a bad setting, but from the outside, it looks exactly like a hardware limit.

The size numbers carry their own warnings. The 13.6 MB figure is one public adapter rather than a fixed rule, since rank, targeted parts, and saved weight formats all shift that footprint. The 3% overhead line is a hosting platform’s own math example rather than a separate test [3][4]. What survives all of it is the ratio. A footprint near a thousandth is what makes a fleet cheap to hold, which is why pricing fine-tunes against rented tokens hinges entirely on local memory costs.

At a glance

  • The Big Shift: A task-specific fine-tune is a small adapter bolted onto a shared base model rather than a massive standalone file. One 80 GB card ran 2,000 adapters at 7.61 requests a second, costing just 5.5% of peak speed for a huge rise in model count.
  • Why It Matters: A team pricing a new graphics card for their twentieth task model is buying the wrong part. The catalog lives in host memory, eating 27.2 GB for 2,000 adapters against nearly 29 terabytes for full models, so the next buy is RAM and disk.
  • What I’d Watch:
  • The live set: The card holds a working set of adapters while the catalog waits in RAM, so swapping speed decides how a larger catalog feels.
  • The packing setting: The measured 100-adapter crash came from copying adapters per batch, which modern servers replace with a maximum-adapter rule.
  • The rank bill: Configured rank reserves memory per slot, meaning a server set for a high rank pays for it on every single adapter.
  • The Catch: The test used fake traffic where a few adapters got most of the calls, the memory crash dates to a 2023 setup, and low-rank adapters measured weaker than full fine-tuning on code and math.

Related reading

Sources

[1] Sheng et al., “S-LoRA: Serving Thousands of Concurrent LoRA Adapters” (arXiv 2311.03285v1), Table 3 and §1 — https://arxiv.org/html/2311.03285v1 [2] Same paper, Table 3 baseline columns and §1 (naive LoRA support in the comparison serving system) — https://arxiv.org/html/2311.03285v1 [3] Hugging Face, “TGI Multi-LoRA: Deploy Once, Serve 30 Models”, 18 July 2024 (adapter size against base size) — https://huggingface.co/blog/multi-lora-serving [4] Same post, adapter memory overhead as a share of the base model — https://huggingface.co/blog/multi-lora-serving [5] NVIDIA, A100 Tensor Core GPU product page (80 GB capacity, memory bandwidth) — https://www.nvidia.com/en-us/data-center/a100/ [6] Biderman et al., “LoRA Learns Less and Forgets Less” (arXiv 2405.09673v2), §2 and §4.1–§4.4 — https://arxiv.org/html/2405.09673v2