GPU for fine-tuning LLMs: memory for full, LoRA and QLoRA training, rented with crypto
The method you choose changes the memory you need by a factor of thirty, and so the GPU you rent, which here you pay for in crypto only. Pick the method first, then the card: here are the rules, the numbers for common model sizes, and the cards to rent for each.
The GPU you need to fine-tune an LLM depends on the method. Full fine-tuning with AdamW in mixed precision needs about 16 bytes per parameter before activations, so even an 8B model needs about 128 GB, two H100s or one B200. LoRA keeps the base model frozen in BF16, about 2 bytes per parameter: an 8B model trains on a 24 GB RTX 4090 or a 32 GB RTX 5090. QLoRA loads the base model in 4-bit, about half a byte per parameter: its authors fine-tuned a 65B model on a single 48 GB GPU, the memory of one L40S.
Memory per method
Memory for the model, gradients and optimizer states, before activations. The cards are the smallest that hold it with some room to spare.
| Method | Rule of thumb | 8B model | 70B model | Cards to rent |
|---|---|---|---|---|
| Full fine-tuning (BF16 weights, AdamW) | About 16 bytes per parameter | About 128 GB | About 1,120 GB | 8B: one B200 or two H100s, sharded. 70B: 8 × B200, or 8 × H200 with optimizer offloading |
| LoRA (frozen BF16 base) | About 2 bytes per parameter, plus small adapters | About 16 GB | About 140 GB | 8B: RTX 4090 or RTX 5090. 70B: two H100s or one B200 |
| QLoRA (frozen 4-bit base) | About 0.5 to 0.6 bytes per parameter, plus adapters | About 5 GB | About 35 to 40 GB | 8B: RTX 4090. 70B: one L40S, or an A100 80GB for longer sequences |
16 bytes = 2 for BF16 weights + 2 for gradients + 12 for the FP32 master copy and AdamW's two moment estimates, as set out in the ZeRO paper. LoRA adapters are usually well under 1% of the model's parameters, so their own optimizer states are small.
What else takes memory
On top of the table come activations: the intermediate values kept for the backward pass. They grow with batch size and, above all, with sequence length, and on long sequences they can exceed the weights.
- Gradient checkpointing recomputes activations instead of storing them. It saves most of that memory for roughly a third more compute time.
- Gradient accumulation gives the effect of a large batch with small ones, so a smaller card can train with the same settings, only more slowly.
- Shorter sequences are the quickest saving when your data allows it.
- Sharding with FSDP or DeepSpeed ZeRO splits weights, gradients and optimizer states across the cards of one machine, which is how full fine-tuning fits on GPU server rental with 2 to 8 cards.
For the memory a model needs once it is trained and serving users, see VRAM per model size.
Estimate before you rent
This small function applies the rules above. Run it with your model size, then add a margin for activations that depends on your sequence length and batch size.
def finetune_memory_gb(params_billion: float, method: str) -> float:
"""Weights, gradients and optimizer states, before activations."""
bytes_per_param = {"full": 16, "lora": 2.0, "qlora": 0.55}[method]
adapters = 0.01 * params_billion * 16 if method != "full" else 0 # ~1% of params, trained in full
return params_billion * bytes_per_param + adapters
for method in ("full", "lora", "qlora"):
print(method, round(finetune_memory_gb(8, method), 1), "GB for 8B,",
round(finetune_memory_gb(70, method), 1), "GB for 70B")Faster card or cheaper card
A training run costs the hourly price times its length. A pricier card is only cheaper per run if it finishes at least this many times faster.
| Cheaper card | Faster card | Hourly prices here | Faster card must be at least |
|---|---|---|---|
| L40S | A100 80GB | $0.87 → $0.99 | 1.14 × faster |
| L40S | H100 80GB | $0.87 → $2.39 | 2.75 × faster |
| A100 80GB | H100 80GB | $0.99 → $2.39 | 2.41 × faster |
| H100 80GB | H200 141GB | $2.39 → $3.29 | 1.38 × faster |
| H100 80GB | B200 180GB | $2.39 → $4.89 | 2.05 × faster |
Run 100 training steps on each candidate, compare the time, and divide each price by its speed. More cards finish sooner but rarely cost less: eight cards for ten hours use the same 80 GPU-hours as one card for eighty.
Renting training GPUs with crypto, without waste
- Test on a small block first. Run a few hundred steps to check memory and speed before you buy the hours for the full run.
- Save checkpoints to disk every few hundred steps. If a run fails, you restart from the last one, not from zero.
- Copy results off the machine before you delete it: a deleted machine's disk is wiped.
- Stop the machine when the run ends. Billing is per minute, so an idle machine left running overnight is the most common waste.
- Upload data once. Keep datasets on the machine's disk between runs instead of copying them for every experiment.
For 70B and larger, H100 rental and the H200 and B200 pages cover SXM machines with NVLink, which matter for sharded training. Every card and block is on the full price list.
- Buy the block that fits the run. Hours are paid in crypto, in USDT, Bitcoin or five other coins, and unused hours never expire, so a block that turns out too large is not lost.
Fine-tuning GPU questions
Why do LLMs use a GPU instead of a CPU?
Training and inference are mostly large matrix multiplications. A GPU runs thousands of them in parallel and reads memory many times faster than a CPU, which turns weeks of CPU time into hours.
Can I fine-tune an LLM on one RTX 4090?
Yes, within limits: LoRA on models up to about 8B, and QLoRA on models up to about 14B comfortably, or around 32B with short sequences. Larger models need 48 GB or more.
Which GPU is best for AI and machine learning work?
The cheapest one that holds your job. For QLoRA and smaller models, an RTX 4090 or L40S; for full fine-tuning and large LoRA runs, H100, H200 or B200 machines with NVLink.
How long does fine-tuning take?
It depends on the model, the data size, the sequence length and the card, so measure a short run and extrapolate. With per-minute billing, you pay only for the time the run actually takes.
Sources
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (Rajbhandari et al.)read 19 November 2026
- QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al.)read 19 November 2026
- LoRA: Low-Rank Adaptation of Large Language Models (Hu et al.)read 19 November 2026
- PyTorch FSDP documentationread 19 November 2026