GPU for LLM inference: memory, speed and cost per model size, paid in Bitcoin or USDT
Choosing a GPU for a language model, which you rent here with crypto only, comes down to two numbers: how much memory the model and its context need, and how fast the card can read that memory. Both can be worked out before you rent anything.
The GPU you need for an LLM is set by memory first: the weights take 2 bytes per parameter in BF16, 1 in FP8 and about half a byte in 4-bit, plus a KV cache that grows with the context. An 8B model in BF16 fits on a 24 GB RTX 4090 ($0.34 an hour), a 32B model in FP8 on a 48 GB L40S, a 70B model in FP8 on one H200 (141 GB, $3.29 an hour) or two H100s, and the largest open models, such as DeepSeek's 671B, need eight H200s or B200s. Speed then follows memory bandwidth: the faster a card reads memory, the more tokens per second it generates.
How much memory each model size needs
Memory for the weights alone. Leave room on top for the KV cache, the runtime and activations, typically 10% to 30% more.
| Model size | BF16 | FP8 | 4-bit | Cards to rent |
|---|---|---|---|---|
| 8B, such as Llama 3.1 8B | 16 GB | 8 GB | About 5 GB | RTX 4090 in BF16 with room for cache |
| 14B | 28 GB | 14 GB | About 8 GB | RTX 4090 in FP8; the 32 GB RTX 5090 for longer contexts |
| 32B | 64 GB | 32 GB | About 18 to 20 GB | A100 80GB in BF16; L40S in FP8; RTX 5090 in 4-bit |
| 70B, such as Llama 3.3 70B | 140 GB | 70 GB | About 35 to 40 GB | One H200 or two H100s in FP8; L40S in 4-bit |
| gpt-oss-120b | — | — | Released in 4-bit (MXFP4); fits one 80 GB GPU, according to OpenAI | One H100 |
| 405B, such as Llama 3.1 405B | 810 GB | 405 GB | About 210 to 230 GB | 8 × H100 in FP8; 8 × H200 in BF16 |
| 671B, such as DeepSeek-V3 or R1 | — | About 700 GB (released in FP8) | — | 8 × H200 (1,128 GB) or 8 × B200 (1,440 GB) |
The A100 has no FP8 Tensor Cores, so run FP8 models on it only through weight-only kernels, or choose BF16 or 4-bit. Mixture-of-experts models such as gpt-oss and DeepSeek still need all their weights in memory, even though each token uses only part of them.
The KV cache grows with context
While generating, the model keeps a key and a value vector for every past token in every layer. Its size per token is 2 × layers × KV heads × head size × bytes per value.
- Llama 3.1 8B has 32 layers and 8 KV heads of size 128, so each token takes about 0.13 MB in BF16. An 8,192-token context needs about 1.1 GB; a full 128,000-token context about 17 GB, which together with the 16 GB of weights is more than an RTX 4090 holds.
- Llama 3.1 70B has 80 layers and 8 KV heads of size 128: about 0.33 MB per token. A 32,768-token context takes about 11 GB, and 128,000 tokens about 42 GB.
Those figures are per conversation. Serving ten users with 8k contexts on the 70B model needs about 27 GB of cache on top of the weights. An FP8 KV cache halves these numbers, at a small cost in accuracy. This is why a 70B model in FP8 fits on an 80 GB H100 but leaves little room for users, while an H200 with 141 GB leaves about 70 GB for them, before the serving software takes its share. The H200 rental page works through that case.
How fast each card can generate
For a single request, each new token requires reading all the weights once, so memory bandwidth sets a ceiling: bandwidth ÷ size of the weights. These are upper limits; real servers reach a fraction of them.
| Card | Memory bandwidth | 8B in BF16 (16 GB) | 70B in FP8 (70 GB) |
|---|---|---|---|
| L40S | 864 GB/s | ≤ 54 tokens/s | Does not fit |
| RTX 4090 | 1,008 GB/s | ≤ 63 tokens/s | Does not fit |
| RTX 5090 | 1,792 GB/s | ≤ 112 tokens/s | Does not fit |
| A100 80GB | 1,935 to 2,039 GB/s | ≤ 121 to 127 tokens/s | No FP8 support |
| H100 80GB | 2.0 TB/s (PCIe) to 3.35 TB/s (SXM) | ≤ 125 to 209 tokens/s | ≤ 29 to 48 tokens/s, little room for cache |
| H200 141GB | 4.8 TB/s | ≤ 300 tokens/s | ≤ 69 tokens/s |
| B200 180GB | 8 TB/s | ≤ 500 tokens/s | ≤ 114 tokens/s |
Bandwidth from NVIDIA's specifications. Serving many users at once raises total throughput far above these single-request figures, because one read of the weights serves every request in the batch.
Cost per million tokens
What a million generated tokens cost at our list prices, depending on the total throughput your server reaches across all users. Measure your own with your model and batch size, then read across.
| Card, per hour | 100 tokens/s | 500 tokens/s | 1,000 tokens/s | 2,500 tokens/s |
|---|---|---|---|---|
| RTX 4090, $0.34 | $0.94 | $0.19 | $0.09 | $0.04 |
| RTX 5090, $0.49 | $1.36 | $0.27 | $0.14 | $0.05 |
| L40S, $0.87 | $2.42 | $0.48 | $0.24 | $0.10 |
| H100, $2.39 | $6.64 | $1.33 | $0.66 | $0.27 |
| H200, $3.29 | $9.14 | $1.83 | $0.91 | $0.37 |
Cost = hourly price ÷ (tokens per second × 3,600) × 1,000,000. A single request is bounded by the ceilings in the table above; batching many requests can reach thousands of tokens per second in total. All prices on price per GPU-hour.
Serve a model on a rented GPU
vLLM serves most open models behind an OpenAI-compatible API with continuous batching. On our PyTorch image it installs with pip; you can also launch the machine straight from the vllm/vllm-openai Docker image. Binding it to 127.0.0.1 and using an SSH tunnel keeps your endpoint private.
# On the rented machine
pip install vllm
hf auth login # gated models such as Llama need an accepted licence
vllm serve meta-llama/Llama-3.1-8B-Instruct --host 127.0.0.1 --port 8000 --max-model-len 16384
# On your own computer: tunnel, then call the API
ssh -N -L 8000:127.0.0.1:8000 root@<address>
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" \
-d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'Which card to rent
- Up to 14B, one user or a few: an RTX 4090 at $0.34, or the RTX 5090 at $0.49 for more context and almost twice the bandwidth.
- 32B, or 70B in 4-bit: an L40S with 48 GB. If you want BF16 quality on a 32B model, rent an A100 80GB instead.
- 70B in FP8 for real traffic: one H200 holds the weights with about 70 GB to spare for cache; two H100s split the model with tensor parallelism. See the H100 page for SXM and PCIe differences.
- 405B and 671B models: eight H200s or eight B200s in one machine.
All of these are whole cards, billed by the minute while the machine runs, as part of our cloud GPU rental. If you also want to adapt a model to your data, memory needs for training are on GPU for LLM fine-tuning.
A model served for weeks fits the large blocks: 1,200 RTX 4090 hours for $384, or 640 H200 hours for $1,894.40. Pay once in USDT, Bitcoin or another of our seven coins, with no card, and the hours never expire. Every block is on the price list.
LLM GPU questions
Do you need a GPU to run an LLM?
Small models run on a CPU, slowly. For anything above a few billion parameters, or for more than one user, a GPU's memory bandwidth is what makes generation fast enough to use.
What GPU does ChatGPT use?
OpenAI does not publish the details of its serving fleet. Services of that size run on large clusters of data-center accelerators, such as NVIDIA's H100, H200 and B200, the same cards rented here, one machine at a time.
Is a GPU required for inference?
Not strictly, but it is the practical choice. Each generated token reads the whole model from memory, and a GPU reads memory many times faster than a CPU.
How do I know a model will fit before renting?
Add the weights (parameters × bytes per parameter) to the KV cache for your context length and number of users, plus 10% to 30% for the runtime. If the total is close to the card's memory, choose the next size up.
Sources
- Llama 3.1 8B model card and configuration (Meta)read 19 November 2026
- Llama 3.1 70B model card and configuration (Meta)read 19 November 2026
- OpenAI, introducing gpt-ossread 19 November 2026
- DeepSeek-V3 model cardread 19 November 2026
- vLLM documentationread 19 November 2026
- NVIDIA H100 specificationsread 19 November 2026
- NVIDIA H200 specificationsread 19 November 2026