Which local AI models fit on which hardware

Reviewed by the Contractor AI Shop research team · Last updated Oct 14, 2026 · How we research

Why this matters: running a model on hardware you own keeps bids, specs and pricing private. Two numbers decide whether that works: memory (does the model fit?) and memory bandwidth (how fast does it write?). This guide gives you the math, real download sizes, fits-on matrices at two context lengths, measured speeds with the software used, and picks per contractor task. Every number links to its source; the full list is at the bottom.

1. Tokens, and tokens per second, in plain words

Models read and write in tokens, small chunks of text. OpenAI's rule of thumb for English is that one token is about four characters, or about three-quarters of a word, so 100 tokens is roughly 75 words. Speeds are quoted in tokens per second (tok/s), and there are two very different speeds:

• Prompt processing (also called prefill): how fast the model reads what you give it, such as a spec section or an RFI log. This depends mostly on raw compute, so numbers run into the thousands.
• Generation (also called decode): how fast it writes the answer. This depends mostly on memory bandwidth, because every new token means reading the model's active weights again. 20 tok/s is roughly 15 words a second, faster than most people read; under about 5 tok/s feels slow for chat but is fine for overnight batch jobs.

So a machine can read a 50-page spec quickly and still write slowly, or the reverse. Quote both when you compare hardware.

2. The memory math

Memory needed = model weights + KV cache + runtime overhead.

• Weights are roughly parameters x bytes per weight: 2 bytes at FP16/BF16, about 1 at Q8, a bit over half at Q4. Ollama lists Llama 3.1 70B at 141GB (FP16), 75GB (Q8_0) and 43GB (Q4_K_M). In practice, use the download size.
• KV cache is the model's working memory for the text it is holding. Per token it is 2 x layers x KV heads x head size x bytes per value, and the total grows in a straight line with context length and with the number of simultaneous users (Together AI, NVIDIA). For Llama 3.1 70B that is about 40GB for one user at the full 128K-token context, according to both Together AI and NVIDIA, almost as much as the 43GB Q4 weights.
• Runtime overhead depends on the engine. VRAMCalculator publishes measured allowances of 0.9GB + 3% for Ollama, 1GB + 3% for LM Studio and 2GB + 5% for vLLM. We use the Ollama figure.
• Macs share memory with the system. By default macOS lets the GPU use about 75% of unified memory, as the llama.cpp maintainer notes; one benchmark there shows a 16GB MacBook Air refusing a 13.5GB model because it exceeded a 12.7GB GPU working-set limit. The limit can be raised, but that is an advanced change that leaves less memory for macOS. The matrices count Macs at 75%.

3. Long documents: how memory grows with context

A spec book is where local AI gets tight. Using OpenAI's ratio, 30,000 words is about 40,000 tokens. The KV cache for that context, per user, at FP16, computed from each model's published configuration:

Computed with the Together AI formula from the config.json files linked in the references (decimal GB). gpt-oss uses a 128-token sliding window on half its layers, so its real cache is smaller than shown. Qwen3's config lists a 40,960-token native context.

Ways to stay inside memory: feed the model the relevant spec sections instead of the whole book (that is what document-Q&A tools such as Open WebUI's retrieval do); set the context length to what you actually use; and consider an 8-bit KV cache, which roughly halves cache size. LLMFit reports Q8 KV cache as near-lossless for most models, with 4-bit cache costing more quality. Long context also slows generation: in MacStories testing summarized by senn-tech, Qwen3.8-27B on an M5 Ultra went from 48 tok/s on a short prompt to 32 tok/s at 128K tokens.

4. Hardware tiers (verified specs)

Specs: Apple Mac mini specs, Apple Mac Studio specs, NVIDIA DGX Spark, DGX Spark user guide, NVIDIA GeForce comparison, NVIDIA RTX Blackwell whitepaper (card power). DGX Spark pricing history: $3,999 at launch, $4,699 from Feb 23, 2026 (NVIDIA forum), $6,950 on NVIDIA's marketplace as of Oct 2026 (listing, tracked by GPUwerk). A GPU card needs a full PC around it, and its price is not included. Apple and NVIDIA prices are US prices and change; check before buying.

The usable memory for each column of the matrices below:

• 16GB Mac: Mac mini M6 base (16GB); about 12GB usable by the GPU by default
• RTX 4090: 24GB GDDR6X VRAM
• RTX 5090: 32GB GDDR7 VRAM
• 32GB Mac: Mac mini M6 32GB; about 24GB usable by default
• 64GB Mac: Mac mini M5 Pro or Mac Studio M5 Max 64GB; about 48GB usable
• DGX Spark: 128GB unified memory, shared with DGX OS
• 128GB Mac: Mac Studio M5 Max 128GB; about 96GB usable
• 256GB Mac: Mac Studio M5 Ultra 256GB; about 192GB usable
• 512GB Mac: Mac Studio M5 Ultra 512GB (Apple: late October 2026); about 384GB usable

5. Fits-on matrix at 8K tokens (everyday chat, email, short documents)

Cells show the highest-quality version that fits at 8,192 tokens of context. A model fits when download size + KV cache at the stated context (FP16 cache, from each model's config.json) + runtime overhead (0.9GB + 3%, the Ollama figure published by VRAMCalculator) is at or below the tier's usable memory. Macs are counted at 75% of unified memory, the default GPU limit noted by the llama.cpp maintainer; NVIDIA GPUs at full VRAM; DGX Spark at its full 128GB, which DGX OS also shares. Models marked "not computed" use newer attention designs, so only weights and overhead are counted; leave extra room. Sizes are Ollama download sizes, checked Oct 14, 2026.

6. Fits-on matrix at 32K tokens (long specs, contracts, agent work)

Cells show the highest-quality version that fits at 32,768 tokens of context. A model fits when download size + KV cache at the stated context (FP16 cache, from each model's config.json) + runtime overhead (0.9GB + 3%, the Ollama figure published by VRAMCalculator) is at or below the tier's usable memory. Macs are counted at 75% of unified memory, the default GPU limit noted by the llama.cpp maintainer; NVIDIA GPUs at full VRAM; DGX Spark at its full 128GB, which DGX OS also shares. Models marked "not computed" use newer attention designs, so only weights and overhead are counted; leave extra room. Sizes are Ollama download sizes, checked Oct 14, 2026.

These matrices are for one user. Each extra simultaneous user adds another KV cache, so an office of five people on one box needs five times the cache column.

7. Measured speeds

Same small model, every machine. The llama.cpp project keeps community scoreboards that run one standard test (Llama 2 7B at Q4_0, reading 512 tokens and writing 128) on each machine. That isolates the hardware:

Sources: llama.cpp CUDA scoreboard (NVIDIA rows) and llama.cpp Apple Silicon scoreboard (Mac rows). Builds differ between rows, so treat small gaps as noise.

Big models people actually run:

Three lessons: software matters (gpt-oss 120B on the Spark went from 11.7 tok/s in Ollama at launch to about 50 to 59 with tuned engines); mixture-of-experts models are much faster than dense models of similar size on bandwidth-limited machines, because only a few billion weights are active per token; and batching multiplies throughput when several people or an agent share one box.

8. Mac vs DGX Spark vs GPU PC

• Bandwidth (writing speed): RTX 5090 1,792GB/s, M5 Ultra 1.2TB/s, RTX 4090 1,008GB/s, M5 Max up to 614GB/s, Mac mini M5 Pro 307GB/s, DGX Spark 273GB/s. The single-model scoreboard above tracks this order closely.
• Capacity (what fits): a GPU stops at its VRAM. When a model spills over into system memory, speed collapses; senn-tech reports a Qwen3.8-27B run on an RTX 5090 falling to 1.5 to 4.6 tok/s once layers spilled over PCIe. The Spark gives 128GB, and Macs go up to 512GB (about 384GB usable by default).
• Software: CUDA (Spark and RTX) runs the widest range of tools, including vLLM, SGLang and TensorRT-LLM, which are built for serving many users. Macs use Metal and MLX; Ollama, LM Studio and llama.cpp run well on them.
• Power and noise: ServeTheHome measured the Spark at 40 to 45W idle and 60 to 90W during LLM inference, never reaching 40dBA outside stress tests (review); Tom's Hardware measured 37.5dBA at 18 inches under load (review). Apple rates the Mac Studio at 480W maximum continuous power and 7dB at idle in its tested M5 Ultra configuration; senn-tech reports sustained full load above 400W with audible fans. An RTX 5090 alone is rated 575W, plus the rest of the PC.
• Price per GB of model memory: see the table in section 4. A 32GB RTX 5090 card is the cheapest route to fast small models; the Spark and Mac Studio are the routes to 70B-class and larger.

9. Recommended picks per tier

Picks are editorial suggestions based on what fits with room for context. Model quality was not benchmarked here; test two candidates on your own bids and documents before you standardize, and always check numbers in an estimate yourself.

10. DGX Spark in depth

The DGX Spark pairs a Blackwell GPU and a 20-core Arm CPU with 128GB of coherent unified memory at 273GB/s. NVIDIA rates one unit for models up to 200 billion parameters, and the user guide says two linked units handle up to 405B. NVIDIA's new 64GB OEM version is rated for models up to 100B.

What fits: everything in the DGX Spark column above, including gpt-oss 120B (65GB), Llama 3.3 70B at Q8 (75GB) and every 30B-class model at FP16. Qwen3 235B-A22B at Q4 (142GB) does not fit on one unit; NVIDIA demonstrates Qwen-235B on two connected Sparks in NVFP4.

Recommended: gpt-oss 120B as the office's main model for document Q&A, estimating help and agents (about 50 to 59 tok/s with SGLang or current llama.cpp); gpt-oss 20B or Qwen3 30B-A3B for fast drafting (61 to 83 tok/s in llama.cpp); dense 70B models only for overnight batch work, since they write at about 2.7 tok/s for one user.

Real-world tips:

• Start with NVIDIA's DGX Spark playbooks and our DGX Spark setup guide.
• Update the inference engine before judging speed. LMSYS's own numbers for gpt-oss 120B rose from 11.7 to about 50 tok/s in three weeks of software work.
• Use vLLM or SGLang when several people or an agent will use it at once; LMSYS measured Llama 3.1 8B going from 20.5 tok/s for one user to 368 tok/s combined across 32.
• Prefer the 4-bit formats NVIDIA optimizes for (MXFP4, NVFP4). NVIDIA says NVFP4 cut memory use by about 40% versus FP8 in its Qwen-235B example.
• Expect slower writing as context grows: in the llama.cpp benches, gpt-oss 120B went from 58.7 tok/s to 42.8 tok/s with 32K tokens already in context.
• Need more than 128GB? Two Sparks connect over the built-in ConnectX-7 200Gbps link (NVIDIA playbook).

11. Mac Studio M5 Ultra with 512GB: confirmed for late October

The 512GB option requires the 36-core CPU and 80-core GPU M5 Ultra, with 1.2TB/s of memory bandwidth (Apple tech specs). With about 384GB usable by default, it would hold, per the matrices above:

• Qwen3 235B-A22B at Q8 (250GB), with room for long context.
• Llama 3.1 405B at Q4 (243GB).
• Several mid-size models at once, for example gpt-oss 120B plus a fast drafting model.
• Not by default: DeepSeek-R1 671B at Q4 (404GB download) exceeds the default GPU limit. It would need the macOS limit raised, which leaves little memory for the system.

No independent tests of the 512GB configuration exist yet. The 256GB M5 Ultra uses the same chip and bandwidth, so its gpt-oss 120B result (110 tok/s writing in oMLX) is a reasonable guide for models that fit in both; larger models will write more slowly because more weights are read per token.

12. Getting started

Find your column, pick one model from section 9, and run it with Ollama (for example ollama run gpt-oss:20b). For a Mac walkthrough see Local AI on a Mac Studio with Ollama. To size a model we do not list, the calculators at VRAMCalculator, LLMFit and CanItRun use the same weights-plus-cache-plus-overhead approach.

References

All accessed Oct 14, 2026.

1. Apple, "Apple introduces new Mac Studio with M5 Max and M5 Ultra," press release, Aug 25, 2026. apple.com/newsroom
2. Apple, Mac Studio product page. apple.com/mac-studio
3. Apple Support, Mac Studio (M5 Max or M5 Ultra) Tech Specs. support.apple.com/en-us/128107
4. Apple, Mac Studio compare page. apple.com/mac-studio/compare
5. Apple, Mac mini Technical Specifications. apple.com/mac-mini/specs
6. Daring Fireball, configurations and pricing for the new Mac minis and Mac Studios, Aug 2026. daringfireball.net
7. The Verge, Mac Studio M5 Max and M5 Ultra launch. theverge.com
8. MacRumors, Mac Studio with M5 Ultra and 512GB launching in October, Aug 25, 2026. macrumors.com
9. NVIDIA, DGX Spark product page. nvidia.com
10. NVIDIA, DGX Spark User Guide, Hardware Overview. docs.nvidia.com
11. NVIDIA Marketplace, DGX Spark listing. marketplace.nvidia.com
12. NVIDIA Developer Forums, "2/23/2026 Price Change Announcement." forums.developer.nvidia.com
13. NVIDIA Blog, "NVIDIA DGX Spark 64GB," Oct 2, 2026. blogs.nvidia.com
14. GPUwerk, DGX Spark price history. gpuwerk.com
15. NVIDIA Technical Blog, "New Software and Model Optimizations Supercharge NVIDIA DGX Spark." developer.nvidia.com
16. NVIDIA, DGX Spark playbooks and "Connect Two Sparks." build.nvidia.com/spark, stacked sparks
17. NVIDIA, GeForce RTX 5090 and RTX 4090 pages and GeForce comparison. RTX 5090, RTX 4090, compare
18. NVIDIA, RTX Blackwell GPU Architecture whitepaper. images.nvidia.com
19. NVIDIA Newsroom, GeForce RTX 50 Series announcement (RTX 5090 at $1,999). nvidianews.nvidia.com
20. llama.cpp, "Performance of llama.cpp on Apple Silicon M-series," Discussion #4167. github.com
21. llama.cpp, "Performance of llama.cpp on Nvidia CUDA," Discussion #15013. github.com
22. llama.cpp, DGX Spark benchmarks. github.com
23. LMSYS, "NVIDIA DGX Spark In-Depth Review," Oct 13, 2025. lmsys.org
24. LMSYS, "Optimizing GPT-OSS on NVIDIA DGX Spark," Nov 3, 2025. lmsys.org
25. oMLX, gpt-oss-120b on M5 Ultra (80c) benchmark. omlx.ai
26. senn-tech, "M5 Ultra: the first measured numbers for local language models," Sep 22, 2026 (summarizes MacStories, Alex Ziskind and Alex Grankin). senn-tech.com
27. ServeTheHome, NVIDIA DGX Spark review, page 4 (power and noise). servethehome.com
28. Tom's Hardware, NVIDIA DGX Spark review, page 4 (power, heat and noise). tomshardware.com
29. Together AI, "KV cache explained." together.ai
30. NVIDIA Technical Blog, "Mastering LLM Techniques: Inference Optimization." developer.nvidia.com
31. NVIDIA Technical Blog, "Accelerate Large-Scale LLM Inference and KV Cache Offload with CPU-GPU Memory Sharing." developer.nvidia.com
32. VRAMCalculator, engine overhead figures. vramcalculator.com
33. LLMFit, context length calculator and "What can I run." context calculator, what can I run
34. CanItRun. canitrun.dev
35. OpenAI Help Center, "Understanding and counting tokens." help.openai.com
36. Ollama library tag pages: llama3.1, llama3.3, qwen3, qwen3.6, gemma3, gemma4, deepseek-r1, mistral-small3.2, gpt-oss
37. Hugging Face model cards and configs (parameter counts, layers, KV heads): Llama 3.1 8B, Llama 3.3 70B, Llama 3.1 405B architecture (Hermes 3), Qwen3 14B, Qwen3 30B-A3B, Qwen3 32B, Qwen3 235B-A22B, Qwen3.6 27B, Qwen3.6 35B-A3B, gpt-oss 20B, gpt-oss 120B, Mistral Small 3.2, Gemma 3 27B, DeepSeek-R1