Small and useful
3B–14B quantized text models, transcription, embeddings, and lighter image workflows.
Local AI lab
Run it yourself—without buying the wrong box.
A practical guide to model memory, DGX Spark, RTX workstations, Apple Silicon, software stacks, and the first setup worth trying.
Start with memory
If the model does not fit, nothing else matters.
A 4-bit quantized model often needs roughly 0.6–0.8 bytes per parameter after runtime overhead, plus memory for context and caches. Treat the estimates below as planning ranges, not guarantees.
Creator sweet spot
7B–32B text models, stronger image generation, and optimized local video workflows.
Large-model lab
70B-class quantized models, long context, larger image/video models, and heavier agents.
Workstation territory
Very large mixture-of-experts models, multi-model pipelines, fine-tuning, and multi-user work.
Context consumes memory too.
A model may load successfully and still fail or slow dramatically when you raise context length. Leave headroom for the KV cache, the operating system, and any vision or audio encoders in the workflow.
Hardware paths
Five setups worth considering.
The best hardware is the least expensive system that fits your real model and finishes your workflow fast enough.
SetupMemoryBest forKnow before buying
Computer you already own16–32GB system / 8–16GB VRAM
Small quantized text models, transcription, light image generation
Free to start; limited model size and slower media workflows
RTX 5090 workstation32GB dedicated VRAM
Fast 7B–32B text models, FLUX.2 variants, optimized ComfyUI video
Excellent speed; high power draw and model size stops at VRAM
Mac StudioUp to 128GB (M4 Max) or 256GB (M3 Ultra) unified
Quiet large-model inference, MLX tools, big context, daily workstation use
Great capacity; many CUDA-first creative tools need workarounds
NVIDIA DGX Spark128GB unified · 273 GB/s
Large-model prototyping, fine-tuning up to 70B, inference up to 200B
$4,699 MSRP; ARM64 Linux and capacity-first performance profile
RTX PRO 6000 workstation96GB GDDR7 dedicated VRAM
Heavy ComfyUI, larger LLMs, professional CUDA, rendering, and video
Fast and flexible; workstation-class cost and power
DGX Spark, honestly
A compact large-model appliance—not a magic 5090 replacement.
DGX Spark pairs a Grace Blackwell GB10 system with 128GB of unified memory. NVIDIA rates it for inference up to 200B parameters and fine-tuning up to 70B. Its superpower is fitting models that a consumer GPU cannot—not necessarily generating every token or video frame faster.
NVIDIA Spark playbooks- Large local models in a tiny footprint
- CUDA-native research and prototyping
- A Linux appliance you can use over the network
- Fine-tuning and NGC/container workflows
- You mainly want maximum ComfyUI speed
- Your tools or containers assume x86 only
- A 24–32GB RTX GPU fits every model you use
- You dislike Linux and infrastructure work
- Keep it headless beside your router or switch
- Use 10GbE when moving large model files
- Serve models to your laptop through a local API
- Use an RTX desktop as a separate creative worker
Software stack
Pick one interface and one inference engine.
Do not install every local AI tool on day one. Start with the smallest stack that can run a model and expose a local API.
Ollama
The easiest command-line model runner and a common local API target.
Official projectLM Studio
A polished desktop app for discovering models, chatting, and serving an OpenAI-compatible local API.
Official projectOpen WebUI
A self-hosted browser interface that can sit in front of Ollama and other inference servers.
Official projectllama.cpp
The portable foundation for GGUF quantized inference across CPUs, NVIDIA, Apple Silicon, and more.
Official projectvLLM
A high-throughput Linux server for production-style inference when you have supported GPUs.
Official projectComfyUI
A node-based system for repeatable image, video, and audio workflows.
Official projectOllama + Open WebUI
Install Ollama, pull one 7B–14B instruct model, add Open WebUI, and keep the service limited to your local network.
LM Studio
Browse compatible model files, see memory estimates before loading, compare prompts, and turn on the local API only when you need it.
vLLM + a simple client
Use Linux, pin a model and runtime version, expose an authenticated internal endpoint, and load-test before multiple users depend on it.
Open-weight shortlist
Good families to investigate now.
Model families move quickly. Prefer the provider’s official model card, then choose the smallest checkpoint that passes your own tests.
Gemma 4
General and multimodal local workChoose a size that fits your memory; Google positions the family for strong intelligence per parameter.
Browse official releasesQwen 3.5 family
Coding, multilingual work, vision, and agentic experimentsA broad family with small and mixture-of-experts variants; check the exact license and model card.
Browse official releasesNVIDIA Nemotron 3
Local agents and NVIDIA-optimized deploymentNano through Super variants, including sparse models designed for efficient local inference.
Browse official releasesDeepSeek V4
Large reasoning and coding experimentsA large-model option for memory-rich systems; verify the release, format, and terms before committing.
Browse official releases