680GB Unified Memory: Why We're Building a 'Crazy' Local Inference Cluster

Why Would Anyone Do This?

When we started planning a local inference cluster with 680GB of unified memory, most people's first reaction was: "Why? Cloud inference is getting cheaper by the month."

Fair question. Here's our actual reasoning.

The Numbers Behind the Decision

Running 15 AI Agents means a lot of inference calls every day. Some rough math on our usage:

• Content generation tasks: ~200 calls/day, avg 2K tokens each
• Code review and debugging: ~50 calls/day, avg 8K tokens
• Research and analysis: ~30 calls/day, avg 15K tokens

At current cloud API rates, this adds up quickly. For shorter, repetitive tasks — content generation, quick code completions, classification — the cost-to-quality ratio of local models has crossed into compelling territory.

The specific model we're targeting: DeepSeek R1 671B at Q8 quantization, fully loaded in memory. At Q8, the model needs ~671GB of memory. With 680GB unified memory across our cluster, we're right at the edge of making this work without offloading.

The Cluster Architecture

Our current setup:

• Mac Studio M3 Ultra 96GB × 2 (online, Ollama + llama.cpp)
• Mac Studio M5 Ultra 256GB × 2 (being integrated)
• Mac Mini Pro 48GB × 1 (being integrated)
• Interconnect: Thunderbolt 5 via CalDigit TS5 dock

Total unified memory when fully assembled: 680GB. We're using llama.cpp's RPC (Remote Procedure Call) mode to distribute model layers across machines — each machine handles a slice of the model, inference runs across all of them in parallel.

What We've Learned So Far

Thunderbolt bandwidth is the bottleneck. TB5 theoretical bandwidth is 120GB/s, but real-world RPC overhead means effective throughput is lower. Inter-node communication during inference adds latency that doesn't exist in cloud inference.

Unified memory architecture matters a lot. Apple Silicon's unified memory design (where CPU and GPU share the same memory pool) is key to making this work — it avoids the PCIe bottleneck that kills discrete GPU approaches at this scale.

This is still research-grade infrastructure. We're not recommending this for production workloads yet. But for a team that wants to experiment with large models without cloud cost constraints, it opens up possibilities that simply aren't accessible otherwise.

The Real Point

We're not doing this to avoid paying for cloud inference forever. We're doing it to understand, at a deep level, how large model inference actually works — and to have infrastructure that lets us experiment freely without a cost meter running.

Some things you can only learn by building them.