The Sweet Spot for Local AI in Business

One way to understand the local AI market is to look at the current hardware and model ecosystem.

So let's start with the hardware real briefly.

I bought my Nvidia DGX Spark around one year ago with 128GB of unified memory, which is one of the key metrics for running local models. But we'll get to that in a bit.

So this year at GTC, I would have expected to see its successor, the next version of the DGX Spark with 256 GB of RAM, which would be the next logical step.

But guess what? Didn't happen.

What we saw instead was a smaller version, a 64GB version of the Spark.

I asked Carter about that, and he said that in the end it's about giving people more options. Models are becoming more powerful and more efficient, and with that, you can run pretty capable models with less memory. Also, it’s an additional option, not a replacement for the 128GB version. If you like, you can still combine multiple Sparks into a cluster if you need more memory later. Start small and scale up instead of buying an entirely new machine.

The other side of the story is that RAM prices have completely gone through the roof:

In simple words, DDR5 RAM today is 10x as expensive as it was two years ago.

When the original DGX Spark launched last year, it cost $3,999 for 128GB of RAM. Today, that same 128GB model costs $6,950. The new 64GB version starts at $4,999. Essentially, you’re paying more for half the RAM.

What kind of computer hardware have you ever had that got almost 75% more expensive in just one year? Welcome to the AI market.

So announcing a smaller version to keep entry-level prices around ~$5K makes sense.

But not every hardware company is going in this direction.

Consider Apple.

Apple recently announced their new Mac Studio which you can order with up to 512GB of unified memory. That beast costs you almost $23,000 USD. This sounds expensive, but if you’re looking at a price-per-GB basis, it’s actually not a bad deal:

And just like the prices, the Spark that I bought last year feels much more powerful today than it did last year.

Why is that?

Time to talk about models.

Let’s do some quick napkin math here:

Now, of course, you can reduce that quite significantly through “quantization” (think compression):

To keep things simple, assume that the more aggressively you quantize a model, the higher the risk of losing some of its intelligence. Although in practice, 4-bit quantization often works surprisingly well.

And of course, that's just the model weights. You also need some additional memory to actually run the model, depending on things like context length and the inference software.

So if we map this to the model ecosystem, you can see that there's actually a gap between models you can easily run locally and models that need serious infrastructure.

Here's how 20 selected open-weight models compare, assuming 4-bit quantization:

Larger models tend to be more capable, although model size alone doesn't necessarily translate to more intelligence. And with that, there’s roughly 4 categories:

As you can see from the chart above, a “prosumer” setup ranging from a 64GB Spark up to a 512GB Mac Studio allows you to comfortably run a large share of these models.

To get to the very largest models, however, you'd need to add RAM much more aggressively. Consider DeepSeek V4 Pro with 1.6 trillion parameters. That's roughly 800GB just for its weights at 4-bit quantization. At that point, you're looking at seriously expensive hardware, typically involving high-end workstations or multi-GPU setups – often exceeding the $50K+ investment range.

On the other hand, a “decent” setup with 128GB lets you run a pretty powerful model like Qwen3.8 Flash Next (the one I'm using right now), or serve multiple smaller models in parallel.

Of course, this is a simplified explanation because RAM is not the only factor (consider memory bandwidth too). And modern Mixture of Experts architectures (MoE) allow much larger models to run more efficiently by activating only a subset of their parameters at a time, although you still need memory for the model weights.

The bottom line is:

If you’re looking into a local AI setup right now, the 128GB range is actually giving you a pretty solid starting point.

Upgrade to 256GB if you're a little bit more ambitious or want to run larger models, longer contexts, or multiple models in parallel.

Which brings me to my next point.

When speaking with Carter around usage scenarios, two broad scenarios appeared for me as to how people can benefit from the Local AI trend right now.

This is the classic ChatGPT or Claude user, except they don’t want to use any of those for reasons (like not wanting to share data, using their credits up all the time, or generally being annoyed that a new model version is shipping every few weeks that disrupts their workflows).

In this scenario, the Power User is generally interacting exclusively with their machine which typically serves a model that is performing very well on their use case (like coding, or office tasks). This is exactly the category where the latest collab between Microsoft and Nvidia fits in – the new RTX Spark laptops, including Microsoft's Surface Laptop Ultra, which come with up to 128GB unified memory, essentially giving you Spark specs in a laptop format. Or you could have a Mac Studio that runs an open-weight model and you’re actually working on that machine.

This is what Carter described is happening at Nvidia's Brev team. Here, a smaller team (~20 people) is sharing two DGX Stations, connected via ConnectX-8. They use them for everything from engineering workloads to tracing production logs. He mentioned DGX Stations, but I guess for lighter workloads, a couple of Sparks could work too.

The architecture behind this is that the machines sit in the background (on your LAN), serving models to the team. People and agents can access those models. And the agents could also work asynchronously.

Imagine a tax advisor leaving work in the afternoon and handing work over to their AI computer, which works through bookkeeping accounts overnight, checking for inconsistencies and preparing reports. The next morning, it simply sends back the results and anything that needs human review.

So where does this leave us?

Even though AI hardware is getting more expensive, running local AI is becoming more useful because models are getting better and more efficient.

You don't need a massive AI cluster to benefit from it. A 128GB machine already gives you a pretty solid starting point – whether you're a power user or a small team.

A key insight that I got from our talk was that your local AI journey can (and should) increasingly look like this:

Stay tuned for more actual, current use cases for Local AI. I’ll cover those in one of my next articles.

If you ask me, there was never a better opportunity to have powerful AI working for you, on your own terms.

See you next Saturday,
Tobias