Local AI FAQs

Earlier this week I sent out a short “Local AI FAQ” in my daily Profitable AI Notes.

I got so much feedback that I decided to expand the format into a full newsletter.

So this one covers the 10 most common questions I get from people I work for or with when it comes to Local AI and open-source / open-weight AI models. (The difference between these terms is one of the top questions.)

The goal of this article is to serve as a practical guide for people trying to understand where Local AI fits into their business – not a technical setup tutorial.

Obviously, this FAQ could go on and on, but I’ve tried to keep each answer short and the whole thing easy to skim.

If you have a question I didn’t answer here, feel free to use the comments under this post and I’ll get back to you this way.

With that said, let’s jump right in!

I’ve written about the importance of Sovereign AI, and running local AI workloads in your business before.

But these days, something interesting is happening around open-weight AI.

When Nvidia CEO Jensen Huang shared Nvidia’s open letter around open weight AI, it quickly got 60M views and counting. In just about 6 days, the open letter that was also officially distributed by Microsoft was signed by over 230 organizations.

For context, the letter contains lines like this:

The core argument here is that open-weight AI has left hobbyist territory and earned its place as strategic (enterprise) infrastructure.

Open models give businesses an exit.

With a proprietary model, the provider can:

Leaving you empty-handed.

Relying completely on proprietary artificial intelligence is a bit like running a business completely on an army of contractors. It works, but the underlying capability never becomes yours.

That’s why I believe at some point you have to call your AI workers back to the office.

Open-weight models allow you to build and own your own AI which you can download, fine-tune, and use as long as you want.

If this reads like rocket science to you, it isn’t.

I recently met a CTO of an entirely non-tech (some people would even call it “legacy”) company with around 200 people who runs 90% of the company’s AI needs on hardware that cost less than a company car would cost for himself. Claude APIs come in just for some edge cases that require frontier intelligence.

I regularly advise clients on the options of local AI once they have a high-volume AI workflow into production.

I’m not saying you should cancel all your AI subscription and go open-source though. That would not make any sense (see the FAQ below).

But if you’re running business-critical workflows with AI, exploring what the world of open AI models has to offer you, is definitely a smart move.

In fact, if you want to try out Local AI, it literally takes about 1 hour and nothing more than a laptop that is less than ~5 years old. Everything else is covered in this LinkedIn Learning course by me which you can take for free of charge using the link in this post.

Anyway, we’re already getting deeper into the matter.

So here are the Top 10 FAQs.

Heads-up: This will be a longer email but since this is a list of FAQs it’s not intended to be read end-to-end (you could though). If you’re in a hurry, feel free to pick the questions that resonate most and skip / skim as you like.

So here are the questions I get asked most by customers my usual responses.

In short: running an AI model on hardware or infrastructure you control.

“Local” could mean:

The important distinction is control over where inference happens.

In short – no.

Open-source describes a model whose code, weights, and license meet open-source criteria. In the most narrow sense, that would also include details about the training process and data (which hardly an “open source” model provides).

That’s why most open-source models area really just open-weight means the model weights are available, but the exact details about how they were trained and not fully disclosed.

Local AI describes where the model runs.

You can:

In practice, most Local AI setups use open-weight) models – but the two terms are not interchangeable.

Let's put it this way.

There are no per-prompt API charges or monthly subscriptions like with cloud models.

However, you still pay for hardware, electricity, and maintenance.

Think of it like owning a car versus taking an Uber.

Once you've bought the car, every trip is cheap. But owning the car isn't free.

Local AI replaces variable usage costs with more fixed infrastructure costs.

What you’re really getting with Local AI isn’t necessarily an AI service that is cheaper, but one that has a more predictable cost structure.

Not if you're actually running it locally.

Once the model is downloaded onto your own machine – regardless of who developed it – it performs inference on your hardware.

Nothing has to leave your computer.

If you're paranoid, you could even unplug your machine from all network connections the model would continue working just fine.

Yes.

Assuming your laptop isn't more than 3-4 years old and has at least ~8 GB RAM.

(In practice, available RAM is usually a much bigger limitation than CPU performance.)

That said, this won’t run GPT-5 class models for you.

But you can absolutely run Small Language Models (SLMs) that are excellent for well-scoped tasks like document extraction, classification, summarization, and many automation workflows.

The rough “class” of a model is measured by its parameter count. It is an imperfect proxy for capability (think of the days when we measured a CPU’s performance by looking at its GHz), but it still helps estimate hardware requirements. SLMs become useful at around 1 billion parameters, whereas frontier models such as GPT-5 supposedly have over 1 trillion parameters.

For example, I can run Google’s Gemma-3 SLM (4 billion parameters) easily on my 2023 MacBook Air with 8 GB RAM. A modern Macbook Pro with 16GB RAM would run Gemma-4 (12 billion parameters). For more advanced models, you would need more bespoke hardware, see below:

There’s not one universal winner.

If you’re looking at benchmarks, you can find a pretty comprehensive comparison of the general “intelligence” of open source models (that you could run locally or anywhere in the cloud) on Artificial Analysis, a research group that tracks model performance across multiple tests.

The “best” model here changes every couple weeks, but most recently models developed in China are leading the pack here.

That said, you shouldn’t look for the best model in a hypothetical benchmark, but the best model that works for your requirements. Which is typically a combination of the following questions:

As a rule of thumb, here are some heuristics I use

If I need to mimic the style of ChatGPT or re-use prompts I’ve written there, I use OpenAI’s gpt-oss models because they “feel” very similar to what many users are used to.

If I need to analyze image or audio files, I typically look at Google’s Gemma models because the family includes multimodality across the board.

If I need agentic use (calling tools, running in loops) on Nvidia hardware I give Nvidia’s Nemotron models a try first, because they are are designed in particular for efficient agentic reasoning and tool use on Nvidia hardware.

I generally do not use the latest and greatest open-weight models like Kimi-K3 or GLM-5.2 (or any other LLMs that lead the benchmarks) because there’s typically no way for me to run them locally.

(See next point)

Personally, I use Local AI less as a private ChatGPT replacement, but more as a private automation engine.

For example, when I build AI solutions for clients I’m running the evaluations for those solutions locally by testing the solution with local LLM judges on my Nvidia Spark.

I have also workflows running for extracting insights from private transcripts, or helping me manage multiple email inboxes.

And the last use case is data analysis where I use local AI models to analyze thousands of records, for example generating insights from surveys or other unstructured datasets.

I never use Local AI as an interface to chat. I use it to make workflows run quietly in the background (mostly using a local installation of n8n as the automation engine.)

I don't think there is one. There are thousands.

Here are some of my favorite use case categories:

Every business has documents. Contracts, invoices, purchase orders, forms, spreadsheets. And every business at some point needs to organize these documents (like routing the right invoice to the right department, or checking whether a contract contains any red flags.)

If you’re consistently processing a few hundreds of documents every week – Local AI might be a great fit, because the data stays on your infrastructure, you have stable costs, and the workflow can run 24/7.

Do your employees regularly need to search through company policies, technical documentation or operating procedures?

A local AI model combined with Retrieval-Augmented Generation (RAG) can easily answer questions without sending confidential company data to an external provider.

Need to pull names, addresses, invoice numbers or product information from PDFs, emails or scanned documents? Or match naming schemas?

Small local models are surprisingly good at structured extraction tasks.

And because they're inexpensive to run, you can process thousands of documents without worrying about API costs.

These are simple AI decisions that happen thousands of times every day.

Good candidates for Local AI!

Many AI tasks don't need an answer in two seconds. They can simply run overnight!

Examples:

Nobody is waiting for the result in real time, so speed isn't the limiting factor. Cost and reliability are.

The common thread through all of these is that they don't require frontier-level intelligence.They require an AI model that's good enough, affordable to run, and close to your data.

That's the happy place for Local AI.

There’s a huge difference between running a well-scoped AI workflow across thousands of records overnight and serving a responsive chat interface to hundreds or thousands of users.

At that point, you are operating a platform rather than merely running a model.

Background automation tasks, on the other hand, hardly need any interface or “user management” at all.

If you do want a local ChatGPT clone, I recommend checking out OpenWebUI which is one of the best “ChatGPT clones” out there.

No.

Local AI is a great choice when you need control, privacy, predictable economics, independence, offline operation, or high-volume processing.

That said, if you’re not sure whether you’ll need any of these, starting with pay-as-you go models in the cloud is often the better choice.

There are rare cases when no data at any circumstance must leave your computer (like in investment due diligence or legal cases), but generally you’re faster prototyping a solution in a secure vendor ecosystem like Microsoft Azure / Copilot or any other trusted provider.

But starting in a vendor ecosystem shouldn’t mean you’re locked in there forever.

You can still choose.

And that’s the whole point.

We’re already 2,000 words in and I feel I’m just getting started.

So let’s wrap this up with one key takeaway for you.

Local AI is not another movement you need to join.

Use cloud AI (and typically proprietary AI models) when you need:

Think of Local AI as another capability you should understand at a high level.

You may decide not to use it today. But knowing what can run privately, cheaply, and independently will help you design better AI systems.

What matters is the mix that works for your business – and whether you chose it intentionally from an economic perspective.

See you next Saturday,

Tobias

PS: If you want to get hands-on and run a Local AI model on your computer in less than an hour, check out my LinkedIn Learning course Run Local AI Workflows with Ollama and n8n – you can watch it for free using the link in this post.