local-llm

In-House LLMs Crossed the Line in 2026. Here's What It Takes

A year ago, running your own LLM on sensitive data was a hobbyist project or a hyperscaler budget line.

Johan Kölhi

Johan Kölhi

— 7 min read
Share
In-House LLMs Crossed the Line in 2026. Here's What It Takes

A year ago, "run the model yourself" was a bad answer for most companies. The open-weight models were a step behind. The hardware was either a gaming rig held together with hope or a six-figure server. So sensitive data went to a cloud API, and everyone signed the data-processing addendum and looked away.

That answer flipped in 2026. Two things happened at once, and together they moved in-house inference from a science project to a real procurement decision.

What actually changed

First, the models caught up. By mid-2026 the capability gap between open-weight and proprietary models had effectively closed for ordinary business work. Open-weight coding scores now sit near the frontier. Mixture-of-experts designs deliver frontier-class quality on a fraction of the active compute. GLM-5.2 is a 744-billion-parameter model with only about 40 billion active per token and a million-token context window, and it competes with closed models from OpenAI and Anthropic.

The remaining lead for closed models is narrow. It lives in long-horizon agentic tool use and the hardest reasoning benchmarks. It does not live in summarisation, extraction, retrieval, or code completion. Those are the everyday jobs most companies actually want done, and open weights now do them well.

Second, the hardware got small and cheap. Desktop-class inference boxes arrived at $2,000 to $5,000. The NVIDIA DGX Spark packs 128GB of unified memory into a shoebox that draws under 200 watts, costs $4,699, and runs 200-billion-parameter models at 35 to 80 tokens per second. A CES 2026 software update added up to 2.5x more throughput on the same hardware. AMD's Strix Halo and Apple's M5 Max reach the same 128GB class for similar money.

Put those together and a mid-sized business can stand up a private LLM for $10,000 to $15,000. That number is the story.

The economics, honestly

Self-hosted inference costs roughly $0.001 to $0.04 per million tokens in electricity. Premium cloud APIs charge $2.50 to $15.00 for the same million tokens. At sustained volume that is up to 18x cheaper to own the metal.

One executive playbook models a $120,000 server breaking even in about four months on high-volume workloads. Another guide puts the monthly comparison at $2,000 on-prem versus $35,000 on cloud for the same throughput.

Do not read those numbers as a blanket win. They only hold at high, steady volume. Cloud still wins for low or spiky traffic and for zero operations. If your workload is a few thousand documents a month, buying hardware is a hobby, not a saving. The honest TCO counts staff, power, cooling, depreciation, and the eval lifecycle, not just the sticker price of the box.

Which models do the job

For general business work, GLM-5.2 is the strongest self-hostable all-rounder right now. DeepSeek V4 leads on reasoning. Kimi handles coding agents.

For a model that fits one 128GB box, look at Qwen 3.6-27B, Mistral Small 4, Devstral Small 2, Gemma 4 for laptops and edge, and Nemotron 3 if you want reproducible open training. GPT-OSS-20B or 120B gives you a US open-weight option if that matters to your buyers.

A rough sizing rule at Q4 quantisation: a 7B model needs about 8GB of VRAM, 14B about 12GB, 32B about 24GB, and 70B upward of 40GB. A quantised 70B fits comfortably in 48 to 64GB. That maps cleanly onto the 128GB desktop boxes.

The European shelf

Most of those names are American or Chinese. If your reason for bringing inference in-house is sovereignty, you probably want to know what Europe offers. The honest answer in mid-2026: more than you think, and it is finally deployable.

Mistral is the obvious starting point and still the only European lab operating at commercial frontier scale. Mistral Large 3 and Mistral Small 4 both ship under Apache-2.0 now, which removes the licensing question entirely. Small 4 fits one 128GB desktop box with room to spare. Devstral Small 24B scores around 90% on HumanEval, which puts it in the same conversation as the Chinese coding models rather than a tier below. Mistral Small 3 runs at roughly 50 tokens per second on 16GB of consumer hardware at Q4. For a company that wants a European vendor, a permissive licence, and hardware that fits under a desk, that is a strong combination.

Behind Mistral sits a genuine public-sector shelf. EuroLLM-22B is the largest in its series, trained from scratch on the MareNostrum 5 supercomputer in Barcelona by a consortium including Edinburgh, Paris-Saclay, Amsterdam, Unbabel and Naver Labs, and it matches or beats global models of similar size on multilingual tasks. Switzerland's Apertus is Apache-2.0, covers over 1,800 languages, and was trained with 40% non-English data by design. Germany's Teuken-7B, Spain's ALIA-40B and Portugal's Amália fill in national coverage. OpenEuroLLM has models landing this year, though nobody involved is claiming frontier scale.

How Europe measures up

Four comparisons, no flag-waving.

On raw benchmarks, the Chinese models lead. Qwen 3 235B-A22B tops the broadest range of open leaderboards. Qwen 3.6-27B leads local coding at 92.1% HumanEval and 77.2% SWE-bench. GLM-5.2 and DeepSeek V4 sit at the top for general work and reasoning. No European model is winning those charts, and pretending otherwise is not a procurement strategy. On licensing, Europe has caught up. Apache-2.0 across Mistral Large 3, Mistral Small 4 and Apertus matches Qwen's Apache-2.0 and DeepSeek's MIT. All of them beat Meta's community licence, which still carries a monthly-active-user threshold that can force a separate agreement. The licensing risk here sits with a US model, not a European one. On European languages, Europe wins, and it is not close. EuroLLM and the OpenEuroLLM projects target all 24 official EU languages as a design goal, not a side effect. If your sensitive documents are in Swedish, Portuguese or Greek, a smaller model that treated your language as a first-class citizen can beat a bigger one that treated it as an afterthought. Test that on your own corpus. The leaderboard gap and the gap on your documents are different numbers. On the thing everyone actually worries about, origin matters less than you think. Download open weights, run them air-gapped, and no data goes anywhere regardless of the flag on the training run. The DeepSeek risk that triggered government bans is a risk of the hosted service, not of weights sitting on your own box. What survives self-hosting is narrower: baked-in censorship and political bias in outputs, and the harder-to-prove worry about backdoors in weights nobody can fully audit. Those are real concerns, and they are the honest argument for a European or US model. "The data goes to Beijing" is not.

So: if procurement needs a European answer, Mistral Small 4 is the default and a good one. If the workload spans EU languages, test EuroLLM-22B and Apertus against it on your own documents. If you want the best quality per box and can defend the origin question to your own risk committee, the Chinese models still win on benchmarks. Most companies land on a mix, and that is a reasonable place to land.

What hardware to buy

Think in three tiers.

Team tier, $2k to $5k. One DGX Spark, Strix Halo, or M5 Max. Runs 24B to 70B quantised models for retrieval, summarisation, and extraction. This handles most "process sensitive documents" work on its own. Workstation tier, $8k to $15k. An RTX PRO 6000 Blackwell workstation. 96GB, roughly 1,792 GB/s, about 32 tokens per second on a 70B model. The real reason to step up is fine-tuning: this is the box that trains on your own data. Production tier, $100k and up. Multi-GPU B200 or MI350X servers with liquid cooling and dense power. This is where the four-month breakeven lives, but it demands capacity planning and people who run it for a living.

One caveat on tooling. NVIDIA gives you the full CUDA stack. AMD means ROCm or Vulkan. Apple means Metal and MLX. If your pipeline assumes CUDA, that assumption has a price.

The part people skip

Local is not automatically secure. You inherit the whole burden: network segmentation, authentication, prompt and output logging, verifying model weights, and controlling which documents your retrieval layer can reach. A misconfigured on-prem box is still a breach, just a differently-shaped one.

Local is not automatically sovereign either. Data residency means the inference runs in Frankfurt. Data sovereignty means that Frankfurt data stays outside the reach of foreign law. An EU cloud region gives you the first. Only owning and controlling the stack gives you the second.

Compliance does not disappear on-prem. GDPR still governs any personal data you train, tune, or run on, and a data-protection impact assessment may be required. The EU AI Act's main obligations apply from 2 August 2026, with fines up to €35 million or 7% of global turnover, and its high-risk duties around auditability and transparency apply no matter where the model runs.

That last point cuts both ways. The European Data Protection Board named on-premise inference the single strongest data-protection control available. So the regulator's preferred answer and the compliance risk both point at the same box. Owning it is the strong position, provided you actually harden it.

So, is it an option now?

Yes, for the first time as a default rather than a stretch. The pattern that works for most companies is hybrid: route the routine 80% of work (summaries, extraction, retrieval, parsing) to a compact local model, and reserve the hardest 20% of reasoning for a frontier API. Sensitive data stays in the building by default and only leaves for the jobs that genuinely need the frontier.

Start with one desktop box and a real workload, not a pilot that processes toy data. Measure quality with your own evals, count the full cost including the people, and decide from numbers. The line moved in 2026. Now the only question is whether your workload sits on the right side of it.

Sources

Read more