Sensitive Data and Local LLMs: Open Source to the Rescue (Just Not the Way You Think)

Here is the instinct almost everyone has. "We can't use a Chinese model, our data would go to China." It feels right. It is also mostly wrong, and the reason why changes how you should think about the whole thing.

The scary facts about DeepSeek are real. Its privacy policy says it stores your personal data in the People's Republic of China. Under China's 2017 National Intelligence Law, firms can be compelled to hand data to the state, with no obligation to tell you. South Korea's privacy regulator found DeepSeek shipped prompts to a Beijing server without consent. Wiz researchers found a public DeepSeek database exposing over a million records, including chat histories and API keys. Governments from the US to Australia to Taiwan have banned it on official devices.

Every one of those facts is about the cloud service. Not the weights.

The pivot most people miss

DeepSeek and Qwen publish their model weights under permissive open licenses. That means you can download them and run them on your own hardware. When you do, the prompts and the outputs never reach a server in China. There is no telemetry channel, no privacy policy, no cross-border transfer, because the model runs inside your own network.

So the sovereignty risk applies to the service, not to the self-hosted weights. Once you air-gap the model, the question "where does my data go?" has the same answer for a Chinese model as for an American one: nowhere. It stays with you.

This collapses the whole China-versus-US-versus-Europe debate. Origin stops being about data leaving the building. It becomes a question of three other things: licensing, built-in bias, and how much you trust a binary blob you did not build.

What origin still tells you

Running the weights yourself does not make origin irrelevant. It moves it.

Chinese models carry baked-in censorship and political bias in their outputs. That is a product quality issue you can test for, not a data leak. Harder to dismiss is the theoretical risk of a deliberate backdoor hidden in the weights. Vitalik Buterin lists exactly this in his self-sovereign LLM threat model. He also makes a sharp point that applies to every model regardless of flag: almost all "open" LLMs are open-weights, not open-source. You get the finished weights, not the training data or code. You cannot fully audit any of them.

American models are trusted by default in Western procurement, but they come with their own friction. Llama 4 is an easy starting point with a huge context window, but its license has a monthly-active-user threshold that can force a separate commercial agreement at scale. That is a lawyer problem, not a security one, but it is real.

So the honest scorecard is this. Self-hosting neutralizes the data question for every origin equally. What is left is licensing terms, output bias, and an unavoidable trust gap that no model fully closes.

Europe now has real options

If you want a European answer, 2026 finally gives you one. Mistral in Paris is the only European lab operating at commercial frontier grade, and its Small and Large models now ship under Apache-2.0. That makes Mistral the pragmatic European pick for deployment.

Behind it sits a growing public tier. EuroLLM-9B benchmarks well and targets the public sector. Teuken-7B came out of a German consortium and covers all 24 EU languages. Spain has ALIA, Portugal has Amália, and the EU-funded OpenEuroLLM and EUROPA projects are pushing toward very large multilingual models trained on European supercomputers.

These rarely top a benchmark. That is not the point. The selling point is control, multilingual coverage, and privacy-conscious design, which is what regulated buyers actually want.

What to actually run for RAG, summary, and compare

Strip away the geopolitics and you still have three real jobs to do. Here is the practical shortlist.

For an all-around default, run Qwen3. It balances quality, sizes, multilingual support, and tooling, and it ships under Apache-2.0. If you want a European deployment, run Mistral Small or Large. For long-document summarization, Qwen3-30B with its 256K context window handles big files well, and GPT-OSS-120B is a strong pick when you want consistent enterprise output. For thin hardware, Gemma 3 runs in 16GB of VRAM and Phi-4 fits in 8GB.

The rule of thumb worth remembering: pick by memory first, then by task. Qwen is the safe default, DeepSeek for logic-heavy work, Mistral for code and European deployment, Gemma for multimodal, Phi for small hardware.

Hardware is no longer the blocker. Consumer GPUs with 24 to 128GB of VRAM run 30B to 70B models after quantization. Ollama on an RTX 4090 or an M4 Pro gets you a working API in an afternoon. vLLM on a 5090 or a multi-GPU box is production grade.

Local is not the same as safe

Here is the part that gets skipped. Self-hosting keeps prompts off external servers. It does not make you secure.

The leak vectors move inside your walls. Runtime logs capture prompts and responses. Telemetry phones home if you leave it on. An API bound to the wrong interface is reachable from the network. A model file you did not verify could be tampered with. The standard advice is now a twelve-point hardening checklist: isolate network access, verify the SHA-256 of model files, bind the API to localhost, encrypt or disable prompt logs, lock down file permissions, and audit any plugins.

Compliance does not disappear either. On-prem removes the cross-border transfer problem, but GDPR still applies. You still need a lawful basis, data minimization, audit logging, and a data protection impact assessment for special-category data at scale. A GDPR-ready local RAG setup that survives a real security review is genuine engineering, not a weekend project.

The takeaway

Open source really is the answer for sensitive data. Just not because it lets you dodge scary foreign models. It is the answer because running the weights yourself takes the single hardest question, where does my data go, off the table entirely.

Once you have done that, the choice gets calmer. Read the license. Test for bias. Harden the box. Then pick the model that does the job. Qwen if you want the best default, Mistral if you want Europe, whatever fits your hardware and your task.

The data never leaves. That was always the point.

Sources

• Vitalik Buterin: My self-sovereign / local / private / secure LLM setup - Credible privacy-first practitioner's threat model, hardware, and model choice for a self-sovereign local LLM setup
• Witness.ai: DeepSeek Security Concerns — What Businesses Need to Know - The key cloud-vs-local distinction for DeepSeek risk plus the shadow-deployment caveat
• SitePoint: Definitive Guide to Local LLMs 2026 — Privacy, Tools & Hardware - Procurement framing, hardware tiers, and Ollama vs vLLM guidance for local LLMs
• PromptQuorum: Local LLM Security 12-Point Checklist - Concrete 12-point hardening checklist showing local does not equal secure
• PromptQuorum: Local RAG for Private Business Data — GDPR-Compliant - GDPR articles that still apply on-prem and when a DPIA is required
• regolo.ai: Practical RAG with Sensitive Documents on EU Infra - Reference RAG architecture where embeddings run locally so documents never leave infrastructure
• BankInfoSecurity: Asian Governments Rush to Ban DeepSeek - Government bans of DeepSeek and the China National Intelligence Law angle
• Finalist: Europe's Open-Source AI Pioneers - Map of European open-source labs and initiatives under the EU AI Act
• Slator: EU Backs Open-Source AI Model Covering 24 EU Languages - OpenEuroLLM, EUROPA, and EuroHPC funding facts for the 24-language EU model
• HuggingFace: Best Open-Weight LLMs to Run Locally in 2026 - Summarization model picks (Qwen3-30B, GLM-4.5V, GPT-OSS-120B) and the six families that matter
• ComputingForGeeks: Open Source LLM Comparison Table (2026) - Licensing detail — Apache-2.0 vs the Llama Community License MAU threshold