Private & Sovereign AI

How Private AI actually works.

A plain-English field guide to hosting, adapting and running your own LLMs — what the terms really mean, where the work is, and why "just fine-tune it" is the most expensive wrong turn in enterprise AI.

Audience: leaders who know the vocabulary and want the mental model behind it — no code required.

The one belief we need to fix

"Fine-tuning is the golden bullet."

It's the most common instinct — and the most expensive. The belief is that if you fine-tune a model on your documents, it "learns your business" and just knows the answers. That is not how it works, and building on that belief burns budget, underdelivers on accuracy, and locks you into a re-training treadmill.

By the end of this deck you'll be able to tell — for any use-case — whether the right tool is retrieval, an agent, fine-tuning, distillation, or a combination. That judgement is the entire difference between an AI programme that compounds and one that stalls.

Fine-tuning is a real, powerful tool — for the right job. The skill is knowing which job that is.

2

Foundational mental model

An LLM is a reasoning engine, not a database.

A database stores facts and returns them exactly. An LLM stores skills — language, reasoning, patterns — as billions of numbers ("weights").
It doesn't "look up" answers. It predicts the next word, one at a time, using those skills plus whatever you put in front of it.
Its knowledge is frozen at training time and blurry — great at reasoning, unreliable at reciting your specific facts.

The analogy we'll use all deckThink of the model as a brilliant new hire with amnesia. Superb reasoning and judgement, fluent in the field — but no memory of your company. You can (a) send them to training to change how they work, or (b) hand them the right files for each task. Most problems are a filing problem, not a training problem.

Almost every enterprise AI mistake starts with treating the model as a memory to be loaded, instead of a mind to be equipped.

3

The two things you can change

You can change the weights, or change the context. That's the whole game.

Path A · change the weights

Training & fine-tuning

Send the new hire back to school.

Permanently alters the model's behaviour and skills
Expensive, slow, needs data + GPUs + expertise
Good for style, format, tone, reflexes — bad for facts
Every update = re-train

Path B · change the context

Retrieval & agents (at run-time)

Hand the new hire the right files for this task.

Feeds the model fresh, specific information per request
Cheap, instant to update, no re-training
Best for knowledge, live data, permissions
Update = change the files, not the model

"Teach our knowledge" sounds like Path A. It is almost always a Path B problem.

4

Before you spend a rupee on GPUs

The cheapest lever is what you say to the model.

Context / prompt engineering — the instructions, examples and formatting you hand the model at run-time. Free, instant, no infrastructure.
A clear system instruction + a few worked examples ("few-shot") often gets you most of the way — before any retrieval or tuning.
It's the same Path B ("change the context") — just the simplest version of it.
Every serious build starts here and only climbs the ladder when prompting genuinely runs out of road.

AnalogyBefore you send your new hire back to school (fine-tuning) or wire them into your systems (agents), first just write them a good brief. Astonishingly often, that's the whole job. Skipping this step is the most common way teams overspend on AI.

Rung 0 of the ladder: never pay to train what a better prompt already does for free.

01

How the model works under the hood

Architecture, and why it decides your GPU bill.

Model architecture · the one that shapes cost

Dense vs. Mixture-of-Experts (MoE)

Dense

every part works on
every word

MoE

a router picks a few
"experts" per word

Dense — all parameters fire for every token. Simple, predictable, easier to fine-tune. (Gemma 3, Phi-4.)
MoE — only a slice fires per token. DeepSeek V3: just 37B of 671B parameters active — frontier quality at a fraction of the compute.
The catch: all experts still sit in GPU memory. MoE saves compute, not VRAM — so it needs big/expensive GPUs even though each answer is "cheap."
By mid-2026, MoE is the dominant design in serious open-weight models.

Why a CTO cares: architecture sets your hardware. MoE = fewer FLOPs but high memory floor (great for busy, batched servers); dense = smaller memory, simpler tuning (great for one GPU). Sources: Raschka "Big LLM Architecture Comparison" 2026; Vinci Rufus, Dense vs MoE, 2026.

6

Inference · what "running the model" really costs

Why serving a model is an engineering problem, not a download.

It generates one token at a time

Each word depends on the last, so the model runs hundreds of times for one answer. Latency and cost scale with output length.

The KV-cache is the hidden GPU hog

To avoid re-reading the whole conversation each step, it caches "attention" in GPU memory. Long context = huge cache — often what actually forces you onto more GPUs.

Batching is where the economics live

One user barely uses a GPU. Serving many requests at once (continuous batching) is what makes cost-per-answer collapse — and why "tokens/sec for one user" is a misleading number.

This is why serving engines exist

vLLM, SGLang, TensorRT-LLM are specialised software whose whole job is packing memory and batching efficiently. Choosing and tuning them is the deployment work.

"Free, open-weight model" ≠ free to run. The GPU-efficiency engineering is the real cost — and the real skill.

7

The lever that halves your GPU bill

Quantization — shrinking the model to run it cheaper

FP16

full size

100% memory

→

FP8

~50%

→

INT4

~25%

Weights are stored as numbers with a certain precision. Quantization stores them with fewer bits — smaller, faster, cheaper to run.
Going from full precision to 4-bit can cut memory to a quarter — fitting a 70B model where only a 20B fit before, or the same model on fewer GPUs.
The trade-off: a small, usually acceptable quality drop — which we measure on the eval harness, never assume.
It's also what makes QLoRA possible (tune a big model on one GPU).

Quantization is the single biggest knob on "predictable, lower GPU cost" — and it's a measured decision, not a default.

02

Fine-tuning, demystified

What it is, what it isn't, what it costs, and the data it demands.

Myth vs. reality · with the evidence

Fine-tuning teaches behaviour, not facts.

17%

base model factual accuracy on a knowledge test

22%

after fine-tuning it on those exact facts — barely moves

82%

when the facts are retrieved and shown at run-time instead

Two peer-reviewed studies land the same verdict: models struggle to absorb new facts through fine-tuning, and retrieval beats it decisively — most of all for the niche, proprietary knowledge enterprises care about. Fine-tuning changes how the model responds; it does not reliably change what it knows.

Sources: Ovadia et al., "Fine-Tuning or Retrieval?" (Microsoft, EMNLP 2024, arXiv:2312.05934); Soudani et al., 2024 (arXiv:2403.01432). Honest caveat: the 82% uses an ideal retriever + small model on a long-tail test; real systems land lower — but the direction is robust and widely replicated.

9

The core technique · in one picture

What "LoRA" actually is

W

billions of
weights
FROZEN

+

A

B

tiny trainable adapter
(~1–2% of the size)

LoRA = Low-Rank Adaptation. Instead of retraining billions of weights, you freeze them and train two small "adapter" matrices bolted on the side.
You end up training ~1–2% of the parameters — so it fits on one GPU, in hours not weeks, for a few dollars.
The adapter is a small file you can swap in and out. One base model can host many adapters (one per task/client).
It changed the economics: fine-tuning went from a data-center project to something a small team does routinely.

LoRA didn't make fine-tuning teach facts — it made behaviour-tuning cheap. That's a big deal, but for the right jobs only.

10

The fine-tuning toolkit · plain English

LoRA → QLoRA → DoRA, and the tools that run them

Method	In one line	When you'd use it
Full fine-tune	Retrain all weights.	Rarely. Max control, max cost — deep behaviour change with lots of data + GPUs.
LoRA	Train a small side-adapter; freeze the rest.	The default. Cheap, fast, swappable. 95% of enterprise fine-tuning.
QLoRA	LoRA + 4-bit compression of the base.	Tune a 30B+ model on a single GPU. The single-GPU default.
DoRA / QDoRA	A smarter LoRA (splits weight into size + direction).	When you want a bit more accuracy at the same cost — the 2026 recommended starting point.

The tools that do it (2026): Unsloth (fastest on one GPU) · Axolotl (multi-GPU, production, reproducible) · LLaMA-Factory (point-and-click UI) · TorchTune (Meta's PyTorch-native). All now support LoRA/QLoRA/DoRA + preference tuning — increasingly complementary, not competing.

Sources: Spheron & TheAIEngineer framework benchmarks, 2026; AppScale "LoRA vs QLoRA vs DoRA," 2026; Turing Post on QDoRA.

11

Two flavours of fine-tuning

Teaching a skill vs. teaching judgement

Instruction tuning (SFT)

"Do it like this"

Show input → ideal output. The model copies the pattern.

Learns a task or format by example
Data = demonstrations of the right answer
What "fine-tuning" usually means

Preference tuning

"This answer is better than that one"

Show good-vs-bad pairs. The model learns what's preferred.

RLHF — the original (used to make ChatGPT helpful)
DPO — a simpler, cheaper modern replacement
GRPO — what powers today's reasoning models

Why a CTO should know this exists: preference tuning is how you align a model to your standards — safety, tone, "our way of answering" — when there isn't one single correct output, only better and worse ones. It's behaviour-shaping, still not knowledge.

The part everyone underestimates

What data does fine-tuning actually need?

// Not documents — labelled examples of the // behaviour you want, hundreds to thousands: { "instruction": "Summarise this claim note for an adjuster.", "input": "<the raw claim note>", "output": "<the ideal summary, in house format, written by an expert>" }

Input→output pairs, not a pile of PDFs. You're teaching a skill by example, so each example must show the ideal answer.
Quality > quantity. A few hundred excellent expert-written examples beat 100k mediocre ones. Garbage in, garbage baked in — permanently.
Someone has to create and label these — usually your scarce domain experts. This is the real cost and timeline, not the GPU.
For tone/preference tuning you also need "good vs. bad" pairs (which answer is better).

Fine-tuning's bottleneck is never the training run — it's producing the curated, expert-labelled examples. Budget for people, not just GPUs.

12

Nuance & the hidden treadmill

Not all models tune the same — and tuning is never "done."

Fine-tuning differs by architecture

Small dense models (Gemma 3, Phi-4) — easiest and cheapest; ideal LoRA targets.
Large MoE models — trickier: you can destabilise the expert "routing," so they need more care and compute.
Reasoning models — often better distilled than fine-tuned (next section).

The maintenance reality

Data changed? The adapter is now stale — retrain.
New base model released? Re-tune to benefit.
No per-user permissions — knowledge is baked in for everyone.
Every tune must be eval-gated, or you ship silent regressions.

This is exactly the "constant re-training" cost clients fear — and it's real if you use fine-tuning for knowledge. Use it only for durable behaviour, and the treadmill mostly disappears.

13

03

The better answer: give it access

Retrieval and agents — how you inject knowledge without touching the weights.

The word behind all the hype

What an "agent" actually is: a loop, not a magic box.

1 · Think

"What do I need to answer this? I should check the claims system."

→

2 · Act

Calls a tool — runs a query, hits an API, searches docs.

→

3 · Observe

Reads the result the tool returns.

↺

4 · Repeat / answer

Loops until it can answer — with citations.

An agent is just a model given tools and allowed to use them in a loop until the job's done. The tools are wired up via MCP (Model Context Protocol) — think of it as a universal adapter (a "USB-C for AI tools") so any model can safely plug into any system through one standard.

Why this matters for us: we build a reusable library of MCP connectors (to databases, CRMs, ticketing, internal APIs) once, then re-deploy them per client. That library is compounding IP — and it's what makes "agentic access to your information" real instead of a slide.

Injecting knowledge at run-time

Classic RAG vs. agentic access — and why agentic wins for regulated data

Classic RAG · good for static document piles

Documents

copy your files

›

Chunk + embed

into a search index

›

Vector DB

a 2nd copy to secure

›

Retrieve top matches

at question time

›

Show to model

as context

Agentic access · good for live, permissioned systems of record

Model plans

what info it needs

›

Calls a tool

query DB · API · search
(via MCP)

›

System of record

answers live, under the
user's own permissions

›

Reads result, repeats

until it can answer,
with citations

Why agentic is usually the better fit for regulated clients: data stays in the source (no second copy of your PHI/PII to govern), it's always live, and access is enforced per-user at the source — the one thing fine-tuning fundamentally cannot do. Use classic RAG as one tool for the genuinely static, unstructured stuff (policy manuals, contracts).

15

Putting it together · the decision ladder

Now the ladder makes sense — climb only as far as the problem needs.

1

Private inference — the model, running in your walls. Solves privacy, residency, predictable cost. Many use-cases stop here.

2

+ Retrieval / agentic access — equip it with your live, permissioned information. Solves ~80% of "make it know our business."

3

+ Targeted fine-tuning — only for durable behaviour (tone, format, tool-use). A scalpel, applied after 1 & 2, never instead of them.

4

+ Continuous LLMOps — evaluation, guardrails, monitoring, upgrades. What keeps it correct, safe and current over time.

The client asks for rung 3. The answer almost always starts at rungs 1–2. Getting that order right is the expertise.

16

Making it concrete · one use-case, all four rungs

Worked example: a claims-triage assistant for a health insurer

Rung	What we do	Why this rung, not fine-tuning
1 · Private inference	Host an open-weight model in the insurer's VPC. No PHI ever leaves.	HIPAA + residency solved on day one.
2 · Agentic access	Agent queries the claims DB, policy rules API, and member records live — each under the adjuster's own permissions — plus searches the static policy manuals.	The "knowledge" is live and per-user. Fine-tuning could never keep balances current or enforce who sees what.
3 · Targeted fine-tune	A small LoRA so every summary follows the insurer's exact adjuster-note format and house tone.	Pure behaviour/format — the one thing fine-tuning is genuinely good at.
4 · LLMOps	Eval harness on real historical claims gates every change; guardrails block PII leaks; dashboards track accuracy & cost.	Makes an agentic system safe enough for a regulated insurer.

The client asked to "fine-tune a model on our claims data." What actually delivered value was rungs 1–2, with fine-tuning as a 5% finishing touch.

04

Distillation: the real game-changer

How you get frontier-class skill at a fraction of the cost.

The technique

What distillation is: a big model teaches a small one.

Teacher

huge, expensive,
frontier-class

→

generates thousands of
worked examples with its
full reasoning shown

Student

small, cheap,
fast

→

trained to imitate
the teacher on
your task

Specialist

teacher-like
skill, student
price

You use the big model not in production, but as a trainer — it produces high-quality demonstrations, and a small model learns to copy them on one narrow job. It's fine-tuning where the labels come from a smarter model instead of humans.

This is the one place "training a model" reliably pays off at enterprise scale — and it directly attacks the token-cost problem.

18

Why it's a game-changer · a real example

DeepSeek-R1: a 671B reasoning model, poured into 7B

671B

teacher size (frontier-class reasoning)

7–14B

student that keeps most of the reasoning skill

~16×

lower latency at the small end — and a fraction of the GPU cost

DeepSeek distilled its giant reasoning model into small Qwen- and Llama-based students. The result beat trying to train those small models directly, and the 7B–70B students retain a high fraction of the teacher's performance while running on modest hardware. Only the tiniest (1.5B) student drops sharply on hard multi-step tasks.

The Coditas play: take a big or frontier model, distil it into a small specialist for a client's one high-volume task (claims triage, ticket routing, doc extraction) → a model that runs on a cheap L40S instead of an H100, answers faster, and never leaves their building. That is how "reduce token cost + predictable bills" actually gets delivered.

Sources: DeepSeek-R1 distillation reports & analyses, 2025–2026 (EmergentMind, XsOne). Student accuracy vs. size is task-dependent; hard-reasoning tasks degrade fastest at very small sizes.

19

05

Doing it for real

Choosing the model, the serving stack, and the operations around it.

Model selection · a process, not a favourite

How we pick the model — a filter funnel per use-case

1

Constraints first licence (MIT vs. restricted), data-residency, hardware budget

2

Capability shortlist size + skill for the task — reasoning, coding, long-context, multilingual

3

Fit the hardware will it run on the GPUs they'll actually buy?

4

Bake-off on THEIR data eval 2–3 finalists on real tasks — the deciding step

The 2026 open-weight field

Generalist, permissive licence: DeepSeek V3.2 (MIT, ~70% SWE-bench), Qwen3, GLM-5.x.
Runs on a single GPU: Gemma 3 27B, Phi-4 14B.
Coding specialist: Qwen3-Coder.
Watch the licence: some carry acceptable-use / revenue clauses that flip "free" to "paid" at scale — we vet every one.

There is no single "best" model. Rankings shift monthly; the winner is whichever passes the bake-off on the client's own task. Model choice is per-engagement, and our IP is model-agnostic.

Sources: Open-weight leaderboards (llm-stats, Vellum, Onyx), July 2026. Snapshot — expect movement.

21

The serving stack · "is there something better than vLLM?"

Pick the engine for the workload — and yes, turnkey options exist.

Option	What it is	Use it when
vLLM	The flexible default. Runs almost any model on almost any GPU.	Your safe starting point; multi-hardware; fast to stand up.
SGLang	Optimised for shared context & multi-step. ~29% faster on those.	Agentic / chat / RAG workloads — often the better pick for us.
TensorRT-LLM	Compiles to a specific NVIDIA GPU. 15–30% more throughput.	Max performance, stable model, NVIDIA-only, can absorb the setup.
NVIDIA NIM	Turnkey, vendor-supported container (SLAs, CVE patching).	Client wants a supported product, not a DIY stack (needs NVIDIA AI Enterprise licence).
Ollama	Dead-simple local runner.	Prototyping & developer laptops — not production scale.

Around the engine we run it on Kubernetes with KServe / BentoML / Ray Serve for scaling, plus a gateway, guardrails and monitoring. Plenty is pre-built and reusable — the value is choosing, wiring and tuning it for their constraints.

Sources: Inference-engine benchmarks (Spheron, Yotta, LeetLLM), 2026; NVIDIA NIM + KServe docs, 2026.

22

Where "in our infra" actually means

Three deployment topologies — increasing isolation, increasing effort

VPC-isolated

Most common. We deploy into their cloud account (AWS/Azure/GCP). No data egress, their identity & keys. This is what most vendors call "private."

easiest · elastic

On-premises

Runs in their datacenter on their own GPUs. Harder: capacity planning, driver management, no cloud burst — but total physical control.

harder · fixed capacity

Air-gapped

No internet at all. Model weights, updates, even patches are shipped in deliberately. Maximum security — and the least-competed, premium tier.

hardest · premium

The trap in the word "air-gapped": most vendors use it loosely to mean VPC-isolated. True air-gap changes everything — you design for no-phone-home from day one. Knowing the difference is part of scoping honestly.

The question every CTO asks · "how much iron?"

GPU sizing — the rough math, in plain terms

Rule of thumb: at 4-bit, a model needs roughly half its parameter count in GB of GPU memory — a 14B model ≈ 8GB, a 70B ≈ 40GB.
Then add the KV-cache for concurrent users and long context — often as much again. This is what quietly drives the GPU count up.
So: a 14B model runs on one mid-range GPU (L40S/RTX); a 70B on a big GPU or two; a giant MoE needs a multi-GPU node.
A typical first private deployment is 1–8 GPUs — not a supercomputer. The Assessment sizes it precisely for their volume.

Indicative GPU rates (2026)

GPU	Rent/hr	Good for
L40S	~$0.72	small/distilled models, single-GPU
A100 80G	~$1.64	mid-size models, RAG
H100	~$2.90	large models, heavy batching

The distillation payoff again: shrink the workload from an H100 to an L40S and the hourly cost drops ~4× — for a task that runs all day, that's the whole ROI story.

Sources: Spheron GPU pricing, Apr 2026; Layer3Labs VRAM sizing, 2026. Rates are volatile spot/marketplace figures — directional only.

A real decision · not everything is bespoke

Build vs. buy — and why we mostly do neither extreme

Buy a platform

Fireworks, Cohere North, NVIDIA NIM. Fast, supported, private-capable. But you rent someone's roadmap and pay per-seat/per-token.

when: speed > control

Build from scratch

Full DIY on open-source. Max control & lowest run-cost, but you own all the plumbing, eval and ops forever.

when: scale & differentiation

Assemble & integrate ← us

Proven open components (vLLM/SGLang, MCP, eval tools) wired into their infra, with our reusable IP. Their asset, no lock-in.

the pragmatic middle

The platforms aren't competitors to fear — they're parts we can assemble. The value is the integration, the eval, and the ops around them.

The whole thing · one diagram

Reference architecture — all of it, inside the client's walls

CLIENT PERIMETER · their cloud / on-prem / air-gapped

⛔ no data egress

① Gateway

one OpenAI-compatible endpoint · auth · guardrails · audit log · routing

GPU inference cluster

vLLM / SGLang / NIM · the model(s) + swappable LoRA adapters

Guardrails & safety

prompt-injection · PII · output validation

② Agent orchestrator

plans, calls tools, composes grounded answers with citations

MCP tool servers

DB · CRM · ticketing · APIs — queried under the user's own permissions

Hybrid search

vector + keyword over the static document corpus

④ Evaluation harness

golden datasets · quality gate in CI on every change

Observability

tracing · cost · latency · drift monitoring

Their IdP + KMS

identity & keys stay theirs

Delivered as reusable Terraform + Helm blueprints. ① ② ④ map to ladder rungs 1, 2 and 4 · fine-tuning (rung 3) produces the swappable adapters shown in the inference cluster.

23

The unglamorous part that is the moat

Evaluation & LLMOps — how you keep it correct and safe

The eval harness (our crown-jewel IP)

A golden dataset of real questions + expert answers, built with the client's SMEs.
Every model swap, prompt change or new adapter must pass the eval before it ships — a gate in the release pipeline.
Mix of automated metrics + "LLM-as-judge" + human spot-checks. Nobody else does this rigorously — so it's how we prove quality.

The 2026 tooling

DeepEval — broad: RAG, agents, safety, 50+ metrics.
RAGAS — lightweight retrieval-quality metrics.
Promptfoo — red-teaming & security.
MLflow / Langfuse / Braintrust — tracking, tracing, versioning.

Continuous, not one-off: pre-production gate → shadow traffic → post-deploy monitoring.

Sources: DeepEval / RAGAS / TruLens comparisons & MLflow agent-eval, 2026.

24

Being honest about what can go wrong

How these systems fail — and how we contain it

Failure mode

Our containment

Hallucination — confident, wrong answers.

Ground every answer in retrieved sources with citations; the eval harness measures it.

Prompt injection — malicious text hijacks the agent.

Input filters, tool allow-lists, and least-privilege per-user permissions at the source.

Data leakage — model reveals what a user shouldn't see.

Access enforced at the system of record, not the model; PII output checks.

Silent regression — an update quietly gets worse.

No change ships without passing the eval gate in CI.

Agent runs amok — wrong or costly tool actions.

Read-only by default, human-in-the-loop for writes, full action tracing.

The whole deck in one table

Myth → reality

What people assume

What's actually true

"Fine-tune the model on our data and it'll know our business."

Fine-tuning teaches behaviour; knowledge comes from retrieval / agents.

"RAG means copying all our documents into a vector database."

Often better: agentic access to live systems, under each user's permissions.

"Open-weight models are free."

The model is free; the GPU efficiency & ops are the real cost and skill.

"Training a custom model is the hard, valuable part."

The value is eval, integration & ops; distillation is where training truly pays.

"Pick the best model and standardise on it."

There's no single best — you bake off per use-case; the IP is model-agnostic.

25

The takeaway

Fine-tuning is a scalpel. Most of the value is in equipping the model — and proving it works.

Retrieve for knowledge. Fine-tune for behaviour. Distil for cost. Evaluate for trust.

That sequence — in that order — is the difference between an AI programme that compounds and one that stalls. It's also exactly what Coditas is built to deliver.