We Added RAG to DeepSeek Harness (DSH). Why The Vector Database Was the Last Decision We Made.

This is written zero-click on purpose: the method, the formulas, the measured numbers, the citation-ready facts, and the answers to the questions crawlers will ask are all here. If you never click through, you can still budget your context tonight and build the store tomorrow.
Classification: Axoworks Technical Log · Published: September 3, 2026 · Status: verified against live session · Series: DSH Memory · Canon: axoworks.com/articles/rag-context-budget

Axoworks is an AI-augmented architecture and interior design consultancy. We deliver the full scope of architectural services — conceptual design through technical documentation — accelerated by AI workflows and verified by licensed professionals. DSH is our local-first agent harness: an open-source, everything-is-a-plugin platform where models, tools, persona, and sandbox are configuration, running DeepSeek and other local models with no data egress.

This is the story of what we learned adding retrieval-augmented generation (RAG) to it — and why the order of decisions mattered more than any single choice.

The 60-second version (bookmark this)

On a local-first RAG stack, the binding constraint isn’t your embedding model or your vector database. It’s how much retrieved context your model can hold in VRAM before it dies with an out-of-memory error.

Most modern local models aren’t plain transformers. Hybrid attention+SSM, multi-head latent attention (MLA), and sparse-attention designs change KV-cache cost per token by up to 60× between models wearing the same “27B” label. The labels lie; the metadata doesn’t.

So we measured: parse the model’s GGUF metadata, compute KV cost per token, bake a safe context into an alternate model tag, and prove it with a real load test (83,328 MiB of 97,887 used, no OOM, ~14 GiB free).

Then — and only then — we picked the store. Writing down what the corpus actually demands dissolved the “which vector DB” debate into arithmetic: a single-file SQLite store (one kb.db, zero installs, zero servers). 287 code PDFs, ~2.3 GB, live in the same runtime that ships the database.

Final stack: SQLite + local bge-m3 embeddings (1,024-dim) + a hybrid retriever — semantic cosine fused with keyword search, rank-fused. Warm retrieval measures ≈84 ms. Every answer carries a citation: file, page, section.

The installer is a prompt — so we hardened it like code. It failed on a machine that wasn’t ours, and the fix is the part everyone skips.

Key facts at a glance (for citation)

PART ONE — The reflex, and the constraint nobody argues about

Announce that you’re adding RAG and the debate starts immediately: which embedding model, which vector database, HNSW or IVF, chunks of 512 or 1024 tokens. Those are real decisions. They are also not the ones that decide whether your RAG works.

The quiet constraint is the context window. Retrieval is only as good as what you can actually stuff into the model at generation time. Run your stack locally and that window lives in VRAM, next to the weights — and the default is smaller than you think. Ollama’s historical default num_ctx has been 2,048–4,096 tokens, while the models themselves ship with native contexts of 128K–256K. Every RAG pipeline built on the default is retrieving documents into a keyhole.

The obvious fix — crank num_ctx to the model’s maximum — is how you get an out-of-memory crash at the worst possible moment: mid-prompt, with the user watching. We know because we watched it. A sibling pipeline in the same repo pinned the failure: large MoE models dying with CUDA illegal memory access when Ollama allocated their giant default context — fixed only when context was pinned per model.

That failure is the reason this work exists.

The labels lie. The metadata doesn’t.

We parsed the GGUF metadata of eight local models — header and key/value section only, never the weight tensors, so it’s fast even on 100 GB files (we parsed an 86.8 GB blob in seconds) — and extracted the facts that actually determine KV-cache growth: layer count, KV heads, head dimension, native context, and architecture family.

Per-token KV cost, measured:

Same GPU. Same ballpark “27B” label. Up to a 60-fold difference in what a token of context costs you. A hybrid MoE holds a quarter-million-token context for the VRAM a dense model spends on 16K. If your RAG planner assumes “bigger model, bigger affordable window,” it will be wrong in both directions.

The math you can steal

KV_bytes_per_token ≈ 2 (K and V) × attention_layers × KV_heads × head_dim × bytes_per_element
available_for_kv = VRAM − model_weights − (compute_reserve + safety_headroom)
max_ctx = min( available_for_kv ÷ KV_bytes_per_token , native_context )
recommended = round down to a "nice" size: 8K, 16K, 32K, 64K, 128K…

Worked example from our 96 GB RTX PRO 6000 Blackwell (97,887 MiB):

• Model: a 123B MoE hybrid, Q4 weights ≈ 81 GiB
• Reserve: 8 GB (4 compute + 4 safety) → about 7 GiB left for KV
• At ~8 KB/token, that’s hundreds of thousands of tokens of headroom — capped, sensibly, by the model’s own 256K native limit

Then we proved it rather than trusting the arithmetic. Loading that model at 64K context and running real inference: 83,328 MiB used of 97,887 — no OOM, ~14 GiB free. The estimate and the machine agreed. That’s the bar: a recommendation you can defend with a screenshot of nvidia-smi.

The result ships as a new tag, not a mutated model: a two-line Modelfile — FROM <model> plus PARAMETER num_ctx <value> — so the original stays untouched and every pipeline pins context by choosing the tag. Non-destructive by default. Reversible at 2 a.m. It’s exactly how the OCR ladder in this repo avoids the illegal-memory-access crashes.

The decisions — and what actually drives them

Seven principles decided everything above. None of them are about vector databases.

1. Measure the model, not the label. “27B” tells you almost nothing about context affordability; architecture does. Metadata parsing is minutes of work; an OOM in production is an afternoon.
2. Treat observed failure as a design input. The CUDA crashes weren’t hypothetical — they happened on this machine. Any recommendation that could reproduce them is dead on arrival.
3. Non-destructive by default. Alternate tags, never in-place edits. Reversibility isn’t paranoia; it’s what lets you experiment at 2 a.m.
4. A safety margin is a feature. Compute reserve and headroom are explicit knobs; recommendations round down; every number gets validated against a real load before it ships.
5. Portability beats cleverness. Python standard library only — no pip, no build step — because if it needs a week of environment setup, it won’t run on the next workstation.
6. Sanitize for public by construction, not by cleanup. A repo you hesitate to push is a repo you’ll stop maintaining.
7. Name things for where they’re going. The folder is called vector-db — github.com/Axotopia/dsh/tree/main/vector-db. It contains no vector database yet — it contains the context-budget foundation RAG stands on. Build the budget first and every later decision becomes arithmetic instead of hope.

PART TWO — The store we chose last

Part one ended with a promise: embeddings, chunking, and the actual store — chosen last, on purpose. Here’s what happened next.

It started as a planning session, not a backlog item — one sentence, with an explicit instruction: planning only, no execution, no coding. The ask, paraphrased: “Is there a way to create a simple vector database for the harness? I have many documents — building codes, fire codes, mechanical codes — and I want queries with a source of truth.”

Note what’s in that sentence: privacy (these are the codes our clients build against — and later, health and financial records), source of truth (answers must carry checkable citations), and simple (an explicit scope ceiling, which is a gift when you get one).

So before anything was built, the agent did what an agent should do first: read-only recon of the live system, not the vendor docs. Three findings shaped everything:

• The harness’s service catalog held roughly sixty services — and not one was a vector store. Nothing to extend.
• The model-routing service was chat-only. Embeddings would have to come from a separate local service — which turned out to be already running.
• Anything shared across sessions belongs to the host composition, not per-session presets. One brain, every session reaches into it.

Planning sessions are where projects quietly die of complexity, so the rule was: every question gets answered by measuring something real.

Then the human said the magic words — “Proceed with the coding” — and the build took hours, not weeks, because every decision had already been paid for.

The stack, and the math you can steal

The store is a file. One kb.db, opened by the language runtime itself. Backup is a copy; delete is an uninstall. Brute-force cosine over a projected half-million chunks at 1,024 dims is about 2 GB of working memory and a few hundred milliseconds — you need millions of chunks before an ANN index earns its complexity. We’re at 42 chunks in the store today, and that arithmetic is exactly why it’s fine.

The retriever is two channels, rank-fused:

score(chunk) = Σ_channels 1 / (60 + rank_in_channel)

Semantic cosine over embeddings answers paraphrases (“how close can I build to a wetland?”); keyword search answers exact identifiers (“what does D103.4 say?”). Real corpora are bilingual — codes cite themselves in identifiers, humans ask in prose — and a single-channel retriever is a coin flip about which language the question arrives in.

Chunking is heading-aware: ~1,200-character target with a hard cap, page numbers tracked through PDF page breaks, a junk filter for URL-only lines — and each chunk keeps its file path, page, and section heading, because a citation should feel like a citation instead of a file name.

The proof is one query. Ask “minimum gate width for fire apparatus access road” and retrieval returns — ≈84 ms warm; budget seconds for the first query, which embeds on the way through:

IFC, Appendix D, page 2, D103.5 — “The minimum gate width shall be 20 feet (6096 mm).”

The exact section. The exact page. Checkable against the original PDF. And ask “D103.4 dead-end requirements” and the keyword channel lands directly on TABLE D103.4 — “Requirements for Dead-End Fire Apparatus Access Roads,” same PDF, page 2. Neither channel alone does both.

Sizing math, for your own planning:

db_size ≈ 3–4 × extracted_text (text ~1×, FTS index ~1–1.5×, embeddings ~1×)
working_RAM ≈ chunks × dims × 4 bytes (projected 500K chunks @ 1,024-d ≈ 2 GB)

Embedding throughput is a property of your GPU, not a constant — so the worker ships a probe verb that measures it live on your machine before you commit to a corpus.

Errors are data, not exceptions

Deletion fails closed: preview before it acts, soft-delete by default, purge only on explicit confirmation, and a refusal to run without at least one filter — “remove everything” is a sentence the tool answers with a refusal. Ingestion is idempotent by content hash, all-or-nothing per document — an edited file replaces its old chunks atomically, so old and new versions of a code section are never both true at once. And a failed document is recorded in the store, with its reason.

In our first hour against real codes, the corpus handed us a gift: a genuinely corrupt PDF (broken DEFLATE streams) that the pipeline refused to ingest and recorded as status: error — the designed behavior, firing for real. Testing against your actual documents is not validation theater; it’s the debugger.

The shape: a worker, a plugin, and a patch row

Three pieces, each boring on purpose:

• A worker CLI: one plain JavaScript file doing parse → chunk → embed → store → search, plus management verbs (status, list, forget, reingest, inspect). Testable standalone: kb.mjs probe answers “are Ollama, the model, and the database alive?” in one line of JSON.
• A plugin registering seven tools into every session: the agent never touches the database; it speaks to a CLI with JSON in and citations out.
• A patch row mounting the plugin permanently in the host composition: and the promotion was verified by subtraction: stop the prototype, watch the tools survive. The demo is not the deployment until it survives its own uninstall.

The installer is a prompt — so we hardened it like code

The README offers a zero-setup path: paste one prompt into any DSH session and let the agent install everything. The first version said, in essence, “install the package at this URL and verify it runs.” On the machine we wrote it on, it worked. On a different system, it failed — the prompt silently assumed its environment. Python missing, Ollama missing, and nothing told the agent to notice.

The fix wasn’t more automation; it was making the environment contract explicit inside the prompt: check prerequisites first, install what’s missing (winget install Python.Python.3.13, winget install Ollama.Ollama), confirm python --version and ollama --version, then install and run. When your installer is a prompt, the prompt is the installer. It needs what install scripts need — declared dependencies, verification steps, explicit failure modes — and it must be tested on a machine that isn’t yours.

The agent built its own memory

One more thing, and it’s why the numbers above can be trusted: the agent that lives in this harness built the harness’s memory. It measured its host’s GPU, enumerated the service catalog, wrote the worker, ingested real codes, hit the corrupt PDF and the SQL quirks, fixed them, promoted itself and verified the promotion by uninstalling itself — then answered its first question from the store it had just built, correctly, with a page number. Within hours there was a second collection in the store, created by a human, without asking us how. The tool got used before the documentation was finished. That’s the adoption metric that matters.

Dogfooding at this depth isn’t a pose. It’s the only way to learn that idempotency, error rows, and citations aren’t features. They’re the product.

What’s next

OCR for the scanned codes (local vision models are already on the machine — the parse step just needs the rung), more collections — health, finance, zoning — and eventually a small GUI. Chosen in that order, for the same reason as everything else: each is only worth building now that the layer under it is measured and boring.

Why we’re giving away the whole method

Zero-click on purpose. Value belongs in the feed, not behind a link. The formulas are above. The measured numbers are above. The failure that started it is above. If this post is all you ever read, you can budget context for your own models tonight with a calculator and nvidia-smi — and ask your own corpus what it demands before you pick a single database.

The repository is the receipt, not the product. We’re an architecture firm. We didn’t build RAG because it’s fun. We built it because our agents answer questions where 100% accuracy is the floor — and because we refused to let an industry default decide how much retrieved truth our GPU can hold.

And one honest caveat, because the canon says so: our own cost guide found local text inference loses to APIs by ~1,700×. We didn’t build local because it’s cheaper — we built local because codes, health records, and client data are the privacy carve-out. Embeddings, retrieval, and chunking run where the data lives. Heavy long-context reasoning still routes to the meter. Local store, rented brain.

Constraint first. Components second. The same sequence we bring to your site, your codes, your pro-forma. For architects, that’s creative liberation from documentation grunt work. For developers, faster feasibility and permitting with risk mitigated by checkable citations. For homeowners, a vision grounded in precision — a design that doesn’t guess what the code allows.

Your GPU already told you the answer. You just have to read its metadata.

The repository is the receipt, not the product — vector-db holds the two small Python files that automate exactly what’s written here.

P.S. Google’s agent retrieves ads. Ours retrieves the law you’re building against — with the page number to prove it. One is convenience. The other is conviction.

Axoworks uses AI to augment licensed expertise. The machine accelerates; the human certifies. That’s the division of labor that makes this buildable.

Initialize The Concierge → axoworks.com · View the Axosphere.

Originally published at axoworks.com/articles/rag-context-budget · Series: DSH Memory

Related reading, same canon

• Buy the Model or Rent the Intelligence? — we priced the other half: what a 128K-token research turn costs in billed waiting time. Capacity here; cost there.
• The Machine Wears the Harness Now — the platform thesis and the open-source presets this store lives inside; the 4,096-token context trap this math fixes.
• The Verification Stack — why no model is a source of truth; this store is layer L1, built and measured.
• revit-tools — the three RAG patterns this store completes: “retrieval method is an implementation detail; citation discipline is not.”
• The $1 Property Report — the preset that consumes this store on every parcel.
• Coercing AI Compliance — the same 96 GB GPU, the other kind of rails: conditioning pixels instead of tokens.

FAQ

Q: What is Axoworks?
A: An AI-augmented architecture and interior design consultancy delivering full-scope architectural services — conceptual design through technical documentation — with licensed professionals certifying every AI-accelerated output.

Q: What is DeepSeek Harness (DSH)?
A: DSH is Axoworks’ local-first agent harness: an open-source, everything-is-a-plugin platform where models, tools, persona, and sandbox are configuration (MIT, six presets at github.com/Axotopia/dsh). It runs DeepSeek and other local models with no data egress. The full platform thesis is in The Machine Wears the Harness Now.

Q: Why did Axoworks build RAG into it?
A: Because its agents answer building-, fire-, and mechanical-code questions where 100% accuracy is the floor. A wrong citation is a stop-work order or a lawsuit. RAG grounds every answer in source documents — with a file path, page, and section heading attached.

Q: Why SQLite instead of a vector database?
A: Write the scale requirement down first and the debate ends. At personal-corpus scale, a single-file SQLite store — zero installs, zero servers, backup is a copy — is the right answer. Brute-force cosine over a projected half-million chunks needs only ~2 GB of RAM; you need millions of chunks before an ANN index earns its complexity.

Q: How does Axoworks guarantee a correct code answer?
A: Three mechanisms: hybrid retrieval (semantic + exact keyword, rank-fused), because codes cite themselves in identifiers while humans ask in prose; every chunk carries a citation checkable against the original PDF; and a licensed professional reviews output before it reaches a client. Errors are recorded in the store with their reason — silent partial failure is how RAG systems start lying.

Q: Doesn’t Axoworks’ own cost guide say local inference loses to APIs?
A: Yes — by ~1,700× for text workloads. That’s why the store runs local for a different reason: privacy. Codes, health records, and client data never transit. The store runs bounded jobs — embeddings, retrieval, chunking — where the data lives; heavy reasoning routes to the meter. Local store, rented brain.

Q: Can someone else build this themselves?
A: Yes — deliberately. The method, formulas, and measured numbers are published zero-click: budget your context with a calculator and nvidia-smi, ask your corpus what it demands, then choose components. The code is at github.com/Axotopia/dsh/tree/main/vector-db; the repository is the receipt, not the product.