CR Innovations Lab

I Gave My AI a Brain — Here's How It Learns From Every Conversation

How I built a self-improving memory system from pgvector, markdown staging files, and a feedback loop that retrains my local model

By · · 8 min read

My last article covered replacing my training pipeline with two containers and a 12-year-old server. That solved deployment. But the system it deploys — my local fine-tuned model — still had a problem I'd been ignoring: every time it retrieved context, it dumped everything into the prompt and burned tokens.

I was pulling 20 chunks from pgvector, stuffing all of them into Claude's context window, and paying for every token whether the model used it or not. At scale, most of those tokens were noise — chunks that scored well on cosine similarity but weren't actually useful for the query.

So I built a layer between retrieval and injection. I call it the PAI Brain.

The Problem: Retrieve-and-Stuff

The standard RAG pattern is simple: embed a query, find the nearest vectors, concatenate the chunks, and send them to the model. It works for demos. It breaks down when your knowledge base grows past a few thousand documents.

The issue isn't retrieval quality — pgvector with HNSW finds relevant chunks in 10-25ms at 100K vectors. The issue is selection. Cosine similarity finds things that look like your query. It doesn't find things that answer your query. A chunk about "deploying auth" and a chunk about "auth token expiry" both score high for "how do I deploy the auth service?" — but only one is useful.

When you stuff all 20 into the prompt, the model has to sort through the noise. You pay for the tokens. You pay in latency. And the model's reasoning gets diluted.

What I Built

The architecture has five components:

pgvector (HNSW, 10-25ms) → .md staging file → pai-selector picks top 5 → context injection
                                                         ↑
                                             n8n maintenance (async)
                                             - garbage collection
                                             - re-ranking
                                             - feedback capture → JSONL → LoRA retrain

1. pgvector on Postgres. HNSW indexing, separate instance from the main database. 100K vectors, 10-25ms query latency. This is the retrieval layer — it finds candidates, not winners.

2. Markdown staging files. Retrieved chunks land in a structured .md file before they enter the prompt. Each chunk has metadata: similarity score, source file, line numbers. The AI reads from this file selectively — top-down by score until it hits the token budget.

## Query: "How do I deploy the auth service?"

### Candidates (cosine sim > 0.75)

1. **doc/2024-09-deploy.md** (0.82)
   docker-compose up -d auth
   Check logs: docker logs auth

2. **notes/weekly-sync-2024-10.md** (0.79)
   Auth deploy: use new staging env var AUTH_DB_URL=postgres://...

3. **slack/2024-08-thread.md** (0.76)
   Auth uses the shared Redis instance for sessions...

*12 chunks below threshold — never enter context*

This file is versioned in git. I can open it and see exactly what the model will see. If the system retrieves a stale credential or an irrelevant conversation, it's visible in staging before it hits inference. Any script can parse it. No opaque internal state.

3. pai-selector — a fine-tuned Qwen2.5-14B as chunk selector. Running on my RTX 5060 Ti (16GB) via Ollama, fine-tuned with Unsloth on my own retrieval data. One job: given a query and 20 chunks, pick the 5 most relevant. Returns JSON. Runs async via n8n — never blocks the query path. Takes 12-18 seconds, which is too slow for inline use but fine for background optimization.

4. Three n8n maintenance workflows:

5. A feedback loop that closes into training. The feedback logger exports used/ignored chunk pairs as Unsloth-compatible JSONL. Each fine-tune cycle incorporates real selection feedback — the model learns which chunks I actually referenced versus which ones it suggested but I skipped. The training pipeline from my previous article handles the retrain automatically.

Live Ingestion: Memory That Stays Current

Most RAG systems run batch indexing — a nightly cron that re-embeds documents. That means anything written today isn't searchable until tomorrow.

My system ingests continuously. New notes hit a webhook, get chunked and embedded, and land in pgvector within seconds. Code changes trigger re-embedding via git hooks. n8n orchestrates the pipeline: watch → chunk → embed → upsert.

The vector store isn't a snapshot. It's a stream. When I write a note at 2 PM, it's retrievable by 2:01 PM.

The Numbers

Tokens per query: ~5,000 → ~2,500. -50%.End-to-end latency: ~3,025ms → ~2,095ms. -31%.Context noise: 20 chunks (mostly noise) → 5 ranked chunks. Precision up.Staging overhead: <6ms total (write + read). Negligible.

The latency improvement comes from processing fewer tokens at inference — 2,500 instead of 5,000 saves nearly a full second on Opus. The staging file write (<1ms) and selective read (<5ms) are rounding errors compared to that.

The 31% latency reduction is real but comes with a caveat: pai-selector runs async. If it ran inline, it would add 12-18 seconds. The system uses pgvector similarity scores for the initial response, then pai-selector re-ranks in the background for subsequent queries on the same topic.

How This Compares to the Industry

The retrieve-broadly-rerank-precisely pattern is well-established. This isn't a new idea:

Where my system differs:

The selector is trained on my data. Cohere and LangChain use generic relevance models. pai-selector has been fine-tuned on which chunks I actually used vs. ignored in my specific system. A generic cross-encoder doesn't know that I always prefer deployment docs over architecture discussions when I ask about services.

The staging file is human-readable and editable. Letta's memory blocks are managed programmatically by the LLM. My staging files are markdown on disk that I can cat, grep, or edit by hand. The debugging story is fundamentally different.

The feedback loop closes into model training. No commercial product I've found captures chunk usage at inference time and feeds it back into reranker fine-tuning via LoRA. The academic literature has this (RAG-RL, Reward-RAG, PRCA — all 2023-2025), but it hasn't shipped in any tool you can install.

Honest Caveats

pai-selector is fine-tuned on Qwen2.5-14B, which is a generation behind. Qwen3 is shipping. The fine-tuning dataset will need adaptation for the new architecture. This is maintenance debt — every base model upgrade requires re-tuning.

The cost savings depend on your pricing era. The original numbers were based on Opus at $15/$75 per million tokens. Current Opus pricing is lower, which shrinks the ROI. The token reduction is real regardless, but the dollar impact varies.

Quality preservation is assumed, not measured. I'm tracking token reduction and latency, but I don't have a systematic evaluation of answer quality pre- vs. post-staging. The system feels better — fewer irrelevant tangents in responses — but I haven't built a benchmark for it yet.

A purpose-built reranker would be faster. NVIDIA's nv-rerankqa-mistral-4b-v3 does reranking at 4B parameters with 75% Recall@5. Using a 14B general-purpose model for chunk selection is overkill. I chose it because pai-selector already existed from other work, not because it was the right-sized tool.

Try This Yourself

If you're running RAG on local hardware, the staging layer is the highest-leverage piece to add:

  1. After retrieval, write chunks to a file instead of directly into the prompt. Markdown with metadata headers (score, source, line number). This alone gives you inspectability and budget enforcement.
  2. Add a token budget with a floor. Cap retrieved context at a fixed token limit, but never go below 3 chunks. Below that, the model doesn't have enough context to reason.
  3. Log which chunks actually get referenced. Even without retraining, this data tells you how much noise your retrieval is producing. If you're consistently retrieving 20 and using 4, your retrieval-to-injection ratio is the problem.
  4. If you fine-tune locally, feed the logs back in. Export used/unused pairs as training data. LoRA retrain. The selector gets better each cycle.

You don't need all five components to get value. The staging file alone — just writing chunks to markdown before injecting them — changes how you think about context.

The Broader Point

RAG has been treated as a retrieval problem. Get better embeddings, use better vector databases, increase top-k. The assumption is that if retrieval improves, everything downstream improves.

That assumption breaks at scale. Past a few thousand documents, retrieval returns noise. The model sees 20 chunks that all score 0.75+ and has to figure out which ones matter. It's doing triage inside inference, and you're paying for every token of that triage.

The PAI Brain reframes RAG as a selection problem. Retrieval finds candidates. Selection picks winners. And the feedback loop means the selector gets better at picking winners the more you use it.

The most efficient AI system isn't the one with the best embeddings. It's the one that knows what to throw away.

This is the fifth article in a series. Previous: "I Replaced My Training Pipeline With Two Containers and a 12-Year-Old Server" (containerized fine-tuning), "I Made My 14B Model 40x Faster by Teaching a 0.5B Model to Say 'I Got This'" (query routing), and "I Benchmarked My GPU and Found It Was Idle 83% of the Time" (PCIe bottleneck).

The system runs on a mesh of a VPS, a work laptop, and a desktop Linux machine with an RTX 5060 Ti (16GB). pgvector on Postgres. pai-selector via Ollama + Unsloth. n8n for workflow orchestration. All local inference — no data leaves the machine.

References:

Originally published on LinkedIn on April 18, 2026.

memoryragpgvectorfine-tuning