CR Innovations Lab

I Built a 4-Source Hybrid RAG System That Fine-Tunes Itself From My Commits — Here's What I Learned

By · · 7 min read

I Built a 4-Source Hybrid RAG System That Fine-Tunes Itself From My Commits — Here's What I Learned

The standard advice for building a RAG system in 2026 is straightforward: spin up a vector database, chunk your documents, embed them, and retrieve the top-k results at query time. Maybe add BM25 for keyword coverage. Slap on a reranker. Call it a day.

That's what 80% of production RAG systems look like right now, and honestly, it works fine for most use cases. But I wanted something different — a personal AI coding assistant that doesn't just retrieve relevant context, but actually understands the structural relationships in my knowledge base and improves its own weights from my actual coding patterns.

Over the past few months, I built exactly that. This post covers what I built, why, and — critically — where the current approach falls short. No hype. Just what works and what doesn't.

The Problem With Two-Source RAG

The two-source hybrid (vector + keyword) approach that's become the industry default has a blind spot: it treats every document as an isolated chunk. It can find documents that are semantically similar to your query and documents that contain your keywords, but it has no concept of how documents relate to each other.

In my personal knowledge system, this matters. I have roughly 14,460 indexed memory entries spanning infrastructure docs, research notes, project files, and codebase snapshots across four Qdrant collections. When I search for "Ollama deployment," I want the Ollama config file — but I also want the Docker Compose that runs it, the monitoring hook that health-checks it, and the training pipeline that depends on it. Those documents share an infrastructure relationship, not just semantic similarity.

Two-source retrieval can't surface that.

What I Built: Four Retrieval Sources, One Merge

My system fuses four heterogeneous retrieval sources into a single ranked result set:

  1. Postgres full-text search — GIN-indexed FTS across 14,460+ memory entries with structured metadata filtering
  2. Qdrant vector search — Semantic similarity using nomic-embed-text embeddings across four collections (memory, codebase, devdocs, research)
  3. Keyword file search — Direct filesystem grep for exact-match patterns the other sources might miss
  4. FalkorDB graph traversal — Multi-hop Cypher queries across a 10,178-node knowledge graph with 14,103 edges

Each source contributes candidates. The merge layer applies source-specific relevance weights, boosts results found by multiple sources, and guarantees that graph-discovered results appear in the output — even if they'd otherwise rank too low on pure relevance score.

Why FalkorDB Over Neo4j

I chose FalkorDB (the successor to RedisGraph) for the graph layer. At my scale — roughly 10K nodes and 14K edges — FalkorDB's sub-millisecond Cypher query performance is a clear win. Independent benchmarks confirm 500x better p99 latency than Neo4j on aggregate expansion queries.

The trade-off is real, though: Neo4j has a vastly larger community, more tooling, and more production history. If something breaks, there are 100x more StackOverflow answers for Neo4j. For inference-time graph retrieval on a personal knowledge base, FalkorDB is the technically correct choice. For enterprise deployments where supportability matters more than raw latency, I'd probably go Neo4j.

The Graph Guarantee

One implementation detail worth sharing: graph-discovered results initially didn't appear in my output at all. The graph traversal returns structurally related documents with a relevance score of 30 (depth-1) or 15 (depth-2), but postgres + keyword results score 60-400. Graph results were consistently outranked.

The fix was a "graph guarantee" — up to two graph-only results always surface in the final output by replacing the lowest-ranked entries. This ensures the structural signal from the graph isn't drowned out by the other sources. It's a simple heuristic, but it's the difference between the graph being decorative and the graph being useful.

The Part That's Actually Novel: Weight-Level Personalization

Here's where things get interesting. Every major coding assistant in 2026 — Cursor, Copilot, Continue.dev, Cody — does codebase-aware suggestions. They index your repo and inject relevant code as context at query time. That's RAG. It's table stakes.

None of them fine-tune a local model on your individual commit history.

My system does. Here's the pipeline:

  1. Post-commit hook captures per-file diffs from every commit across my active repos
  2. Batch processor converts raw diffs to ChatML training format, using the model itself to generate natural-language descriptions of each change
  3. QLoRA fine-tuning runs biweekly on Desktop-Linux (RTX 5060 Ti, 16GB VRAM) via Unsloth, training Qwen2.5-14B on the accumulated commit data
  4. Automated deployment exports to GGUF, deploys to Ollama, runs a smoke test
  5. n8n orchestration triggers the whole pipeline on the 1st and 15th of each month

The result: my local model's weights actually change based on how I write code. It's not just retrieving my code as context — it's internalized my patterns, my naming conventions, my architectural preferences.

Honest Caveats

I need to be upfront about the limitations:

QLoRA quality ceiling. QLoRA achieves roughly 80-90% of full fine-tuning quality. Published research on consumer GPUs confirms measurable accuracy degradation from strict VRAM limits. My fine-tuned model is good, but it's not as good as it would be with a full fine-tune on an A100.

Catastrophic forgetting risk. Biweekly retraining on a narrow corpus — just my own commits — risks the model becoming increasingly specialized in my patterns while losing general reasoning ability. I don't have a diverse instruction dataset mixed in to counteract this. This is probably the single biggest technical risk in the pipeline, and it's something I need to address with evaluation benchmarks and diverse data mixing.

No formal evaluation metrics yet. I can't quantitatively prove the retraining cycle makes the model better rather than worse. I need a held-out evaluation set and benchmark tracking across retraining cycles. That's on the roadmap.

Qwen2.5-14B is defensible but not cutting-edge. When I built this pipeline, Qwen2.5-14B was the leading option for my VRAM tier. Qwen3-14B has since surpassed it on most benchmarks. The architecture supports model swaps, so upgrading is straightforward — but I'm currently one generation behind.

The Metadata Layer: QX-1

One piece I haven't seen elsewhere: a structured metadata taxonomy applied across both the vector store and the relational database.

I call it QX-1. Every memory entry gets tagged with domain (Work, Personal, Infrastructure, etc.), action type, and target. These tags exist in both Postgres (as columns with WHERE clause filtering) and Qdrant (as payload filter conditions). When I search, I can narrow retrieval to specific knowledge domains before the vector similarity even runs.

This sounds simple, but it eliminates an entire class of noise. A query about "Docker deployment" doesn't return my personal journal entries that happen to mention Docker metaphorically. The domain filter catches that before the embedding model even gets involved.

Currently, about 74% of my Qdrant points are QX-1 tagged. The enrichment runs via a reusable script that syncs Postgres tags to Qdrant payloads.

How This Compares to the Industry

Let me be precise about what's differentiated and what isn't:

Commodity (everyone does this):

Uncommon at personal scale:

Not available in any commercial tool:

The enterprise world has all of this and more — NVIDIA's documented GitOps-driven LLMOps pipelines that trigger LoRA fine-tuning on DGX Cloud from git commits. But bringing that capability to a single developer running on a single consumer GPU, with no cloud dependencies, is a different engineering challenge entirely.

What I'd Do Differently

If I were starting over:

  1. Start with LightRAG instead of a custom graph. Microsoft's GraphRAG and its successors (LightRAG, Fast GraphRAG) handle entity extraction and community detection automatically. My file-centric graph works but lacks the semantic richness of LLM-extracted entity relationships.
  2. Add a reranker. I'm using source-weighted relevance scoring for fusion, but a cross-encoder reranker (Cohere Rerank or a fine-tuned Mistral) would improve precision significantly. This is the lowest-hanging fruit I haven't picked.
  3. Mix diverse instruction data into retraining. To counteract catastrophic forgetting, I should combine my commit data with a diverse instruction dataset (like OpenHermes or WildChat) during each retraining cycle.
  4. Build evaluation from day one. A held-out test set tracking model quality across retraining cycles should have been step one, not a future roadmap item.

The Takeaway

The gap between enterprise AI infrastructure and personal AI infrastructure is closing. The components are all open-source — Ollama, Qdrant, FalkorDB, Unsloth, n8n. The models are competitive — Qwen2.5-14B on 16GB VRAM matches GPT-4-class performance on many coding tasks. The techniques are documented — QLoRA, hybrid retrieval, graph RAG.

What's still hard is the integration. Getting four retrieval sources to merge gracefully, building a training pipeline that's actually automated end-to-end, keeping a knowledge graph in sync, managing the retraining cycle without catastrophic forgetting — these are engineering problems, not research problems. And they're solvable by a single developer with consumer hardware.

The commercial tools will catch up. Cursor, Copilot, and the rest will eventually offer local fine-tuning from user data. When they do, I'll switch — their UX will be better than my shell scripts.

Until then, my model actually knows how I write code. Not because it retrieved an example from my repo. Because it learned.

Corey Ryan is a developer building personal AI infrastructure. The system described in this post runs on a Tailscale mesh connecting a work laptop (orchestration), a desktop Linux machine (GPU training), and a home server (graph database). The full architecture is open and documented — reach out if you want to compare notes.

Originally published on LinkedIn on April 9, 2026.

ragfine-tuninghybrid-searchpersonal-ai