CR Innovations Lab

I Replaced My Training Pipeline With Two Containers and a 12-Year-Old Server

How I built an automated fine-tuning system across three machines, then watched it break in two new ways on the first real run

By · · 11 min read

My last article was about making a 14B model 40x faster with query routing. That solved inference. But every two weeks, I still had to retrain the model by hand.

The pipeline was a 200-line bash script. It SSH'd into my GPU machine, ran docker run with a dozen flags, piped outputs through sed, and prayed. If training succeeded, it docker cp'd the GGUF into an Ollama container and ran ollama create. If anything failed, I got a Slack message that said "error" with no context.

I ran this script 6 times over three months. It broke in a new way every single time.

So I rebuilt it as a containerized, multi-machine pipeline that my CLI can trigger with one command. Then I ran it for real, and it broke in two new ways I hadn't predicted. Both failures taught me more than the design phase did.

The Problem With "Just Run a Script"

My fine-tuning setup is straightforward: QLoRA on a Qwen2.5-14B base model, using corrections captured from daily interactions. Training takes 77 minutes on an RTX 5060 Ti. Export to GGUF takes 65 minutes. Deploy to Ollama takes 7 minutes.

The orchestration around it was where everything broke.

VRAM conflicts: Ollama, Whisper STT, and Chatterbox TTS all share the same 16GB GPU. Training needs the full card. Stopping services, verifying VRAM is actually free, killing orphaned CUDA processes — 30 lines of brittle nvidia-smi parsing.

No rollback: if the new model produced garbage, the old GGUF was already overwritten.

Export bugs: Unsloth's save_pretrained_gguf() hangs on my hardware. Every time. I had to use a separate export script — a workaround I rediscovered on three separate occasions before I finally wrote it down.

Manual SSH: the script ran on my laptop, SSH'd to the GPU server, and managed state through temp files. Every connection was a point of failure. Escaping nested quotes through SSH was a recurring nightmare.

The script worked. But "works if you babysit it" isn't a pipeline. It's a chore.

What I Built: Two Containers, Three Machines

The new system has three components.

Trainer runs on Desktop-Linux (RTX 5060 Ti). GPU-only: QLoRA fine-tuning. Ephemeral — starts, trains, exits.

Deployer also runs on Desktop-Linux. Watches for trained models, deploys to Ollama, runs smoke tests. Persistent — always running.

Preprocessor runs on my MSI-Server, a 12-year-old laptop with a quad-core i7 and 29GB of RAM. In MSI mode, it handles everything that doesn't need a GPU: JSONL validation, deduplication, LoRA merge, GGUF quantization, model evaluation. In local mode, the trainer handles export itself.

The trainer starts, trains, exports, writes a signal file, and exits. The deployer watches for that signal. When a new GGUF appears, it copies it into the Ollama container, registers the model, smoke-tests identity with the production system prompt, restarts the other GPU services, and writes an acknowledgment. If anything fails, it writes a failure signal and sends me a voice notification.

Why the Oldest Machine Does the Hardest Job

This came directly from benchmarking.

In my GPU idle time article, I found the GPU spends 83% of its wall-clock time waiting during a training pipeline. The post-training work — merging LoRA weights into the base model (40 minutes), converting to f16 GGUF (7 minutes), quantizing to Q4_K_M (18 minutes) — is pure CPU/RAM work. During those 65 minutes, the GPU sits at 0% utilization.

That's nearly half the total pipeline where the $400 GPU card is doing nothing, while a $0 (already owned) 12-year-old laptop could be doing the work instead.

So I split the pipeline. GPU-only work stays on the GPU machine. CPU/RAM work moves to MSI-Server. The tradeoff is network transfer latency — LoRA adapters (~200MB) need to SCP between machines. But that transfer is async. The GPU is free to resume serving inference the moment training finishes, instead of being locked for another hour of CPU work.

The Signal Protocol

The containers coordinate through files, not APIs. This is intentional.

Trainer writes  →  /shared/.ready     (GGUF + Modelfile ready)
Deployer writes →  /shared/.ack-ok    (deployed + smoke test passed)
Deployer writes →  /shared/.ack-fail  (deployment failed, reason logged)
Trainer polls   →  .ack-ok or .ack-fail, then exits

Why files instead of HTTP or message queues? File-based signals survive container restarts. If the deployer crashes and comes back, the .ready file is still there. If the trainer crashes mid-training, no signal is written and the deployer does nothing. The failure mode is always silence, and silence is safe.

Redis or RabbitMQ would add a dependency, a failure mode, and 500MB of container image for a problem that touch .ready solves.

What Actually Happened

I'd built the architecture and tested individual components, but the first real end-to-end run was April 16 — 480 training examples, live data, full pipeline.

Training took 77 minutes. Loss dropped from 1.886 to 0.089 across 360 steps. The preflight check caught an orphaned CUDA process holding VRAM and killed it automatically — the old script would have OOM'd 40 minutes in. Export started: LoRA merge (40 minutes), f16 GGUF conversion (7 minutes), Q4_K_M quantization (18 minutes). All three phases completed. A 9.0 GB GGUF was sitting on disk, ready to deploy.

Then the Python process hung.

The export script finished all its work. The GGUF existed. The Modelfile was written. But the Python process entered an uninterruptible sleep state trying to clean up the CUDA context. It would never exit on its own.

This killed the pipeline. The entrypoint script uses set -e, so when I manually killed the hung process, the kill signal cascaded to the parent shell. The trainer container died before writing the .ready signal.

I had to manually copy the 9 GB GGUF into the shared volume and write the .ready signal by hand. The deployer picked it up within 3 seconds and completed deployment cleanly. That's the part the architecture got right — even with a manual intervention, I only had to write one file, and the rest of the pipeline handled itself.

The fix: wrap the export script in timeout --signal=KILL 2700. If the GGUF file exists after the kill, continue to signaling. The export takes about 65 minutes; the timeout gives headroom while catching the indefinite hang.

The Identity Bug

The deployer's original smoke test ran ollama run rai-qwen "Who are you?" and checked for a non-empty response. The model responded: "I'm Qwen, a large language model created by Alibaba Cloud."

That's the base model identity bleeding through. Without a system prompt, the fine-tuned model defaults to its pre-training identity. The smoke test was passing on response length, not response content.

The fix: changed the smoke test to call the Ollama API with the production system prompt. Now it verifies that the model identifies correctly, knows its user, and responds in the right voice. After the fix, the model nailed every identity test — "Are you ChatGPT?" got "No. I'm WILLAI running on the rai-qwen model." Build-vs-buy questions got "Use the library. Auth is a solved problem and a minefield."

The lesson is broader than this pipeline: a smoke test that checks "is the response non-empty?" will pass on a completely wrong model. You have to test what makes your model yours.

What Made the Difference: Data, Not Architecture

This was the third training run overall, first using the containerized system.

Runs 1 and 2 used 367 and 408 examples. Both had plenty of tool-use and coding data but almost no identity or persona examples. The model sometimes knew who it was, sometimes reverted to base Qwen. It failed identity questions about half the time.

Run 3 added 174 examples specifically covering identity, greeting style, decision-making voice, and anti-patterns to avoid. The model needed to be explicitly taught its identity the same way it was taught to use tools. After that, 7 out of 7 identity tests passed.

The container system's contribution was operational, not quality. The preflight check caught an orphaned CUDA process. The validation script verified all 480 examples before training started. When the export hung, the signal protocol meant I only needed to write one file to resume. Those are the things that make the difference between "works on the third try" and "works on the first try."

CLI Orchestration

The whole pipeline is wired into BossCLI, my terminal interface:

/train run       # Local mode: full pipeline on Desktop-Linux
/train run msi   # MSI mode: GPU on Desktop-Linux, post-processing on MSI
/train status    # Check pipeline state

Under the hood, /train run SCPs the training data to Desktop-Linux and runs docker compose --profile training up trainer -d via SSH. The trainer handles the rest.

The model itself can trigger training through a train_pipeline tool. When enough corrections accumulate (threshold: 50), the system auto-triggers a retraining cycle. It's not self-improving in any grand sense — it's a counter and an SCP command. But it closes the loop on what was previously a manual step.

How This Compares to the Industry

The pattern — ephemeral trainer, persistent deployer, file-based signals — isn't new. Kubernetes has had this for years with PyTorchJob and KServe. SageMaker, Kubeflow, and Vertex AI all automate this loop end-to-end.

But those platforms assume cloud infrastructure. The cost gap is where it gets interesting.

SageMaker charges $8-11 per run on a g5.xlarge. You get automatic model swapping and monitoring, but you're paying per-minute for GPU time. OpenPipe charges ~$2.25 per run ($1.50/M tokens) and adds A/B routing and auto-retraining from production data — the most complete commercial implementation of the continuous training loop. Together AI is $0.72-4.35 per run for LoRA, no deployment automation included.

On the local side, the training frameworks — Axolotl, Unsloth, LLaMA-Factory — all stop at "here are your weights." Every one of them. The gap between "training finished" and "model is serving traffic" is a boundary none of them cross, because deployment has too many possible answers: Ollama, vLLM, llama.cpp, TGI. The platforms that do cross it all assume your model lives on their infrastructure.

My pipeline costs ~$0.13 per run in electricity. An RTX 5060 Ti costs $500; amortized over 200 runs, that's $2.15/run in hardware, comparable to a single OpenPipe job. After 200 runs, it's electricity only. The MSI-Server was $0 incremental — it was already in a closet.

The piece I haven't found anywhere else — in products, papers, or blog posts — is the CPU/GPU split across separate consumer machines. Enterprise training uses CPU offload within a single node (DeepSpeed ZeRO-Infinity). Enterprise pipelines use preprocessing clusters feeding training clusters. But a 12-year-old laptop handling LoRA merge and quantization while the GPU machine resumes inference? That's not a product anyone sells. The closest is smolcluster, an open-source library for distributed training on heterogeneous consumer hardware, but even that focuses on distributed training, not splitting post-training CPU work to a separate node.

Honest Caveats

These are real issues from the first run, not hypothetical risks.

The export script hangs after completion. Python enters an uninterruptible sleep during CUDA context cleanup. The GGUF is written, the process just won't exit. I've worked around it with a timeout wrapper. It works, but it's a workaround for a bug I can't fix upstream.

Smoke tests need the full system prompt. If the system prompt changes and the deployer isn't updated, the smoke test either false-passes or false-fails. I should template it from a shared config.

Quantization takes 18 minutes, not 3. My original SOP estimated 3 minutes for Q4_K_M on a 14B model, based on an 8B estimate that got copy-pasted forward. This kind of inherited wrong number is how "it should take an hour" becomes "it actually takes three hours."

MSI mode is still untested end-to-end. Local mode ran successfully. The full MSI chain — training on Desktop-Linux, LoRA transfer, merge, quantize, GGUF return, deploy — hasn't been tested with real training outputs yet. My bet is SCP path quoting or the LoRA merge running out of MSI's 29GB on a 14B model.

The Pattern Worth Stealing

If you're running any kind of ML pipeline on local hardware:

  1. Separate GPU work from CPU work. They don't need to be on the same machine or in the same container. If your GPU is idle during post-processing, you're wasting the most expensive component in your system. In my pipeline, 65 of 167 total minutes are pure CPU work.

  2. Use file-based signals for container coordination. It's debuggable (ls /shared/), survives restarts, and has zero dependencies. When my export hung and I had to manually intervene, writing one file resumed the entire deploy pipeline.

  3. Make training triggerable from your existing tools. If you have to SSH into a machine and type commands to start training, you won't train often enough. Wire it into whatever CLI or UI you already use.

  4. Ephemeral trainers, persistent deployers. The thing that does GPU work should start, do the work, and exit. The thing that manages deployment should always be running. This is PyTorchJob + KServe on Kubernetes, but it works the same with Docker Compose and a shared volume.

  5. Test identity, not just output. My first deploy looked successful until I asked the model who it was and got "I'm Qwen by Alibaba Cloud." If you're fine-tuning for a specific persona or use case, your smoke test needs to verify that persona, not just check for a non-empty string.

The Broader Point

Fine-tuning local models has gotten remarkably accessible. The training tools work. The serving tools work. The models are open and capable.

What hasn't gotten easier is the plumbing between them. Stopping services before training. Verifying VRAM. Exporting formats. Deploying without downtime. Rolling back failures. These are boring problems, and they determine whether you retrain your model once and forget about it, or continuously improve it from your own data.

A training script that runs once is a demo. A training pipeline that handles its own failures is infrastructure. And infrastructure is what turns a personal AI project into a personal AI system — not the model weights, but the loop that keeps them improving.

The question I'm more interested in now isn't whether the pipeline works. It does. The question is what happens when the model can reliably trigger its own retraining, and corrections compound across dozens of cycles instead of three. That's the experiment the infrastructure was built for.

This is the fourth article in a series. Previous: "I Made My 14B Model 40x Faster by Teaching a 0.5B Model to Say 'I Got This'" (query routing), "I Benchmarked My GPU and Found It Was Idle 83% of the Time" (PCIe bottleneck), and "I Built a 4-Source Hybrid RAG System That Fine-Tunes Itself From My Commits" (hybrid RAG + weight personalization).

The system runs on three machines connected via a Headscale mesh VPN: a work laptop (orchestration), a desktop Linux machine with an RTX 5060 Ti (GPU training + deployment), and a 12-year-old MSI laptop (CPU preprocessing). Total hardware cost for the two "servers": $0 incremental — both were already owned.

References:

Originally published on LinkedIn on April 17, 2026.

fine-tuningdockerpipelineslora