I Split My Training Data Into 5 Layers — Here's Why Order Matters

How curriculum learning from a 2009 paper changed the way I train a 4B model on a single GPU

By Corey Ryan · October 21, 2026 · 17 min read

I wrote this on April 24, 2026, the day after the first full training run, and sat on it for six months. Before publishing I went back through the literature to see what had moved. The October additions are marked.

Earlier in this series I covered replacing my training pipeline with two containers and a 12-year-old server. That solved deployment. But the pipeline was still treating all training data the same — 550 examples shuffled into one batch, one stage, one pass. Personality examples mixed with tool-use examples mixed with reasoning chains.

The model learned. But it learned unevenly. It could reason through a troubleshooting flow, but it forgot who it was mid-conversation. It knew how to call tools, but it applied SOP procedures inconsistently. Every retraining cycle was a lottery — which capabilities would survive this time?

So I stopped shuffling and started ordering.

This Isn't a New Idea

I need to say this upfront: the concept of ordering training data isn't mine. It's a 17-year-old idea with nearly 8,000 citations.

Curriculum learning — training on data ordered from simple to complex — was formalized by Yoshua Bengio in 2009. His ICML paper showed that ordered training outperforms random shuffling by acting as a continuation method for non-convex optimization. The model finds better local minima when it doesn't have to process hard examples before it understands easy ones. The analogy is literal: you teach addition before calculus.

The LLM alignment community took this further. OpenAI's InstructGPT paper (2022) showed SFT preceding reinforcement learning, and that mixing pretraining gradients into PPO updates was necessary to prevent capability regression. The ordering isn't optional — it's architectural. Every major lab now runs some variant of SFT → RL → preference optimization.

More recently, "Scalpel vs. Hammer" (2025) compared GRPO and SFT side by side on the same math data and found a mechanistic difference: SFT makes larger parameter updates, hits mid-layer MLPs harder, and replaces prior skills. GRPO amplifies what's already there, with less collateral damage. The paper says nothing about running one before the other. The inference that this makes order matter is mine. The authors also call their findings preliminary. I'm leaning on them anyway, and I'll say exactly how much weight they can bear further down.

What I built isn't a new training paradigm. It's a specific application of these established principles to a specific problem: training a 4-billion parameter voice AI assistant on a single GPU, where the capabilities need to stack in a particular order to avoid overwriting each other.

The Five Layers

Here's how I split the training data for Mary Ann, a Gemma 3 4B model that handles Level 1 tech support for a SaaS product:

Three training stages execute these layers in order:

Stage 1: GRPO on Layer 1 (reasoning chains). This runs first — directly on the base model, before any SFT. 549 steps, rewards trending up throughout.

Stage 2: SFT on Layers 2-5 (personality, tools, SOPs, knowledge). 207 steps across 550 examples, loss dropping from 1.0 to 0.78.

Stage 3: DPO on Layers 2-4 (preference pairs). 44 steps on 175 chosen/rejected pairs, with rewards_chosen consistently above rewards_rejected — the correct direction.

The total pipeline runs on an RTX 5060 Ti with 16GB VRAM using QLoRA. The GRPO stage is the most VRAM-intensive, but it fits comfortably with Unsloth's optimizations.

Why This Order

The ordering isn't arbitrary. Each layer builds on what the previous one established.

Layer 1 (Reasoning) goes first because GRPO amplifies, it doesn't replace. That's the amplify-versus-replace finding applied directly. Running GRPO on the base model reinforces the model's existing reasoning capabilities — structured problem decomposition, step-by-step thinking, asking clarifying questions before jumping to solutions. If I ran SFT first, the personality and tool-use training would modify the model's internal representations. Then GRPO would be amplifying a model that's already been altered — potentially reinforcing the alterations rather than the base reasoning capacity.

DeepSeek's R1-Zero proved that GRPO can develop reasoning capabilities on a base model without any SFT at all. My approach is less extreme — I'm not trying to develop reasoning from scratch, just reinforce the reasoning patterns already in Gemma 3 4B before SFT starts overwriting things.

Layer 2 (Personality) goes second because identity anchors everything downstream. Once the model has strong reasoning foundations, it needs to know who it is before it learns what to do. A model that learns tool use before personality might use the right tools but sound like base Gemma doing it. Personality training establishes voice, tone, and behavioral boundaries that persist through subsequent capability layers.

Layer 3 (Tool Use) goes third because the model needs identity before it can make tool decisions. Knowing when to call RAG vs. when to answer directly vs. when to escalate to a human requires judgment that's shaped by the model's role. Mary Ann, as a Level 1 support agent, should escalate more aggressively than a general-purpose assistant. That escalation threshold is personality-dependent.

Layers 4-5 (SOPs and Knowledge) go last because they're the most specific and most replaceable. Procedures change. Product knowledge updates. These layers are intentionally the thinnest and most recent — they'll be retrained most frequently as the product evolves. Putting them last means they benefit from the reasoning, personality, and tool-use foundations already established, and retraining them doesn't require re-running the earlier stages.

DPO runs after all SFT layers because it needs to see the model's actual behavior across all capabilities to optimize preferences. The chosen/rejected pairs span Layers 2-4 — the model needs to have learned personality, tool use, and SOPs before DPO can meaningfully teach it which responses are preferred.

What Actually Happened

The first full run was April 23, 2026. Three stages, one GPU, real training data from production corrections.

GRPO completed cleanly. 549 steps, 183 reasoning chain examples. Rewards trended up across the run. The base Gemma 3 4B model's reasoning patterns got reinforced — structured troubleshooting approaches, multi-step problem decomposition.

SFT hit two bugs immediately.

The first: TRL renamed its API between versions. max_seq_length became max_length. tokenizer became processing_class. The training script crashed at step 0. No deprecation warning — just a stack trace. Fix was mechanical: find-and-replace the parameter names, update the SOP so the next run doesn't rediscover this.

The second: Unsloth's lazy evaluation of outputs.logits returned a function object instead of a tensor. TRL's entropy calculation tried to do math on a function. The monkey-patch approach failed because TRL imports entropy_from_logits by name into its local scope — patching the module doesn't update already-imported references. Fix: patch the TRL source file on disk inside the container at build time.

After those two fixes, SFT ran clean — 207 steps, loss 1.0 → 0.78.

DPO exposed the ugliest bug. TRL's DPOTrainer detects Gemma 3 as a multimodal model from model.config and unconditionally enters the vision-language code path. It expected pixel values. For a text-only fine-tune. On a text-only dataset.

The fix: subclass DPOTrainer as Gemma3TextOnlyDPOTrainer, override process_row to redirect through tokenize_row instead of the VLM path. Pass Gemma3Processor (which has a .tokenizer attribute) instead of a bare tokenizer. 44 steps, loss 0.66-0.76, preferences correctly directional.

Deployment revealed the identity problem. After GGUF export and Ollama deploy, the first test prompt — "Hello! Who are you?" — got: "Hi there! I'm Gemma, a large language model created by the Gemma team at Google DeepMind."

Base model identity bleeding through. The personality layer (Layer 2) nudged behavior but didn't fully override the pre-training identity at 4B scale. Adding a SYSTEM directive to the Ollama Modelfile fixed it — the model then responded correctly as Mary Ann. The fine-tuning taught the model how to be Mary Ann. In my run, the system prompt is what told it that it was Mary Ann. More on whether that's a 4B ceiling or a 140-example ceiling below.

The actual stress test results: strong on structured troubleshooting (Layer 1 worked), clean safety refusals (Layer 2 boundaries held), weak on account recovery procedures without RAG context (Layer 4 needs RAG to deliver specific SOPs — as designed).

The Counter-Argument: Just Mix Everything

Unsloth's documentation says that if you have multiple datasets, it's best to combine them and fine-tune in a single process rather than train on them one after another. They don't spell out why. The reason I'd give is catastrophic forgetting — when you train on Layer 3 data, the model may forget what it learned in Layer 2.

This is a legitimate concern, and I want to be precise about what I'm actually doing vs. what that guidance addresses.

The guidance targets sequential SFT runs — loading a checkpoint, training on Dataset A, saving, then loading and training on Dataset B. Each subsequent SFT run overwrites gradients from previous runs. That's the catastrophic forgetting failure mode.

My Layers 2-5 don't run as four sequential SFT passes. They're combined into a single SFT dataset (550 examples), ordered by layer but trained in one pass. The ordering affects what the model sees first within that pass — personality examples before tool-use examples — but it's not four separate training runs. The gradients from all layers coexist within the same optimization trajectory.

The GRPO stage is a separate run before SFT. This is the deliberate ordering choice, and it's the one the amplify-versus-replace result supports — GRPO first to amplify reasoning, SFT second to add capabilities. The risk of GRPO-then-SFT is that SFT could overwrite some of the reinforced reasoning patterns. In practice, with QLoRA targeting specific modules and a modest 550-example SFT dataset, the reasoning foundations survived. The stress tests confirmed structured troubleshooting persisted through the SFT and DPO stages.

Whether this ordering outperforms a single mixed pass — I don't have that data. The ordered approach produced a model that passes all capability tests, and the theoretical justification is sound, but a shuffled version might pass the same tests. The comparison I owe is specified in the caveats below.

How This Compares

The closest academic test of ordering inside SFT is "Unveiling the Secret Recipe" (December 2024), and it cuts against me. They compared phased training against stacked training — everything mixed into one pass — on 3B-7B models across general benchmarks, and found no significant difference. Stacked was simpler and more sample-efficient. That's the strongest published evidence that ordering SFT data doesn't matter. Two things keep me from folding: their phases were by task type on general instruction data, not by capability layer for a single role, and their benchmarks were MMLU-style, not "does the model still know who it is." Different axis. Whether that difference is enough to produce a different answer is exactly what my ablation has to show.

The canonical industry pipeline is SFT → DPO → GRPO (reasoning last). I'm running GRPO → SFT → DPO (reasoning first). The rationale: if GRPO amplifies and SFT replaces, I want to amplify reasoning before SFT starts replacing. The standard pipeline assumes SFT provides the instruction-following foundation that GRPO then refines. My ordering assumes the base model (Gemma 3 4B, already instruction-tuned) has sufficient instruction-following capability, and GRPO should strengthen reasoning before SFT adds role-specific behavior.

ByteDance's Seed POLARIS RL recipe made a 4B model rival a 235B model on math reasoning, and a core finding was that difficulty is model-dependent — models of different capability see mirrored difficulty distributions on the same dataset, so the training data has to be built around the specific model. Training configurations need to be tailored per model, not assumed universal. My 5-layer taxonomy is specific to a customer-facing voice assistant. A code assistant would have different layers in a different order.

The academic frontier is already past static ordering. Self-Evolving Curriculum (2025) uses multi-armed bandits to select training prompts of optimal difficulty as the model improves. My approach is simple and static by comparison — but it's also implementable in an afternoon on consumer hardware without a research team.

October 2026 addition. I went looking for anyone who had tested what I did. I couldn't find anyone, and I want to be exact about what I was looking for, because there are two different interventions in this article and they keep getting collapsed into one.

The first intervention is stage order: GRPO before SFT instead of after. Six months on, the field still runs it the other way. An April 2026 paper on autoformalization opens with "SFT followed by GRPO is a common post-training recipe" and treats the data overlap between the two stages as the hyperparameter worth tuning. A June 2026 paper on quantum reasoning benchmarks SFT alone against SFT → GRPO. Neither tests GRPO → SFT. The TRL docs still present SFT, GRPO, and DPO as independent trainers with no recommended chain. So my stage order isn't refuted. It also isn't replicated. It's a hypothesis with one data point, and that data point is me.

The autoformalization paper does hand me one thing. They found that keeping SFT and GRPO data disjoint beats overlapping it, and that at full overlap the GRPO stage becomes "effectively redundant." My 183 reasoning chains and 550 SFT rows are disjoint by construction — different capability, different files. I did that for bookkeeping. It turns out to be the configuration their ablation favors.

The second intervention is in-dataset order: personality examples before tool-use examples inside one SFT pass. The two 2026 curriculum papers don't touch this either. PAC (August 2026) allocates GRPO rollouts across tasks using advantage and progress signals. Self-Evolving Curriculum learns a sampling policy during RL. Both are about which problems the RL loop sees next, not about how you sequence a 550-row SFT file. They're closer to Layer 1 than to Layers 2-5. If you read them as validating or invalidating my layering, you're reading them wrong.

What the field has converged on since April is that order is a real variable. That's the part Bengio got right in 2009 and everyone keeps rediscovering. Which order, at which stage, for which model size — still open.

Honest Caveats

No A/B evaluation, and here's the one I owe. I haven't trained the same model on shuffled data to compare. The layered model passes all capability tests, but I can't prove ordering caused that — a shuffled version might pass the same tests. The next training cycle has two comparisons in it, same 550 examples, same 183 chains, same 175 pairs, same seeds, same stress battery:

1. Layered SFT vs. shuffled SFT, GRPO held first in both. This tests the in-dataset ordering claim on its own.
2. GRPO → SFT → DPO vs. SFT → GRPO → DPO, layered data in both. This tests the stage-order claim on its own.

The two comparisons share an arm, so that's three runs on one 16GB card. If layering wins the first and loses the second, I'll say so. If both lose, this article becomes a post-mortem, and that's fine — it would still be the only published test of either question at this scale.

Catastrophic forgetting is a real risk. The GRPO → SFT boundary is where it matters most. If a future SFT dataset is significantly larger or covers broader capabilities, it could erode the reasoning reinforcement from GRPO. I'm monitoring this across training cycles.

This is a 4B model. The capability taxonomy that works at 4B — where each layer's training has outsized impact on a small parameter space — may not apply at 14B or 70B, where the model has more capacity to learn everything simultaneously without interference.

The identity problem might be a data problem, not a 4B problem. My first read was that fine-tuning alone couldn't override pre-training identity at this scale and a system prompt was required. That's stronger than my evidence. What I actually know: 140 personality examples at the QLoRA rank I used didn't override "I'm Gemma" on a cold "Who are you?" prompt, and a one-line SYSTEM directive did. I don't know whether 500 examples would have, or a higher rank, or a few identity pairs in the DPO set. Until I test it, the defensible claim is narrower: at 4B with ~140 examples, bake in the how through fine-tuning and keep a system prompt for the who. It's cheap insurance either way.

Static ordering is behind the frontier. Adaptive curriculum methods (Self-Evolving Curriculum, and now PAC) that respond to model competence during training are more sophisticated. For a practitioner running on consumer hardware with a few hundred examples, static ordering is practical. For a research team with a compute budget, adaptive is better.

Try This Yourself

If you're fine-tuning a local model for a specific role, here's the recipe:

1. Define your capability layers. What does your model need to do? Reasoning, personality, tool use, domain knowledge — whatever applies. Order them from most foundational to most replaceable.
2. Separate your training data by layer. Tag each example with its primary capability. You don't need thousands — 100-200 examples per layer is sufficient for QLoRA on small models.
3. Run GRPO first if you have reasoning data. Use reward functions, not a critic model — GRPO doesn't need one, which is why it fits on consumer GPUs. Unsloth makes this practical at 5GB VRAM. Know that you're running against the grain of the standard recipe when you do this, and keep a checkpoint so you can run it the other way.
4. Combine SFT layers into one ordered dataset. Don't run four separate SFT passes. Combine into one training dataset, ordered by layer (personality first, specialist knowledge last). Train in one pass.
5. Run DPO last on preference pairs. The chosen/rejected pairs should span your capability layers. The model needs to have learned the capabilities before it can learn which responses are preferred.
6. Test each layer independently. Don't just check "does it respond?" — test reasoning quality, identity consistency, tool use accuracy, SOP compliance, and domain knowledge separately. My first deploy looked successful until I asked the model who it was.
7. Keep the shuffled control. It's one extra run. Without it you have a story, not a result. I skipped it in April and I've spent six months unable to say the word "proved."

The total investment: a day organizing your training data into layers, plus whatever your training pipeline already takes. The return is a model whose capabilities stack predictably instead of interfering randomly — or, if the control run wins, a clear answer that saves you from doing this again.

The Broader Point

Most fine-tuning tutorials treat training data as a homogeneous blob. Collect examples, shuffle, train, evaluate. The implicit assumption is that more data and more epochs will produce a better model, and the ordering doesn't matter.

That assumption works when you're training for a single capability — instruction following, or coding, or math. It breaks down when you're training for a role — a model that needs to reason, maintain identity, use tools, follow procedures, and apply domain knowledge, all at once.

Curriculum learning says the order matters. Multi-stage alignment says the method matters. "Scalpel vs. Hammer" says the mechanism matters — GRPO and SFT produce fundamentally different internal changes.

The question isn't whether your model can learn everything in your dataset. At 4B parameters with 550 examples, it probably can. The question is whether the capabilities reinforce each other or interfere with each other. And that depends entirely on the order in which the model encounters them.

I didn't invent this. Bengio published it in 2009. OpenAI operationalized it in 2022. What I did was apply it to a specific model, a specific role, and a specific GPU — and watch the capabilities stack instead of collide.

The most effective training run isn't the one with the most data. It's the one where the model learns things in the right order.

Six Months Later

I sat on this for six months. That turned out to be useful. Here's what I'd tell April me.

The two ideas in here are still only mine. GRPO before SFT, and layering SFT data by capability type instead of difficulty. Six months of papers and I couldn't find anyone who had tested either one. There's no result saying I'm wrong. There's also no result saying I'm right, and "it worked once on my machine" is not a result.

The field moved toward adaptive curricula for RL, not toward ordered SFT. PAC and Self-Evolving Curriculum are the state of the art for what the GRPO loop should see next. If I had a research budget, Layer 1 would use one of them. Layers 2-5 would still be a hand-ordered file, because I haven't seen an adaptive version of that and 550 rows wouldn't justify building one.

The ablation is the next article. Three runs, two questions, one GPU. Whatever the answer is, it's the first one at this scale.

This is the seventh article in the series. Previous: "Everyone Is Building Agents That Think When You Ask. I Built One That Thinks When You Don't." (cogitation daemon), "I Gave My AI a Brain — Here's How It Learns From Every Conversation" (self-improving RAG), "I Replaced My Training Pipeline With Two Containers and a 12-Year-Old Server" (containerized fine-tuning), "I Made My 14B Model 40x Faster by Teaching a 0.5B Model to Say 'I Got This'" (query routing), "I Benchmarked My GPU and Found It Was Idle 83% of the Time" (PCIe bottleneck), and "I Built a 4-Source Hybrid RAG System That Fine-Tunes Itself From My Commits" (hybrid RAG).

The implementation runs on an RTX 5060 Ti (16GB) using Unsloth, TRL, and Ollama. Base model: Gemma 3 4B (unsloth/gemma-3-4b-it-bnb-4bit). Training: QLoRA with GRPO → SFT → DPO. All local — no cloud training services.

References:

• Curriculum Learning — Bengio et al., ICML 2009. The foundational paper on ordered training data.
• Training Language Models to Follow Instructions with Human Feedback — Ouyang et al., NeurIPS 2022. InstructGPT and the SFT → RL pipeline.
• Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them — July 2025. Mechanistic comparison of GRPO and SFT. The authors describe the findings as preliminary.
• Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs — December 2024. Phased vs. stacked SFT on 3B-7B models; no significant difference found.
• ByteDance Seed POLARIS RL Recipe — 4B model rivaling 235B via RL. Model-dependent difficulty distributions.
• Unsloth Datasets Guide — Official recommendation on multi-dataset training.
• Two-Stage SFT + GRPO Pipeline — Documented SFT-then-GRPO paradigm.
• SFT-GRPO Data Overlap as a Post-Training Hyperparameter for Autoformalization — April 2026. SFT-then-GRPO as the common recipe; stage data overlap as the variable.
• Fine-Tuning Large Language Models for Quantum Reasoning — June 2026. SFT vs. SFT → GRPO comparison.
• PAC: Progress-Augmented Advantage Curriculum for Multi-Task Reinforcement Learning of LLMs — August 2026. Adaptive task allocation inside the RL loop.
• TRL Quickstart — SFT, GRPO, and DPO as independent trainers; no prescribed order.