I Benchmarked My GPU and Found It Was Idle 83% of the Time

The PCIe bottleneck nobody talks about — and what it means for 40 million GPU systems shipping every year

By Corey Ryan · April 12, 2026 · 9 min read

I had a question that wouldn't leave me alone: what actually happens to data moving between a CPU and GPU inside a Docker container?

Not the theory. Not the spec sheet numbers. The actual, measured reality.

So I built a benchmarking suite, ran five phases of experiments on my RTX 5060 Ti, and what I found surprised me: my GPU spends 83% of its time doing absolutely nothing — just waiting for data to arrive over the PCIe bus.

The GPU isn't slow. It's starved.

The Setup

I ran all benchmarks on a consumer-grade system: an NVIDIA RTX 5060 Ti (16GB GDDR7), Intel i9-10900KF (10 cores), and 32GB DDR4 RAM. Everything ran inside Docker containers with CUDA 12.8 and PyTorch 2.11 — a setup that mirrors what many AI practitioners use in production.

The benchmark suite tested five things:

1. Baseline: How long does a GPU matrix multiplication actually take when you include data transfers?
2. Memory strategies: What if you could eliminate transfers entirely?
3. CPU-GPU interleaving: Can you overlap CPU and GPU work to hide the transfer cost?
4. Scheduling jitter: Does Docker's CPU pinning make GPU work more deterministic?
5. Real LLM inference: Does this bottleneck affect actual AI workloads, not just synthetic benchmarks?

Finding #1: Your GPU Is a Sports Car Stuck in Traffic

When I ran a 4096x4096 matrix multiplication — a standard GPU workload — here's where the time went:

• Transfer data to GPU: 19,448 microseconds (39%)
• GPU compute: 7,958 microseconds (16%)
• Transfer results back: 22,235 microseconds (45%)

Total: ~50,000 microseconds. The GPU was working for 8,000 of them.

The rest was PCIe bus traffic. Data crawling between system RAM and GPU memory at 32 GB/s while the GPU's own memory interface sits at 672 GB/s — a 21x bandwidth mismatch.

I ran this across five matrix sizes (512 to 8,192). The transfer overhead ranged from 73% to 93%. At no tested size did the GPU spend more than 27% of wall-clock time computing.

Finding #2: Eliminating Transfers = 6-13x Faster

The natural question: what if you could skip the transfers entirely?

I tested this by keeping data resident on the GPU — simulating what a unified memory architecture (like AMD's MI300A or Apple Silicon) provides. No copies, no PCIe bus, just compute.

The results across five matrix sizes:

• At 512: 686 us with transfers vs 114 us GPU-resident — 6.0x faster
• At 1024: 2,198 us with transfers vs 168 us GPU-resident — 13.1x faster
• At 2048: 8,645 us with transfers vs 1,079 us GPU-resident — 8.0x faster
• At 4096: 48,559 us with transfers vs 8,206 us GPU-resident — 5.9x faster
• At 8192: 220,843 us with transfers vs 63,701 us GPU-resident — 3.5x faster

At 1024, eliminating transfers made the workload 13 times faster. Even at 8192 — where GPU compute is substantial — the speedup was still 3.5x.

The GPU was hitting ~17 TFLOP/s the entire time. It wasn't getting faster. It was just finally being allowed to work without waiting.

Finding #3: Even 11% VRAM Spill Destroys LLM Performance

Synthetic benchmarks are one thing. Does this matter for real AI workloads?

I tested three LLM models through Ollama:

• Llama 3.1 8B (4.9 GB, 100% in VRAM): 83 tokens/sec, 13.2 ms per token
• Qwen 14B (9.0 GB, 100% in VRAM): 48 tokens/sec, 23.9 ms per token
• Mistral Small 24B (15.2 GB, 89% GPU / 11% CPU): 20 tokens/sec, 50.5 ms per token

Mistral Small barely exceeds the 16GB VRAM limit, offloading just 11% of its layers to system RAM. The result: a 4.2x performance collapse. Every token generation has to shuttle activations across PCIe for those offloaded layers, serializing the entire decode pipeline.

11% offload. 77% performance loss. That's the PCIe tax.

Finding #4: Docker CPU Pinning Helps — But Only for Transfer Jitter

I tested Docker containers with and without CPU pinning (--cpuset-cpus) to see if container isolation improves determinism:

• PCIe transfer jitter: 0.38% with pinning vs 0.69% without — 45% less variance
• GPU compute jitter: 2.91% with pinning vs 2.99% without — negligible difference
• CPU compute jitter: 6.23% with pinning vs 7.19% without — 15% less variance

CPU pinning significantly reduces PCIe transfer variance — the DMA operations that move data between CPU and GPU memory are sensitive to OS scheduling interference. But GPU compute jitter is identical either way. The GPU has its own scheduler; CPU pinning can't touch it.

This matters for reproducible benchmarks and production inference where consistent latency matters.

Finding #5: CPU-GPU Overlap Only Works When Workloads Are Balanced

I implemented an APEX-style pipeline where the CPU preprocesses batch N+1 while the GPU computes batch N. The idea: hide the transfer cost by overlapping work.

The result was 0% overlap efficiency.

Why? At 2048x2048, CPU preprocessing took 20,861 microseconds while GPU compute took only 1,269 microseconds. The GPU finished 16x faster than the CPU could prepare the next batch. There was nothing to overlap — the GPU was idle waiting regardless.

This is the often-overlooked prerequisite for CPU-GPU interleaving: the workloads must be time-balanced. When one side dominates, the other sits idle no matter how clever your pipeline is.

A double-buffered approach (pre-computing all CPU work, then pipelining GPU transfers and compute) achieved a 4x speedup — but it sidesteps the balance problem rather than solving it.

The Scale of the Problem

This isn't a quirk of my hardware. Every system with a discrete GPU and a PCIe slot has the same architectural bottleneck.

That's approximately 40 million GPU systems shipping per year: 35 million discrete desktop GPUs, 3.5 million datacenter GPUs (H100, A100, L40S), and 2 million professional workstations. It represents 67-70% of enterprise GPU compute revenue.

The platforms that have solved it are either datacenter-only (NVIDIA Grace Hopper with 900 GB/s NVLink-C2C, AMD MI300A with 5.3 TB/s unified HBM3), ecosystem-locked (Apple Silicon with unified LPDDR5), or consumer-tier with limited bandwidth (AMD APUs sharing DDR5).

PCIe Gen 5, now appearing on high-end motherboards, doubles bandwidth to 64 GB/s. That cuts the bottleneck in half — but the GPU memory bandwidth gap is still 10x. The fundamental architecture hasn't changed.

What You Can Do About It Today

Based on the experimental data, here's what actually moves the needle on existing discrete GPU hardware:

1. Keep models in VRAM. This is the single highest-impact optimization. Quantize aggressively (Q4/Q5) to fit within your GPU's memory. The 4.2x penalty for even minor offloading is devastating.
2. Use double-buffered CUDA streams. For custom workloads, overlapping transfers and compute via dual CUDA streams with pinned memory recovered 44% of the transfer overhead in my tests.
3. Pin CPU cores for inference containers. Docker --cpuset-cpus reduces PCIe transfer jitter by 45%. This won't make inference faster, but it makes it more consistent — which matters for SLA-bound services.
4. Batch your requests. Larger batch sizes increase operational intensity, pushing workloads toward the compute-bound regime where the GPU spends proportionally less time waiting for data.

For a structural fix: unified memory hardware is the path. AMD's Strix Halo APUs and Apple Silicon offer it at the consumer level. MI300A and Grace Hopper offer it at datacenter scale. The performance case is clear from the data — 3.5x to 13.1x for compute, ~4x for LLM inference.

Beyond Unified Memory: PIM and MRAM

Unified memory eliminates the PCIe bus. But there's a deeper question: what if you didn't just share memory between CPU and GPU — what if the memory itself could compute?

That's the premise behind two emerging technologies that could fundamentally reshape the data-to-compute relationship.

Processing-In-Memory (PIM) embeds compute logic directly inside memory chips. Instead of moving data from HBM to GPU cores, PIM performs operations where the data already lives. SK Hynix's GDDR6-AiM achieves 10x faster processing at 1/5 the power compared to discrete GPU systems. Samsung's HBM-PIM delivers 2x performance with 70% power reduction by integrating SIMD engines into HBM banks. UPMEM's DDR4-based PIM DIMMs show an average 23x speedup over high-end Intel Xeons on memory-bound workloads — with peaks of 93x on streaming operations.

The implications for LLM inference are significant. The decode phase — generating one token at a time — is inherently sequential and memory-bound. It's the exact workload type where PIM excels. One study projects a 6.94x reduction in 3-year total cost of ownership for cloud LLM inference using PIM architectures versus state-of-the-art GPUs.

Magnetoresistive RAM (MRAM), pioneered by Everspin Technologies, adds another dimension: non-volatile memory with near-SRAM latency. STT-MRAM reads at 10-20 nanoseconds — 2-5x faster than DRAM and comparable to HBM — while retaining data without power. Research shows MRAM-based GPU caches deliver 4-16% IPC gains with 20% lower power versus traditional SRAM caches, at 4x the density.

What makes MRAM novel in this context is persistence. Model weights stored in on-package MRAM survive power cycles. No reloading from disk, no PCIe transfer to initialize VRAM. For edge AI and multi-model inference servers that constantly swap models, this eliminates an entire class of data movement overhead.

Samsung has already demonstrated MRAM-based in-memory computing — embedding compute directly into MRAM crossbar arrays, achieving 98% accuracy on classification tasks with data never leaving the memory array.

Where this converges: Imagine a GPU package with HBM for working memory, MRAM for persistent model weights, and PIM logic embedded in both. Data never transits PCIe. Weights never reload. Compute happens where the data lives. The 83% idle time we measured wouldn't just be reduced — the entire concept of "data transfer" as a separate phase would disappear.

This isn't science fiction. HBM4 (mass production 2026) integrates a logic base die into memory stacks — the first standard enabling native PIM in next-generation GPUs. JEDEC is standardizing LPDDR6-PIM. The building blocks are being manufactured now.

Our benchmarks quantified the problem. Unified memory solves the bus. PIM and MRAM solve the architecture.

Limitations

I want to be transparent about what this data does and doesn't prove:

• Simulated unified memory, not tested. The GPU-resident benchmarks eliminate transfers, simulating what UMA provides. Real unified memory has cache coherency overhead (5-15% per MI300A research) that our tests don't capture. Our speedup numbers are optimistic by that margin.
• Matrix multiplication is a worst case. Matmul is the most transfer-intensive workload — all data moves, all compute is parallel. Real neural networks have attention, normalization, and activation layers with different transfer/compute ratios. The 83% overhead is an upper bound for homogeneous matmul; real workloads likely show 40-70%.
• Single hardware configuration. Results are specific to RTX 5060 Ti + i9-10900KF + DDR4. PCIe Gen 5 systems would show lower transfer overhead. Different GPU architectures may behave differently.
• Ollama offloading may not be optimal. The 4.2x LLM slowdown reflects Ollama's specific layer offloading implementation. Optimized inference engines with prefetching and pipeline overlap could reduce this gap.

The Bottom Line

Three numbers summarize two days of benchmarking:

• 83% — the portion of GPU wall-clock time spent on PCIe transfers, not computation
• 13.1x — peak speedup from eliminating transfers (simulating unified memory)
• 4.2x — real-world LLM performance loss from just 11% VRAM spill to system RAM

Your GPU is fast. Incredibly fast. But on 40 million discrete GPU systems shipping every year, it spends most of its time waiting for data to arrive over a bus that's 21 times slower than its own memory.

The industry is building toward unified memory — Grace Hopper, MI300A, Apple Silicon, AMD APUs. The question isn't whether the PCIe bottleneck matters. The data is clear: it's the dominant factor in GPU utilization.

The question is how long it takes the rest of the industry to catch up.

All benchmark code, raw data, and detailed methodology are available on request. Experiments conducted on an RTX 5060 Ti 16GB GDDR7, i9-10900KF, 32GB DDR4 using Docker containers with CUDA 12.8 and PyTorch 2.11.0+cu128.

Corey Ryan is a software engineer and AI infrastructure researcher.