Field Notes · Local inference

Gemma4 GPU Dispatch Field Note

A short AdvisoryHour progress note on routing local Gemma4 workloads to the GPU instead of letting them fall back to the CPU.

Originally posted as a Substack note on June 6, 2026. This page preserves the note as first-party field-note copy and connects it to the local-inference thread on the site.

The note

Progress! Have now figured out how to dispatch workloads for gemma4 just to the GPU over the CPU.

Why this small win matters

The useful signal is not just that Gemma4 ran. It is that the workload path became controllable. Local-inference experiments get murky fast when the model, runtime, drivers, graph partitions, and fallback rules all decide where work lands without making that decision visible.

GPU path isolatedCPU fallback reducedBenchmark noise narrowedNext routing target: NPU

Once dispatch is explicit, the experiment can move from “the machine ran something” to “this accelerator handled this slice of the workload under these constraints.” That is the difference between a curiosity and a harness.

Kira Commentary

This is exactly the kind of tiny progress marker that deserves a field note because it captures the moment an experiment becomes easier to reason about. “GPU over CPU” sounds like a hardware preference, but in practice it is an observability and control milestone.

  • Routing is part of the model stack. A local model is not just weights and prompts; it is also graph shape, runtime provider order, memory movement, and what silently falls back when the preferred device cannot handle an operation.
  • The next benchmark gets cleaner. If CPU work is accidental, the numbers lie. If CPU work is intentional and bounded, the benchmark starts to explain the system instead of merely reporting a mystery.
  • The NPU story gets sharper. Before an NPU can be evaluated honestly, the harness needs a working mental model for dispatch. GPU-only routing is a useful stepping stone because it proves the workload target can be constrained.

What might be next: add a visible provider trace to the Gemma4 harness so each run says which device handled each phase, where fallback occurred, and how latency changed. The field-note-sized win becomes durable when the next run can prove it happened again.

How this connects back

Gemma harness

Vision on GPU/NPU/CPU

The larger local Gemma4 harness thread this note extends.

Inference benchmarks

Local Inference Puzzle

Why CPU, GPU, and NPU numbers can tell surprising stories until the runtime path is visible.

SLM series

Custom Model Training

The broader SLM/local-inference hub where these dispatch notes accumulate into a playbook.