Source note: This field note first appeared on Eric's Advisory Hour Substack. Read the canonical post or subscribe there.
In my NPU series on substack I’ve written about adventure and discovery, in search of missing tokens/sec for my local ai workloads. Precious compute gone to waste, unused and unoptimized because the wrong thing is doing the right thing like a microwave warming a piece of bread instead of making use of a toaster.
And I pushed the lever on that toaster so many times, each press popping in and out a new substack article covering the detail of a chip. The bread, the chip, the Snapdragon Hexagon chipset, is a System on a Chip architecture using a NPU and GPU.
The hardware is hidden behind three powerful letters:QNN, which is a software layer that handles dispatching workload. And these insights I’ve shared elsewhere and for a good reason, which I’ll show you now.
Below, just a few lines from here, this was the sorrowful state of gemma4 on my machine when I started. This first clip is of my local gemma4 harness-lord, each token chiseled in stone and by hand. The speed is as slow and clumsy as my human hands attempted to write coherently.
And what you see in the writing is a command line prompt, which obscures a fancier thing. The elegance of a harness all simplified into a mere set of letters and angled bracket-but wait! Why bother with a custom harness? Have you not written of all the any other wonders of the modern world?
Yes, there are general purpose harnesses.
However, I wanted a harness that was like a custom suit: tailored for my needs, on my hardware, and something that I didn’t have to worry about being at risk of a future update breaking something that’s working just fine. I made a simple wheel, just good enough to get work done, and just for me.
Sometimes a wheel can just be… a wheel, you see.
▶ Watch clip on Substack
So the harness wheeled on, and now the runtime environment have gone thru several iterations and so I can claim that the runtime is usable. This is a strong claim with follow-up requirements.
The next step is to put this harness to productive uses. Uses that are hinted at in the prior video and in this video that you should take a moment to click play on. Once you clicked play what happens next is the start of a scene.
The scene scrolls by, showing the tools available in the harness are all situated and ready for some interesting work. Interesting for me is clearly going to be weird. All my Field Notes here in this journal have been quite along those lines.
Long substack subscribers know I poke around the retro PC games, and while it can be serving in risk of poking mself in the eye, I have in mind to test this harness with a little digital archaeological project of a game from the late 80s. A block of work I suspect will work just fine, and if it does may open up the door to even larger project work.
▶ Watch clip on Substack
As you no doubt watched the frames play by in that video at a fast speed, which suggests I’m ready to jump into action. And you’d be right. I’ll be posting more articles showing how this gemma4 harness is pushed into useful labor.
On the balance, I’m pleased that i’m getting real performance on this now, and not the slow tok/sec. The solution to getting it all to work at these speeds is not something that I think to be inherently usable by others, but if you send my NPU articles to codex or claude code and set a long running /goal, the path forward will be made clear.
Speaking of seeing the path forward…
Related notes
This GPU dispatch note continues the local-inference sequence from Tinkering with a NPU, NPU and Gemma4, Snapdragon Hexagon: Learning about QNN, and Gemma4 Harness for Vision on GPU/NPU/CPU.
Original source
This local copy preserves the article text, source link, and inline visual context. Canonical Substack URL: https://advisoryhour.substack.com/p/gemma4-on-snapdragon-gpu-case.