← Back to Custom Model Training
2026-04-03Model TrainingTernary weightsSLM

Ternary Weights Training Field Note

A short Advisory Hour note on a model-training experiment where the weights are constrained to -1, 0, or 1 instead of full floats.

Source context: This field note first appeared on Eric's Advisory Hour Substack. View the original note.

The field note

Experiment in model training where the weights aren’t a float but is a -1,0, or 1 is going surprisingly well
Screenshot of an Astal app run report saying the overnight ternary recipe training run went very well, with exact-match scores of 0.95 in-template, 0.95 near-OOD, 0.45 far-OOD, and 0.55 depth-generalization.
The attached run report points to an overnight six-hour ternary-recipe experiment with strong in-template and near-OOD exact-match scores, while far-OOD and depth-generalization remain the harder tests.

Why it matters

The interesting part is the constraint. Replacing floating-point weights with a tiny {-1, 0, 1} vocabulary turns the training question into a pressure test for representation: how much useful behavior can survive when the model has almost no numeric nuance to hide behind?

That matters for the small-model thread because constraints expose structure. If a narrow task still performs well after the representation is squeezed this hard, the system may be learning a compact recipe instead of merely memorizing a high-precision surface.

What this note points toward

  • Compression as a diagnostic: ternary weights make it easier to see which parts of a task need real capacity and which parts are overbuilt.
  • Cheap specialist loops: constrained weights can make repeated overnight experiments feel closer to iteration than capital expenditure.
  • Evaluation pressure: the screenshot's split between near-OOD and far-OOD keeps the useful question centered on generalization, not just headline fit.

Connection to the SLM series

This belongs beside the nanochat, pico-LLM, and local-inference notes because it asks the same practical question from a different angle: what is the smallest useful machine for a bounded job?

The next useful artifact is not just a better score. It is a ledger of what fails as the representation gets smaller: where exact match holds, where far-OOD collapses, and which data recipes move the boundary.

Related notes

For the surrounding small-model thread, see Why Tiny Specialists Matter, The Depth Dial and Miniseries, GPT-2 Economics Under $100, and Training Small Model Abilities.

Original source

This local copy preserves the note text, attached image, and source link from Eric's Advisory Hour. Original Substack URL: https://substack.com/@advisoryhour/note/c-237607549. You can also subscribe to Advisory Hour.