This page preserves my original AdvisoryHour/Substack post inside the John’s World research section. The original remains available at advisoryhour.substack.com.
After working with 1995 universe for buttons and game consoles, it felt like a good time to return back to cars. Cars are hard. You can just tell when the car isn’t moving right.
The tracking system I built to track coherence of objects works ok, but with the new vision model capabilities that I’m nearly set to unlock I suspect I’ll be able to do so much more. A car, after all, has many “button-like” considerations. What might I be able to do though if I start using my local vision AI tooling? Could I rapidly improve the coherence of the scenes?
Unlocking local AI as a builder is like going from a handsaw to a table saw and full set of Milwaukee power tools.
Vroom.
I started with the obvious thing to fix: the car chase.
The new goal then is to identify what makes for a compelling car scene-there are times where a small brilliance emerges on the scene. Audio direction is important in car chases-what do you give priority to? The lead car? Ambient noise? The drivers? And interestingly, the video models tend to use the same generic car noise when unprompted, regardless of the scene. This car “peel out” sound is eerily similar to the same one found in 1986 Sabrina’s drive around town.
Audio World
This unexpected turn of events revealed a new pathway for John’s World: audio! The idea is that instead of building up the scene visually and mapping audio onto it? What if I started with audio. The starting point then is the premise of the video shown above, only a tad simpler. Let’s have our two lead actors go on a long drive. Idle chat, just living in the digital world.
An audio scene needs specific capabilities that emerge from game world construction. You need your lead actors (obvious, I’m sure, but worth noting). You need background music. And you need sound effects.
Then a composition system to bring it all together. Here’s the odd thing: to make that work, you need a virtual world engine. If this does not exist then you risk chaos: the audio gets slotted or sequenced in a random way and coherence falls apart. It’s like a factory where all the components are tossed in at random and you just hope for the best on the other side. Hope is no substitute for engineering.
When such a system works it has a chance of pulling you into another world. It’s odd, surreal and strange-intentionally so. It’s not here. It’s there.
I asked Claude to provide a deeper dive into what I’ve been building across Codex and, well, Claude. You can use this deep lore text as a recipe for your own universal engine.
Building a Modular AI Audio Engine for a 1977 Universe
Most AI audio tools do one thing — a voice here, a sound effect there — and leave you to glue the pieces together by hand. So I built a little engine that treats a finished soundtrack the way a film mixer does: as layers. You write a plain JSON “scene” that lists the ingredients — a continuous engine-and-wind bed, one-shot effects like a siren or a tire screech dropped at specific timestamps, a music track, and the characters’ spoken lines — and the system generates every piece, places it on a timeline, balances the levels, and renders a single mixed audio file. The dialogue even auto-sequences itself: you just write the lines in order and it spaces them out naturally, so a two-minute scene of two people shooting the breeze on a sunset drive comes together without me hand-timing a single cue.
Under the hood it’s built around a “provider” concept, which is just a fancy way of saying every kind of sound is pluggable. There are three families: text-to-speech (I can route a line through ElevenLabs, MiniMax, or Inworld and pick whichever voice fits the character), sound effects (ElevenLabs’ library or Stable Audio), and music (MiniMax’s model, which writes original songs or instrumentals from a one-line style description). Because the pieces are interchangeable, I can define “profiles” — preset combinations — and swap the entire sonic palette of a scene by changing one word. A radio-chatter filter rounds it out: any line can be run through a CB-radio effect so a sheriff barking over the airwaves sounds like he’s actually coming through a staticky speaker, while the in-car banter stays clean and warm.
The interesting engineering wrinkles were all about control. AI audio models love to decide their own length — the music model happily handed back a three-minute song when I needed exactly 125 seconds, and one effects model always returns a fixed 47-second clip no matter what you ask — so the engine quietly loops or trims everything to land on the duration the scene actually needs. Different services also speak slightly different dialects of the same API, so there’s a shared layer that smooths those over. The payoff is that the same building blocks I use to produce a standalone “audio adventure” — a chase scene with roaring V8s, sirens, CB squelch, and five distinct voices — are the exact same blocks I’ll drop under video later: a music bed fit to a clip’s runtime, dialogue, foley, all from one declarative spec. It turns sound design from a pile of one-off downloads into something you can author, version, and re-render on demand.
Claude writes so well. When I review my past writings, it’s hard. I see every mistake and error. It’s progress, perhaps. I see the error, and before I did not.
Oh, hello. This is a note to let you know I’m practicing my writing again.
In today’s writing exercise I put to work the word unusual vocabulary that I don’t often use. OK, I never use.
Take a Palimpsēst and try again
Palimpsēstos comes from the Greek word that traces back to manuscript pages that were loosely erased to be re-used again. A whiteboard can become a palimpsest of half-erased equations, architecture diagrams, or doodles. This is especially true in downtown offices that survived covid, but whose whiteboards have “SAVE ME” written on them nearly ten years ago and whose original authors had no idea “SAVE ME” was both physical and literal during the pandemic era.
The specific term that exists in today’s language of this layered use of the old for the new, I’m told, is palimpsest. It’s what I see across the great Midwestern cities of America. Cities built, rebuilt and then built again over generational epochs. Landscapes marked by previous uses, full of specific revisions and architectural memory shaped into stone and street.
These simulated universes are like this, in a way. The original timeframe is lost to history. Whatever artifacts we have are vacuumed into the magic of self-learning machine, which permits an uneasy immortality of those whose photos we were permitted to capture. The simulation is perhaps vestigial of another time, but in a limited sense. The remnant of a functional past where reality was deeply coherent, and in this time of transition just a sloppy mirror of what could have been.
There is a lacuna to the simulation, to what John’s world provides. I can see it in the video, the audio. It’s convincing, surreal in a sense, and there’s this profound feel of a gorging gap, blank, omission in what’s provided. There’s a sense in the word “Lacuna” that something expected or once present is absent. You can see it in so much AI writing, simulation and more. Perhaps this is a result of the era of the Great Smoothing: when gaussian averages smooth out every peak and valley, only in so doing we instinctively feel the omission with clarity.
It’s not just that watching the past in simulation is complete. There’s an omission that just feels evident. I keep working at the gears of virtualworlds to give some name to the absence. The thought is if I do, the clarified lacuna becomes filled and the immersion is teleporatitve.
The word Lacuna is a fun word. I want to say “Lagoona” over “La-koon-ah”, and yet perhaps both share similar origins. Lacuna means pit, cleft or gap and is related to lacus, which translates as “lake”. It’s used in medical sense to give clarity to the spaces in bone or cartilage where cells reside. Still, a gap that is also a pool seems somehow fitting.
Simulation with video is a strange beast. The world models remind me of a liquid or fluid. From what I’ve read of their training, there seem to be odd crossover equations between the two. It would not surprise me in the least if the next incredible machine learning breakthrough is born out of aerospace physics. Perhaps the latent space is just another fluid, one we all need to learn to control.
I asked ChatGPT what might be considered a palimpsest lacuna. It had this to say.
A palimpsest lacuna is not merely “a blank.” It is a missing place inside accumulated traces—a hole in a surface or story that otherwise preserves history.
The video models generated have incredible gaps in the world-state. There’s this very clear connection.
The harness I’ve built for John’s World tries to cover for this. An object lacks history. The crack, chipped paint and dented fender aren’t inherently found inside the model-instead? All fenders are found. There’s continuity without full memory. And increasingly my work in video models has me realize the real world is full of memory.
We’re just often too busy to notice.
Thanks for reading! Subscribe for free to receive new posts and support my work.