City Explorer: What challenging /goal taught me about myself

The obvious and first learning is that /goal taught me action reveals information. But it’s deeper than that, and I feel it is a good idea to take you on a journey with me and city-explorer.

To do that, we need to talk about the Codex feature known as goal, or /goal. The simple explanation: it runs the AI tool in a loop, until some defined end or you exhaust your token budget.

There are a lot of people talking about how they’re using /goal to achieve impressive new software tiers. Long running loops with the goal feature is impressive-but what I haven’t seen people discuss is how /goal raises questions within the human operator. “Using /goal, I realized something about myself.” might even sound strange to the reader. My goal in this article is to surface questions by way of showing how the questions emerged.

City-Explorer is in several days of constant runtime.

I paused Windows updates this week just to let it grind out more, and to make use of the new vision API I discussed in other articles. There’s something that happens with long running software goals like this. My best explanation is that your interaction with the computer and the software produced is like that of the garden outside your house. You periodically check on it, sometimes weed and prune, and otherwise it carries on.

In the Pre-AI era of writing code, producing software is handled by taking an idea, mapping it down to specific work items and then sending your agentic delivery team (human or AI) to get results around the listed work items. The work items are structured in common approaches that include the task to be done, the acceptance criteria, and possibly reference screenshots of what you want to achieve. This, in a nutshell, is how nearly every software company or software manufacturing company works. However, I believe in the power of Sutton’s The Bitter Lesson, which is an essay that anyone writing software should take to heart because of how broadly it appears to apply to so many different software creations.

“The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin.”

Sutton goes on to explain that the reason for this is Moore’s law — the continued exponential fall in the cost per unit of computation — which means that over time, more computation becomes available. In the long run, the only thing that matters is how well a method can take advantage of that growing computational power. This often means simpler, scalable approaches like search and learning outperform methods that rely heavily on human-curated domain knowledge.

The “bitter” part of the lesson comes from the fact that many researchers have been slow to accept this, because it challenges the anthropocentric assumption that human expertise is the key to AI success. Instead, Sutton argues that time spent on complex, knowledge-based designs is time not spent on methods that can scale with compute.

I claim his essay isn’t just applicable to AI.

Suttons’ mental model for the Bitter Lesson can be applied to other systems, and having the lesson in mind can change your orientation to problems. For example, do you really need to take a work item and break it down into stories? Do you need an epic, a kanban, and for however many cards of work to achieve that? You might discover you don’t-maybe a general purpose search algorithm that just writes code by itself and searches for the solution is “good enough”.

Is the problem you’re solving fundamentally about search or learning? A tremendous number of problems can be twisted into being just about search or learning. There’s a very large amount of software that misrepresents itself as a product or feature, when the activity involved is really about search or learning.

Whether I’m right or not on this point isn’t the debate I’m looking to have in this article. I shared this as an explanation for a decision that would kickstart the city-explorer project. It’s important you understand why I structured /goal in a particular way, and why I eschewed other methods for it.

Two projects were the inception behind the idea of City-Explorer. One was the poorly named “John’s World,” which is short for “John Doe’s World”. An experiment that I keep extending and expanding over time. John’s World is a universal engine. It has a temporal system I use to achieve coherence, and slings together multiple types of models to create scenes.

The second was the javascript build of an old MMO called EverQuest. I re-built the game from first principles so it’d run in a browser. So, yes, the entire Client and Server architecture in javascript, ran it in the browser-and I’m still shaking my head at how absurd that one is. You can find coverage of that in my past articles.

The experiments brought me an idea that I could build more challenging scenes. Scenes with far more components. A structured layout that allowed for not just a small setup, but big ones-ones so big I can’t explore them all.

In John’s World, when I started this city-explorer project, I was in the midst of figuring out multi-verse #1986. And so I wondered: what if the AI were directed to build an entire city. Not just a small city, but all of NYC.

/goal Build a threeJS replica of 1986 NYC.

The goal is still running.

Initially the world came into being with juts terrain and water. The early builds of the city were like every other vibe coded city that you’ve seen on the internet. I let /goal continue to work as I pondered the problem. How could I get goal to produce a bigger volume of city? And clearly it was having trouble with the terrain.

Then one day I awoke with an idea. In the codex CLI you can send steering commands as /goal operates. I sent in a prompt: I noticed the terrain is just a noise distribution solve. Since we’re basing this on a real place, why don’t you use an image generator to create a height map?

And then I stepped away from the computer again and went about my day. When I returned a few hours later I had… the broad outline of NYC. The terrain had been fully replaced. Instead of mountains and ponds everywhere, I saw the structure of a real place forming. Wow, this is cool!

The goal is still running.

In the notes I posted you may see a visual aesthetic. This I attribute to an idea I had some time after getting the terrain going. I took screencaptures of the scene and went to ChatGPT. I had chatGPT upscale the images to high fidelity AAA level art. As I watched Codex work, I realized that there wasn’t a viable outcome for the model to work towards. I thought maybe it could use reference art.

As the city was coming together, I had this thought. What if the city streets had a working traffic light system and the cars obeyed the traffic lights? So I dropped into the still running /goal and asked Codex to build out a fully working traffic light system for the road construction is was putting in place so I could see how that would turn out.

It’s pretty wild watching cars move around the scene, coming to a stop on red, and then moving on green.

The goal is still running.

You can take a ride around the city on the train system.

While the original /goal was still running, I had Codex “take a break” to build a train system with traincars from original threeJS geometry. Mind you this is 5.5. The first versions of the train were not great, they resembled jenga blocks assembled at random around the entire city.

I left Codex working and I woke up one day with a new idea. What if I asked Codex to build a bezier curve and procedural mesh generation system and sent in a few links of best practices for building train simulators? My next step was simple. I gathered material, dropped that material into the still running /goal session, and asked Codex to take that info and build out a full train system. And then the entire train system around the scene was fully replaced, and a working train simulator was put in place.

I would drop in additional commands: an interior view so I could take a ride, a mechanism for me to board and deboard the train, and a complex track.

The track, along with the world, is massive.

Other vehicles.

There are barges, boats and ferries. Did I mention how large this world is? It’s absurd. And you know what?

The goal is still running.

In addition to all of this, there’s also an entire active research subsystem the /goal prompt maintains and curates over time. I chuckled while writing this. Instead of solving an automated AI researcher for RSI, it’s a cheeky solution of an automated AI researcher that builds threeJS replicas of NYC.

The research is fascinating. You can see the AI at work, pulling these things together in interesting ways.

You probably are catching on now: the goal is still running.

So many.

It doesn’t go smoothly. /goal is like sending a thousand monkeys into a bookstore in order for them to build a car. Maybe you’ll get a car. Maybe you’ll get art. Maybe.

What if the software isn’t ever done and just runs forever?

Going on five days of /goal just running has me rethink the relationship to software. If you had 24x7 /goal loops running on software, in pursuit of a goal, you can just “drop in” and improve the plan or let the AI continue to work. What if that’s how software should be built? Not with breaking it into units of work, but just one, never-ending goal that gets feedback and context over time. Not so much a factory, but a garden? Seasons where a new idea takes root and flowers, and later on a winter when the entire situation needs a major adjustment?

Why didn’t I think to bring up these ideas until I built the project? Could I have thought of the various subsystems and techniques required to build the city… before I had goal build it all out?

This one vexes me so. Why didn’t height map, procedural mesh systems, vehicles, and all the rest occur to me at the very beginning? Why did it take seeing the scene for me to “bolt on” more ideas? What other “big ideas” am I just forgetting to have the scene construction consider-ideas that even the robust research catalog hasn’t resolved?

WHY does this work?

This is a massive replica of NYC with buildings, traffic lights, water, train system, barge, cars and more. It shouldn’t work. It does. What other ideas have I written off as “out of reach of computers” that should be dusted off, tossed to a never-ending /goal, and see just what happens next?

It’s something I’m thinking about. I’ll share more of city-explorer later on.

Thanks for reading! Subscribe for free to receive new posts and support my work.

The goal is still running.