Local AI: When TCO is cash positive?

Or as the Jetson might say…

Might it one day pay in FCF, too?

I’m seeing the total savings numbers of my local AI reach to levels where it’s hard to wrap my head around. I’m expecting a rapid payback sooner than predicted. My current estimate has the payback period towards the back half of this year or perhaps sometime Q1 if I just use it for vision tasks alone.

ROI ahead of schedule: success.

My mind is bouncing around with new “all-day-ai” ideas, those 24x7 ideas where it’ll get to payoff even faster. In this article, I want to brainstorm with you about the implications of this hardware existing. I built a… “thing” that literally saves me money-and helps with quality of harder tasks.

I shared one of my hardware projects is a customized Jetson Nano Orion. I’ve consistently been handed surprises with what I can do with this little thing. If you tried my Week3 submission called “nasapd”, then these textures might look familiar. I needed some unusual looking stone, worn by time. Not complex shaders, just good old fashionied textures that I’d later on assemble into a texture atlas.

A texture atlas is such a great unlock for threeJS titles. You have one larger image, but it contains all the smaller textures you need. Then clever UV manipulation allows you to scale out from that one image to all sorts of textures on an object. I’ve tested the performance of this and it’s substantial enough that “texture atlasing” is a regular capability in my toolbox.

The Jetson has it’s own API and site these days. You can build one yourself-I’ve written on how to do it. There’s a few more capabilities now in this that I’ve not exhaustively tried-for example? The hardware supports sound effects, music and now speech generation. All on this little workhorse at 7 watts.

It’s great having local AI on a dedicated device. It means you can just do concept art generation all-day-long, and have a gallery of ideas to review, or different material concepts.

What begins to emerge is that there’s definitely opportunities for ambient art creation. I mentioned in other posts about my kitchen virtual world experiment. It’s a little 4.3” LCD that plays an infinite series of episodes telling the stories of it’s characters. These art capabilities of the jetson have me think it may be time to get a dedicated display built for this hardware.

Sure, it’s slow. Who cares?

AI generation on this device is slower and the quality is task dependent. Does anyone really care if it takes longer to complete a verifiable task? For example, a text completion response for 256 tokens might take 161.9 seconds if it’s using the cpu pipeline over gpu.

I just find a lot of tasks do not need to be done fast. With any luck, the tech industry will emerge from the nightmare that the all too familiar software hustle for faster, faster, faster was never the right ball to chase. Faster makes sense in some domains, but not all domains. Sure, there are times one needs to move fast. However, in my study of the market it’s become clear the market has a calculatable absorption rate of anything new, or even in terms of speed up existing.

Challenges or the “What’s Next”

• Cooling even a small device isn’t without it’s challenges. At 7W, the overcurrent issues I was facing when running the clock at 15W are fixed and the device does seem to run cooler overall. The downside of the current mount for the device is that it’s resting on a non-conductive surface in my office. Should I build a dedicated display, I’ll have to contemplate both a cooling and mount solve.
• Steering the content/media for consistency in theme is something I’m still in pursuit of. It’d be nice to pick a generated art, such as the miniature above and get several more generations of characters that “are in this style”.
• 24x7 research task test run. I’ll be standing up a longer research task run with this device. I’m not expecting it to work in any meaningful way, but it’ll be a good runway of “ok, so I can also do something like this or that”

Someone asked me over the weekend if I’d ever consider selling a preconfigured Jetson like what I have setup. It’s an interesting and complicated thought-maybe?

It does seem to be pretty unusual to have a box that not only pays for itself, but conceptually could be net-positive in free cash flow over the lifetime of the hardware. Except what I’d say is that this is generally true for any local AI. What’s great about a local hardware setup is all the measured insight stays local, the capabilities reduce TCO of AI in general, and there’s a path for ROI where the hardware generates cashflow? But really: having the AI closer, on the edge, to your project is really nice. It’s the difference between home-made bread and store bought bread.

For sure, more experiments ahead.