I was on board with the first half ;-)

if the sentiment after using AI as much as you both have, how do you square that sentiment of “determinism is better than non” with the love of AI? do you think those struggles you are seeing will grow/continue?

I’ve had AI write lots of deterministic scripts yep, a great way to replay things and save meaningful output from the non-deterministic word vomit machine. It’s a tool, a powerful tool trained on the patterns of humanity, there’s plenty to like about that and still plenty to be critical about.

I think this tool, in this shape, isn’t perfect and has room to grow. I’m happy enough with it, but do hope it gets better.

thanks for the reply! i’m also curious about the “determinism is better” angle with the idea of giving these llms “skills” and the way you refer to them as “puppies that can be trained”. doesn’t thinking of them as trainable mean you want determinism from their output that you’ll never get? 1/

isn’t “whether or not a bot uses a skill” non-deterministic like everything else it does? your experiences you recount on the pod show that skills don’t produce determinism, but by writing them in the first place, aren’t we saying that’s what we want to achieve? 2/

if non-determinism was acceptable us, why would we write skills at all knowing that there will be times the skills aren’t used? is “hopefully most of the time” good enough to justify both writing out all the skills we hope they use and enough to justify using these tools in the first place? 3/

ex: you want the tool to never post in the General channel on discord, knowing that there’s no guarantee that it won’t. you have been frustrated by the fact that it ignores your skill telling it not to. would you use any other tool randomly doesn’t do what you told it to? what makes ai different?

Agree it sounds like maybe predictability is what I’m after, but it’s quality too. Skills are an attempt to provide patterns on the fly in the shape of the results you want. Puppies that are really good at mimicking, more examples give more deterministic results / things in the shape you want

Taste is too specific to be hopefully most of the time, gets tedious and you can nip the tediousness with hints

Ai is malleable in ways, especially when there are layers of malleable memory that is front loaded with each request. Front loaded info that you put there, not someone else. Things that’ll make all your future interactions easier because you have a layered set of shapings

Since it’s a computer though, there’s an amount of expectation, you can easily think the malleable memory is something other than a skill.. which is to say it’s nudging to the autocomplete engine that can’t actually follow every rule. The line though is interesting..

how much following and how much autocomplete do you want out of your LLM

that’s kinda what i’m getting at. i have a suspicion that us devs will hit a wall with llms we won’t be able to get over. we live in a deterministic world. seems like a disconnect to use a non-deterministic tool to build determinism. and we expect determinism in most tools we call useful 1/

if we come to an equilibrium of “use ai to build deterministic scripts”. that will mean we’re using less AI in the future right? cuz the more scripts we have, the less we need new ones? and maybe the less we need AI just to run those scripts? do we need to spend $ on tokens for ‘npm run…’? 2/

i’m kinda struggling with the whole idea that a tool can be good even though sometimes it doesn’t what we say it’s job is supposed to be. that’s not true or any other tool imo. most tools we use have defined purposes. we have to assign AI its purpose while knowing it can’t always do its purpose 3/

is that like calling something a hammer knowing it sometimes can’t drive nails? and does that mean we have to constantly search for new purpose? or does it mean that AI isn’t really right for any purpose since it’ll never be deterministically good at anything we assign it to do?

Indeed, but much of what we deliver isn’t inventive, they’re applied patterns. Knowing which pattern goes where is our job, it’s good at applying patterns and maybe good at knowing which pattern. Patterns have always been part of our job, but not all of it.