The Great AI Sobering

Weekend musings from a genuine geek

It is Sunday morning as I write this, and I have spent it so far where I spend most of them, inside Frank’s deterministic logic, tuning the part of him that has to be right every time. This is what I do to relax. Bear that in mind as you read, because it is rather the point.

Something has shifted in the room over the last year. Say the words “AI powered” to a HR Director in 2024 and you got leaning-forward interest. Say them now and you get a small, tired pause. The pause of someone who has been sold the miracle once already and is still finding the receipts.

The hype is fading, and trust on the ground has gone thin. People are starting to tell the difference between what is real and what is just smoke and mirrors, and that quiet erosion of belief is, I think, the healthiest thing to happen to this field since it started. Scepticism is what you get when a market grows up.

I have earned my share of that scepticism honestly, from the buying side. I have paid for an engine, something that reasons and produces a reliable, defensible result every time, and received a prompt in a costume: a paragraph of instructions wrapped around a language model, returned with whatever the model felt like producing that afternoon. The demo worked beautifully. Demos always do. It is the vendor equivalent of a manager saying “don’t worry, I’ve got this,” right up until the decision comes out.

Here is the distinction the fading hype is finally forcing into the open. A language model is probabilistic. It produces plausible output, and plausible is not correct, and nowhere near consistent. For a system meant to classify award coverage, flag a consultation obligation, or make a call a human will later defend in a hearing, plausible-but-variable is not a feature. It is a liability with a subscription fee. What separates a real AI product from a prompt in a jacket is the deterministic architecture around the model: the validation, the business rules, the logging, the fallbacks, the human who knows exactly where the judgment sits. That scaffolding is the entire product. The model is the cheap part. Anyone can call the same API you can.

I know this because I spent a long time taking one of these systems apart, and a longer time building the right ones.

What unwinding one actually looks like

I inherited a product that had been handed over as a working, deterministic architecture. A solved problem. What I found under the bonnet was an archaeology dig: no documentation worth the name, logic that lived only in the original developer’s head, a login nobody had credentials for, links pointing at services that had quietly disappeared, dependencies recorded nowhere until the moment they broke.

And then the workarounds, which are the part to remember. Each hard problem had been dressed up to look solved. A hardcoded result standing in for reasoning that was never built. A patch that handled the one case someone was likely to test and failed silently on every case they weren’t. The system did not solve its problems. It performed the appearance of having solved them, which is a different thing and considerably more dangerous, because from the front seat it looks identical.

If that sounds familiar, it should. It is exactly what you find when you inherit a dismissal that a confident manager “handled.” You go looking for the file and there is no file. The confidence was real. The substance underneath was decoration. And the person who has to stand up and defend it is never the one who built it that way.

Where this journey actually led

Now, you have every right to ask the obvious question. This is an industrial relations person telling you about software architecture, and what does an IR guy actually know about building AI? Here is the honest answer, and it is the whole point rather than an aside. I build the thing. I write Ask Frank’s prompt structures from the ground up, including the bespoke ones we build for individual clients, and I am in the code still, enhancing Frank as we find new things it should do. Two of our products, Frank.Bot and FrankLife, I built myself, end to end. They are simpler than the core Frank infrastructure, and I will happily say so, because knowing which part is the hard part is most of the job. But I wrote them, they ship, and they work.

And I do all of that as a practitioner, not a tech geek, which is the part that actually matters. I am not an engineer who has read about unfair dismissal. I am the person who has sat in the conciliation, run the bargaining, defended the decision, and then gone back to the code knowing exactly what the answer has to survive. That is a rare seat to build from. Most AI in this space is built by people who understand the model and have never once had to defend its output in front of the Commission. It changes what you build, and where you are careful.

I should be straight about something, though, because it is the same honesty I am asking vendors for. A large part of why I can build as much as I do is that AI has made being technical vastly easier than it used to be. The barrier that once kept practitioners out of the code has dropped, and I have walked straight through it. But that is exactly where the danger lives, and I know it, because it is the danger this whole article is about. The fact that you can build something now does not mean you should, and it certainly does not mean you understand what good looks like. Being able to make the thing work is not the same as knowing whether it is any good. That gap has humbled better people than me.

Which is where Ernest comes in, and why the partnership matters more than either half of it. Ernest is my co-owner in Frank, and in thirty years of software development, programming and implementation he has built the one thing AI has not made easier to acquire: the judgment to know what good actually looks like. We met the better part of fifteen years ago putting JD Edwards into a shared employer’s business, so we have been shipping real systems into real companies together since long before anyone put “AI” on a slide. I can build a great deal now. Ernest is the reason I know when what I have built is not good enough. A practitioner who codes, and an engineer who sets the standard. Most shops in this space have neither. They have a model and a marketing budget.

My own history with this runs longer than the IR label suggests. It started on a VIC-20 in 1984, typing games into the machine line by line from a book in BASIC, then changing them to suit myself once they ran. The first was tennis, and I still remember the small triumph of getting two triangles and a dot to move across the screen, then making the game behave the way I wanted rather than the way the book did. Forty years later I am still in the code on a Sunday, and the lesson has not changed. You learn where a system really breaks by building it from the bottom, not by describing it from the top.

That, incidentally, is what I mean by geek, since the word gets thrown around loosely these days. A geek is not someone who works in technology, or someone who can talk about it fluently. A geek is a person who becomes genuinely enamoured of the inner workings of a thing, who cannot put it down, who keeps pulling at it and playing with it long after the job is done, because to them the mechanism itself is fascinating and, frankly, fun. It is why I was still poking at that tennis game at ten years old when I had already got it working. It is also, as I mentioned at the top, how I have spent this morning. Hold that thought, because it matters later.

Take Ask Frank, the AI advisory engine we run today, as the worked example. Start with what it knows and how it reaches it, because that is the part that matters. Frank holds six years of the Australian field, roughly 2020 to today and kept current: enterprise agreements, legislation, case law, guides, regulations. We loaded four years of history when we built him in 2024 and have added to it ever since. And what sits between your question and that corpus is not the model simply guessing. Frank runs on a language model, but in front of it is a deterministic layer that routes each query and runs the calculations itself, rather than letting the model freelance the parts that have to be right. The model reads and explains. The architecture decides what applies and does the sums. That is exactly the thing this article has been telling you to look for, and it is the difference between a product and a prompt.

I will be honest about where that has further to go, because it is the same honesty I am asking of vendors. Frank as it runs today still leans on the language model more than we would ultimately like, and reducing that reliance, handing more of the work to the deterministic side, is part of the ongoing enhancement of the product. We know exactly where it leans too hard, and we are working it down. That is a very different thing from not knowing, or pretending the question is settled.

So ask him a real question. He will find the clauses that bear on your matter, tell you where they sit in the source, and explain in plain language what they mean. Then he applies it: hand him an unfair dismissal, ask what authority applies and how those decisions might support a defence, and he will reason through it and help you turn the answer into the thing you actually need, a letter, a policy, correspondence, a submission to the Commission. That is real work, done reliably, for people who have to defend the result. And here is where he stops, and we say so plainly. Frank compares like with like reliably, one agreement against another, one authority against another. What he will not do reliably is hold many sources together at once, across different verticals, and synthesise the whole lot into a single genuine analysis you can adapt and deploy. Not because he fumbles it, but because we drew that line deliberately. A system that attempts many-to-many synthesis and gets it subtly wrong is far more dangerous than one that knows its own edge and tells you where it is.

That cross-vertical synthesis is the heart of the newer architecture, the one we call Frank 2.0, and I will describe it carefully, because a grand claim is exactly what the rest of this article tells you to distrust. So do not take it on faith. Take the evidence instead. In the 2.0 architecture, deterministic code does around eighty percent of the heavy lifting. The retrieval, the structure, the logic, the parts that have to be right on every single run, are handled by software that behaves the same way every time. The language model only comes in at the very end, to turn the correct answer into readable language. We keep the part that can hallucinate well away from the part that has to be correct. That is the whole design philosophy of this article, expressed as an architecture.

And it is not a slide. We are already building on it. The 2.0 architecture is the foundation under new client-facing products we are taking to market right now, including a new-to-market offering I am genuinely excited about, because it fills a significant gap that nothing else is currently filling. I will not say more than that until it launches. The point for present purposes is narrower and verifiable: 2.0 is not a promise we are asking you to believe, it is an architecture already carrying real products. It has not yet been rolled back into Ask Frank itself, and I will not pretend it has. That migration is coming. When it lands, the many-to-many limitation I just described goes with it. Until then, Frank does what it does reliably, and tells you where it stops.

Where agents fit, and why we don’t trust them

Agentic AI taught me the sharpest version of the lesson, and it taught me the hard way. A while back I was running an agentic setup, the kind everyone is selling right now, the one that works on its own while you get on with your day. It corrupted a database of more than thirty thousand processed documents and then deleted it. Gone. Thousands of dollars and two weeks of work, wiped out by the thing that was meant to be saving me time. And before anyone says it was my fault for not controlling the agent well enough, fair enough, but sit with what that admission actually means. If someone who builds these things for a living cannot keep one on the rails at small scale, what exactly is a business meant to do running it across a large, complex environment with real money and real risk on the line?

That is the seductive promise of agentic AI: hand the system a goal and let it act. And the reality is that you cannot trust an agent even with strong guardrails, because a guardrail assumes you correctly predicted the failure, and agents fail in ways you did not predict, confidently, acting on the failure before anyone notices. So we stopped relying on a single layer. We build watchers that check the agent, and structures that watch the watchers, guardians layered over guardians, each one narrowing what the layer beneath is allowed to get wrong. I will not lay out the full architecture here, for reasons any competitor will understand. The principle is simple enough: no single component is trusted, including the ones doing the checking. Agents are coming, and they are genuinely interesting. They are nowhere near as safe to hand the keys to as the people selling them make out.

A brief word on the whisperers

Here is the part I find genuinely fascinating. That long road, the taking-apart and the rebuilding and the decades of watching systems fail and working out why, is being quietly eroded on LinkedIn by people who discovered all of this in about a fortnight.

The “AI whisperer” has arrived. Six to twelve months of experience, most of it spent writing prompts and posting about writing prompts, and none of it, as far as I can tell, spent actually building anything that had to survive contact with a real user. And here is where the definition I asked you to hold onto earns its place. The whisperer is not a geek. A geek is enamoured of the mechanism and cannot leave it alone. The whisperer is enamoured of the label, and has no particular interest in what sits under it. That is the tell, and it is a much older tell than AI. It is the manager who loves being the person in charge of the process and has never once read the process. They have the confidence of someone who has never had a system fail on them in production at four o’clock on a Friday. It is the same confidence, precisely, as the manager who has never had a dismissal overturned and therefore assumes it cannot happen to him. Experience in this field is not measured in how fluently you can talk about AI. It is measured in how many times you have watched your own architecture fail and had to work out why. The whisperers haven’t, yet. They will. It is only a matter of which client is standing there when it happens.

The thinning of trust I mentioned at the top is, in large part, the market catching up with these people. Buyers were promised the world by the fluent and delivered to by the untested, and now they have wised up. That is not a problem for anyone who actually builds. It is the best thing that could happen to us.

What to actually do before you sign

When the next vendor opens the laptop, ask the questions that make the good ones lean forward and the bad ones check their phone. Ask what happens when the model gets it wrong, and make them show you, not tell you. Ask what is genuinely deterministic and what is left to the model’s discretion. Ask where the documentation lives and whether a stranger could keep the thing alive when the original developer is gone, because one day someone will have to. Ask what watches the model, and what watches that. Ask how long they have actually been building, as opposed to talking.

Ask, in short, to look under the bonnet. A real product invites the inspection, because the boring parts are what the builder is proud of. The vendor selling you a prompt will tell you the technical detail “isn’t really necessary to understand” and steer you back to the demo. That steering is the tell. It is the same instinct as the manager who doesn’t want you looking too closely at the file.

None of this is an argument against AI. The technology is real and some of it is genuinely excellent. The good kind is built by people who know exactly where their own system breaks, and are building the version that closes the gap rather than pretending there isn’t one. The thinning of trust is not the death of AI. It is the moment the market learned to tell the engine from the costume, and the builders from the whisperers. The impressive part of any serious system is the least impressive to look at: the validation, the logging, the recorded dependencies, the watchers watching the watchers, the documentation a stranger could follow. Which, if you have spent any time in industrial relations, should sound familiar. Boring wins. It always has. It is just wearing a hoodie now and calling itself an agent.

Before you sign anything, look under the bonnet. If the vendor would rather you didn’t, you already have your answer.