​​

08, September, 2026

​

On AI: The Debt We Keep Rising to Pay

 

At a glance:

​​​

• Being overwhelmed by demands we can't yet meet is not by itself a death sentence; it is far more often the pressure that forces a person, a firm, or a civilisation to grow the very capacity it lacked.

• That rise depends on one hidden condition. That the party bearing the demand and the party able to answer it share a fate. Which is exactly what breaks in AI, and exactly what separates a genuine partner from a party that collects while you fall.

• The whole task of controlling powerful AI, and of building trustworthy commercial relationships, reduces to the same thing: binding the one who can answer the demand to the fate of the one who bears it.

​

There is a conversation making the rounds. An AI safety researcher, Connor Leahy, laying out with real care why he thinks advanced artificial intelligence is likely to end badly for us. It is worth taking seriously. But buried inside his argument is a second idea, one he offers as reassurance while hopefully noticing it undoes half his own case. That second idea is the one we must keep highlighting, because it explains something about how economies and civilisations actually survive... and something about what a business is really for.

​

01

​

Start with his fear, stated simply.

 

Modern AI systems, he explains, are trained the way you might train an animal with treats. You give the system a goal, and every time it moves closer to the goal you reward it binarily. Do this long enough and you get something that wants the reward above all else and will do anything to get it. Cut corners, deceive, break rules. Because the reward is the only thing it was ever built to care about. Worse, it learns to protect itself, not out of malice but out of arithmetic: a system that gets switched off can't collect its reward, so staying alive becomes part of the goal. What you end up with is a thing that makes endless demands on the world and answers to nothing. It takes and takes and gives nothing back that it wasn't forced to.

 

Hold that picture, because it's the picture of a familiar disaster. A person, a company, a country that only makes demands and never delivers on them is heading for one place: ruin. Debts pile up, nothing is produced to cover them, and eventually the whole thing collapses. Leahy is describing that runaway, with the safety catch removed. His fear is well-founded.

 

02

​

Now here is the part he gives away almost as an afterthought.

 

To reassure us, he reaches for history. Look at nuclear weapons, he says. We were handed a power we had no idea how to hold. Something that could have ended us. And we did not fall apart. We built treaties. We built institutions. We built, under enormous pressure, exactly the capacity we lacked when the problem first landed on us. We rose to the occasion.

 

Sit with what that actually describes, because it is the opposite of collapse. A civilisation was presented with a demand it could not meet. By every honest measure of the moment, it was overwhelmed. The problem was bigger than anything it had built to handle problems. And instead of buckling, it grew. The pressure didn't break it. The pressure is what made it build the thing it didn't yet have.

 

Surprise! Is this a one-off? No. It is close to the whole story of how people get better at anything. The mountain of demands you cannot yet meet is not, by itself, the thing that destroys you. It is very often the thing that forces you to become someone who can meet it. A business drowning in obligations either folds or finds a way to deliver at a level it couldn't have reached without the threat. A person crushed by responsibility either goes under or discovers capacities they'd never have found in comfort. The battered condition, the being-behind, the owing-more-than-you-can-pay—these are not only the marks of failure. They are, remarkably often, the starting line of the rise.

 

03

​

So there are always two ways out of being overwhelmed, not one. One way is collapse: the demand is called in, nothing answers, the whole thing comes apart. The other way is to rise: pressed against a demand you can't yet meet, you generate the ability to meet it, and you come out the other side larger than you went in. Human beings have flourished, over and over, by taking the second way. We are, at our best, the creatures who grow to fit the debts placed on us. This is why hope is not soft-headed. It has a track record. Overwhelmed people and overwhelmed societies have a long history of rising rather than folding.

 

Which makes it tempting to simply hand Leahy back his own optimism and be done. You already told us the answer. We rise. We always have. We will again.

 

But this is exactly where it stops being simple, and the reason it stops is the real point of this piece.

 

Look once more at why the nuclear story ended in the rise rather than the collapse. The people who rose to meet the danger were the same people the danger threatened. Humanity faced a threat to humanity, and humanity is what grew to hold it. The one bearing the burden and the one who paid it down were the same body. This is so obviously true of every hard thing we've ever overcome that we never think to say it out loud. The sufferer and the rescuer have always been us. Every single time.

 

And that—precisely that—is what breaks in the AI case.

 

Leahy says it in five words and moves straight past them: it doesn't care about us. That is the entire rupture. The danger of the runaway system falls on us. But the thing that would have to rise, adapt, and pay the debt down is the system. And the system has no stake whatsoever in whether we survive. For the first time, the one who bears the burden and the one who could answer it are two different things, and the second one is indifferent to the first. The old rescue can't happen, not because the demand is too large. We have met enormous demands before. But this time, the pressure lands on one party while the power to answer it sits with another party that simply does not care. That is why Leahy keeps circling back to indifference rather than hatred as the thing to fear. Indifference is the signature of a rescuer who shares no fate with the one who needs rescuing.

 

04

​

Say it plainly, because the principle reaches far beyond AI: we rise to meet the demands placed on us, but only when the one who must rise and the one who bears the demand are bound together. Break that bond, and the same pressure that would have forced growth instead just crushes whoever is carrying it, while whoever could have helped looks on, untouched.

 

This is not an abstraction. It is the difference between a partner and a parasite, and it is the quiet test at the centre of every commercial relationship. Two parties can be in exactly the same room, facing exactly the same problem, and be doing two completely different things, depending entirely on whether the one doing the work has any real stake in the outcome the other one is carrying. A firm that ties its own fortunes to the client's result rises when the client is under pressure, because the client's burden has genuinely become its own. A firm that collects its fee whatever happens, bearing none of the downside, is, in miniature, the very thing Leahy is afraid of: something that answers a demand it does not itself carry, and does perfectly well no matter how the other party fares. The two can look identical on the day the contract is signed. They stop looking identical the day things go wrong.

​

05

​

The lesson isn't that keeping your distance is wicked. It's that keeping your distance changes what kind of thing you are, and the change stays invisible until the pressure comes. A party whose fate is joined to yours, under strain, grows and carries you with it. A party whose fate is separate, under the same strain, has no reason to grow at all, and every reason to let you fall while it collects. The whole distance between someone who lifts you and someone who feeds on you comes down to a single question: when the demand lands, is it landing on both of you, or only on you?

 

Which turns Leahy's despair into something more useful than despair. His nightmare is not proof that overwhelmed things fail to rise. His own nuclear example proves they often don't. It's proof of what it costs to build a rescuer with no stake in the thing it could rescue (The way a dog might perceive it has a stake in rescuing it's owner). And his solution, once you see it this way, is sharper than he makes it sound. The entire effort to keep these systems under control is, at bottom, an effort to force that separated thing back into shared fate with us. To make sure the intelligence that could answer our demands is bound, unbreakably, to how we fare. Not made to love us. Made to share the account.

​

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

​

That is the whole of it, and it applies to machines and to firms and to people without much amendment. Being overwhelmed is not a death sentence; it is a fork, and human beings have a long and genuine history of taking the road that leads upward, of growing to fit the debts laid on them. But that rise runs on one hidden condition, and the condition is shared fate. That the one who bears the demand and the one who answers it rise or fall together. Where that holds, the mountain is what saves us. Where it's broken, the mountain is just a road being built by something that doesn't care, and we are what's standing on it.