– by zayne

Freedom Invites Chaos

It’s a common trope in science fiction that the introduction of a new technology, especially one that is powerful and disruptive, will lead to chaos and destruction. This is perhaps an overly pessimistic view, but one that is not without merit. Time and time again, we see that security is seldom the first priority of new technology, and in the wild west of the Internet, hackers will always rise up to the intellectual challenge of breaking into the latest and greatest.

When Robert Morris created the infamous Morris worm in 1988, he simply wanted to see if it could be done. This is perhaps not too different from when an innocent software engineer who hasn’t done any serious hacking before decided to poke into production apps hosted by Lovable.

Code generation tools like Lovable have empowered a new generation of non-technical users to create software, and they have done so by providing AI with absolute freedom to fuel the creativity of their human collaborators. But just like how the wild west of the Internet had tons of competing browsers — none of them secure — before browser standards became a thing, we are headed into a new wild west of AI-generated code where the rate of code generation far outstrips the human ability to review and secure it.

The freedom to create has always invited chaos — this is a fundamental truth of software development, and much of the recent work in the security space has been about trying to rein in that chaos by making it intuitive to only create a strict subset of all possible software and hoping that this subset is secure. The idea of “secure by default” frameworks is a good example of this. The fundamental difference this time is that where previously the number of insecure systems scaled linearly with human capital, the number of insecure systems will soon scale exponentially with the number of GPU cores.

Scale is the Enemy

Great, the amount of code in the world is going to explode. Surely the same AI that generated the code can also be used to secure it, right?

Let’s take a step back and look at what a vulnerability is. A vulnerability can only exist if there is a difference between the intended behavior of a system and the actual behavior of that system. So to prove a vulnerability, you are showing that

Given a system S, there exists an input I such that S(I) is not equal to T(I), where T is the intended behavior of S.

and presumably that mismatch gives the attacker some kind of advantage over the system.

So surely it is way easier to prove that a system is insecure than to prove that it is secure. This doesn’t even take into account the fact that you can’t expect an AI agent to have the full context of the system in its working memory, for a codebase of any reasonable size. Any gaps in context when forming abstractions (e.g. through summarization) will lead to a gap in understanding, and thus a gap in reasoning about security. It’s the same challenge that humans face when writing code, and why vulnerabilities exist in the first place.

To find vulnerabilities, on the other hand, is much easier. Perhaps performing novel security research (the kind that takes years, like James Kettle’s work on HTTP request smuggling) is still a bit too much to ask of AI agents just yet, but in reality, most vulnerabilities are repetitions of the same mistakes. This variant analysis is something that AI agents can be good at.

The models will only get more intelligent. At Hacktron, we’ve been consistently impressed by the ability of AI agents to find and exploit vulnerabilities in complex codebases that would have taken us days to fully understand. This progress is not going to stop anytime soon.

I believe that AI will expand the capabilities of threat actors exponentially, at a time when the number of true vulnerabilities in the world is also exploding. Where previously nation-state espionage scaled only with human capital, it will, in a few years, be a far more worthwhile endeavour to simply run a cluster of AI hackers targeting systems that don’t have an army of SOC operators behind them.

Which systems are these? The natural answer is tech startups and mid-sized companies. And these are precisely the companies that aren’t going to be prepared for the day that this happens. North Korea has so far only been able to afford targeting a handful of carefully selected targets, but this is likely going to change when it becomes much more cost-effective to cast a wider net.

The Opportunity

There are two potential futures here:

1. No one except nation-states build AI hackers, and X years from now, country Y unleashes their AI hackers on the world. This future sucks.
2. Someone builds an AI hacker as a service, the world integrates it into their security stacks, realises how scary good it is, wakes up, catches up to it, and we are all prepared for the day that country Y unleashes their AI hackers on the world. This future is much better.

The second future is the one that I am betting on. That’s why I believe in Hacktron. Every software company in the world is going to need to be protected round the clock, and the best way to do that is to have a team of AI hackers working for you.

The next few years are going to be very interesting, and I believe we will slowly start to see this generational threat unravel. Let’s hope that we are all prepared for it.