Marcus Povey

Recently, I gave some training to my team on the use of AI within our development team. During the training, I had one slide dedicated to the security issues introduced by AI and the increasing use of LLMs and other tools. On this slide, I said that AI security was terrible…

.. and when I said in the training session that AI security is terrible, I meant absolutely horrific.

Given this raised more than a few eyebrows of surprise in the room, I thought I’d dig into this a little further. As we rush with excitement towards this glorious future where we are all watched over by machines of loving grace, I thought I’d go back and expand on that slide, and pump the brakes a little. We all know that company that’s just wired their CRM, email, customer database and internal wiki into an AI agent because a competitor did it first…and I imagine several of you dear readers work for a company that has done just that, I can practically hear the nervous shuffling.

The attraction is real

Speaking personally, the use of AI has been absolutely transformative. Development tasks that were previously high effort, whether because of complexity or just the amount of effort involved, have become almost effortless. Even simpler developments have shifted from being measured in days and weeks worth of effort to hours, and in some cases, minutes.

This has caused me to have more than a few “oh shit” moments, but also a lot of excitement as suddenly previously impractical projects (think that idea you’ve been kicking around but can’t justify the time to build) suddenly become viable. MVP side projects become something that can be built in a quiet morning while drinking coffee rather than something that needs to compete with other commitments for a spare weekend.

I’ve started looking at things quite differently, and thinking to myself: “What is now possible that wasn’t before?”

The security landscape hasn’t caught up

It probably shouldn’t come as a surprise to anyone, given how blindingly fast things are moving right now, but the security industry is sprinting to try and keep up. AI in all its forms has opened up a plethora of new and exciting exploit surfaces that nobody had previously had to consider, and a lot of seemingly robust systems are being stress tested in new and interesting ways.

Some things to consider:

• OWASP now has an entire Top 10 just for LLMs which didn’t exist two years ago, because simply the attack surface didn’t exist two years ago. The fact that prompt injection is number one, with no known deterministic fix, should bother you.
• MITRE ATLAS is growing at a spectacular rate, jumping from 66 techniques to 84 in a few months last year, indicating new attacks being discovered in the wild at an alarming rate.
• The NCSC’s most recent AI guidance stresses that finding vulnerabilities doesn’t automatically improve security if teams lack a process to manage, prioritise and fix what the tools surface.

The point being, every major IT security body in the world right now has had to build entirely new frameworks from scratch in the last couple of years, as brand new vistas of attack open up to be exploited. To me, that is not a sign that the technology, while undoubtedly useful, is ready to be wired into everything in your organisation.

The risks of wiring AI into everything

Ok, so let’s look at a handful of specifics, bearing in mind that this is far from an exhaustive list. New and exciting vulnerabilities are being discovered all the time, but here are just a handful that you might encounter:

Excessive Agency: When you give an agent access to your tools, you’re creating a new attack surface. Effectively, an agent is an autonomous bit of software that takes its instructions from a prompt, and if ordered to do so could quite easily do something malicious. A simple chatbot could potentially be instructed to exfiltrate data, send emails or delete records. MITRE ATLAS now has quite extensive case studies on this, so this is far from theoretical.

System Prompt Leakage: Most organisations and developers think the system prompt is a security boundary. It isn’t, and OWASP is quite clear on this. API keys, internal sensitive information, anything… if it ends up in the prompt, it has been exposed.

Supply chain opacity: Do you know the provenance of that code the AI just wrote? Or what that library it downloaded does? Or, that RAG data source, is that something that can be trusted? Each of these could be its own topic of discussion, but the general theme is one of provenance. Can it be trusted, and can it be audited? Because, it turns out that supply chain poisoning is alarmingly common, and with one poisoned component your AI is suddenly not doing what you think it’s doing.

Pump the brakes

I am the last person to be saying that you shouldn’t use these tools, they’ve been transformative in terms of what I, as well as my team, have been able to do and create.

Threat model before integration… which is a fancy way of thinking about what an attacker could do if they controlled the AI’s input. Think of an agent as autonomous software running on your system with whatever permissions you’ve given it, so anything controlled by the AI could potentially be used by an attacker. So don’t wire it up to the nuclear launch system, that was a great movie, but let’s not actually do this.

Minimum privileges only: following on from the previous point, if the AI doesn’t need the privileges to nuke your production database, don’t give it that access. Boring, basic, security hygiene is good to practice, even in this new context where you’re dealing with non human actors.

Keep the human in the loop. Draw a clear line between the AI that advises a human, and an AI that acts. I use AI all the time, but the times where I let it run autonomously are exceptionally limited, and only where an error would not cause significant problems. Again, don’t give the agent root access to your production database. For developers, you should also check and review the output before hitting “commit”, AI can produce truthy looking code but there are attack vectors (compromised supply chains, “poisoned” models, etc) which can introduce critical vulnerabilities if not properly audited.

Your chat/system prompt is a public document. Meaning, if your API keys or passwords somehow enter the chat prompt (whether by you cut ‘n’ pasting them in or because the AI has analysed the file in your repo containing them), these are now public. Congrats. I hope you didn’t have production details commented out in your development .env file.

We have been here before (sort of)

I have been building systems for a long time now, and I’ve seen this pattern before. Something new and shiny comes out, there’s a rush to adopt it, and security debt accumulates leading to a fairly painful reckoning further down the line. AI isn’t uniquely dangerous in this regard, but it is moving at a pace much faster than anyone can hope to keep up with.

So what should we do?

So, rather than wiring up a full agentic workflow in your core business because you heard your competitor has done so, or adding AI magic sprinkles in every new product offering, take a beat and think.

There is a difference between reacting and responding, and while you may still choose to proceed, at least do so having considered the potential risks, as well as the exciting opportunities. Proceed deliberately, and you will have better AI and fewer incidents.

Oh, and if this is something you’re trying to figure out where to draw the line, this is exactly what I help organisations think through!