Gandalf reveals the biggest GenAI security threats to enterprises

What can companies learn from Gandalf? We asked this question to David Haber, CEO and co-founder of an AI security company called Lakera, which recently secured $20 million in Series A funding to deliver “real-time GenAI security.”

We were not talking about the famous wizard from Lord of the Rings, but about an educational game called Gandalf in which players attempt to trick a large language model into revealing a secret password using prompt injection techniques (attacks that use malicious input to manipulate an AI's behavior or output).

Gandalf has now been played by more than a million people, with 200,000 of those players successfully completing seven levels of the game by manipulating models and causing them to perform unintended actions – giving Lakera deep insights into GenAI hacking.

“You have to imagine Gandalf as the biggest red team in the world or a live stream of the world’s creativity when it comes to how to manipulate genetic AI systems,” says Haber The stack. “When Anthropic publishes a new paper describing a new vulnerability or type of attack targeting GenAI models, people try it on Gandalf within minutes of publication.”

“It's a great example of how accessible hacking has become and how easy it is to manipulate artificial intelligence systems,” he adds.

Gandalf was originally going to be called Yoda until someone sent around the company a meme showing the wizard from The Lord of the Rings and his famous words: “You can't get through here.” The game was so popular that it became one of Lakera's main marketing channels.

“A large part of our business is a direct result of people playing Gandalf, recognizing their security vulnerabilities and turning to Lakera to protect themselves from this new breed of cyber risk.”

Act quickly to combat a growing threat

Haber describes GenAI as “the biggest technological transformation we as humans are likely to experience in our lifetime,” comparing the rapid pace of adoption to the relatively slow adoption of the cloud. As a reminder, ChatGPT was launched just 20 months ago. About a year after its release, more than 90% of Fortune 500 companies were using GenAI.

“We are just beginning to understand the threat landscape around AI – but we know that existing security solutions and tools do not take these new risks into account,” adds the Lakera CEO. “So we are in a race to catch up, with the technology being adopted quickly, and we need to protect companies from being exposed to major risks.”

GenAI can make corporate data accessible to large numbers of people – and then allow malicious actors to maliciously access that information by typing in a cleverly worded comment.

Lakera customers also reported that their chatbots had leaked internal data, harassed customers, given incorrect advice, behaved erratically, or shared data with the wrong people inside and outside the organization.

“Today, anyone who can speak can also hack,” says Haber.

He highlights two “big concerns” for companies. The first is instant attacks enabled by a “whole universe of phrases that can be used to trick the model into performing unintended actions.”

“It has been shown that these artificial intelligence models are essentially Turing-complete machines, meaning attackers can make them do absolutely anything if they find the right words and terminology,” he warns.

The second threat is data loss, where GenAI models lose confidential customer or company data.

He adds: “Because we link these models to all sensitive company data and people enter credit card information or names and addresses, companies have to make sure that this does not get out to the public or even the model provider. This is a big problem.”

Lessons for companies

The most important conclusion from Gandalf is that security leaders should take the GenAI threat seriously and prioritize containing it through the use of appropriate defense products.

“Security solutions must be alive and breathing – just like the underlying technology of GenAI,” advises Haber. “They must continuously evolve.”

Rule-based solutions “won't be enough,” he adds, because they're “too simple for the complexity of the data” that models process. Organizations should also avoid using one GenAI model to secure another GenAI model, as both could fall victim to the same exploits or attacks.

Instead, Haber advises organizations to create custom models that are architecturally different from the models they monitor and ensure they are continuously updated with the latest data and threat intelligence.

“GenAI presents entirely new risks that traditional security measures do not cover,” concludes Haber. “This means that organizations have blind spots and we are in a race to protect companies from these major new risks.”

The threat posed by GenAI is undoubtedly great. But organizations must face it like Frodo or face the consequences. Because, in the words of Gandalf himself: “It is the small things, the everyday actions of ordinary people, that keep the darkness at bay.”