September 26, 2023
With National Coding Week behind us, the development community has had its annual moment of collective reflection and focus on emerging technologies that are shaping the industry. Among these, large language models (LLMs) and “generative AI” have become a cornerstone for applications ranging from automated customer service to complex data analysis.
Recent research shows that generative AI is a critical priority for 89% of tech companies in the US and UK. However, the genuine buzz surrounding these advancements masks a looming threat: prompt injection vulnerabilities.
While LLMs promise a future streamlined by artificial intelligence, their current developmental status—in what can best be described as “beta” mode—creates a fertile ground for security exploits, particularly prompt injection attacks. This overlooked vulnerability is no trivial matter, and it raises the critical question: Are we doing enough to insulate our code and applications from the risks of prompt injection?
The critical challenges of generative AI
While the benefits of LLMs in data interpretation, natural language understanding, and predictive analytics are clear, a more pressing dialogue needs to center around their inherent security risks.
We have recently developed a simulated exercise, challenging users to convince an LLM chatbot to reveal a password. More than 20,000 participated, and the majority succeeded in beating the bot. This challenge underscores the point that Al can be exploited to expose sensitive data, iterating the significant risks of prompt injection.
Moreover, these vulnerabilities don’t exist in a vacuum. According to a recent industry survey, a staggering 59% of IT professionals voice concerns over the potential for AI tools trained on general-purpose LLMs to carry forward the security flaws of the datasets and codes used to develop them. The ramifications are clear: organizations are rushing to develop and adopt these technologies, thus risking the propagation of existing vulnerabilities into new systems.
Why prompt injection should be on developers’ radar
Prompt injection is an insidious technique where attackers introduce malicious commands into the free text input that controls an LLM. By doing so, they can force the model into performing unintended and malicious actions. These actions can range from leaking sensitive data to executing unauthorized activities, thus converting a tool designed for productivity into a conduit for cybercrime.
The vulnerability to prompt injection can be traced back to the foundational framework behind large language models. The architecture of LLMs typically involves transformer-based neural networks or similar structures that rely on massive data sets for training. These models are designed to process and respond to free text input, a feature that is both the greatest asset and the Achille’s heel of these tools.
In a standard setup, the “free text input” model ingests a text-based prompt and produces an output based on its training and the perceived intent of the prompt. This is where the vulnerability persists. Attackers can craft carefully designed prompts—either through direct or indirect methods—to manipulate the model’s behavior.
In direct prompt injection, the malicious input is straightforward and aims to lead the model into generating a specific, often harmful, output. Indirect prompt injection, on the other hand, employs subtler techniques, such as context manipulation, to trick the model into executing unintended actions over a period of interactions.
The exploitability extends beyond simply tweaking the model’s output. An attacker could manipulate the LLM to execute arbitrary code, leak sensitive data, or even create feedback loops that progressively train the model to become more accommodating to malicious inputs.
The threat of prompt injection has already manifested itself in practical scenarios. For instance, security researchers have been actively probing generative AI systems, including well-known chatbots, using a combination of jailbreaks and prompt injection methods.
While jailbreaking focuses on crafting prompts that force the AI to produce content it should ethically or legally avoid, prompt injection techniques are designed to covertly insert harmful data or commands. These real-world experiments highlight the immediate need to address the issue before it becomes a common vector for cyberattacks.
Given the expanding role of LLMs in modern operations, the risk posed by prompt injection attacks is not a theoretical concern – it is a real and present danger. As businesses continue to develop and integrate these advanced models, fortifying them against this type of vulnerability should be a priority for every stakeholder involved, from developers to C-suite executives.
