Indirect prompt injection (IPI) is an evolving threat vector targeting users of complex AI applications with multiple data sources, such as Workspace with Gemini. This t…
Posted by Adam Gavish, Google GenAI Security TeamIndirect prompt injection (IPI) is an evolving threat vector targeting users of complex AI applications with multiple data sources, such as Workspace with Gemini. This technique enables the attacker to influence the behavior of an LLM by injecting malicious instructions into the data or tools used by the LLM as it completes the user’s query. This may even be possible without any input directly from the user.IPI is not the kind of technical proble…
OpenClaw AI agents pose identity and data risks if deployed with broad cloud permissions. Learn how to find and secure these apps before an attacker does.
We are opening our advanced Client-Side Security tools to all users, featuring a new cascading AI detection system. By combining graph neural networks and LLMs, we've reduced false positives by up to 200x while catching sophisticated zero-day exploits.
This year, AI agents took the center stage – as a defensive capability, but more pressingly as a risk many organizations haven't caught up with
Thank you to everyone who participated in the Cybersecurity Framework Profile for Artificial Intelligence (Cyber AI Profile) Workshop in January! The input we received on the Preliminary Draft during this workshop has been invaluable and is informing the development of the next draft of the NIST Cyber AI Profile. We are working toward publishing a full workshop summary soon that captures themes and highlights from the event. In the interim, we would like to share a preview of what we heard… Bac…
Prepared by: Sudhanshu Chauhan 🔗 (Director, RedHunt Labs) and Devang Solanki 🔗 (Security Researcher, RedHunt Labs) There is a loop in cybersecurity. A new technology enters the enterprise. Adoption moves faster than governance. Security teams hear about it during incident response instead of during planning. And the exposure window, that gap between deployment and visibility, gets exploited before anyone realizes it even existed. We saw it with open cloud storage buckets. We saw it with expos…
Cloudflare AI Security for Apps is now generally available, providing a security layer to discover and protect AI-powered applications, regardless of the model or hosting provider. We are also making AI discovery free for all plans, to help teams find and secure shadow AI deployments.
Cloudflare AI Security for Apps is now generally available, providing a security layer to discover and protect AI-powered applications, regardless of the model or hosting provider. We are also making AI discovery free for all plans, to help teams find and secure shadow AI deployments.
Posted by Nathan Parker, Chrome security team Chrome has been advancing the web’s security for well over 15 years, and we’re committed to meeting new challenges and opportunities with AI. Billions of people trust Chrome to keep them safe by default, and this is a responsibility we take seriously. Following the recent launch of Gemini in Chrome and the preview of agentic capabilities, we want to share our approach and some new innovations to improve the safety of agentic browsing. The primary ne…
Posted by Lyubov Farafonova, Product Manager, Phone by Google; Alberto Pastor Nieto, Sr. Product Manager Google Messages and RCS Spam and Abuse; Vijay Pareek, Manager, Android Messaging Trust and Safety As Cybersecurity Awareness Month wraps up, we’re focusing on one of today's most pervasive digital threats: mobile scams. In the last 12 months, fraudsters have used advanced AI tools to create more convincing schemes, resulting in over $400 billion in stolen funds globally.¹ For years, Android …
Posted by Elie Bursztein and Marianna Tishchenko, Google Privacy, Safety and Security TeamEmpowering cyber defenders with AI is critical to tilting the cybersecurity balance back in their favor as they battle cybercriminals and keep users safe. To help accelerate adoption of AI for cybersecurity workflows, we partnered with Airbus at DEF CON 33 to host the GenSec Capture the Flag (CTF), dedicated to human-AI collaboration in cybersecurity. Our goal was to create a fun, interactive environment, …
AI in cybercrime is evolving fast, fueling AI phishing attacks, AI scam calls, AI voice cloning scams, and even AI deepfake scams. From Dark LLMs to next-gen AI phishing tactics, we break down how criminals exploit AI today and what you can do to stay protected.
Posted by Adam Gavish, Google GenAI Security TeamWith the rapid adoption of generative AI, a new wave of threats is emerging across the industry with the aim of manipulating the AI systems themselves. One such emerging attack vector is indirect prompt injections. Unlike direct prompt injections, where an attacker directly inputs malicious commands into a prompt, indirect prompt injections involve hidden malicious instructions within external data sources. These may include emails, documents, or…
The NICE Workforce Framework for Cybersecurity ( NICE Framework) was revised in November 2020 as NIST Special Publication 800-181 rev.1 to enable more effective and rapid updates to the NICE Framework Components, including how the advent of emerging technologies would impact cybersecurity work. NICE has been actively engaging in conversations with: federal departments and agencies; industry; education, training, and certification providers; and international representatives to understand how Ar…
A remote prompt injection vulnerability in GitLab Duo allowed attackers to steal source code from private projects, manipulate code suggestions, and exfiltrate confidential information. The attack chain involved hidden prompts, HTML injection, and exploitation of Duo's access to private data. GitLab has since patched both the HTML and prompt injection vectors.
Our new LLM-powered chatbot is designed for efficiency and security. Discover how Group-IB AI Assistant enhances threat intelligence workflows and provides security teams with instant insights — without compromising privacy.
Card testing attacks exploit stolen credit card details through small, unnoticed purchases to verify active cards for larger fraud. Cybercriminals use bots, proxies, and automation to evade detection, making real-time fraud prevention challenging. Learn how these attacks work and how to protect against them.
A vulnerability in GCP's Vertex AI service allows privilege escalation and unauthorized access to sensitive LLM models. Attackers can exfiltrate these models by exploiting misconfigurations in access controls and service bindings. By exploiting custom job permissions, researchers were able to escalate their privileges and gain unauthorized access to all data services in the project. In addition, deploying a poisoned model in Vertex AI led to the exfiltration of all other fine-tuned models, posi…
The rapid proliferation of Artificial Intelligence (AI) promises significant value for industry, consumers, and broader society, but as with many technologies, new risks from these advancements in AI must be managed to realize it’s full potential. The NIST AI Risk Management Framework (AI RMF) was developed to manage the benefits and risks to individuals, organizations, and society associated with AI and covers a wide range of risk ranging from safety to lack of transparency and accountability.…
GitHub Copilot Chat VS Code Extension was vulnerable to data exfiltration via prompt injection when analyzing untrusted source code. The vulnerability allowed attackers to access previous conversation turns and append information from the chat history to an image URL, which was then automatically retrieved by Copilot, sending the data to the attacker.
An Indirect Prompt Injection attack can cause the LLM to return markdown tags. This allows an adversary who’s data makes it into the chat context (e.g via an uploaded file) to achieve data exfiltration of the victim’s data by rendering hyperlinks.
A vulnerability in Google Bard allowed for prompt injection and data exfiltration through its Extensions feature. By injecting malicious instructions into shared Google Docs, an attacker could force Bard to render images with exfiltrated chat history data in the URL. The exploit bypassed Content Security Policy using Google Apps Script.
In Azure AI Playground, a Prompt Injection attack could cause an LLM to return markdown tags. This would have allowed an adversary whose data makes it into the chat context (e.g., via an uploaded file) to achieve exfiltration of the victim’s data by rendering hyperlinks. However, the severity of this issue is low, as there were no integrations that could pull remote content. This means Indirect Prompt Injection was not possible, and it would require the victim to copy the malicious prompt from …
In Vertex AI Studio, a Prompt Injection attack could cause the LLM to return markdown tags. This could have allowed an adversary whose data makes it into the chat context (e.g., via an uploaded file) to achieve exfiltration of the victim’s data by rendering hyperlinks. However, the severity of this issue is low, as there were no integrations that could pull remote content. This means Indirect Prompt Injection was not possible, and it would require the victim to copy the malicious prompt from el…
