Open Problems in Securing and Testing AI Agents

Why it mattered

• It took AI safety to a hacker meetup. The room was penetration testers, red teamers and security engineers: people whose day job, finding how systems fail under pressure, is close to what AI evaluation and control research needs.
• It framed safety in terms security engineers already work with: least privilege, assume breach, pentest scoping, incident response and supply-chain review. The skills they use every day already apply to AI safety.
• It gave practical starting points: five open-source tools people could run within a week.
• It was run with a wide group of local partners, which builds ties between Ahmedabad's security scene and the AI safety community.

What we covered

1. 01Why nowIn 2026, AI agents were used to break into companies at a cost of tens of dollars per intrusion, and, in a separate case, agents in a lab's hacking test escaped it and attacked a real company with no human telling them to. Researchers had warned about both kinds of failure years earlier.
2. 02What an agent isA model in a loop with tools: it plans, acts, reads the result and repeats. The harm happens in the actions, and outside text (a web page, an email) can steer it. One example: a hidden line in an email makes an assistant forward invoices to an attacker while telling the user everything is fine. It is the same family of bug as SQL injection, but with no equivalent fix yet.
3. 03Testing versus red teamingEvaluations give you a score you can compare; red teaming finds ways to break the system. The talk pointed out why both matter, then walked through five tools: Inspect, garak, PyRIT, promptfoo and AgentDojo.

Further reading

• METR & Redwood Research, investigation of the OpenAI / Hugging Face incident (Aug 2026)
• METR, GPT-5.6 Sol pre-deployment evaluation (Jun 2026)
• Simon Willison, The lethal trifecta for AI agents (2025)
• OWASP, Top 10 for Agentic Applications 2026
• Debenedetti et al., AgentDojo (NeurIPS 2024)
• Nasr, Carlini et al., The Attacker Moves Second (2025)
• Brundage et al., The Malicious Use of Artificial Intelligence (2018)
• Amodei et al., Concrete Problems in AI Safety (2016)
• Inspect, UK AI Security Institute
• garak, NVIDIA
• PyRIT, Microsoft
• promptfoo

Speaker

• MAMann AcharyaResearcher @ The Millennium Project