As artificial intelligence transitions from content-generating Generative AI (GenAI) to autonomous, multi-step Agentic AI, the landscape of enterprise opportunities — ranging from operations automation to fraud detection — is expanding rapidly. However, this evolution introduces significant “double-edged sword” risks, where AI serves as both a powerful tool for defenders and a sophisticated weapon for attackers.

This talk explores the critical security challenges inherent in AI-based systems, including model hallucinations, prompt injection, data poisoning, and the emergence of malicious generative frameworks such as MalGEN. The session also discussed the global regulatory landscape, highlighting key frameworks such as India’s RBI ethical principles, the EU AI Act, and NIST’s AI Risk Management Framework. Attendees gained insights into a Secure AI Development Lifecycle, emphasizing human-in-the-loop (HITL) oversight, continuous red-teaming, and robust explainability standards.