Dynamic Threat Modeling for Continually Learning AI Systems

Abstract

AI systems that use continual learning paradigms pose novel and emerging security requirements that are difficult to apprehend with conventional, static threat modeling efforts. In contrast to traditional models, which will remain unchanged after being deployed, continually learning AI systems are dynamic, evolving as new data emerges, presenting an ever-changing attack surface that can be actively exploited by adversaries, who might attempt to poison the data used to train the model (data poisoning), use evasion techniques to obfuscate data during the inference process (adaptive evasion), or capture the model being trained (model inversion). Existing models like STRIDE, MITRE ATLAS, and OWASP MAESTRO are good at their entry point but fail to offer real-time customization, continuous observation and resilience designing.
The current paper proposes a framework of Dynamic Threat Modeling (DTM) that is specifically tailored towards the continuous learning AI systems. The framework places much importance on the adaptivity, feedback mechanism, data-centric security, human-in-the-loop governance to assure proactive identification and neutralization of new threats. It is also proposed how to do it through methodological integration with MLOps pipelines, adversarial red-teaming, and emerging threat intelligence. Examples of DTM within healthcare, finance and autonomous systems provide examples of the practical implications of DTM regarding safeguarding of critical infrastructures.
It is concluded that dynamic threat modeling is critical to technical resilience, to building trustworthy AI adoption, and to setting regulatory requirements. The transformation of the static to adaptive solutions proposed in this work brings about building blocks in achieving secured, open, and stable AI ecosystems that will be able to endure the nature of adversarial challenges that are also dynamic.