I recently came across MAESTRO—billed as a “novel threat modeling framework designed specifically for the unique challenges of Agentic AI.” I fancy myself a bit of a collector of threat modeling frameworks, so of course I decided to dig into the writeup to see what innovative ideas it brings that are uniquely applicable to the world of agentic AI systems. TL;DR—I don’t think its approach, the actual “framework” for modeling, is particularly novel. Rather, what this whitepaper usefully introduces (if anything) is a multi-layered, AI-specific, attack/threat catalog.
Comparing existing frameworks
To illustrate the need for MAESTRO and distinguish it from other established threat modeling methodologies, the author (Ken Huang) first runs through a couple of the more well-known frameworks, enumerating the respective strengths, weaknesses and gaps related to AI. In this exercise, I think the paper fails to understand the modular quality of any given framework (more on this shortly *), but correctly highlights the ridgidity of any one framework’s “steps”, and the infeasibility of using them to-the-letter in a practical sense.
* For example, it’s called out that PASTA is “complex and resource intensive” which is not conducive to modern development. Absolutely, definitely agree here. But then it goes on to say that PASTA doesn’t specifically focus on AI vulnerabilities. Huh? PASTA (and frankly most other actual threat modeling frameworks—*cough* not STRIDE *cough*) give a lot of latitude in terms of attack generation (among other things)—i.e. there’s no reason you can’t use an AI-specific threat catalog (e.g. MITRE ATLAS) with PASTA.
As another example, the paper suggests that LINDDUN is inadequate for threat modeling AI systems because it is narrowly scoped to privacy-specific threats. Again, I think the paper fails to understand that LINDDUN has this specificity for a reason. It isn’t that LINDDUN isn’t good for AI systems, but rather LINDDUN isn’t a general-purpose (bring-your-own-threat-classification) threat modeling framework. If you are uniquely interested in privacy-related threats, LINDDUN is probably still a perfectly applicable methodology, even in the context of agentic AI systems.
As a final example, the paper suggests VAST is inadequate to evaluate AI systems because of some gap related to AI-specific risks. What? VAST is a very simple, and most notably, abstract framework, and as such allows for a lot of liberty in terms of the types of threats you can consider. Again, I think this speaks to a fundamental misunderstanding of the model (VAST) that MAESTRO is ultimately being compared with.
As an added note, there’s a lot of other models that this paper does not attempt to cover. Granted, these other models may not be as well-known, even if they could be more applicable in the AI context.
Getting into MAESTRO
Enough talk about other models, let’s get into what MAESTRO really is. To understand MAESTRO, let’s take a look at the framework’s stated principles and its methodology for modeling.
MAESTRO’s Principles
MAESTRO’s principles are meant to be tailor-made for conducting practical security assessments against agentic AI systems. They are also meant to be unique and differentiating with respect to other “competing” methodologies. These principles are listed below…
- Extended Security Categories: Expanding traditional categories like STRIDE, PASTA, and LINDDUN with AI-specific considerations.
- Multi-Agent and Environment Focus: Explicitly considering the interactions between agents and their environment.
- Layered Security: Security isn’t a single layer, but a property that must be built into each layer of the agentic architecture.
- AI-Specific Threats: Addressing threats arising from AI, especially adversarial ML and autonomy-related risks.
- Risk-Based Approach: Prioritizing threats based on likelihood and impact within the agent’s context.
- Continuous Monitoring and Adaptation: Ongoing monitoring, threat intelligence, and model updates to address the evolving nature of AI and threats.
After a cursory review, these principles seem perfectly adequate for assessing agentic AI systems—no comment there. But I don’t think these principles are particularly novel juxtaposed with other existing frameworks. As I covered earlier, many methodologies provide the space to plug-in an attack/threat catalog of your choosing. Sure, threat classification models like STRIDE or threat modeling frameworks like LINDDUN that have more rigid threat categories exist, but most methodologies allow you to generate threats with much greater latitude. Understanding system layers and environmental context is nothing unique either. This just sounds like the classic step of application decomposition, i.e. understanding the data flow, the use cases, the actors, mitigating controls, etc… The remaining three principles just cover threat generation, risk analysis and revisiting the model. So… really nothing new to add.
To be clear, these aren’t bad principles. It’s just not groundbreaking stuff.
The Approach
Speaking of nothing groundbreaking, let’s analyze MAESTRO’s “step-by-step approach”, i.e. the actual methodology. The steps are listed below…
- System Decomposition: Break down the system into components according to the seven-layer architecture. Define agent capabilities, goals, and interactions.
- Layer-Specific Threat Modeling: Use layer-specific threat landscapes to identify threats. Tailor the identified threats to the specifics of your system.
- Cross-Layer Threat Identification: Analyze interactions between layers to identify cross-layer threats. Consider how vulnerabilities in one layer could impact others.
- Risk Assessment: Assess likelihood and impact of each threat using the risk measurement and risk matrix, prioritize threats based on the results.
- Mitigation Planning: Develop a plan to address prioritized threats. Implement layer-specific, cross-layer, and AI-specific mitigations.
- Implementation and Monitoring: Implement mitigations. Continuously monitor for new threats and update the threat model as the system evolves.
Seem familiar? That’s because it is. Application decomposition, threat generation, risk assessment, risk treatments and validation would describe a lot of other models. The only difference here is that the threat generation is focused on AI-specific threats across these defined layers… but other models (i.e. PASTA) would also accommodate for this. So in short, the “model” is not novel. If there’s value here (and I think there could be), it’s in the layered threat catalog. Let’s get to that…
7-Layer Reference Architecture, i.e. the Attack Catalog
What I do find interesting and useful from the MAESTRO writeup is the layer-by-layer breakdown of AI-related threats. I won’t regurgitate them here so I would encourage you to read through the writeup to see the listing/breakdown of attacks.
Though other AI-specific threat catalogs exist (and will likely continue to be developed) (e.g. ATLAS), I do like the way MAESTRO breaks it down by layers.