CATEMS: A Lifecycle Model of Compromised AI Teammate Attacks, Effects, and Mitigation Strategies for Enhanced Resilience

Abstract

Human-AI teams are gaining traction as they combine the unique strengths of humans and AI. Very little research has examined the unique vulnerability of compromised AI teammates that work directly against the team’s shared goals. Compromised AI attacks combine the potency of technological attacks with teammate betrayal, potentially devastating the team’s ability to execute its goals. Such attacks by a malicious actor may misalign mental models, individual situational awareness, and shared situation awareness. This misalignment could potentially result in the complete loss of team coordination, endangering performance, resiliency, and security. Given the severity of a compromise, this paper presents a model to explain the types of attacks, when they may occur, their effects on the team, and potential approaches for attack identification and recovery.