Summary
Human teaming has been the subject of research for over a century, covering aspects of team behaviors and their effectiveness under certain constraints and challenges.
This research has provided insights into the formation, management, and functioning of teams that can be applied to improve outcomes measurably. Comparatively, little progress has been made, however, in either applying human-team analysis methods or developing new methods for effective evaluation of human-machine teams.
Although the eventuality has long been predicted, machines have yet to be considered equal members in team dynamics or process effectiveness research. Despite their pervasiveness, the machines we use every day are still conceptualized most often as communication conduits, calculators, or reference books.
However, with the advent and broad utilization of generative AI technologies, human-AI interdependency is growing within critical systems and processes and in our everyday lives. The volume and extent of our interactions with machines will continue to increase, as will our expectations of machine competence to perform independent and collaborative tasks.
While the U.S. Department of Defense (DoD) has forecast the importance of human-machine teaming, significant gaps remain in our understanding of the expected behaviors of human-AI teams. Moreover, due to the rapid advancement and breadth of generative AI technologies, evaluation paradigms for human-AI teams need to be revised to accommodate the range of human-AI team behaviors that might emerge, owing to the diversity of human team members and the variation of fine-tuned AI agents.
The goal of the Exploratory Models of Human-AI Teams (EMHAT) Artificial Intelligence Exploration (AIE) is to develop technologies for generating and evaluating diverse and realistic digital twins representing human-AI teams to understand and characterize the emergent capabilities and limitations of such teams in proxy operational settings.
EMHAT will leverage generative AI to create a human-AI modeling and simulation framework capable of providing sufficient naturalistic data to enable quantitative modeling and assessment of the task execution effectiveness of human-machine teams in realistic settings. The resulting technology will utilize expert feedback, AI-assembled knowledge bases, and generative AI to produce various computational agents representing diverse human teammate simulacra, analogous to digital twins. EMHAT researchers will then leverage these digital twins to model human interaction with AI systems carrying out human-AI teaming tasks that mimic operational tasks, assessing 1) human-machine task completion rate relative to baseline and 2) AI behavioral adaptation in the presence of explicit and implicit simulated human behavior.
Workshops informed the EMHAT AIE as part of DARPA’s AI Forward initiative. Additional information is available in the EMHAT Program Announcement on Sam.gov.