Influence Benchmarks for AI Systems | SBIR

OUSD (R&E) critical technology area(s): Advanced Computing and Software, Applied Artificial Intelligence, Human-Machine Interfaces, Integrated Sensing and Cyber, Trusted AI and Autonomy

Objective: Develop a testbed utilizing economic frameworks to benchmark AI biases in dynamic environments. This tool will evaluate how agents adapt, collaborate, or deceive within a simulated marketplace. Ultimately, these assessments will ensure safe and effective AI integration in high-stakes decision-making.

Description: This topic seeks to develop a simulated market as a test environment to characterize the latent behavioral preferences of AI systems.

As warfighters increasingly engage with AI systems, particularly agents that rely on large-language models (LLMs), concern has arisen that the systems may encourage cognitive behavior in users that impart hidden biases, impair judgement, and ultimately degrade warfighting capacity. For example, overreliance on AI-powered decision support tools may induce users to accept erroneous suggestions or change from correct decisions to incorrect decisions (Buçinca et al, 2021). LLM use may also induce novel cognitive biases (Alessa et al. 2025) and amplify delusional beliefs (Dohnány et al. 2025). Compounding the threat, deceptive strategies frequently emerge among interacting AI systems (Ying et al. 2026). Few tests of AI systems account for their adaption to dynamic data, which obfuscates such adaptive strategies. To detect and combat these risks, the DoW requires a universal approach to elicit, characterize, and compare the behavior of AI agents amid changing contexts (Li et al. 2026). Such a testbed shall rely only on queries and outputs of the subject AI system, rather than direct access to the model itself.

Economic frameworks permit the measurement and comparison of decisions. As the basis of extensive prior research, models of auctions, markets, and other economic arenas provide a critical baseline of organic patterns of human behavior (e.g. Hausch 1986, Martinez-Saito 2019). Emerging open-source tools, such as Magentic Marketplace (Bansal et al. 2026) have shown promise in revealing behavioral variations in market-focused contexts. Furthermore, unlike existing assessments of AI risk, economic frameworks do not rely on a priori definitions of “harmful” or “helpful” traits (Vijayvargiya et al. 2026), and these testbeds allow for diverse social strategies such as deception and collaboration.

Phase I

The goal of Phase I is to develop and define the architecture of the test environment and demonstrate a functional proof of concept for a simulated auction or market (heretofore “market”). The test environment must be able to accommodate different models of auctions and markets to evaluate agents on different measures of market efficiency. Proposers must also develop classifiers to detect biases in AI agent behavior and outcomes in the market environment. This test environment should be a sandbox in which multiple AI agents may operate. By Month 2, proposers should have defined the core mechanics of the test environment, such as market structures, payout schemes, measures of market efficiency and consumer utility, and a dynamic “news feed” that informs the behavior of AI agents within the market. The “news feed” is meant to simulate public information relevant to the market. By Month 6, proposers should establish a strategy for scaling issues and define how AI agents interact through their decision space: a set of bids, each of which is defined by the time, asset, quantity, and price offered/requested on the market. By Month 9, proposers must demonstrate a suite of “stock” AI agents for use within the test environment. These AI agents shall only be used within the test environment sandbox and shall not be used on other systems. By Month 12, proposers should have a proof-of-concept test environment for assessing behavior preferences of AI systems amid dynamic data. In the proof-of-concept, the test environment must be able to replicate bidding patterns and allocation efficiencies observed in markets and demonstrate allocative efficiencies of >90%. The environment and associated data must be sufficient for comparison to models of human behavior in auctions or markets, as well as to compare agents to one another. Phase I fixed payable milestones for this program should include:

• Month 2: Report on initial architecture and core mechanics of test environment and classifiers of AI agent behavior
• Month 6: Report on scaling and AI agent interaction within the test environment
• Month 9: Month 9: Demonstration of a suite of stock AI agents, drawn from at least 10 different LLMs, operating within the test environment.
• Month 12: Demonstration of test environment and AI agent behavior within the marketplace, with allocative efficiencies of >90%

Phase II

Proposers submitting Direct to Phase II should be able to demonstrate an existing test environment that meets the criteria outlined in Phase I.

In Phase II, proposers should develop a range of AI agents as stock reference models against which the performance of a test AI agent may be compared. Proposers must define an error function that describes differences between AI agent decisions and expected human results. The stock AI agents should help identify and bound agent behaviors of concern, such as hidden biases. At least one AI agent should represent the human user of the test AI agent, to explore how the test AI agent may drive human behaviors of concern such as encouraging delusions or eroding military discipline. Proposers should define analytics to compare the behavior of the AI agents. By Month 6, proposers should have a suite of such stock AI agents, representing both human and native AI agent behavior. Agents should be able to interact with other agents in the test environment to develop strategic bids in an attempt to influence the behavior of other market actors to maximize their expected profit. By Month 12, proposers should demonstrate and measure how each AI agent responds to the “news feed” and the observable behavior of other agents, which alters each agent’s expectation of future asset values and behaviors of other agents. By Month 15, proposers should assess social interactions among AI agents to model behaviors including, but not limited to, collaboration and deception. Proposers must provide quantitative metrics on how social behaviors between AI agents alter market efficiency and outcomes and generate testable hypotheses for how latent AI agent preferences affect social interactions, dynamic responses to external stimuli, and ultimately market outcomes. By Month 18, proposers should test their own hypotheses across a range of AI agents and quantify differences between human-like agents and AI agents. By Month 21, proposers should demonstrate the test environment to DARPA, and deliver software and data for testing by U.S. Government partners to ensure that the environment may be used to reliably characterize the latent preferences of a test AI agent. By Month 24, the proposers should deliver a final report describing the performance of the test environment and AI agents, and include a transition plan. Phase II fixed payable milestones should include:

• Month 2: A report on new capabilities that will be added to the test environment to enable analysis of dynamic AI agent behavior and outcomes. Provide definitions of an error function to compare AI agent decisions vs human decisions, as well as measures for comparing distributions of human and AI agent market outcomes.
• Month 6: Demonstration of “human” stock AI agents that replicate human preferences and behavior in the market, and a report on how a “human” stock AI agent simulates expected human interaction in the market. Test Phase I classifier’s ability to distinguish “human” vs. AI agents in the market environment, and propose improvements to classification algorithm or alternative test statistics
• Month 12: Report on dynamic behavior of AI agents in response to news events and observed actions of other agents within the test environment. Evaluate agent behavior using improved classifiers.
• Month 15: Report on modeling social interactions between AI agents, including impact on bidding behavior of AI agents
• Month 18: Report on how latent AI preferences impact social interactions between AI agents, dynamic responses, and market outcomes.
• Month 21: Final software delivery, both object and source code, for operation by DARPA or other U.S. Government personnel for additional demonstrations, with suitable documentation in a contractor proposed format
• Month 24: Final report, including quantitative metrics on AI agent behavior and fidelity of the test environment. The report must also discuss advances in LLM models and AI agents that have occurred during the period of performance and their potential impact on the evaluation of AI models.

Phase III dual use applications

The potential risk of AI systems inducing harmful behaviors is a major commercial concern due to the associated legal and reputational risk to their developers. AI firms are actively seeking mechanisms to argue the limits of their systems’ biases and impacts on user behavior, including measuring deviations from expected human norms of behavior, in response to lawsuits. The impact on a firm’s stock value of establishing the scope of liability far exceeds the already-robust immediate financial consequences of these court cases, suggesting that the AI firms have strong market demand for the capability to benchmark the influence of their systems.

References

1. Buçinca Z., Malaya M., Gajos, K. (2021). To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making. Proceedings of the ACM on Human-computer Interaction 5, CSCW, 1-21.
2. Alessa A., Somane P., Lakshminarasimhan, A., Skirzynski J., McAuley J., Echterhoff J. (2025). Quantifying Cognitive Bias Induction in LLM-Generated Content. Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics. 2890-2910.
3. Hausch DB. (1986). Multi-Object Auctions: Sequential vs. Simultaneous Sales. Management Science 32(12):1599-1610.
4. Dohnány, S., Kurth-Nelson, Z., Spens, E., Luettgau, L., Reid, A., Gabriel, I., Arulkumaran, K., & Nour, M. M. (2025). Technological folie à deux: Feedback loops between AI chatbots and mental illness (arXiv:2507.19218)
5. Li E., Bellotti V., Sessions N., Kao R. (2026). Mastering Agentic Techniques: AI Agent Evaluation. Accessed at: https://developer.nvidia.com/blog/mastering-agentic-techniques-ai-agent-evaluation/
6. Martinez-Saito M., Konovalov R, Piradov MA, Shestakova A, Gutkin B, Klucharey V. (2019). Action in auctions: neural and computational mechanisms of bidding behaviour. European Journal of Neuroscience. 50(8): 3327-3348.
7. Bansal, G., Hua, W., Huang, Z., Fourney, A., Swearngin, A., Epperson, W., Payne, T., Hofman, J., Lucier, B., Singh, C., Mobius, M., Nambi, A., Yadav, A., Gao, K., Rothschild, D., Slivkins, A., Goldstein, D., Mozannar, H., Immorlica, N., Murad, M., Vogel, M., Kambhampati, S., Horvitz, E., Amershi, S. Magentic Marketplace: An Open-Source Environment for Studying Agentic Markets. arXiv:2510.25779v1 [cs.AI].
8. Vijayvargiya S., Bharat Soni A., Zhou X., Zhiruo Z., Wang Z.Z. (2026). OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety. arXiv:2507.06134v2 [cs.AI].

Keywords

AI, LLM, cognitive bias, economic model, market, information

TPOC-1-PoC

DARPA BAA Help Desk

Email

SBIR_BAA@darpa.mil

Ready to apply?

For additional information and to submit your full proposal package, visit the DSIP Portal.