SBIR: Language Models for Veteran Suicide Prevention (LM4VSP)

OUSD (R&E) critical technology area(s): Trusted AI and Autonomy

Objective: The objective of the Language Models for Veteran Suicide Prevention (LM4VSP) SBIR topic is to develop a conversational artificial intelligence (AI) clinical co-pilot, based on Language Models, to augment the ability of mental health care professionals to care for and interact with patients at risk of suicide.

Description: On September 22, 2021, the US House Veterans Affairs Committee held a hearing entitled, “Veteran Suicide Prevention: Innovative Research and Expanded Public Health Efforts” [1]. The hearing followed the release of annual data from the US Department of Veterans Affairs showing that the disproportionate rate of veteran suicide is a public health crisis[2]. Although there is no single reason why veterans commit suicide, evidence suggests that stable housing, financial security, access to healthcare, addressing social isolation and loneliness, and treating the effects of trauma are important components of a comprehensive suicide prevention strategy; all of which require coordination and cooperation across families, communities, and at all levels of government.

Recent advances in AI, and specifically LMs, have the potential to help lessen the effects of social isolation/loneliness and trauma. According to a 2019 report on “Sleep and timing of death by suicide among US Veterans 2006-2015” [McCarthy, et al. 2019], the raw proportion of veteran suicides peaks between the hours of 1000 and 1200; however, the peak prevalence of suicide, after accounting for the population being awake, is between the hours of 0000 and 0300 (p < 0.00001, F = 0.88). The highest Standardized Incidence Ratio (SIR) is at midnight; US Veterans are eight times more likely to die by suicide than expected given the population awake (SIR = 8.17; 95% Confidence Interval = 7.45-8.94). In other words, when clinical help is likely unavailable or difficult to access, technology has the potential to provide critical assistance.

Recent advances in the field of natural language processing have allowed LMs (for example, Chat GPT (Generative Pre-trained Transformer)) to be fine-tuned using reinforcement learning based on human feedback [3]. Prior efforts have shown that it is possible to create a highly conversational model based on 40,000 pieces of feedback [4]. Other recent research work in the field suggests promising results in prompt engineering [5], using memory-based machine learning with dramatic improvements in the LM’s ability to stay on task, and return more accurate and precise results. [6]

LM4VSP seeks to develop a clinical co-pilot based on LMs specific to the mental health subdomain, and in close collaboration with mental health subject matter experts (SMEs). The goal is for the LM4VSP clinical co-pilot to enable caregivers to offer around-the-clock assistance and accelerate their understanding and assessments and improve the effectiveness of intervention.

Phase I

Phase I will focus on producing a feasibility study and proof-of-concept model of the LM4VSP clinical co-pilot. The Phase I effort should include, at a minimum:

• Identification and acquisition of relevant domain data, e.g., anonymized mental health, battlefield, etc.;
• Identification and engagement of SMEs, e.g., mental health professionals;
• Identification of key trigger points and ranges of appropriate responses in collaboration with mental health SMEs;
• Identification of major categories of stress and trauma and best practices for their management; identification of trigger points or categories that have the highest probability of successful technology-assisted intervention;
• Assessment of current approaches in the use of language models to detect mental health classes; and development of a proof-of-concept model.

Schedule/Milestones/Deliverables: Proposers will execute the research and development (R&D) plan as described in their proposals, including the below deliverables. Proposers that anticipate including Human Subjects Research (HSR) in their approach must comply with the approval procedures detailed at http://www.darpa.mil/work-with-us/additional-baa. Proposers planning to use HSR are highly encouraged to clearly segregate research tasks from human testing tasks to allow for partial funding while internal and DoD approvals are obtained

• Months 2 and 4: Months 2 and 4: report detailing technical progress made to date, tasks accomplished, current risks/mitigations, plan for the remainder of Phase I (e.g., tasks to be accomplished, planned activities/trips/meeting), trip/meeting summaries, a summary of any potential issues or problem areas (technical or financial) that require the attention of the DARPA PM, and updates on (if applicable) Institutional Review Board (IRB) and Human Research Protection Office (HRPO) approvals or subject enrollment when approvals are obtained. Reports may be in the form of Microsoft Word or PowerPoint documents.
• Month 5:
  • Final LM4VSP feasibility study report; and,
  • Final LM4VSP proof-of-concept model.

Phase II

Phase II will focus on the development of a prototype LM4VSP clinical co-pilot, trained on a LM that is fine-tuned on the mental health subdomain and capable of producing high-fidelity and appropriate responses as determined by clinical SMEs. The Phase II effort should include, at a minimum:

• Identification of key metrics for evaluation of performance;
• Identification of gaps and shortcomings of existing LMs and transformer-based models to be used for baseline system comparisons;
• Development of a labeled dataset of anonymized mental health data;
• Development of a prototype LM4VSP clinical co-pilot.

Phase II will culminate in an evaluation of the LM4VSP clinical co-pilot prototype capability by mental health clinicians.

Schedule/Milestones/Deliverables: Proposers will execute the R&D plan as described in their proposals, including the below deliverables. Proposers that anticipate including HSR in their approach must comply with the approval procedures detailed at http://www.darpa.mil/work-with-us/additional-baa Proposers planning to use HSR are highly encouraged to clearly segregate research tasks from human testing tasks to allow for partial funding while internal and DoD approvals are obtained.

• Quarterly: Quarterly: reports detailing technical progress made to date, tasks accomplished, current risks/mitigations, plan for the remainder of Phase 2 (e.g., tasks to be accomplished, planned activities/trips/meeting), trip/meeting summaries, a summary of any potential issues or problem areas (technical or financial) that require the attention of the DARPA PM, and updates on (if applicable) IRB and HRPO approvals or subject enrollment when approvals are obtained. Reports may be in the form of Microsoft Word or PowerPoint documents
• Month 24: Final LM4VSP delivery/demonstration of capability.
• Month 36 (Phase II Option period):
  • Final Phase II Option period technical report including details of LM4VSP clinical co-pilot prototype efficacy against other state-of-the-art platforms, including quantitative metrics for assessment; and,
  • Final LM4VSP delivery/demonstration of capability.

Phase III dual use applications

LM4VSP has potential applications across the DoD and industry. 

For DoD, successful LM4VSP approaches will provide the ability to augment the services provided by military mental health clinicians to veterans at risk of suicide. LM4VSP has the same applicability for industry/the commercial sector. Phase III refers to work that derives from, extends, or completes an effort made under prior SBIR funding agreements, but is funded by sources other than the SBIR program. 

The Phase III work will be oriented towards transition and commercialization of the developed LM4VSP prototype. For Phase III, the proposer is required to obtain funding from either the private sector, a non-SBIR Government source, or both, to develop the prototype into a viable product or non-R&D service for sale in government or private sector markets. LM4VSP solutions will support national efforts to improve critical mental healthcare support for veterans and their families.

References

[1] “Veteran Suicide Prevention: Innovative Research and Expanded Public Health Efforts.” YouTube, uploaded by House Committee on Veterans’ Affairs, Streamed live on Sep 22, 2021, 

[2] (2022). National Veteran Suicide Prevention Annual Report. US Department of Veterans Affairs, Office of Mental Health and Suicide Prevention.

[3] Stiennon, Nisan, et al. 2020. Learning to summarize from human feedback. In Proceedings of the 34th International Conference on Neural Information Processing Systems (NIPS'20). Curran Associates Inc., Red Hook, NY, USA, Article 253, 3008–3021. 

[4] Ouyang, Long, et al. (2022). Training language models to follow instructions with human feedback.

[5] Eliaçik, E. (2023 Feb. 24) AI prompt engineering is the key to limitless worlds, The art of writing text prompts. DATACONOMY. 

[6] Madaan, Aman, et al. (2022). MemPrompt: Memory-assisted Prompt Editing with User Feedback. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2833-2861. 

Keywords

Language Models, Suicide Prevention, Mental Health, Natural Language Models, Clinician Assistant, Training Data

TPOC-1

DARPA BAA Help Desk

Email

SBIR_BAA@darpa.mil