CODORD: Human-AI Communication for Deontic Reasoning Devops

Human-AI Communication for Deontic Reasoning Devops (CODORD) intends to create new, automated techniques for humans to author knowledge about deontics (obligations, permissions, and prohibitions) into an expressively flexible logical language by using natural language (i.e., English). 

CODORD has the potential to enable automated deontic reasoning with high assurance (i.e., verifiability/explainability and correctness) to assess compliance with orders, regulations, laws, operational policies, and ethics. 

Automated deontic reasoning could be critical for enabling autonomous systems to:

• accurately apply rules of engagement in dynamic contexts
• interpreting and maintaining the integrity of commander's intent across the chain of command and in operations planning
• analyzing intelligence
• negotiating among subtle but crucial differences in legal and command authorities in allied relationships during joint operations.  

FAQs 

As of Nov. 28, 2024

22Q: Are you open to proposals that use a system different from the ones listed in the solicitation? 
22A: We encourage the community to consider systems other than those specifically mentioned in the solicitation; however, to be successful in CODORD such systems must exhibit the four practical requirements listed in section I.A of DARPA-PA-24-04-01, i.e., sufficient expressiveness, computational scalability, strong explainability, and availability of a commercial-quality open-source toolset for the logical language.

21Q: There is an inconsistency in the Total KA Labor Time metric in Table 1 and Figure 1 of the CODORD Disruption Opportunity (DO), DARPA-PA-24-04-01. Which takes precedence? 
21A: Table 1 is correct. This has been addressed in a recent amendment to the solicitation.

20Q: Where can I find specifics on the “necessary nondisclosure agreements, security regulations, export control laws, and other governing statutes applicable under the circumstances” with which non-US organizations must comply, as described in Section 7.1.2 of the Program Announcement (PA) for Disruptioneering, DARPA-PA-24-04? 
20A: In addition to export control laws and Article XVI for the Prohibition on Contracting for Certain Telecommunications and Video Surveillance Services or Equipment in the Model OT document, proposers should take appropriate measures to be compliant with all applicable U.S. laws.

19Q: Is a US citizen employed by a foreign university considered a US citizen for purposes of compliance with “necessary nondisclosure agreements, security regulations, export control laws, and other governing statutes applicable under the circumstances” with which non-US organizations must comply, as described in section 7.1.2 of the PA for Disruptioneering, DARPA-PA-24-04?
19A: A US citizen is a citizen regardless of employer. However, the citizenship of an individual within a proposer organization does not remove the need for a foreign organization to comply with Section 7.1.2 of the Disruptioneering PA.

18Q: Are UARCs/FFRDCs eligible to participate as performers?
18A: 

1. Per Section 7.1.1 of the Program Announcement for Disruptioneering (DARPA-PA24-04): “Federally Funded Research and Development Centers (FFRDCs), University DARPA-PA-24-04-01 FAQs 2 Affiliated Research Centers (UARCs), and Government entities that are interested in participating in a Disruption Opportunity due to specific expertise or core competencies should contact the Agency Point of Contact (POC) listed in the specific DO prior to submitting an Abstract (if applicable) or Proposal to discuss related opportunities.”
2. Entities interested in receiving awards under Disruptioneering must also be able to accept an Other Transaction for Prototype award issued under the authority of 10 U.S.C. § 4022.
3. Finally, per section 7.2 of DARPA PA-24-04, Organizational Conflicts of Interest: Without prior approval or a waiver from the DARPA Deputy Director, a contractor cannot simultaneously provide scientific, engineering, technical assistance (SETA), advisory and assistance services (A&AS), or similar support and also be a technical performer. As part of the proposal submission, all members of the proposed team (including any potential subcontractors or consultants) must affirm whether they (their organizations and individual team members) are providing SETA or similar support to any DARPA office(s) through an active award or subaward. All facts relevant to the existence or potential existence of organizational conflicts of interest (OCI) must be disclosed in Volume 3: Administrative & National Policy Requirements.
   
   DARPA currently considers entities conducting Independent Verification and Validation and/or Test and Evaluation activities on any DARPA programs to be engaged in support similar to SETA or A&AS. If you are engaged in these activities, then you are deemed to have a conflict of interest (COI) if applying to be a performer, either as a Prime awardee or subcontractor. In that case, receiving an award as a Prime or sub under DARPA-PA-24-04 would require submission of an Organizational Conflict of Interest (COI) Mitigation Plan with the proposal for DARPA’s approval, and subsequent waiver signed by the DARPA Deputy Director.

17Q: Must non-US subcontractors comply with “the “necessary nondisclosure agreements, security regulations, export control laws, and other governing statutes applicable under the circumstances” with which non-US organizations must comply, as described in section 7.1.2 of the PA?” 
17A: Yes. It is the prime contractor’s responsibility to ensure that their subcontractors comply with all such requirements.

16Q: What role do fact sentences and background sentences play in CODORD Test and Evaluation? 
16A: The background non-fact sentences and the fact sentences (in addition to the foreground non-fact sentences) are knowledge that will also need to be authored, via each performer’s novel authoring technique and/or the SOA manual authoring. For context, during the CODORD Test & Evaluation team’s conducting of the independent evaluations, and by performers in their own somewhat-similar testing as they develop their techniques. Their number will be within the same order of magnitude as that of the foreground non-fact sentences and will vary per particular use case test problem. Use cases will have significant reuse of knowledge across multiple test problems within that use case domain. See sections I.C and I.D, including Figure 1 (“CODORD schedule”), of the CODORD Program Announcement, DARPA-PA-24-04-01.

15Q: Is there guidance on the precision of performer's chosen domains? For example, building codes are written in highly technical language while philosophical approaches to the trolley problem are written at a general conceptual level. 
15A: Performers will be evaluated on use case domains chosen by the T&E team. These domains will be programmatic, rather than chosen by each individual performer. These use case domains will be described in a largely self-contained manner, rather than require knowledge authors to be previously familiar with highly technical language. That said, performers may develop additional test examples in additional domains for their own development purposes and/or to demonstrate (as part of their quarterly progress reporting) in a manner separate from the independent evaluations.

14Q: Will competitive proposals need to propose knowledge authoring systems that input unrestricted natural language, rather than restricted natural language that significantly limits vocabulary or grammatical structures? 
14A: Systems should meet the evaluation criteria, specified in Sections I.C and I.D of the CODORD DO. Restricted natural language in past systems for knowledge authoring has tended to have disadvantages in meeting those evaluation criteria.

13Q: To encompass concepts such as belief or intent, where much knowledge is implicit, will approaches to knowledge authoring need to support reasoning about implicit knowledge? 
13A: Yes. Knowledge authoring in high-expressiveness logical languages largely must deal with drawing conclusions about implicit beliefs, intents, etc. – but based on what is authored as explicit knowledge.

12Q: Should proposals presuppose an existing ontology for representing knowledge in a given use case domain? 12A: No. The technique for knowledge authoring should be able to be used, reasonably cost-effectively, to generate whatever background logical knowledge (including ontological) is needed to answer correctly the test problems’ focal queries. That said, it is DARPA-PA-24-04-01 FAQs 4 possible that moderate amounts of background logical knowledge (including ontological) will be supplied by the T&E team for a given use case domain and its test problems.

11Q: Must the open-source logical language produced by the generating process match that used by the reasoning process? 
11A: Yes. The logical reasoning toolset associated with that logical language will be run in order to perform semantic assessment, notably of correctness on use case domains’ test problems. T&E evaluations will, moreover, rely on the interoperability of performers’ generating processes – and their resulting knowledge in the logical language – with the overall T&E testing framework, which will include the reasoning process. See Sections I.B. and I.C. of the DO.

10Q: In regard to the user interface (UI) aspect of the performer’s novel technique for natural language based knowledge authoring: is that expected to be novel, or can/should proposers plan to use an existing user interface? 
10A: Performers on the program may elect to rely on existing UIs or develop their own, for purposes of internal development, and/or demonstrations to DARPA, and/or the independent evaluations conducted by the T&E team. Additionally the T&E team will use a common UI, developed by T&E as part of the common testing framework described in Section I.C of the DO, during performer evaluations at months 9, 12, 18, and 21.

9Q: How many awards will be made? 
9A: Multiple awards are anticipated.

8Q: Will the PM consider meetings with individual proposers? 
8A: The PM will not be holding discussions with potential proposers. All communication must be conducted through the CODORD@darpa.mil email address, and responses will be duplicated in this FAQ to ensure equitable distribution of information.

7Q: Can a single team submit more than one abstract? 
7A: Yes.

-- New Q&A --

6Q: Can an organization be included as a subcontractor in multiple proposals? 
6A: Yes, however, each proposal should contain a discussion of overlap and management issues that may arise if multiple proposals are selected for funding. DARPA-PA-24-04-01 FAQs 5

5Q: Are service-run universities (e.g. Naval Postgraduate School, Air Force Institute of Technology) eligible to participate? 
5A: Yes, provided certain conditions are met. See Appendix A of the Program Announcement for Disruptioneering, DARPA-PA-24-04.

4Q: Where are the slides and recording from the CODORD Information Session? 
4A: These can be found at https://www.darpa.mil/work-with-us/opportunities

3Q: What is the funding structure for CODORD? 
3A: All awards will be made in the form of an Other Transaction (OT) for Prototype project. The total award value for the combined Phase 1 base and Phase 2 option is limited to $2,000,000. This total award value includes Government funding and performer cost share if required or proposed. Proposers must only propose an OT agreement with fixed payable milestones. Fixed payable milestones are fixed payments based on successful completion of the milestone accomplishments agreed to in the milestone plan. See Sections I and I.D of the CODORD Program Announcement, DARPA-PA-24-04-01, and Section 2.3 of the Program Announcement for Disruptioneering, DARPA-PA-24-04.

2Q: Can a foreign entity participate as a prime, subcontractor, or individual (e.g. a graduate student at a US university)? 
2A: Non-U.S. organizations and/or individuals may participate to the extent that such participants comply with any necessary nondisclosure agreements, security regulations, export control laws, and other governing statutes applicable under the circumstances. See section 7.1.2 of the Program Announcement for Disruptioneering, DARPA-PA-24-04.

1Q: Can a traditional defense contractor be a prime? 
1A: Yes, provided certain conditions are met. See Appendix A of the Program Announcement for Disruptioneering, DARPA-PA-24-04.