Program Summary
The goal of the Mathematics of Sensing, Exploitation and Execution (MSEE) program is to explore and develop high-impact methods for scalable autonomous systems capable of understanding scenes and events for learning, planning, and execution of complex tasks. The program is exploring powerful mathematical frameworks for unified knowledge representation for shared perception, learning, reasoning, and action. One of the central concepts in MSEE is to exploit methods based on minimalist generative grammar, similar to human language, to represent and process visual scenes and actions. Data-driven methods for spatial, temporal, and causal parsing of information are being developed for semantic understanding of scenes and events in unstructured environments along with cognitive processing methods for exploitation and manipulations. The foundational premise of the program is that a comprehensive mathematical framework to describe an integrated SEE system would allow for detailed analysis of its potential performance and serve as a guide to prototype design. Methods will be demonstrated in use cases motivated by defense applications such as intelligence, surveillance and reconnaissance (ISR) and vision-guided robots to perform repairs.
The MSEE program aims to address a growing difficulty: The amount of data collected by DoD sensor systems exceeds the ability of human analysts and current automated decision systems to extract actionable information. This data deluge is pervasive throughout the military and applies to single and multi-modal sensing platforms. Today, evaluation methods rely on feature detection and category classification using individual pipelines for different tasks due to the lack of an effective unified representation. Hence, under the current paradigm, the semantics derived from sensor outputs do not emerge until an analyst has assimilated the data.
As a result of this dynamic, three challenges emerge:
- data's worth can only be evaluated after analysts have interpreted it, and with knowledge of how it will be used;
- prior knowledge, including that which may have accrued during previous processing, is not used during the production of sensor products; in other words, sensors process signals as if they were seeing the world anew at every instant; and,
- in the presence of multiple sensors, analysts must reconstruct a unified view out of sensor output products that were not, in general, designed for integration.
A new approach to sensing is required to confront these shortcomings. An effective representation for recognizing objects, attributes and actions, and for parsing spatial-temporal relational information, would result in scalable platforms capable of autonomous learning, inference, reasoning, planning and execution of complex tasks.
The goal of the Mathematics of Sensing, Exploitation and Execution (MSEE) program is to capture the economy and efficiency that would derive from an intrinsic, objective-driven unification of sensing and exploitation. The foundational premise of the program is that a comprehensive mathematical framework to describe an integrated SEE system exists. Such a theoretical description would allow for detailed analysis of its potential performance, serve as an invaluable guide when constructing a prototype to demonstrate the effectiveness of the approach, and enable quantitative determination of the effective utility of various sensors and sensing modalities.
MSEE includes three planned phases. The goal of Phase I is to create the mathematical foundation for a representation-centric model. The goal of Phase II is to refine the representation constructed in an initial software system able to answer queries related to the content of sensor data from a single modality. The goal of Phase III is to develop a fully integrated, modular system that demonstrates quantitative and qualitative analyses of systems that integrate sensing/perception, exploitation, and execution, with multi-modal sensor data. This implementation-ready software must be To date, performers on MSEE have pursued fundamental research into the nature of stochastic modeling and knowledge representation. These basic tools are being used to build prototype systems. Progress in these areas should greatly advance DoD's ability to build high-performance systems in a number of areas including ISR and supervised robust autonomous systems.