Semantically-Aware ISR | SBIR

OUSD (R&E) critical technology area(s): Trusted AI and Autonomy, Advanced Computing and Software, Human-Machine Interfaces, Integrated Sensing and Cyber

The technology within this topic is restricted under the International Traffic in Arms Regulation (ITAR), 22 CFR Parts 120-130, which controls the export and import of defense-related material and services, including export of sensitive technical data, or the Export Administration Regulation (EAR), 15 CFR Parts 730-774, which controls dual use items. Offerors must disclose any proposed use of foreign nationals (FNs), their country(ies) of origin, the type of visa or work permit possessed, and the statement of work (SOW) tasks intended for accomplishment by the FN(s) in accordance with the Announcement. Offerors are advised foreign nationals proposed to perform on this topic may be restricted due to the technical data under US Export Control Laws.

Objective: Develop and demonstrate a mission-aware semantic communications capability for low-SWaP tactical Intelligence, Surveillance, and Reconnaissance (ISR) platforms that reduces transmitted multimodal ISR data while preserving mission-relevant context under degraded, disrupted, intermittent, and limited communications. Phase II prototypes should demonstrate threshold = 90 percent (10X) reduction versus full-frame mission-video baselines with preserved mission utility, with objective = 95-99 percent reduction in mission-suitable scenes, traceable and object-agnostic selection of regions and events, real-time embedded execution, and a path to = 2 W incremental compute power on representative Group 1 and Group 2 unmanned aircraft systems or surrogates.

Description: Effective ISR in contested theaters depends on the ability to extract and transmit decision-relevant information from sensors under severely constrained communications. Modern multimodal sensor payloads, including electro-optical (EO), infrared (IR), event-camera, low-light, and platform-telemetry streams, produce data volumes orders of magnitude beyond what tactical communications links can carry, particularly when those links are degraded, intermittent, jammed, emission-constrained, or forced to single-digit kilobits per second by electronic warfare. The operational consequence is binary: an ISR platform either delivers mission-relevant evidence to the operator under these conditions, or it provides limited operational value.

Small, unmanned aircraft systems (sUAS) embody this challenge in its most acute form. Representative Group 1 platforms such as RQ-11B Raven and RQ-20 Puma-class systems carry EO and IR video for dismounted operators with single-digit-watt payload compute budgets. Representative Group 2 platforms such as ScanEagle-class systems provide broader reconnaissance with greater range, endurance, and modular sensors, but remain size, weight, and power (SWaP) constrained. Across both classes, full-motion EO and IR video routinely exceeds available tactical link capacity, particularly when multiple unmanned systems operate simultaneously.

Three concurrent technical failure modes define this problem space and rule out incremental approaches:

• Mission intent is vague and operator-defined. Operators do not request a single class label. They request answers to compositional, context-dependent tasks such as identifying indicators of hostile staging along a ridgeline, judging whether a gathering is consistent with prior pattern-of-life, or noticing whether access patterns to a structure have changed in the last twenty minutes. There is no fixed object dictionary, and what is mission-relevant in one engagement is clutter in the next.
• Regions of interest are not necessarily objects. Mission-relevant regions are frequently not detectable as discrete objects in any single frame. Relevant evidence may be a vehicle stopped where vehicles do not normally stop, a crowd dispersing unusually fast, a heat signature appearing at an unusual time, a person loitering near an asset, or a pattern of vegetation disturbance accumulated over many frames. Object detectors and large vision-language models are weak in this regime because the evidence is distributed across space and time rather than localized in a single frame.
• The platform cannot host heavy artificial intelligence. Group 1 payload compute is typically constrained to single-digit watts; Group 2 platforms allow modest growth but remain size, weight, and power constrained. Large vision-language models and conventional deep-learning trackers are not deployable at this power envelope and are additionally fragile under domain shift, adversarial obscuration, and data-poor conditions characteristic of low-density and low-prior threats.

This topic seeks a new class of mission-aware semantic communication systems that can approach two orders of magnitude data reduction in mission-suitable ISR scenarios while preserving mission-relevant context and semantic content. The required capability is not video compression, object detection, or target tracking; it is an onboard, real-time, low-power information-processing layer that understands operator intent, reasons over spatial and temporal context, captures behavioral evidence distributed across frames, and emits only the compact semantic packets needed for receiver-side reconstruction. Semantic packets shall combine high-fidelity regions of interest, tracklets, scene-change descriptors, temporal evidence, low-rate background context, confidence, uncertainty, and traceable explanations sufficient for the operator to act on the evidence at substantially lower bandwidth.

Phase II prototypes should demonstrate the following capabilities:

• Mission-intent grounding. Translate vague natural-language or structured operator objectives into compact, executable mission cards that identify relevant cues, contextual relationships, temporal patterns, negative evidence, and uncertainty thresholds. Support operator updates over a constrained link without mission-specific retraining during flight.
• Open-world spatiotemporal reasoning with adaptive ROI coding. Identify regions, events, and behaviors that are mission-relevant even when object categories are unknown, low-resolution, occluded, or camouflaged, by reasoning over multiple frames, tracks, scene context, motion fields, and temporal change rather than per-frame detections. Allocate bits dynamically among high-fidelity ROI patches, event summaries, background context, descriptors, and metadata such that the transmitted payload is reconstructable into operator-usable products at the receiver.
• Multimodal semantic fusion. Extend reasoning beyond single-modality EO video. By end of Phase II, demonstrate EO video plus at least one additional modality or metadata source: long-wave infrared, low-light video, event-camera streams, platform telemetry, gimbal or inertial state, acoustic cueing, or RF or link-state metadata.
• Traceability and operator trust. Each semantic transmission decision should include a compact evidence chain comprising mission clause, cue type, temporal support, spatial support, confidence, and uncertainty. The receiver should enable operator audit of why information was transmitted, suppressed, or summarized. This is a hard requirement intended to distinguish the deliverable from black-box neural compression.
• Low-SWaP embedded execution. Operate on edge hardware compatible with small UAS payload constraints. Threshold = 5 W incremental power during interim testing; final objective = 2 W average incremental power excluding camera and radio. Implementations may use quantized neural networks, vector symbolic or hyperdimensional accelerators, FPGA, low-power NPU, microcontroller co-processing, neuromorphic devices, or custom hardware-software co-design. Representative embedded targets include the NVIDIA Jetson Orin Nano in low-power profiles, Hailo-8L, Coral Edge TPU, AMD Kintex UltraScale+ FPGA fabrics, and ARM Cortex-M-class microcontrollers for symbolic primitive execution.
• Robustness under contested operation. Maintain performance under motion blur, compression artifacts, clutter, low contrast, sensor noise, partial occlusion, camera motion, lighting change, intermittent links, packet loss, and adversarially confusing backgrounds. Robustness shall be characterized quantitatively, not asserted.
• Cyber posture and supply chain integrity. Deliver a software bill of materials, secure update path, documented interfaces, and cybersecurity assessment suitable for transition review. The deployable system shall not introduce avoidable vulnerabilities into tactical networks or mission systems.

Approaches of interest may combine lightweight neural perception with mathematically structured spatiotemporal reasoning, including vector symbolic architectures, hyperdimensional computing, temporal graphs, probabilistic scene memory, causal or relational models, neurosymbolic methods, or other compact online-reasoning approaches. Such methods are relevant because they can support compositional representation, efficient similarity search, noise tolerance, online learning, and traceable reasoning under tight power budgets. The Government does not prescribe a specific algorithmic stack; proposers may employ any combination of techniques meeting the performance, traceability, robustness, and SWaP requirements.

Approaches not of interest include solutions relying solely on conventional video compression without mission-aware reasoning; solutions relying solely on object detection or tracking with a fixed taxonomy; solutions relying solely on large vision-language models that cannot execute onboard; approaches requiring cloud processing, continuous high-bandwidth reachback, or extensive per-mission retraining during flight; and approaches requiring replacement of existing UAS flight controllers, autopilots, radios, or ground-control stations. Weapon-release, lethal engagement, and named-person identification capabilities are explicitly excluded from this topic. The deliverable system shall not autonomously make lethal engagement decisions, shall not identify named persons, and shall not serve as a weapon-release authority. The intended use is bandwidth-efficient ISR sensing, operator cueing, and semantic preservation under constrained communications.

Phase I

This topic is soliciting Direct to Phase II proposals only. Proposers shall provide documentation that Phase I-equivalent feasibility has been achieved outside the SBIR program. The projected technical readiness level at start of Phase II shall be 4 or above for the perception, reasoning, and semantic encoding subsystems. Feasibility documentation shall include evidence of the following:

• A prior prototype that ingests EO video and produces ROI or semantic transmissions with measured data reduction relative to full-frame video, including at least one task where ROI identification depends on behavior change, persistence, or context rather than per-frame object class.
• Preliminary measurements showing a credible path to = 90 percent reduction in transmitted ISR video data while preserving mission-relevant information.
• Documented edge deployment of at least one critical block of the perception or reasoning pipeline on an embedded inference target representative of small UAS payload compute, with measured (not simulated) power and latency.
• Preliminary comparison against relevant baselines such as H.264, H.265, or AV1 compression, object detection and tracking pipelines, learned video compression, and lightweight vision-language perception.
• A path-to-platform analysis identifying at least two specific Group 1 or Group 2 sUAS targets or surrogates with payload compute element, interface control document or SDK access status, integration risks, and a regulatory, cybersecurity, intellectual property, and commercialization posture sufficient to support Phase III transition.

Phase II

Phase II will develop, integrate, test, and demonstrate a mission-aware semantic communication prototype for low-SWaP ISR platforms. The base period of performance is 18 months. A 6-month option period may be exercised to extend the system to multi-UAS semantic coordination, small-satellite or high-altitude-platform downlink emulation, ruggedized payload packaging, and Government-led field demonstration support. Phase II should mature the technology to a field-relevant prototype that ingests live or recorded UAS video, reasons over mission-defined spatiotemporal context, generates semantic packets, transmits under constrained bandwidth, reconstructs mission-relevant information at the receiver, and operates at low power on embedded hardware.

Phase II prototypes should integrate with two representative platform classes: (1) Group 1 short-range reconnaissance small UAS such as RQ-11B Raven or RQ-20 Puma, and (2) Group 2 longer-endurance tactical UAS such as ScanEagle. Government-furnished platforms are not required for proposal submission. Performers may use representative surrogate airframes, software-in-the-loop, hardware-in-the-loop testbeds, recorded UAS video, captive-carry tests, and emulated degraded links. By end of Phase II base, performers should demonstrate integration with both representative classes or Government-approved surrogates. Phase II performance should meet the threshold and objective metrics in Table 1. Final thresholds will be negotiated at Phase II kickoff.

Table 1. Phase II Performance Metrics.

Mission utility should be evaluated using a combination of operator scoring, ROI/event recall and precision, track continuity, time-to-cue, semantic reconstruction quality, and downstream task performance relative to the original sensor stream.

Phase II fixed payable milestones for this program should include:

• Month 1: Kickoff and System Requirements Review. SRR package defining mission scenarios, target platform classes, sensor and link interfaces, mission-card schema, semantic packet schema, embedded hardware down-select, evaluation metrics, baseline comparison methodology, risk register, cyber and safety plan, power measurement plan, and integration plan for two representative UAS classes.
• Month 3: Mission-intent and semantic architecture. Initial mission-intent interface and semantic packet format demonstrated on recorded ISR video. Architecture report defining the mission-card representation, spatiotemporal reasoning approach, ROI and event scoring pipeline, receiver-side reconstruction concept, and power and latency budget.
• Month 6: Object-agnostic spatiotemporal reasoning demonstration. ROI and event selection on = 3 recorded ISR-style scenarios including = 1 driven primarily by temporal or behavioral cues rather than object class. Quantitative comparison against object-detector-only and conventional video-compression baselines. Demonstrate = 90 percent data reduction in at least one scenario while preserving mission-relevant ROIs.
• Month 9: Embedded low-SWaP prototype. Port semantic reasoning and packetization to representative embedded hardware. Measured latency, memory footprint, compute utilization, thermals, and power. Threshold = 5 W on COTS embedded hardware; credible path to = 2 W incremental module power. EO and IR ingestion or fusion in laboratory testing.
• Month 12: Group 1 UAS integration demonstration. Integration with Group 1 short-range reconnaissance platform or surrogate. Live, captive-carry, or hardware-in-the-loop processing of EO video under constrained bandwidth. Mission-intent-driven ROI transmission, receiver-side reconstruction, and traceability of transmitted regions. Demonstrate = 90 percent transmitted-data reduction in small-unit ISR scenarios.
• Month 15: Group 2 UAS integration and constrained-link testing. Integration with Group 2 medium-range reconnaissance UAS or surrogate. Broader-area ISR with EO plus one additional modality or metadata source. Adaptive operation under emulated link budgets at 1 Mbps, 100 Kbps, 10 Kbps, and 1 Kbps tiers, and on at least one withheld mission scenario. Power optimization report toward the 2 W objective.
• Month 18: Phase II base capstone and technical data package. Capstone demonstration across two representative UAS classes or approved surrogates. Achieve = 90 percent data reduction at iso-utility, with 20 to 30 dB stretch in mission-suitable scenes, mission-utility preservation, object-agnostic ROI detection, receiver-side reconstruction, explainable ROI selection, constrained-link adaptation, and measured low-SWaP operation. Deliver final base-period report, software, API documentation, semantic packet specification, test data package, power and latency report, software bill of materials, cybersecurity assessment, user manual, transition plan, and commercialization plan.
• Month 21 (option): Cooperative multi-UAS semantic sharing. Extend the system to semantic coordination across two or more UAS feeds. Demonstrate cross-platform track and ROI consistency, semantic de-duplication, and prioritized transmission across overlapping or complementary mission regions.
• Month 24 (option): Operationally realistic capstone and productization package. Operation in a more realistic ISR scenario with dynamic mission-intent updates, constrained links, multimodal evidence, and onboard low-power execution. Support at least one Government-organized live demonstration where feasible. Productization package: ruggedization plan, hardware bill of materials, manufacturing cost estimate, expanded cybersecurity assessment, transition-partner feedback, and Phase III roadmap.

Phase III dual use applications

Phase III work should be oriented toward transition and commercialization of the resulting mission-aware semantic communications capability. Funding shall be obtained from the private sector, a non-SBIR Government source, or both. The expected product form is a small hardware-software module, embedded software development kit, or sensor-pipeline plugin integrable with ISR payloads, ground control stations, small UAS, unattended sensors, small satellites, and tactical communications systems. Department of Defense applications include small UAS ISR, route reconnaissance, perimeter security, convoy overwatch, maritime and littoral surveillance, force protection, contested logistics, manned-unmanned teaming, sensor-to-shooter support, and low-bandwidth intelligence dissemination. Candidate transition partners include Service science and technology organizations supporting Group 1 and Group 2 sUAS programs, DARPA and Service portfolios concerned with command, control, communications, computers, combat systems, intelligence, surveillance, reconnaissance, and targeting under degraded, disrupted, intermittent, and limited (DDIL) conditions, the Defense Innovation Unit Blue UAS ecosystem, and the Office of Naval Research and Air Force Research Laboratory ISR portfolios. Commercial applications include wildfire monitoring, search and rescue, law-enforcement overwatch, critical infrastructure inspection, border and port security, offshore energy monitoring, precision agriculture, wildlife monitoring, disaster assessment, and satellite or high-altitude remote sensing. Productization may include embedded software licenses for drone original equipment manufacturers, FPGA and system-on-chip accelerator intellectual property, rugged plug-in payload processors, EO and IR payload software development kits, semantic packet and reconstruction application programming interfaces for command-and-control systems, and ground-station analytics that receive, reconstruct, and audit semantic ISR packets. A small-satellite or high-altitude-platform variant applies the same technology to downlink reduction in remote-sensing imagery.

References

1. Gündüz, D.; Qin, Z.; Aguerri, I. E.; Dhillon, H. S.; Yang, Z.; Yener, A.; Wong, K. K.; Chae, C.-B. Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications. IEEE Journal on Selected Areas in Communications, vol. 41, no. 1, pp. 5-41, 2023.
2. Shao, J.; Zhang, X.; Zhang, J. Task-Oriented Communication for Edge Video Analytics. IEEE Transactions on Wireless Communications, vol. 23, no. 5, pp. 4141-4154, 2024.
3. Shao, J.; Zhang, X.; Zhang, J. Task-Oriented Communication for Edge Video Analytics. IEEE Transactions on Wireless Communications, vol. 23, no. 5, pp. 4141-4154, 2024. 3. Yun, S.; Hassan, R.; Masukawa, R.; Imani, M. MissionHD: Data-Driven Refinement of Reasoning Graph Structure through Hyperdimensional Causal Path Encoding and Decoding. arXiv:2508.14746, 2025.
4. Yun, S.; Chen, H.; Masukawa, R.; Errahmouni Barkam, H.; Ding, A.; Huang, W.; Rezvani, A.; Angizi, S.; Imani, M. HyperSense: Hyperdimensional Intelligent Sensing for Energy-Efficient Sparse Data Processing. Advanced Intelligent Systems, 2024.
5. Schlegel, K.; Neubert, P.; Protzel, P. A Comparison of Vector Symbolic Architectures. Artificial Intelligence Review, vol. 55, no. 6, pp. 4523-4555, 2022.
6. U.S. Navy and NAVAIR. Group 1 Small Unmanned Aircraft Systems (RQ-11B Raven, RQ-20 Puma) and U.S. Air Force ScanEagle fact sheets.
7. Defense Innovation Unit. Blue UAS Cleared List, U.S. Department of Defense.

Keywords

Semantic communications; mission-aware ISR; low-SWaP AI; edge AI; unmanned aircraft systems; EO/IR sensing; spatiotemporal reasoning; hyperdimensional computing; vector symbolic architectures; object-agnostic perception; region-of-interest

TPOC-1-PoC

DARPA BAA Help Desk

Email

SBIR_BAA@darpa.mil

Ready to apply?

For additional information and to submit your full proposal package, visit the DSIP Portal.