Defense Advanced Research Projects AgencyTagged Content List

Data Analysis at Massive Scales

Extracting information and insights from massive datasets; "big data"; "data mining"

Showing 115 results for Data RSS
Throughout DARPA’s history, artificial intelligence (AI) has been an important area of groundbreaking research and development (R&D). In the 1960s, DARPA researchers completed some of the foundational work in the field, leading to the creation of expert systems, or the first wave of AI technologies. Since then, DARPA has funded developments in the second wave of AI – machine learning – which has significantly impacted defense and commercial capabilities in areas such as speech understanding, self-driving cars, and image recognition.
Rapid comprehension of world events is critical to informing national security efforts. These noteworthy changes in the natural world or human society can create significant impact on their own, or may form part of a causal chain that produces broader impact. Many events are not simple occurrences but complex phenomena composed of a web of numerous subsidiary elements – from actors to timelines. The growing volume of unstructured, multimedia information available, however, hampers uncovering and understanding these events and their underlying elements.
The Department of Defense (DoD)’s Joint Logistics Enterprise, which spans both supply chain and logistics operations, provides the means to muster, transport, and sustain military power anywhere in the world at a high level of readiness. To operate successfully in an increasingly contested global security environment, however, the logistics enterprise needs to change how it operates. In particular, the enterprise needs to overcome its reliance on thousands of disparate legacy information systems, which can’t provide the status of millions of military parts, supplies, and pieces of equipment, which are stocked and shipped around the world.
Computing performance has steadily increased against the trajectory set by Moore’s Law, and networking performance has accelerated at a similar rate. Despite these connected evolutions in network and server technology however, the network stack, starting with the network interface card (NIC) – or the hardware that bridges the network/server boundary – has not kept pace. Today, network interface hardware is hampering data ingest from the network to processing hardware. Additional factors, such as limitations in server memory technologies, memory copying, poor application design, and competition for shared resources, has resulted in network subsystems that are creating a bottleneck within the network stack and are throttling application throughput.
U.S. forces operating in remote, under-governed regions around the world often find that an area’s distinct cultural and societal practices are opaque to outsiders, but are obvious to locals. Commanders can be hindered from making optimal decisions because they lack knowledge of how local socio-economic, political, religious, health, and infrastructure factors interact to shape a specific community.