Defense Advanced Research Projects AgencyTagged Content List

Data Analysis at Massive Scales

Extracting information and insights from massive datasets; "big data"; "data mining"

Showing 95 results for Data RSS
Popular search engines are great at finding answers for point-of-fact questions like the elevation of Mount Everest or current movies running at local theaters. They are not, however, very good at answering what-if or predictive questions—questions that depend on multiple variables, such as “What influences the stock market?” or “What are the major drivers of environmental stability?” In many cases that shortcoming is not for lack of relevant data. Rather, what’s missing are empirical models of complex processes that influence the behavior and impact of those data elements.
Military commanders responsible for situational awareness and command and control of assets in space know all too well the challenge that comes from the vast size of the space domain. The volume of Earth’s operational space domain is hundreds of thousands times larger than the Earth’s oceans. It contains thousands of objects hurtling at tens of thousands of miles per hour. The scales and speeds in this extreme environment are difficult enough to grasp conceptually, let alone operationally, as is required for commanders overseeing the nation’s increasingly critical space assets.
More than 500,000 pieces of manmade space debris—including spent rocket stages, defunct satellites, and fragments as small as flecks of paint—currently hurtle around the Earth at roughly 17,000 miles per hour. At those speeds, impacts involving even the smallest of those items can damage satellites and spawn chain reactions of collisions, increasing the amount of orbital flotsam and creating “minefields” in space that can remain unpassable for centuries.
As the complexity and volume of global digital data grows, so too does the need for more capable and compact means of processing and storing data. To address this challenge, DARPA has announced its Molecular Informatics program, which seeks a new paradigm for data storage, retrieval, and processing. Instead of relying on the binary digital logic of computers based on the Von Neumann architecture, Molecular Informatics aims to investigate and exploit the wide range of structural characteristics and properties of molecules to encode and manipulate data.
The U.S. government has always had an interest in developing and maintaining a strategic understanding of events, situations, and trends around the world. In recent years, however, information complexity has exceeded the capacity of analysts to glean meaningful or actionable insights as data pours in from disparate sources, across a variety of genres, and a mixture of structured and unstructured forms, from military intelligence to social media to accurate and inaccurate news.