Disco.
c. 2009–2012
From scientific computing to the tools of discovery.
The path to Disco began with biological data. At Northwestern’s Biomedical Informatics Center, work on microarray processing and early next-generation sequencing brought scientific questions into contact with the practical limits of computation. Applying Hadoop to bioinformatics opened a way to distribute that work across many machines. It drew attention upstream, toward the tools themselves: how much easier could it be to express an analysis, carry it out at scale, and understand the result?
Making the value visible.
In 2009, that interest led to the Data Insight team at Nokia Research Center in Palo Alto. It was a tiny team on a mission to convince a vast company to make better use of its data. The work combined large-scale processing with striking visualizations that made the value visible. An analysis needed to reveal something worth paying attention to, and its presentation needed to make people want to look.
Data came from some of Nokia’s largest services, including Maps and its proxy browser. Working at that scale meant building the means to investigate it: distributing computation, organizing the results, and making them accessible to further questions. The team’s mission gave the infrastructure a purpose. Better tools could help people inside the company see possibilities in data they already had.
A small interface to a large machine.
Disco brought together Python’s expressiveness and Erlang’s approach to reliable distributed systems. An analysis could be written as a few functions while the framework distributed tasks, tracked their progress, and recovered work when a machine failed. At Nokia Research Center, Disco ran on an 800-core cluster, with the framework handling datasets measured in tens of terabytes. Keeping the interface small made that computational reach available without requiring every experiment to become an infrastructure project of its own.
Contributions to Disco and the collaborative design of Discodex carried the work from processing into indexing. Discodex organized raw data into distributed indexes that could be queried without repeatedly scanning the entire collection. Replicated storage kept datasets and results available for reuse. These pieces supported a longer conversation with the data, from an initial computation through more focused questions and visual exploration.
From URL trends to Nokia Trends.
Analysis of URL trends opened a path from the research tools to a consumer product. Nokia Trends turned that work into a standalone web application, bringing patterns of interest on the web to an audience far beyond the research center. It found a worldwide following and became one of the top web applications in Nigeria and India.
The effort to demonstrate the value of data had produced something people chose to use for themselves. The work now spanned the whole distance from processing service data to designing a public-facing experience. Infrastructure, analysis, and presentation each shaped what the others could achieve.
The beginnings of Quasi Convex.
This period helped establish a lasting interest in the relationship between powerful computation and intuitive tools. It runs through viz and vzi as well: giving an idea a direct route into code, and giving the result a form that invites exploration. Disco’s open-source collaboration extended that opportunity to people working on questions well beyond Nokia.
The work led to a deeper study of operations research and convex optimization, extending the interest in understanding data into the mathematics of making better decisions. Quasi Convex takes its name from that exploration. The connection reaches back to this small team in Palo Alto, and the possibilities that opened up when the tools themselves became part of the inquiry.