AdRoll.
c. 2014–2018
Real-time pricing and the machinery of scale.
Advertising turns computation into a market decision. Every bid commits resources, every outcome provides evidence, and the market keeps moving while that evidence arrives. The work at AdRoll connected pricing, feedback control, and large-scale data processing, with hands-on engineering from optimization algorithms to storage engines and distributed services.
Pricing a moving market.
The pricing system continuously adjusted each campaign’s buying threshold, or VCPX, against its budget and performance goals. Incoming market data accumulated in price buckets and rolling time windows, maintaining a compact view of what different buying levels could achieve. The controller recalculated those levels as conditions changed, issuing updates when constraints changed or pricing error crossed a configured tolerance.
Buying decisions shaped the evidence available to make the next decision. Below the current threshold, the market was visible; above it, information was sparse. The design accounted for that uneven exposure, the need for exploration, and the delay between spending and observing results. Budget pacing and performance targets became explicit control constraints in a changing market.
Making billions of histories queryable.
A week of funnel data contained more than 256 billion events across 2.6 billion cookie identifiers. Daily compressed logs ran to 10–20 terabytes. Understanding the advertising funnel meant tracking overlapping populations through targeting, bidding, impressions, clicks, and conversions, then asking where opportunities were lost.
Contributions to the underlying event store were paired with a new funnel analytics engine written in C. Event types became compact bit masks; each population used whichever dense or sparse representation required less space. Memory-mapped storage and composable set queries made those populations practical to compare. Partitioning by identifier produced disjoint sets whose counts could be combined across machines without double counting.
Distributed computing, reduced to essentials.
A purpose-built microframework expressed data pipelines as targets, dependencies, and executable recipes. Workers coordinated through shared storage without a dedicated master process. Each worker selected work according to its available resources, and completed tasks could generate new tasks as the computation unfolded.
The same pipeline could run locally during development or across a cluster processing enormous datasets. Familiar tools, including Python, SSH, and object storage, supplied the foundation. The framework kept the machinery small enough to understand and adapt while handling dynamic, distributed workloads.
Engineering the whole path.
The work extended into Erlang service discovery, adaptive connection pools, request batching, and deadline-aware dispatch. Separating logging and prediction services from the bidders allowed those workloads to scale independently, with explicit control over queues, load, and timeouts. Performance depended on decisions at every level: how a byte was represented, where a task ran, and how quickly new information could influence the next purchase.