THE PRODUCT PROBLEM
Ensuring seamless and reliable query processing at Meta's scale was becoming increasingly difficult as the demand for Presto, an SQL query engine, grew. The business needed to maintain high availability and performance for interactive query use cases, where users expect immediate results, while also accommodating automated batch processes. The challenge was to deploy new Presto versions across numerous global clusters without disrupting user experience. Failure to address this could lead to user dissatisfaction and potential competitive disadvantages, as real-time data insights are crucial for decision-making in a fast-paced environment.
THE DECISION
The decision to automate the deployment and management of Presto clusters involved a significant tradeoff between availability and deployment speed. By using a load balancer system, the Gateway, they ensured that no new queries were sent to clusters during updates, but this required careful orchestration to avoid taking down too many clusters at once, which would affect availability. Aligning PM, engineering, and operations was crucial, as the process had to ensure both technical feasibility and user satisfaction. In hindsight, while the automation streamlined operations, it required continuous monitoring and adjustments to maintain balance, highlighting the complexity of scaling infrastructure.
THE LESSON
This experience underscores the importance of dynamic collaboration between PMs and engineers when scaling infrastructure at large organizations. It reveals that while automation can solve many operational challenges, it also introduces new layers of complexity that require ongoing cross-functional alignment. PMs must remain agile, ready to pivot strategies as technological and user needs evolve, and understand that the balance between speed and reliability is a moving target. This case illustrates that successful scaling is not just about technical solutions but also about strategic decision-making and stakeholder alignment.