THE PRODUCT PROBLEM
The anti-fraud platform's Counter Service was struggling with its storage solution, which was becoming a bottleneck as the service scaled to handle billions of requests daily. The business needed a storage system that could support real-time fraud detection with high query volumes, maintaining low latency and high reliability. The existing wide-column database, while reliable, was not optimized for future scalability and flexibility. Competitive pressures demanded a system that could adapt quickly to evolving fraud patterns without downtime, ensuring Grab stayed ahead in the fast-paced ride-hailing market.
THE DECISION
The decision to migrate to Aerospike was driven by the need to enhance scalability and decouple storage from business logic, which would facilitate future changes. This required alignment across PM, engineering, and operations to ensure a seamless transition with no service disruption. The tradeoff involved the complexity of a gradual migration and the risk of potential data integrity issues. Alignment was challenging due to the need to balance innovation with the operational stability of a core service. In hindsight, while the migration was successful, the complexity of maintaining dual storage paths during the transition revealed the importance of robust monitoring and rollback mechanisms.
THE LESSON
This migration highlights the critical role of PM/engineering collaboration in managing technical debt while pursuing innovation. A PM advice blog might not emphasize the importance of decoupling business logic from storage, but this case shows how it can reduce future migration efforts and enhance system agility. The experience underscores the necessity of detailed planning and cross-functional alignment in large-scale infrastructure changes, emphasizing the need for PMs to deeply understand technical constraints and opportunities to drive strategic decisions effectively.