🔗 Two database migrations and a divorce

Jack Ellis narrates the journey of migrating Fathom Analytics to a new database structure while eliminating significant technical debt. This complex project, which involved moving over 65 billion rows, was accomplished by a single engineer. Ellis discusses the challenges faced, particularly concerning cookie laws, user privacy, and the evolution of their data ingestion model. A pivotal moment arose when they concluded that traditional methods of tracking unique visitors could not align with emerging privacy standards. The team innovated with daily salts and hashing, paving the way for a privacy-first approach in analytics. However, years of technical debt complicated their architecture, leading to issues like broken queries and inability to scale. The article provides detailed insights into the considerations and decisions that shaped this significant migration, reflecting on lessons learned and how they evolved their analytics solutions for better performance and privacy.