The Evolution of Data Processing at Spotify
In Spotify Creator we strive to provide artists an accurate view of what their fan base is. We crunch numbers for 100 Million platform monthly active users daily and compute dashboards for millions of artists. As a result we process terabytes of data daily and condense it to about ~200 GB of data. In this talk we will discuss the evolution of our pipelines and how we made them more resilient to the irregularities of data as well as external failures.
November 3, 2016