Go to content

Sean Braithwaite - Mechanics of Data Pipelines

Further information: https://berlinbuzzwords.de/17/session/mechanics-data-pipelines This talk focused the topic on how to model data pipelines as retroactive, immutable data structures. It covers the topic of how do you build a data pipelines for a growing organization where different teams depend on each others data and need to be able to re-process data when errors occur upstream. I draw comparisons between the microservice architectures for both stream and batch processings and provide some guiding principals towards building resiliant systems based on experience scaling out infrastructure at SoundCloud. Speaker: Sean Braithwaite https://twitter.com/brapse

June 11, 2017