Second generation "workflow managers" for big data by Alex Van Boxel
The big data landscape is a fast moving. You quickly outgrow cron as scheduling you jobs and need a workflow manager. But the first generation doesn't cut it anymore. Now you need an agile and extendable tool. The new open source contenters in this space are all Python based, moving away from describing workflows in XML or special DSL's. Having your workflows described in code makes it more versatile. We'll have a look at one of the new open source workflow managers Luigi from Spotify. Then have a quick look at the feature set of the others like Pinball from Pinterest and Airflow from AirBnB.
November 9, 2015