Apache Airflow is a wonderful product, possibly one of the best when it comes to orchestrating workflows. Airflow supports different executors for running these workflows, namely,LocalExecutor SequentialExecutor & CeleryExecutor. Out of these, only CeleryExecutor supports distributed execution of these tasks. A typical multi-node cluster setup using CeleryExecutor looks like the following and here is an excellent resource explaining the setup. There…

Continue reading Running Airflow on top of Apache Mesos