Skip to content

Category archive for: Big Data

Running Airflow on top of Apache Mesos

Apache Airflow is a wonderful product, possibly one of the best when it comes to orchestrating workflows. Airflow supports different executors for running these workflows, namely,LocalExecutor SequentialExecutor & CeleryExecutor. Out of these, only CeleryExecutor supports distributed execution of these tasks. A typical multi-node cluster setup using CeleryExecutor looks like the following and here is an excellent resource explaining the setup. There…

Continue reading Running Airflow on top of Apache Mesos

The views expressed on this blog are those of the author and do not necessarily reflect the views of his past or future employers in any manner