Apache Airflow is a wonderful product, possibly one of the best when it comes to orchestrating workflows. Airflow supports different executors for running these workflows, namely,LocalExecutor SequentialExecutor & CeleryExecutor. Out of these, only CeleryExecutor supports distributed execution of these tasks. A typical multi-node cluster setup using CeleryExecutor looks like the following and here is an excellent resource explaining the setup. There…
Tech Talkies
Ramblings of an Inquistive Mind