This post is not intended to be an exhaustive tour of Prefect’s features, but rather a guide for users familiar with Airflow that explains Prefect’s analogous approach. We prepared this document to highlight common Airflow issues that the Prefect engine takes specific steps to address. We know that questions about how Prefect compares to Airflow are paramount to our users, especially given Prefect’s lineage. We open sourced the Prefect engine a few weeks ago as the first step toward introducing a modern data platform, and we’re extremely encouraged by the early response! Disappointingly, those observations remain valid today. The seed that would grow into Prefect was first planted all the way back in 2016, in a series of discussions about how Airflow would need to change to support what were rapidly becoming standard data practices. It simply does not have the requisite vocabulary to describe many of those activities. ![]() Airflow got many things right, but its core assumptions never anticipated the rich variety of data applications that has emerged. Processes are fast, dynamic, and unpredictable. Compute and storage are cheap, so friction is low and experimentation prevails. Today, many data engineers are working more directly with their analytical counterparts. However, Airflow’s applicability is limited by its legacy as a monolithic batch scheduler aimed at data engineers principally concerned with orchestrating third-party systems employed by others in their organizations. It introduced the ability to combine a strict Directed Acyclic Graph (DAG) model with Pythonic flexibility in a way that made it appropriate for a wide variety of use cases. ![]() ![]() Airflow is a historically important tool in the data engineering ecosystem, and we have spent a great deal of time working on it.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |