Data Pipelines with Apache Airflow (2 ed)

Data Pipelines with Apache Airflow have empowered countless data engineers to create more robust and efficient data platforms. Now, in its fully updated second edition, this book has been revised to cover Airflow 3, incorporating all the latest features of Apache Airflow, including the Taskflow API, deferrable operators, and Large Language Model integration. Packed with real-world examples and practical scenarios, this guide will take you from an Airflow beginner to an expert in no time.

Through a combination of theory and hands-on examples, this book teaches you how to simplify and automate your data pipelines, reduce operational complexity, and seamlessly integrate every technology in your stack. Blending reference material with tutorials, you'll explore techniques illustrated with engaging, real-world use cases—from training machine learning models for generative AI to optimizing delivery logistics.

In Data Pipelines with Apache Airflow, Second Edition, you'll learn how to:

• Master the foundational principles of Airflow architecture and workflow design.
• Schedule complex data pipelines using the Dataset API and timetables, including irregular schedules.
• Develop custom Airflow components tailored to your specific needs.
• Implement thorough testing strategies to ensure your pipelines run smoothly.
• Adopt industry-leading practices for building and maintaining Airflow workflows.
• Effectively deploy and manage Airflow in production environments.
• Orchestrate workflows in container-native environments.
• Build and deploy Machine Learning and Generative AI models using Airflow.

This comprehensive guide is designed to equip you with the tools, knowledge, and skills needed to master Apache Airflow and create powerful, efficient data pipelines for modern data engineering challenges.