Inside the Custom Framework for Managing Airflow Code at Wix with Gil Reich

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Konten disediakan oleh The Data Flowcast. Semua konten podcast termasuk episode, grafik, dan deskripsi podcast diunggah dan disediakan langsung oleh The Data Flowcast atau mitra platform podcast mereka. Jika Anda yakin seseorang menggunakan karya berhak cipta Anda tanpa izin, Anda dapat mengikuti proses yang diuraikan di sini https://id.player.fm/legal.

7M ago 31:02

MP3•Beranda episode

Efficient orchestration and maintainability are crucial for data engineering at scale. Gil Reich, Data Developer for Data Science at Wix, shares how his team reduced code duplication, standardized pipelines, and improved Airflow task orchestration using a Python-based framework built within the data science team.

In this episode, Gil explains how this internal framework simplifies DAG creation, improves documentation accuracy, and enables consistent task generation for machine learning pipelines. He also shares lessons from complex DAG optimization and maintaining testable code.

Key Takeaways:

(03:23) Code duplication creates long-term problems.

(08:16) Frameworks bring order to complex pipelines.

(09:41) Shared functions cut down repetitive code.

(17:18) Auto-generated docs stay accurate by design.

(22:40) On-demand DAGs support real-time workflows.

(25:08) Task-level sensors improve run efficiency.

(27:40) Combine local runs with automated tests.

(30:09) Clean code helps teams scale faster.

Resources Mentioned:

Gil Reich

https://www.linkedin.com/in/gilreich/

Wix | LinkedIn

https://www.linkedin.com/company/wix-com/

Wix | Website

https://www.wix.com/

DS DAG Framework

https://airflowsummit.org/slides/2024/92-refactoring-dags.pdf

Apache Airflow

https://airflow.apache.org/

https://www.astronomer.io/events/roadshow/london/

https://www.astronomer.io/events/roadshow/new-york/

https://www.astronomer.io/events/roadshow/sydney/

https://www.astronomer.io/events/roadshow/san-francisco/

https://www.astronomer.io/events/roadshow/chicago/

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

82 episode

#Tech #Airflow