Skip to main content

Data pipelines with Celery: modular, signal-driven and manageable

Level:
intermediate
Duration:
30 minutes

Abstract

Writing pipelines for processing large datasets has its challenges – processing data within an acceptable time frame, dealing with unreliable and rate-limited APIs, and unexpected failures that can cause data incompleteness. In this talk we’ll discuss how to design & implement modular, efficient, and manageable workflows with Celery, Redis, and signal-based triggering.

We’ll begin by exploring the motivation behind segmenting pipelines into smaller, more manageable ones. The segmentation simplifies development, enhances fault tolerance, and improves modularity, making it easier to test and debug each component. By leveraging Redis as a data store and Celery’s signals, we introduce self-triggering (or looped) pipelines that efficiently manage data batches within API rate limits and system resource constraints. We will look at an example of how we did things in the past using periodic tasks and how this new approach, instead, simplifies and increases our data throughput and completeness. Additionally, this facilitates triggering pipelines with secondary benefits, such as persisting and reporting results, which allows analysis and insight into the processed data. This can help us tackle inaccuracies and optimise data handling in budget-sensitive environments.

The talk offers the attendees a perspective on designing data pipelines in Celery that they may have not seen before. We will share the techniques for implementing more effective and maintainable data pipelines in their own projects.


The speaker

Marin Aglić Čuvić

Marin Aglić Čuvić

Marin Aglić Čuvić is a software engineer adept in building data pipelines using Python Celery and Apache Airflow. He aims to improve the scalability, maintainability, and developer experience on the projects he works on. He is also passionate about data engineering. Other than work, Marin likes writing articles in which he takes a deep dive into the technologies he uses or is passionate about.