FlowerPower: Data Pipeline Orchestration

Welcome to the official documentation for FlowerPower, a powerful Python library designed to help you build, configure, schedule, and execute data processing pipelines with ease.

FlowerPower streamlines complex data workflows by integrating the modularity of Hamilton for pipeline logic and the robustness of Redis Queue (RQ) for asynchronous job processing.

Get Started

Ready to dive in? Our Quickstart Guide will walk you through installing FlowerPower and running your first pipeline in just a few minutes.

Core Concepts

FlowerPower is built around a few key concepts that make it both powerful and flexible:

• Modular Pipeline Design: Define your data transformations as a collection of simple Python functions. FlowerPower, using Hamilton, automatically understands their dependencies and assembles them into a Directed Acyclic Graph (DAG).
• Configuration-Driven: Separate your pipeline logic from its execution parameters. Environments, data sources, and pipeline settings are all managed through clear and simple YAML files.
• Job Queue Integration: Scale your data processing by offloading tasks to a distributed job queue. FlowerPower provides a seamless interface for sending, managing, and monitoring asynchronous jobs with RQ.
• Unified Project Interface: Interact with your pipelines through the method that suits you best—a Python API (FlowerPowerProject), a command-line interface (CLI), or a web-based UI for visualization and monitoring.
• Extensible I/O: Easily read from and write to various data sources with built-in and custom I/O plugins, ensuring your pipelines can connect to any data, anywhere.

A Note on Hamilton and RQ

FlowerPower acts as an orchestrator, not a replacement. You will still write your pipeline logic using Hamilton's function-based syntax and interact with job queue concepts from RQ. FlowerPower's role is to connect these two ecosystems, providing a structured project environment and simplifying their combined use.