Awesome Pipeline

URL: https://github.com/pditommaso/awesome-pipeline
Type: Curated awesome list / themed directory
Category: Themed Directories
Tags: awesome-lists, devops, workflow

Summary

Awesome Pipeline is a curated GitHub list of workflow and pipeline toolkits. It aggregates frameworks, libraries, and related tools for building, managing, and executing pipelines (e.g., for data processing, scientific workflows, ML, CI/CD, and HPC environments). Inspired by the "Awesome Sysadmin" collection, it serves as a specialized directory within the broader awesome-lists ecosystem.

Features

• Curated collection of pipeline toolkits

  • Focused specifically on workflow and pipeline frameworks and libraries.
  • Draws from multiple domains: DevOps, data engineering, scientific computing, ML, and HPC.

• Emphasis on workflow/pipeline orchestration

  • Includes tools for defining, scheduling, and running complex workflows.
  • Covers both linear and branching workflows, as well as parallel and distributed execution.

• Diverse technology stack coverage

  • Python-based systems (e.g., Airflow, Balsam, Autosubmit, etc.).
  • Domain-specific languages and scripting frameworks for pipelines (e.g., Bds / BigDataScript).
  • Container-native and Kubernetes-focused engines (e.g., Argo Workflows).
  • Specialized scientific workflow managers (e.g., Anduril, AiiDA).

• Examples of listed tools
  (Non-exhaustive; drawn from the visible portion of the list)

  • ActionChain – Workflow system for simple linear success/failure workflows.
  • Adage – Describes workflows that are not completely known at definition time.
  • AiiDA – Workflow manager with strong focus on provenance, performance, and extensibility.
  • Airflow – Python-based workflow system originally created by Airbnb.
  • Anduril – Component-based workflow framework for scientific data analysis.
  • Antha – High-level language aimed at biology workflows.
  • Argo Workflows – Container-native workflow engine for parallel data/ML/CI jobs on Kubernetes.
  • Autosubmit – Python experiment and workflow manager for complex Cloud and HPC workflows.
  • AWE – Workflow and resource management system with CWL support.
  • Balsam – Python-based high throughput task and workflow engine.
  • Bds (BigDataScript) – Scripting language designed for data pipelines.

• Open-source and community-driven

  • Hosted on GitHub under the pditommaso/awesome-pipeline repository.
  • Supports contributions via standard GitHub workflow (issues, pull requests), with a CONTRIBUTING.md file.

• Part of the "awesome" ecosystem

  • Styled and organized as an "awesome" list, making it familiar to users of other awesome collections.
  • Aims to highlight maintained, noteworthy, and widely useful pipeline-related projects.

Typical Use Cases

• Discovering and comparing pipeline and workflow engines for new projects.
• Exploring tools suitable for data pipelines, ML orchestration, or scientific workflows.
• Surveying options for Kubernetes-native, HPC-focused, or domain-specific workflows.
• Using as a reference directory when evaluating or teaching workflow technologies.

Pricing

• Free – The repository and its contents are freely accessible as an open-source curated list. There are no pricing plans for using the list itself. (Individual listed tools may have their own licenses and pricing, which are not defined in this repository.)