MLOps - Workflow orchestration

Super clear comparison of available products

https://ljvmiranda921.github.io/notebook/2021/05/15/navigating-the-mlops-landscape-part-2/

• Quadrants :
  • whole vs piecemeal
  • artifacts are software-centered vs model-centered

Broad comparison of data ( pipeline ) orchestrators

https://ploomber.io/blog/survey/

Best ratings overall, excluding Ploomber:

• Metaflow
• Prefect

General purpose

Airflow - 24k

N8n-io - 19k

https://n8n.io/

• has desktop app

Node-RED - 18k

https://nodered.org/

Luigi - 15k

Argo - 9k

Container-based, on Kubernetes

Prefect - 7.8k

• Run flow locally

• Create API key

• Install agent

• Switch to cloud backend

• Authenticate to cloud

• Register flow

• Start flow from UI

Specialized for ML

Metaflow

DVC pipelines