Any advice on a good tool to use for orchestrating ML pipelines? I've tried some (e.g. Kedro), but none fully satisfied me. We use Hydra for experiment iteration and configuration, because of it's extreme flexibility. However, we would like to place the experimentation phase inside a wider pipeline.
- VertexAI pipelines which is basically the managed version of KubeFlow Pipelines (you could go for KubeFlow if running on prem or in a different cloud than GCP).
- MLFlow Pipelines were recently released, would not necessarily jump in but worth trying out.
- Airflow is nice but it is not the best option when it comes to ML specifically and is more meant for Data Engineering Pipeline Orchestration.
- ZenML is an new ML Pipeline Framework that abstracts the tool you use for orchestration from you and allows plugging in different ones.
Hi!
Any advice on a good tool to use for orchestrating ML pipelines? I've tried some (e.g. Kedro), but none fully satisfied me. We use Hydra for experiment iteration and configuration, because of it's extreme flexibility. However, we would like to place the experimentation phase inside a wider pipeline.
Hi Marco, sorry for taking this long to reply.
There are many, but I like:
- VertexAI pipelines which is basically the managed version of KubeFlow Pipelines (you could go for KubeFlow if running on prem or in a different cloud than GCP).
- MLFlow Pipelines were recently released, would not necessarily jump in but worth trying out.
- Airflow is nice but it is not the best option when it comes to ML specifically and is more meant for Data Engineering Pipeline Orchestration.
- ZenML is an new ML Pipeline Framework that abstracts the tool you use for orchestration from you and allows plugging in different ones.