ML - 3-pipeline architecture

Also known as Feature / Training / Inference (F/T/I)

1. Feature pipeline

Transforms your data into features & labels, which are stored and versioned into a feature store

2. Training pipeline

Ingests a specific version of the features & labels and outputs the trained models, which are stored and versioned inside a model registry

3. Inference pipeline

Takes a specific version of the features and models and outputs the prediction to a client

Advantages

• it defines a transparent interface between the 3 components
• the ML system is built with modularity from the start
• components can be divided between teams, if needed
• every component can use the best stack of technologies available for the job
• every component can be scaled, deployed and monitored independently
• the feature pipeline can be either batch, streaming or both
• it guarantees that the model can move out of a notebook and into production

See also

• Data pipelines