MLOps is not a thing from the future. Trust us.
Just like DevOps, MLOps it’s not only about technology, it’s about processes, their operationalization and essentially the people involved. In this blog post, we will be sharing a maturity model that supports us – Link Redglue, on how to assess the maturity of Machine Learning initiatives inside an organization.
Maturity Model
The maturity model on MLOps it’s a model that we’ve been curating at Link Redglue, watching industrial standards (there are a few of them, including Azure) together with our experience on the technology and along with the reality of our customers. This model shares a set of requirements that should be fulfilled by each perspective, namely: people, processes and technology.
The main objective is to ensure that everyone involved in a MLOps strategy have the same understanding of what is the current status of the organization and eventually knows what the next step will be. Look at it as a reality check for your organization.
| Level | Name | State | People | Process | Technology |
| 0 | No MLOps | Difficult to impossible to manage ML lifecycle.
Releases are difficult and painful |
Teams organized around skillset with silos: Data Engineers and Data Scientists | Data it’s manually prepared
No tracking Manual “everything” |
Manual builds and deployments
Manual training and testing of the models No centralized tool for model tracking |
| 1 | DataOps, no MLOps | Some DataOps pipelines in place made by Data Engineering team
Releases still manual Limited feedback on model’s performance |
Teams organized around skillset with silos: Data Engineers and Data Scientists | Data pipelines gather data and prepare it automatically
No version control of experiments “Single model file” |
Automation of data preparation
Automated builds and deployments Manual training and testing of the models |
| 2 | Automated training | Training environment is fully managed
Easy to reproduce the model Releases still manual |
Data scientists working with data engineers to convert experimentation into job scripts | Data pipelines gather data and prepare it automatically
Code and trained models are version controlled (experiments are tracked) |
Automated model training
Manual testing of the models Model registry tracking and management |
| 3 | Automated model deploy | Releases are automated and faster
CI/CD releases pipelines |
Data scientists working with data engineers to convert experimentation into pipelines | Release management policy in place
Automated releases Code and trained models are version controlled (experiments are tracked) |
Automated model training
Manual testing of the models Model registry tracking and management Possible model test performance |
| 4 | Full MLOps | Full System automation and monitoring
Feedback from production models CI/CD releases pipelines
|
Data scientists working with data engineers to convert experimentation into pipelines | Retraining triggered automatically | Automated model training
Automated testing of the models Model registry tracking and management Centralized performance metrics of the deployed model |
Maturity Model – Considerations
There are a few notes to take regarding this maturity model:
- Level 0/1 is the most common level on organizations (so far). DataOps is for most of them something that they are very aware of, but they are just starting;
- Only Level 2 introduces the concept of Model Registry (Like MLFlow);
- Level 3 is the sweet spot in most of the cases where automatic retraining is not necessary;
- Level 4 is an ambitious goal and it takes time, maturity and a mindset oriented for the 3 Cs (Continuous Integration, Continuous Delivery and Continuous Training).
Maturity Level 4
Bellow is a conceptual representation of MLOps maturity level 4 that includes a model registry, a feature store, performance monitoring of the models and automatic retraining.
If you want to know more regarding MLOps and our experience drop us a message
