Becoming an MLOps expert (4)
Through this blog series, we've already learned that if you are investing in data science, you should be looking at Machine Learning Operations (MLOps). But getting started isn't always easy. Before jumping in, it's important to ensure you are well supported on your journey and setting yourself up for success. You will need to understand your organisation's level of maturity in both data and AI, to identify any gaps that exist across people, processes and technology that you may need to invest in
How do I assess my data maturity?
ML is dependent on good data - as the saying goes, 'garbage in, garbage out' When assessing your data maturity there are several questions you need to ask to see what gaps exist:
- How easy is it for a data scientist to access your data?
- How easy is it to add a new dataset? Or to modify or expand the underlying schema?
- How well-defined-and well-established is your data function?
- Do you have strategies and processes around data collection and use? Are you automating these processes?
- Have you defined ownership and roles, with business stakeholder engagement?
How do I assess my AI maturity?
Those who are yet to get started on any ML projects need to get on the ladder fast to start reaping the benefits. For those that have started along the path to ML enlightenment, it's helpful to compare against the following four levels of maturity to see how far you are from ML treasure island:
Experimental: You have some data scientists in your organisation, or have outsourced a project, to trial and experiment with what an ML project might look like. However, you have nothing to prove the value of the algorithm.‚ÄØ
Tactical: You have good foundational skills in the team for data wrangling and analytics, have put some measurements in place and have proved the value of your ML project. However, the focus has been on narrow, simple projects or use cases and you lack the knowledge to properly engineer solutions for live/production environments.
Strategic: You have several ML projects/models deployed and maintained in production which are being used to support everyday business operations. There is some centralised management of the advanced analytics team, and you have some experience of integrating your model into existing workflows and systems.
Transformational: This is ML treasure island, your ideal final stage. Your organisation looks at ML as a strategic investment and seeks to leverage it across business functions - not just as a one-off solution. Everything - from data to experimentation and PoCs to ML deployment and optimisation - is optimised holistically and everyone recognises the value.
How do I build a strategy and roadmap?
Wherever you are on your journey, you will need an ML strategy and roadmap. You need to look across your business and rate opportunities, assessing impact, value, and feasibility. At this stage, look for patterns. The more you can find commonality across different projects and areas of the business, the more you will be able to reuse models, data or methods and accelerate your projects. This will help you to design your roadmap in a more efficient way.
How do I get the most from my data?
Invest in data and data pipeline infrastructure. Within a project, this supports easier transition through the phases from discovery to deployment to support and update. The more critical change here is for that data to be accessible for the next project and other data consumers. The data needs to be in a cloud environment that is accessible, and there needs to be a governance process that reduces the effort of getting access.
What else can I reuse?
MLOps implies evolving to more standardised assets. In some respects, we want "everything as software". There are thousands of tasks across several phases in an ML project. As a result, there are several shared assets and scripts that can be reused from data-wrangling tasks, to standing up databases or virtual machines, to broadening test cases and processes for different model types - such as Infrastructure as a Service (IaaS) or Infrastructure as Code (IaC), such as Terraform. `
Why is versioning so important?
I have talked previously about transparency and auditability. In effect, you need to carefully version almost every step of the process from start to finish – from the datasets and data versions to the model/model configurations, through to the test combinations and so on. This facilitates internal and external understanding and explainability, both in a general sense (how/why does the model add value?), as well as down to the specific sense (which model configuration was used?) within one project. Again, we want asset reuse and knowledge transferred. When there are many ML projects, hundreds of developers and data scientists' versioning enables communication, transparency and explainability across projects and across time.
How do I achieve great all-round performance?
Great all-rounders will have various non-functional requirements - such as speed, scalability, latency or UX - woven in. Technical debt associated with the discovery phases can be purged out through model selection to ensure a better fit with the live environment requirements and software. As you have more models supporting more parts of the business, a more sophisticated view of cost-benefit also starts applying. First, you are moving to having more ML models delivering benefit. Also, the "cost to experiment" and "cost to serve" profiles become both more predictable and lower (per model).
Do you need help?
I have introduced MLOps over several weeks, touching upon the benefits, challenges, and some practical steps to help you get started. However, there is a real depth to ML and MLOps. Many organisations are early in their ML journey, and almost all are grappling with scaling their ML processes. I would always recommend getting advice from people who have been there and done that on ML and MLOps.
Are you interested in finding out how we have operationalised Data and AI solutions for our customers? Check out our‚Machine Learning Operations (MLOps) services page to read our latest case study or speak to an expert.
Sources:
1.Amazon SageMaker, Developer Guide
3.Microsoft MLOps Maturity Model