A client in the medical technology field desires to implement a monitoring system for their machine learning model on the cloud.

Grafana, Prometheus, Azure Monitor, TorchServe, Docker

The medical technology client aims to deploy their machine learning models on Azure cloud for active use by consumers. In a production environment, the behavior of the model can change due to differences in the new data compared to the training data, which can impact its accuracy and performance. To prevent unexpected degradation of critical production models, the client requires a production monitoring system. To effectively manage the end-to-end model lifecycle, observability is essential. Datics was approached to provide a solution to address the existing challenges in the client’s infrastructure and help them achieve their goal.

While major cloud service providers like Azure and AWS offer integrated monitoring solutions like Azure Monitor and Cloudwatch, the integration of machine learning and the cloud is still a relatively new concept. This specific application of cloud technology requires more than what is offered by the integrated monitoring solutions. As a result, the client’s data scientists face challenges in obtaining information about the model’s performance in production, hindering their ability to improve the models. They have been using the Pytorch framework to get machine learning metrics, but this makes it difficult to correlate hardware specifications with the model’s performance as the metrics cannot be viewed together.

Datics Offers End-to-End Solution for Machine Learning Model Monitoring on the Cloud

Datics has found the solution for the client’s need for monitoring their machine learning models in production on the cloud. After researching various monitoring products, the company found Grafana and Prometheus to be the best tools for collecting and visualizing metrics. Both tools are affordable and flexible, with Grafana offering customizable dashboards and native support for a range of datasources, and Prometheus allowing for monitoring of cloud-native applications and infrastructure.

Datics also offers a solution using the TorchServe platform, as the client’s data scientists are already working with the PyTorch framework. TorchServe can collect important model performance metrics and make it available to Prometheus through its API interface. The entire monitoring system is packaged inside a Docker container and can be deployed on any cloud platform. With this solution, the client’s data scientists can have a complete picture of their model’s performance and ensure the continued success of their business.

With the integration of Azure Monitor and Prometheus, the client’s data scientists have gained complete visibility into their cloud-based machine learning models. Using Grafana’s customizable dashboards, the team can now view hardware metrics from Azure Monitor alongside model metrics from Prometheus, providing a complete picture of the performance and health of their models. This solution has not only improved the accuracy and performance of the models, but it has also empowered the IT staff to identify and resolve critical issues in the production environment quickly. The monitoring system, housed in a Docker container, is ready for deployment on any cloud platform, ensuring the client’s end-to-end model lifecycle is fully observable and manageable.