jomit/real-time-prediction

Real-time predictions on manufacturing quality using Azure Machine Learning

Real-time predictions on manufacturing quality using Azure Machine Learning

High level reference architecture:

Business Case

Improve quality and reduce production scrap by predicting the quality of a production run near real time using the machine sensor data coming out of the PLC's / Historian.

Prerequisites

• Active Azure Subscription

• Install Azure CLI

Steps

Training Pipeline

• Create Azure Blob storage Account and upload data\train.csv in a container named mldatasets

• Create a new Data Science Virtual Machine and verify that the Jupyter notebook server is running on port 8000.

• Deploy the Azure ML Workspace using the mlworkspace\template.json template file

  • az group deployment create --resource-group <resource-group-name> --template-file <path-to-template>

• Upload all the notebooks from mlnotebooks folder to the Jupyter notebook server using the Web UI.

• Run all notebooks as per the sequence to build and deploy the initial model.

  • Make sure to update the parameters in the notebook as per the instructions.

  • For production deployments use Azure Kubernetes Service to deploy the scoring api

Inference Pipeline

• Create an Event Hub Namespace and an event hub inside it named qualityprediction

• Create a new Cosmos DB Account and add a new container with following details:

  • Database id : MLPrediction
  • Container id: qualitypredictions
  • Partition key: /batchid

• Create an Azure Function App with Runtime Stack as Nodejs

  • Add new Azure Event hub trigger function with following parameters:

    • Name: PredictQuality
    • Event Hub Connection: <Use the event hub created above>
    • Event Hub Name: qualityprediction
    • The event hub cardinality: One
    • Event parameter name: eventHubMessage

  • Click on Integrate in the function and add a New Output of type Azure Cosmos DB with following parameters:

    • Document parameter name: predictionResultDocument
    • Database name: MLPrediction
    • Collection name: qualitypredictions
    • Azure Cosmos DB account connection: <Choose the Cosmos DB Account created above>

  • Updated the function code with function\index.js. Make sure to replace the scoring api url.

• Setup Edge Gateway \edgegateway which is a nodejs app. (For more complex edge requirements please use Azure IoT Edge along with Azure IoT Hub)

  • Rename the sample-dot-env file to .env

  • Update the eventHubConnectionString and eventHubName values in the .env file

  • (Optional) If integrating with local SQL DB

    • Update the sql related values in the .env file
    • Update code in app.js and sqlserverdb.js

Additional Resources

• Scalability, Monitoring and Security considerations for using Azure Kubernetes Service to deploy ML Models

• Scaling Azure Funcions

• Scaling Azure Event Hubs