azure/openai-at-scale

Simple ChatGPT UI application

OpenAI at Scale

🌏 English | 日本語

✨ OpenAI at Scale is a workshop by FastTrack for Azure in Microsoft team that helps customers to build and deploy simple ChatGPT UI application on Azure.

---

🎯 Features

• Chat UI
• Configure system prompts and hyperparameters
• Authenticate with Azure Active Directory and get user information from Microsoft Graph
• Collect application logs with Azure Log Analytics
• Store prompt log data to Azure Cosmos DB

---

🚀 Getting Started

⚒️ Prerequisites

To run locally

• OS - Windows 11, MacOS or Linux

⚠ For Windows client user, please use Ubuntu 20.04 LTS (Windows subsystem for Linux) to run this application.
⚠ GitHub Codespaces is supported as Linux envionment.

• Azure CLI (v4.28.1 or higher)
• Node.js (v16.20 or higher)
• Python (v3.9 or higher)
• git client
• Docker Desktop or any other Docker environment
  • Docker is used for Visual Studio Code Dev Container.

To run on Azure

• Azure subscription
  • Resources
    • Azure OpenAI Service
    • Azure Active Directory application
    • Azure Log Analytics
    • (Optional) Azure Cosmos DB

⚠ Free account is not supported

• Role
  • Contributor role or higher for Azure subscription
  • Permission to create Azure Active Directory application
  • Permission to create Azure OpenAI Service

1. Creating Azure OpenAI Service 🧠

There are some ways to create Azure OpenAI Service, we recommend to use Azure Portal if you are not familiar with Azure CLI.

Before started you need to choose a location for your service. You can find the list of available locations here -> supported location is here : Supported locations

• Azure Portal : You can follow the official document to create Azure OpenAI Service.
• Azure CLI : You can use the following command to create Azure OpenAI Service.

command example

# Set environment variables
export SUBSCRIPTION_ID=<your subscription id>
export RESOURCE_GROUP=<your resource group name>
export LOCATION=eastus #hard coded to avoid confusion, you can change any avaiable location.
export AZURE_OPENAI_SERVICE=<your openai service name>
export AZURE_OPENAI_CHATGPT_DEPLOYMENT=<deployment name of your gpt-35-turbo model>

az login #check your subscription id
az account set --subscription $SUBSCRIPTION_ID
az group create --name $RESOURCE_GROUP  --location $LOCATION
az cognitiveservices account create \
    --name $AZURE_OPENAI_SERVICE \
    --kind OpenAI \
    --sku S0 \
    --resource-group $RESOURCE_GROUP \
    --location $LOCATION \
    --yes
az cognitiveservices account deployment create \
   -g $RESOURCE_GROUP  \
   -n $AZURE_OPENAI_SERVICE \
   --deployment-name $AZURE_OPENAI_CHATGPT_DEPLOYMENT \
   --model-name gpt-35-turbo \
   --model-version "0301"  \
   --model-format OpenAI \
   --scale-settings-scale-type "Standard"

2. Creating Azure Active Directory application 🔑

Follow the steps in register your application to register your application.

• Select Single-page application (SPA) as platform type
• The Redirect URI will be http://localhost:5000 and http://localhost:5173 for local development
• Keep the Application (client) ID and Directory (tenant) ID for later use

(Optional) 3. Creating Azure Cosmos DB 🪐

You can create a Azure Cosmos DB account by following instructions on the Azure docs here and please make sure you enable Analytical Store, more details can be found here.

• Select Core (SQL) as API
• Container name will be chat_log and partition key will be /chat_session_id

4. Deploying to local environment 💻

Environment variables

You need to create .env files from .env.sample to set up your environment variables before you spin up your application.

• app/frontend/.env
  • This file will be used for authentication function by Azure Active Directory SDK.

# Azure Active Directory application
VITE_CLIENTID="<your client id>"
VITE_TENANTID="<your tenant id>"

• app/backend/.env
  • This file will be used for accessing to Azure OpenAI Service and Azure Cosmos DB.

# Azure OpenAI Service
AZURE_OPENAI_SERVICE="<your Azure OpenAI Service endpoint>"
OPENAI_API_KEY="<your Azure OpenAI Service key>"
AZURE_OPENAI_CHATGPT_DEPLOYMENT="<your model deployment>"


# (Optional) Azure Cosmos DB
AZURE_COSMOSDB_ENDPOINT="https://<account_name>.documents.azure.com:443/"
AZURE_COSMOSDB_KEY="<your Azure Cosmos DB access Key>"
AZURE_COSMOSDB_DB="< your Azure Cosmos DB database name>"

⚠ Please use Azure Key Vault to configure environment variables in production environments.

command examples to get environment variables from Azure CLI.

export RESOURCE_GROUP=<your resource group name>
export AZURE_OPENAI_SERVICE=<your openai service name>
export AZURE_OPENAI_CHATGPT_DEPLOYMENT=<deployment name of your gpt-35-turbo model>
export OPENAI_API_KEY=`az cognitiveservices account keys list \
-n $AZURE_OPENAI_SERVICE \
-g $RESOURCE_GROUP \
-o json \
| jq -r .key1`

Python environment

Python is required to run the backend Flask application.

Install Python libraries

cd app/backend
python -m venv ./backend_env
source .backend_env/bin/activate  #bash
pip install -r requirements.txt

Start Backend (Flask)

cd app/backend
flask run --debug #hot reload
#python ./app.py 

Node.js environment

Node.js is required to run the frontend React application.

Install Node.js packages

cd app/frontend
npm install

Start Frontend (React)

For development

npm run dev

For production

npm run build

It is used to optimize and reduce the size of all application files which are deployed in app/backend/static folder.

5. Deploying to Azure ☁️

Deploy to Azure App Service

⚠ Before you run following command, you must run npm run build on app/frontend to set frontend files to backend static dir.

• Example of Azure App Service

  • Deploy app to Azure App Service with easist way.

    cd app/backend
    az webapp up --runtime "python:3.10" --sku B1 -g <Resource Group Name>

  • Deploy Azure App Service Plan and Web App separately.

    • You can deploy an app with above command but the command doesn't allow to change detailed App Service Plan and Web App settings. So if you want to change these settings you can deploy it separately with following command.

    • Create Azure App Service Plan resources

      az appservice plan create -g <Resource Group Name> --is-linux -n <App Service Plan Name> --sku <SKU Name> --location eastus

    • Create WebApp Resource on above App Service Plan

      az webapp create -g <Resource Group Name> -n <WebApp Name> -p <App Service Plan Name> -r "python:3.10"

      ⚡️completely optional: if your system needs to add private endpoint and/or VNET integration, you can add it here with following options.

      • VNET Integration

        # you need create vnet/subnet before execute this command
        az webapp create -g <Resource Group Name> -n <WebApp Name> -p <App Service Plan Name> -r "python:3.10" --vnet <VNET Name> --subnet <Subnet Name>

      • Private Endpoint

        # you need create vnet/subnet webapp before execute this command
        az network private-endpoint create \
          -n <PE Name> \
          -g <Resource Group Name> \
          --vnet-name <VNET Name> \
          --subnet <Subnet Name> \
          --connection-name <Private Endpoint Connection Name> \
          --private-connection-resource-id /subscriptions/SubscriptionID/resourceGroups/myResourceGroup/providers/Microsoft.Web/sites/<WebApp Name> \
          --group-id sites

    • Update redirectURI at aadConfig.ts and rebuild frontend app

      • Update redirectURI with following FQDN, which is webapp endpoint.

        az webapp config hostname list -g <Resource Group Name> --webapp-name <WebApp Name> -o json | jq '.[0].name'

      • Rebuild frontend

        cd app/frontend
        npm run build

    • Before deployed webapp, you must change the environment variable with application settings of Azure App Service.

      az webapp config appsettings set --name <Web App Name> -g <Resource Group Name> --settings SCM_DO_BUILD_DURING_DEPLOYMENT="true"

    • Deploy demo app to WebApp

      cd app/backend
      zip -r deploy.zip .
      az webapp deploy -g <Resource Group Name> -n <Webapp Name> --src-path deploy.zip --type zip

    • After deployed webapp, you must change the environment variables with application settings of Azure App Service.

      az webapp config appsettings set --name <Web App Name> -g <Resource Group Name> --settings OPENAI_API_KEY=<KEY> AZURE_OPENAI_CHATGPT_DEPLOYMENT=<Deployment Model Name> AZURE_OPENAI_SERVICE=<OpenAI Service Name>

6. Configuration ⚙️

Collect application logs with Azure Log Analytics

• Example of Log collection

  • Deploy Azure Log Analytics workspace

  export APP_SERIVCE=<your app service name>
  export LOCATION=<azure datacenter region - eastus, japaneast, etc...>
  export RESOURCE_GROUP=<your resource group name>
  export WORKSPACE=<your log analytics workspace name>
  export DIAGSETTINNG_NAME=<your diagnistics setting name (arbitary)>
  
  az monitor log-analytics workspace create --name $WORKSPACE  --resource-group $RESOURCE_GROUP --location $LOCATION

  • Enable diagnostics setting

  export RESOURCE_ID=`az webapp show -g $RESOURCE_GROUP -n $APP_SERIVCE --query id --output tsv | tr -d '\r'`
  export WORKSPACE_ID=`az monitor log-analytics workspace show -g $RESOURCE_GROUP --workspace-name $WORKSPACE --query id --output tsv | tr -d '\r'`
  
  az monitor diagnostic-settings create \
    --resource $RESOURCE_ID \
    --workspace $WORKSPACE_ID \
  -n $DIAGSETTINNG_NAME \
  --logs '[{"category": "AppServiceAppLogs", "enabled": true},{"category": "AppServicePlatformLogs", "enabled": true},{"category": "AppServiceConsoleLogs", "enabled": true},{"category": "AppServiceAuditLogs", "enabled": true},{"category": "AppServiceHTTPLogs", "enabled": true}]'

(Optional) Store prompt log data to Azure Cosmos DB

The logging chat on Azure Cosmos DB section explains in detail on how chat messages can be logged into Azure Cosmos DB and used in deriving insights further downstream.

---

🙋🏾‍♂️Question and Feedback

You can ask question and feedback about this repo on GitHub Issues.

---

📚 Resources

• ChatGPT + Enterprise data with Azure OpenAI and Cognitive Search
  • This repo is based on this sample code.

---

🤝 Contributing

We are welcome your contribution from customers and internal Microsoft employees. Please see CONTRIBUTING. We appreciate all contributors to make this repo thrive!

openai-at-scale's Issues

npm run build SyntaxError: Unexpected reserved word

Please provide us with the following information:

This issue is for a: (mark with an x)

- [X ] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Node.js environment
Node.js is required to run the frontend React application.

Install Node.js packages
cd app/frontend
npm install
Start Frontend (React)
For development

npm run dev

Any log messages given by the failure

~/openai-at-scale/app/frontend$ npm run build

[email protected] build
tsc && vite build

file:///home/user/openai-at-scale/app/frontend/node_modules/vite/bin/vite.js:7
await import('source-map-support').then((r) => r.default.install())
^^^^^

SyntaxError: Unexpected reserved word
at Loader.moduleStrategy (internal/modules/esm/translators.js:133:18)
at async link (internal/modules/esm/module_job.js:42:21)

Expected/desired behavior

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy

Versions

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

getting "The API deployment for this resource does not exist"

Please provide us with the following information:

This issue is for a: (mark with an x)

- [X] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Followed guide

Any log messages given by the failure

getting this error:

Error: The API deployment for this resource does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.

Expected/desired behavior

response

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)

Versions

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

Anonymous User keep logging after refreshing the page

We have successfully deployed the Chat UI app and consigured SAML for Authentication.

We can with our AD username and password with SAML authentication however when we refresh the page the login changes to Anonymous User.

Looks like something in the code causing this issue

[Discuss]how can we provide deployment procedure?

Attendee need to deploy own services on Azure such as below.

In case attendee run demo locally...

1. Azure Open AI Service(mandatory)

If attendee need to run services on Azure...

1. App Service or Azure Container App or Azure Kubernetes Service
2. Keyvault?
3. ... etc.

Feature Request: Use the Authentication function of the App Service to require user authentication.

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [x] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

I would like to add an authentication function for AzureAD. I have configured the following article as a reference, but do I need to add functionality on the application side?

Configure your App Service or Azure Functions app to use Azure AD login

Any log messages given by the failure

AADSTS9002325: Proof Key for Code Exchange is required for cross-origin authorization code redemption.

Expected/desired behavior

We would like to have unauthenticated users see the Microsoft account login screen and authenticated users be able to use the application.

OS and Version?

macOS (Big Sur)

Versions

Mention any other details that might be useful

The Authentication setting screen of App registrations is attached for your reference.