Welcome to the Cognitive Services Kitchen Sink application. It gives you an overview of how different Azure Cognitive Services are embedded in a TypeScript application using Rest API calls or the JavaScript SDK.
First of all, to use this application, FORK the repository using the fork button on the top right corner of the repository.
Then CLONE this repository to your laptop using the command git clone and the information you find under Code on the top right corner of the repository.
git clone https://github.com/azuredevcollege/cognitive-services-kitchen-sink.gitAfter that, follow the setup instructions for your IDE.
- Install VSCode + the VSCode Extensions Volar (and disable Vetur) and TypeScript Vue Plugin (Volar).
- TypeScript cannot handle type information for
.vueimports by default, so we replace thetscCLI withvue-tscfor type checking. In editors, we need TypeScript Vue Plugin (Volar) to make the TypeScript language service aware of.vuetypes.
Using the Terminal open the cloned repository using VSCode by typing:
code cognitive-services-kitchen-sinkNow that you have the code ready in VSCode, install all necessary packages.
npm installTo test the app locally, you can run the following code.
npm run devDisclaimer: The Speech Service only works by running the web application online. We will get to this later on.
Now that the environment is set up, lets move onto the challenges.
Go to Settings and then Actions , General and under Workflow Permissions make sure that you have set Read and write permissions. If not, change your settings and save.
After the setup from the compile and hot-reload step is complete, the site is not yet published to GitHub Pages (published online). In the next two steps, we will run a workflow to create a new gh-pages branch, and deploy the content from this branch to our GitHub page.
- Activate GitHub Pages and run the workflow to create gh-pages branch:
First, we need to commit something to the main branch, so that the Pages workflow runs. For this, make a simple change to the README file by clicking on the pencil icon. It can be anything you like, adding a space will suffice. Commit your changes directly to the main branch.
Go to Actions and you should see the Pages workflow running.
Once it is done, we can move to step 2.
- Change source to gh-pages:
After the workflow is done, a new branch called gh-pages has been created, which we will use as a source for our GitHub page deployment. Go to Settings, Pages , and set the source to the gh-pages branch and save. You will see under Actions that the Pages and Deployment workflows runs.
Optional: In the meantime, check out this website to learn more about GitHub Pages.
Once it is done, go to your GitHub page. The URL should be YOUR_GITHUB_NAME.github.io/cognitive-services-kitchen-sink/. But you can also find the link on the Settings page of your GitHub Repository.
-
From now on, every time you push a commit to your remote repository (GitHub), the GitHub page will be updated.
You push changes by:
git add *adding the changes in the files of the directory where you run this command to staging for the next commitgit commit -m "comment on what you want to commit"prepares the staged content like a snapshot of the current state of your repositorygit pushpublish and upload the changes you staged and commited to the repo in GitHub. This ought to trigger a GitHub action which will rebuild the GitHub page with the newly added Features
-
In order to be able to use the Cognitive Services on the web application, we need to connect them to a Cognitive Services resource. Therefore, go to the Settings page of your web application and paste in your multi-service Cognitive Services subscription key and select the service's region. Moreover, also paste your Face Service subscription key and endpoint. If you haven't created those resources yet, first deploy them on the Azure Portal.
You might not have a multi-service Cognitive Service deployed in Azure. To do so repeat the steps from the last challenge in the Azure portal, search for Cognitive Services and create the service. This service combines Vision, Language, Search, and Speech Cognitive Services with a single API key. This allows us to use several functionalities without having to enter too many keys.
-
Go to the different pages Speech, Translator, Computer Vision, Form Recognizer and Language to try out the cognitive services in action (We will try out the Custom Vision service in the next challenge.).
Info: The Speech service only works using the GitHub page. It does not work using the local site.
-
Open the source code in Visual Studio code and check it out. Under src > views you can find the different pages of the cognitive services. See how these services are deployed using the JavaScript SDK or Rest API calls written in JavaScript.
As you can see on the drawer on the left, there is a service which you haven't tried out yet - Custom Vision. Now, you will get the chance to test the image classifier which you created in an earlier challenge.
Firstly, go to your flower classifier project in the custom vision studio and click on performance. Then go to the newest training interation and click on publish. You can choose your own Model name or leave the default one. Moreover, choose the custom vision resource which you created earlier. It should end with -Prediction.
Once published, you can find the prediction endpoint under Prediction URL. Copy the endpoint for classfying image URLs and paste it into the space Azure Custom Vision Prediction Endpoint on the settings page of your app.
Moreover, you will need the Azure Custom Vision Published Name and Iteration id. You can find both pieces of information in the custom vision studio on the Performance tab of your published training iteration as Published as and Iteration id.
Lastly, input the Custom Vision Prediction Key, which you can find in the Azure Portal. When you created the Custom Vision resource, another prediction resource was created for you automatically called RESOURCENAME-Prediction. Go to this resource and copy the key from the Keys and Endpoint page.
Now that you have set up the image classfier, go to the Custom Vision page of your app and test it out by copying an image url. You should receive a response.
So far, you can use the Translator Language services in the web application to translate to English, German, French, Hindi and Spanish and analyze the sentiment. Try adding more languages to the translator and language services by changing the source code under src > views > TranslateView.vue and LanguageView.vue.
To program the Translator, we used Rest API calls written in JavaScript. For the language service, we wrote the code using the JavaScript SDK.
Challenge number 4 is a little more tricky. Remember trying out the language service and doing a sentiment analysis in challenge 1? You had to select the language of the text you wrote to do the sentiment analysis on.
This time we will implement automatic language detection. Under src > views > LanguageView.vue alter the source code in such a way that it can detect the language automatically. Check out the JavaScript SDK documentation, which will help you get there.
Challenge 5 - Give the probability of recognized objects and text for Computer Vision (detect objects & text recognition)
Currently, the Computer Vision Service displays 5 information:
- Description of image with an accuracy confidence
- Object position and object tag (here add the confidence)
- Text position and detected content (here add the confidence)
To understand which part of the response from the corresponding APIs is important for you take a look at the entire JSON result and slice it to your needs.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent