azure-samples/azure-search-knowledge-mining

Azure Search Knowledge Mining Accelerator

page_type	languages	products	description	urlFragment
sample	csharp javascript html powershell	azure azure-ai-search	Welcome to the Knowledge Mining Solution Accelerator!	azure-search-knowledge-mining

Knowledge Mining Solution Accelerator

Important

As of November 15, 2023, Azure Cognitive Search has been renamed to Azure AI Search.

About this repository

Welcome to the Knowledge Mining Solution Accelerator! This accelerator provides developers with all of the resources they need to quickly build an initial Knowledge Mining prototype with Azure AI Search. Use this accelerator to jump-start your development efforts with your own data or as a learning tool to better understand how you can use AI Search to meet the unique needs of your business.

In this repository, we've provided you with all of the artifacts you need to quickly create a AI Search Solution including: templates for deploying the appropriate Azure resources, assets for creating your first search index, templates for using custom skills, a basic web app, and PowerBI reports to monitor search solution performance. We've infused best practices throughout the documentation to help guide you. With AI Search, you can easily index both digital data (such as documents and text files) and analog data (such as images and scanned documents).

Note: This guide uses the AI enrichment feature of AI Search. AI enrichment allows you to ingest many kinds of data (documents, text files, images, scanned docs, and more), extract their contents, enrich and transform it, and then index it for exploration purposes. To learn more about this feature, see the AI Enrichment in AI Search doc.

Once you're finished, you'll have a web app ready to search your data.

Prerequisites

In order to successfully complete your solution, you'll need to gain access and provision the following resources:

• Azure subscription - Create one for free
• Visual Studio 2019 or later - Community edition or higher
• Postman for making API calls
• Documents uploaded to any data source supported by Azure Search Indexers. For a list of these, see Indexers in Azure AI Search. This solution accelerator uses Azure Blob Storage as a container for source data files. You can find sample documents in the sample_documents/ folder.

The directions provided in this guide assume you have a fundamental working knowledge of the Azure portal, Azure Functions, Azure AI Search, Visual Studio and Postman. For additional training and support, please see:

• Introduction to Azure AI Search
• AI enrichment in AI Search documentation

Process overview

Clone or download this repository and then navigate through each of these folders in order, following the steps outlined in each of the README files. When you complete all of the steps, you'll have a working end-to-end solution that combines data sources with data enrichment skills, a web app powered by Azure AI Search, and intelligent reporting on user search activity.

00 - Resource Deployment

The contents of this folder show you how to deploy the required resources to your Azure subscription. You can do this either through the Azure portal or using the provided PowerShell script.

Alternatively, you can automatically deploy the required resources using this button:

If you need to deploy to Azure Government, check out the Azure Government version of the Knowledge Mining Solution Accelerator.

01 - Search Index Creation

This folder contains a Postman collection that you can use to create a search index. The collection is pre-configured to take advantage of out-of-the-box AI Search functionality.

We recommend using this collection to create an initial search index and then iterating by editing the postman collection and adding custom skills as needed.

02 - Web UI Template

This folder contains a basic Web UI Template, written in .NET Core, which you can configure to query your search index. Follow the steps outlined in the Web UI Template README file to integrate your new search index into the web app.

03 - Data Science & Custom Skills

This folder contains examples and templates to add your own custom skills to your solution. These custom skills help to align the solution to the needs of your particular use case. This step is entirely optional and may be skipped if not needed.

For additional samples and information on custom skill development, see the Custom skill documentation. .NET Azure Function Custom Skills have moved to the Power Skills repository.

04 - Reporting

This folder contains pre-built PowerBI reports that you can use to monitor your solution and to understand user search behavior. They leverage data captured through Application Insights and can be modified to meet your particular business objectives. This step is entirely optional and may be skipped if not needed.

Sample Documents

This folder contains a small data set in a variety of file formats that you can use to build your solution if you don't have another data set available.

Workshop

Become an Azure AI Search expert in a day! This folder contains a self paced workshop that teaches you everything you need to know. Most developers with Azure familiarity should be able to complete the majority of the modules in the training within 8 hours.

License

Please refer to LICENSE for all licensing information.

azure-search-knowledge-mining's Issues

Postman collection Environment variables is not correct

Looks like someone may have overwritten the Postman Environment variables file with the actual collection. So you are unable to import the env. settings file currently. Please fix.

Container Registry and Image Name Not Supplied

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [x] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

In 02 - Web UI Template under 2. Run the UI, the following property is not supplied.: ACR_NAME/IMAGE_NAME.
It is not possible to run the pre-built docker image without the reference to its container registry and name.

Any log messages given by the failure

None

Expected/desired behavior

That the correct container registry and name be supplied by Microsoft.

OS and Version?

Windows 11

Versions

None

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

Any roadmap of Web UI for nodejs or another language?

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [x] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Any roadmap of Web UI for nodejs or another language?

link of Postman documentation needs update

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [x ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

on the https://github.com/Azure-Samples/azure-search-knowledge-mining/tree/master/01%20-%20Search%20Index%20Creation the link to postman documentation needs to be updated.

the link is https://learning.postman.com/docs/postman/launching-postman/introduction/

Any log messages given by the failure

Expected/desired behavior

link should work

OS and Version?

Windows 10

Versions

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

Module 4 instructions in the workshop extremely confusing

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [x] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

The section titled "Add a new record to the Shaper Skill, and add a new Table Projection" is extremely confusing. When I even try to add 'new diseases input into the Shaper Skill' with what I can gather I get the error pasted below:

Any log messages given by the failure

{
"error": {
"code": "",
"message": "Enricher definitions create a cyclic dependency"
}
}

Expected/desired behavior

The instructions should clearly describe the steps to add new disease inputs and new table projections.

OS and Version?

All OSes/versions

Versions

Current version

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

FileNotFoundException: Could not find file '/app/wwwroot\css\navbar.css' when click 'Customize' menu item

Please provide us with the following information:

This issue is for a: (mark with an x)

- [x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

click 'Customize' menu item

Any log messages given by the failure

FileNotFoundException: Could not find file '/app/wwwroot\css\navbar.css'.
Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, string path, bool isDirectory, Func<ErrorInfo, ErrorInfo> errorRewriter)
Microsoft.Win32.SafeHandles.SafeFileHandle.Open(string path, OpenFlags flags, int mode)
System.IO.FileStream.OpenHandle(FileMode mode, FileShare share, FileOptions options)
System.IO.FileStream..ctor(string path, FileMode mode, FileAccess access, FileShare share, int bufferSize, FileOptions options)
System.IO.File.AsyncStreamReader(string path, Encoding encoding)
System.IO.File.InternalReadAllTextAsync(string path, Encoding encoding, CancellationToken cancellationToken)
CognitiveSearch.UI.Controllers.AdminController.ReadCssColorProperties(string section) in AdminController.cs
CognitiveSearch.UI.Controllers.AdminController.Customize() in AdminController.cs
Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor+TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, object controller, object[] arguments)
System.Threading.Tasks.ValueTask.get_Result()
System.Runtime.CompilerServices.ValueTaskAwaiter.GetResult()
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask actionResultValueTask)
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, object state, bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(ref State next, ref Scope scope, ref object state, ref bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, object state, bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(ref State next, ref Scope scope, ref object state, ref bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Logged|17_1(ResourceInvoker invoker)
Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

Expected/desired behavior

No error

OS and Version?

Azure App Service

Versions

241628

Mention any other details that might be useful

Clicking on the other two menu items are fine. Local debug in Visutal Studio is fine.

---

Thanks! We'll be in touch soon.

Cognitive Search UI Template error

This issue is for a: (mark with an x)

- [x ] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Generated the Azure services in previous modules
Updated the appsettings.json
Ran the program and as soon as the web page load and I type a search value in the search field and press enter, the program stops with the following error

Any log messages given by the failure

'Exception User-Unhandled' - System.UriFormatException: 'Invalid URI: The URI is empty' on line 413 in file DocumentSearchClient.cs where it is trying to create a new BlobContainerClient

Expected/desired behavior

OS and Version?

Windows 10

Versions

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

Document not opening while clicking in it

While I am clicking in any doc i m getting this error:

and the error is: "Cannot read property 'metadata_storage_path' of null"

Update from azure.search 11.4.0-beta.7 to 11.4.0 breaks Classes

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ X ] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ X ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

I have copied the Cognitive Search UI solution several times today and tried to use it as a basis for a new app by copying the code over side by side. I couldnt make sense why the class "SearchResults" on the "DocumentSearchClient.cs" file was breaking. After reloading the source project a few times to compare and contrast, I noticed that the package is using the 11.4.0-beta.7 of Azure.Search.Documents. I did a clean install of that solution and did an upgrade to 11.4.0 and it instantly reproduced the same error.

Any log messages given by the failure

"SearchResults" does not contain a definition for 'Answers' and no accessible extension method 'Answers'
'SearchQueryType' fores not contain a definition for 'Semantic'

Expected/desired behavior

It would be great to have a working example that was using the stable relase instead of a beta release. I even tried to manually find the beta release just to get the example working in my own app, but couldn't find it.

OS and Version?

Windows 11

Versions

Visual Studio 2022

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

Clinical Trials Example SAS URI not working

Please provide us with the following information:

This issue is for a: (mark with an x)

- [x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Attempt to access the URL in module 0 of the course, step 4, copy the example files from:
https://kmworkshop.blob.core.windows.net/clinical-trials-small?sv=2019-10-10&st=2020-09-14T19%3A21%3A00Z&se=2022-09-16T19%3A21%3A00Z&sr=c&sp=rl&sig=CtoGmOSkbap8ZOslI9EYeWs148gw4NdYpynU0Eyoj4s%3D

The URL does not work in Storage Explorer when you attempt to add it, and it does not work in a browser, failing with an authentication failure.

Any log messages given by the failure

In the browser you will receive the following error:

AuthenticationFailed Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. RequestId:688c96da-401e-0048-7534-5ed8ae000000 Time:2021-06-10T20:10:28.8938011Z Signature did not match. String to sign used was rl 2020-09-14T19:21:00Z 2022-09-16T19:21:00Z /blob/kmworkshop/$root 2019-10-10 c

Expected/desired behavior

Expect to access the sample files so they can be copied to the student storage account

OS and Version?

Windows 10

Versions

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

CustomEntityLookup not working

• [ X] bug report -> custom-entity-lookup failing on post request

Minimal steps to reproduce

Right click on CustomEntityLookup in the Solution Explorer and choose "Set as StartUp Project"
Press F5 - NOTE: You may need to allow the function to run
Once the function is running it should supply you with the URL to use for POST calls. Copy this URL.
Used postman to send a POST request

Any log messages given by the failure

Postman Error 500
[2020-10-30T10:55:21.780] Executing 'custom-entity-lookup' (Reason='This function was programmatically called via the host APIs.', Id=9aac505f-12f4-4c98-84ec-2c927c19769c)
[2020-10-30T10:55:21.840] Custom Entity Search function: C# HTTP trigger function processed a request.
[2020-10-30T10:55:21.998] Executed 'custom-entity-lookup' (Failed, Id=9aac505f-12f4-4c98-84ec-2c927c19769c, Duration=329ms)
[2020-10-30T10:55:22.000] System.Private.CoreLib: Exception while executing function: custom-entity-lookup. CustomEntityLookup: Method not found: 'System.String System.IO.Path.Join(System.String, System.String)'.

Expected/desired behavior

Reponse

OS and Version?

WIndows 10

Versions

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

Multiple Index and Storages with different buttons at home page.

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ X ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

The steps are the same as the Workshop 1 and 2

Expected/desired behavior

Hi, firstly I would like to thank you because of how great this project is. I have been using it for a while now and now I am trying to work on a future step on my own, but I am not familiarized that well with the code structure of the repository. What I would like to do is to have four different buttons on the home page (that I know how to do in the Home HTML) but what I would like to ask is how can I link that button to one storage and just display the files within that Blob Storage and the other buttons to the next storages. I think it has to do something with the storage Controller, but I am not sure.

Thanks in advance

OS and Version?

Windows 10.

Thanks! We'll be in touch soon.

Workshops Module 1 and 2

I didn't have Locations-field at all in the index definition. I had Organizations-field. Well guessed they are probably the same. But then in Module 2 the VS solution didn't work, it opened the web page but didn't bring any results.

Search instance can't see provisioned Cognitive Services instances from ARM template

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [x] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

1. Deploy ARM template via Deploy to Azure to Azure Service (location e.g. eastus, but same in westeurope, tiers all S0 or compareable)
2. Goto Search Service and select Import Data, pick sample dataset (or any other), hit next
3. Under Attach Cognitive Services, the provisioned Microsoft.CognitiveServices/accounts will not show up

Azure Cognitive Search can't find the CogServices service:

But it exists in the RG:

Creating a new CognitiveServicesAllInOne resource shows up properly in the Search instance.

Any log messages given by the failure

No

Expected/desired behavior

Provisioned Microsoft.CognitiveServices/accounts from ARM template to show up in Search Instance

OS and Version?

Windows 10, but should be irrelevant (all in browser)

Versions

Irrelevant

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

Pre-reqs blob not copying metadata

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting

Hi
I'm getting the following error when trying to copy the blob container data to my blob storage area. Any ideas what might be causing this? When I get to stage 2 and completed the UI build using Visual studio the autosuggest works but it doesn't return any search results on the page. I'm assuming this second bug is caused by no metadata being migrated.

"The properties and metadata for this container cannot be determined and will not be copied to the new container. Proceed with the copy?"

Minimal steps to reproduce

Any log messages given by the failure

Expected/desired behavior

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)

Versions

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

Getting an error while opening Web UI Template

When i try to excecute the Web UR Template from visual studio 2019 i receive an error .
An unhandled exception occurred while processing the request.
TypeLoadException: Could not load type 'Microsoft.AspNetCore.Mvc.Internal.IHttpResponseStreamWriterFactory' from assembly 'Microsoft.AspNetCore.Mvc.Core, Version=2.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'.

Support complex types

This issue is for a:

- [ ] bug report
- [x] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Use a complex type in your index, Web UI template will stop working.

Expected/desired behavior

Support for complex types is generally available in api-version=2019-05-06.

Support complex types by:

• Upgrading the SDK to .NET SDK 9.0
• Change ToSearchField logic in SearchSchema.cs
  

Versions

latest (master)

Struggling with Docker and Step 2

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [x] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

I struggled to connect Docker with the resources that I provisioned. On the "02 - Web UI Template", I was unable to run this command:

"docker run -d --env-file .env -p 80:80 kmworkshop.azurecr.io/web-ui:latest"

I tried to use the visual studio path instead of using the Docker path, however, the instructions on the Visual Studio path weren't clear either. It didn't provide any instructions on how to ensure that your Web Services app was connected to the resources provisioned earlier. Maybe I'm misunderstanding the instructions there, but I'd love more clarity around that.

Any log messages given by the failure

Expected/desired behavior

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)

Versions

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

Az powershell vs AzureRM

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [x] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Hi, I really love the repo but we should probably use Az cmdlet instead of AzureRM (which is no longer the default powershell modelue, "Az" runs in Cloud shell, is opensource and has maintenance vs the legacy AzureRM that will die next year). The changes needed are minimal, ie replace az where AzureRM (see full code below)

Any log messages given by the failure

Expected/desired behavior

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)

Windows 10 Enterprise 1903

$PSVersionTable.PSVersion
Major Minor Build Revision

---

5 1 18362 145

Version Name Repository Description

---

2.5.0 Az PSGallery Microsoft Azure PowerShell - Cmdlets to manage r...

Versions

Mention any other details that might be useful

More info on Az (vs AzureRM) here: https://docs.microsoft.com/en-us/powershell/azure/migrate-from-azurerm-to-az?view=azps-2.5.0 and here
https://azure.microsoft.com/en-us/blog/azure-powershell-cross-platform-az-module-replacing-azurerm/

###This is the new code for deploy.ps1

Copyright (c) Microsoft Corporation. All rights reserved.

Licensed under the MIT License.

PARAMETERS SECTION

$resourceGroupName = "";
$subscriptionId = "";
$location = "SouthCentralUS";
$searchSku = "basic"; # "free", "basic", or "standard"

Resource names should be lower case

$searchServiceName = "";
$webappname = "";
$cogServicesName = "";
$appInsightsName = $webappname + "insights";

Storage account names can only include numbers and letters

$storageAccountName = "";

END PARAMETERS SECTION

sign in

Write-Host "Logging in...";
Login-AzureAccount;

select subscription

Write-Host "Selecting subscription '$subscriptionId'";
Select-AzSubscription -SubscriptionID $subscriptionId;

Register RPs

$resourceProviders = @("microsoft.cognitiveservices", "microsoft.documentdb","microsoft.insights","microsoft.search","microsoft.sql","microsoft.storage");
if($resourceProviders.length) {
Write-Host "Registering resource providers"
foreach($resourceProvider in $resourceProviders) {
Register-AzResourceProvider -ProviderNamespace $resourceProvider;
}
}

#Create or check for existing resource group
$resourceGroup = Get-AzResourceGroup -Name $resourceGroupName -ErrorAction SilentlyContinue
if(!$resourceGroup)
{
Write-Host "Resource group '$resourceGroupName' does not exist.";
if(!$location) {
$location = Read-Host "please enter a location:";
}
Write-Host "Creating resource group '$resourceGroupName' in location '$location'";
New-AzResourceGroup -Name $resourceGroupName -Location $location
}
else{
Write-Host "Using existing resource group '$resourceGroupName'";
}

Create a new storage account

Write-Host "Creating Storage Account";
$storageAccount = New-AzStorageAccount -ResourceGroupName $resourceGroupName
-Name $storageAccountName -Location $location
-SkuName Standard_LRS `
-Kind StorageV2

Create a cognitive services resource

Write-Host "Creating Cognitive Services";
$cogServices = New-AzCognitiveServicesAccount -ResourceGroupName $resourceGroupName
-Name $cogServicesName -Location $location
-SkuName S0 `
-Type CognitiveServices

Create a new search service

This command will return once the service is fully created

Write-Host "Creating Search Service";
$searchService = New-AzResourceGroupDeployment -ResourceGroupName $resourceGroupName
-TemplateUri "https://gallery.azure.com/artifact/20151001/Microsoft.Search.1.0.9/DeploymentTemplates/searchServiceDefaultTemplate.json" -NameFromTemplate $searchServiceName
-Sku $searchSku -Location $location
-PartitionCount 1 `
-ReplicaCount 1

Create an App Service plan

Write-Host "Creating App Service Plan";
$appService = New-AzAppServicePlan -Name $webappname
-Location $location -ResourceGroupName $resourceGroupName
-Tier Standard

Create a web app.

Write-Host "Creating Web App";
$webApp = New-AzWebApp -Name $webappname
-Location $location -AppServicePlan $webappname
-ResourceGroupName $resourceGroupName `
-WarningAction SilentlyContinue

Create an application insights instance

Write-Host "Creating App Insights";
$appInsights = New-AzResource -ResourceName $appInsightsName
-ResourceGroupName $resourceGroupName -Tag @{ applicationType = "web"; applicationName = $webappname}
-ResourceType "Microsoft.Insights/components" -Location $location
-PropertyObject @{"Application_Type"="web"} `
-Force

Setting App Insights Key in Web app

Write-Host "Connecting App Insights with Web App";
$appSetting = @{'APPINSIGHTS_INSTRUMENTATIONKEY'= $appInsights.Properties.InstrumentationKey}
$updateSettings = Set-AzWebApp -Name $webappname -ResourceGroupName $resourceGroupName -AppSettings $appSetting

---

Thanks! We'll be in touch soon.

SAS token has expired for clinical-trials-small

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [x] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Copy the storage container that holds clinical trials from a read only location to your Storage Account.

1. Open Azure Storage Explorer and select Managed Accounts, Add and Account..., Use a shared access signature (SAS) URI. The Display name will autofill. Select Next
2. Add the following URI, then select Next, then select Connect

At this step the URI provided gives a SAS token has expired error

Any log messages given by the failure

The SAS token has expired.

Expected/desired behavior

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)
windows 10

Versions

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

Module 2 System.NullReferenceException: 'Object reference not set to an instance of an object.'

Please provide us with the following information:

This issue is for a: (mark with an x)

- [x ] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Run CognitiveSearch.UI at end of Module 2. Type phrase in search bar when web app is launched. Exception is thrown in Visual Studio, search is not completed in browser. No other files change except for appsettings.json.

Any log messages given by the failure

Exception thrown: 'System.NullReferenceException' in CognitiveSearch.UI.dll
Object reference not set to an instance of an object.

Expected/desired behavior

Search in browser completed, search results appear.

OS and Version?

Windows 10

Versions

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

Drop down selection from view is not showing in the controller

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [ x] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Any log messages given by the failure

Expected/desired behavior

I have this code added in the search.cshtml file
the selected value from drop down is not getting in the the controller parameter

HomeController

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)

Versions

Windows 10

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

Many of the links on the Readme (under workshop) appear to be broken

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [X] regression (a behavior that used to work and stopped in a new release)

Most of the workshop links are broken and will render a 'not found'. The readme has this link: https://github.com/Azure-Samples/azure-search-knowledge-mining/workshops/Module%200.md but the correct link appears to be: https://github.com/Azure-Samples/azure-search-knowledge-mining/blob/master/workshops/Module%200.md

Minimal steps to reproduce

The workshop link for Module 7 works but others do not. Displays a 'not found'.

Any log messages given by the failure

Expected/desired behavior

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)

Versions

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

Deploying Search UI to dotnet core environment

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

try to run application in dotnet core environment

Any log messages given by the failure

startup configuration for application does not load in .net core environment.. can provide exact errors if you are unable to replicate. Proposed fix below anyhow.

Expected/desired behavior

successfully load configuration / appsettings

OS and Version?

Linux - Ubuntu 16.04, dotnet core 3.1

Mention any other details that might be useful

In startup.cs if you add the following logic, it will fix the issue
public Startup(IConfiguration configuration)
{
Configuration = configuration;
if(string.IsNullOrEmpty(Configuration["OrganizationName"])) OR A BETTER CHECK IF THE CONFIG LOADED OK
{
var builder = new ConfigurationBuilder()
.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
.AddEnvironmentVariables();
Configuration = builder.Build();
}

    }

---

Thanks! We'll be in touch soon.

'metadata_title' is not a retrievable field

This issue is for a: (mark with an x)

- [x ] bug report -> please search issues before submitting
- [ ] feature request
- [x] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

To fully understand cognitive search and indexing I decided I would recreate all of the resource groups and indexes by myself in the Azure portal. I created my index/indexer following the guide, step by step. This created no issues.

I ran the UI locally on my computer and I could open the homepage. When I went into search or tried to search something It would crash and produce the following code.

Any log messages given by the failure

'metadata_title' is not a retrievable field. Only fields marked as retrievable in the index can be used in $select.
Parameter name: $select
Status: 400 (Bad Request)
Content:
{"error":{"code":"","message":"Invalid expression: 'metadata_title' is not a retrievable field. Only fields marked as retrievable in the index can be used in $select.\r\nParameter name: $select"}}

Mention any other details that might be useful

The fix is to make sure that "metadata_title" field in the index set to retrievable. This is missing in the documentation as followed;

In the picture above "metadata_title" is missing. When I created my indexes "metadata_title" was automatically created.

Filtering on Edm.DateTimeOffset field not working in .NET SDK

This is an issue I found in Web UI Template. I have a search index with Collection(Ed.String) (field name : organizations) and Edm.DateTimeOffset (field name : Date) fields. I'm trying to filter based on my Date field, please find the screenshot below which has the search query

This query returns some documents when I tried it in the search explorer in Azure portal, but here when I call it in the SDK I'm getting result as null. I tried this with 'organizations' field, that one is working as expected in SDK also. This issue seems to be only for Edm.DateTimeOffset.

API Version : 2020-06-30

Steps to reproduce :

1. Create a search index with Edm.DateTimeOffset field and make it filterable
2. Try to filter documents based Edm.DateTimeOffset field using Azure Search SDK for .NET

Workshop 2

Please provide us with the following information:
Hi, I'm right now doing the guide and following step by step, but encountered the same problem as in the https://github.com/Azure-Samples/azure-search-knowledge-mining/issues/43.

The thing is it does show the webpage but when I do the search it does not display any results or any of the search fields as it shows at the main ReadME file.

I was trying to debug it as you commented on the issue, but it does show to be iterating correctly at the fields and everything, I am not sure how to address the issue.

thanks

---

This issue is for a: (mark with an x)

- [x ] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Any log messages given by the failure

None that I am aware of.

Expected/desired behavior

The search field should be able to display with the fields in the searchmodel.cs file

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)
windows 10
Thanks! We'll be in touch soon.

Knowledge Mining Workshop: metadata_storage_path is Base64 encoded twice

Please provide us with the following information:

This issue is for a: (mark with an x)

- [X] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

1. Complete Module 0 and Module 1;
2. Open your index's Search explorer and query for * (or anything else);
3. Look the Results panel, this is a extract of my first result:
   "metadata_storage_name": "NCT00943111.pdf",
   "metadata_storage_path": "YQBIAFIAMABjAEgATQA2AEwAeQA5AHMAWgAzAEIAdABaAG0AdAB0AGQAMwBOADAAYgAzAEoAaABaADIAVQB1AFkAbQB4AHYAWQBpADUAagBiADMASgBsAEwAbgBkAHAAYgBtAFIAdgBkADMATQB1AGIAbQBWADAATAAyAE4AcwBhAFcANQBwAFkAMgBGAHMATABYAFIAeQBhAFcARgBzAGMAeQAxAHoAYgBXAEYAcwBiAEMAOQBPAFEAMQBRAHcATQBEAGsAMABNAHoARQB4AE0AUwA1AHcAWgBHAFkAMQA1"
4. Take the metadata_storage_path value and use some tool to decode Base64 (like https://www.base64decode.org), then you should get:
   aHR0cHM6Ly9sZ3BtZmttd3N0b3JhZ2UuYmxvYi5jb3JlLndpbmRvd3MubmV0L2NsaW5pY2FsLXRyaWFscy1zbWFsbC9OQ1QwMDk0MzExMS5wZGY15
5. As it is still encoded, decode this value once more. The result will be correct now:
   https://lgpmfkmwstorage.blob.core.windows.net/clinical-trials-small/NCT00943111.pdf5

Because metadata_storage_path is encoded twice, details.js is unable to preview any file (line 24 will fail to decode the path).

I suspect it has something to do with the way the Index is created, because I have no problem to preview files with Knowledge Mining Solution Accelerator deployment.

Any log messages given by the failure

This file cannot be previewed. Download it here to view: Download

Expected/desired behavior

The Index field metadata_storage_path be encoded only once.

OS and Version?

Windows 10.

Versions

Latest.

Bootcamp link dead

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [ X ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Read the README.md file, and look for the Bootcamp link.

Any log messages given by the failure

404 error

Expected/desired behavior

A working Link

Text refers to a skills section that has been moved

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [x] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

The link https://github.com/Azure-Samples/azure-search-knowledge-mining/tree/master/02%20-%20Web%20UI%20Template refers to a skill subfolder that has moved to https://github.com/Azure-Samples/azure-search-knowledge-mining/tree/master/03 - Data Science and Custom Skills/Azure Function Custom Skills . Please remove the text saying: "The Cognitive Search Template contains two projects"... for more clarity

Any log messages given by the failure

Expected/desired behavior

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)

Versions

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

Azure search inbuilt cognitive skill extracting the wrong entity-location name.

Some location are not right. And same things happening with other enitity extraction as well

Module 3: Unable to test Azure Function locally, 'Output type of Class library cannot be started directly' error.

Please provide us with the following information:

This issue is for a: (mark with an x)

- [x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Make CustomEntityLookup as Startup Project and run locally.

Any log messages given by the failure

With the current code and Output Type of Class Library, error thrown is "A project with an Output Type of Class Library cannot be started directly. In order to debug this project, add an executable project to this solution which references the library project. Set the executable project as the startup project".

Expected/desired behavior

Azure Function to run and be tested locally.

OS and Version?

Windows 10

Versions

Mention any other details that might be useful

I tried changing the output type from Class Library to Console/Windows application, but received the error: 'Program does not contain a static 'Main' method suitable for an entry point. I added a static Main at the end of the CustomEntityLookup class in CustomEntityLookup.cs, but Azure Function did not run and could not be tested.

---

Thanks! We'll be in touch soon.

Search queries are not returning results from Cognitive Search Service Indexer and Index

Please provide us with the following information:

This issue is for a: (mark with an x)

- [X] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Followed the Azure search knowledge mining steps, ran into issues in Cognitive UI wherein results are not displayed in UI of scanned documents or search query is not functioning as expected.

Any log messages given by the failure

DataPlaneApiArgumentException : Unknown field '' in facet specification.
Parameter name: facet.
Stack:
at Microsoft.WindowsAzure.Search.Core.Services.FacetParser.ParseFacet(CustomerData facetSpecification) in d:\dbs\el\mbas\Source\Search\Product\SearchCore\Services\FacetParser.cs:line 85
at Microsoft.WindowsAzure.Search.Core.Services.FacetParser.ParseFacets(CustomerData[] facetSpecifications) in d:\dbs\el\mbas\Source\Search\Product\SearchCore\Services\FacetParser.cs:line 45
at Microsoft.WindowsAzure.Search.Core.Services.QueryBuilder.BuildSearchQuery(ApiSearchQuery apiSearchQuery) in d:\dbs\el\mbas\Source\Search\Product\SearchCore.WebApi.Windows\Services\QueryBuilder.cs:line 172
at Microsoft.WindowsAzure.Search.Core.Controllers.QueryControllerImpl.<>c__DisplayClass16_0.b__0(Index index) in d:\dbs\el\mbas\Source\Search\Product\SearchCore.WebApi.Windows\Controllers\QueryControllerImpl.cs:line 90
at Microsoft.WindowsAzure.Search.Core.Controllers.QueryControllerImpl.d__241.MoveNext() in d:\dbs\el\mbas\Source\Search\Product\SearchCore.WebApi.Windows\Controllers\QueryControllerImpl.cs:line 233 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.WindowsAzure.Search.Core.Controllers.QueryControllerImpl.<SearchAsync>d__16.MoveNext() in d:\dbs\el\mbas\Source\Search\Product\SearchCore.WebApi.Windows\Controllers\QueryControllerImpl.cs:line 86 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.WindowsAzure.Search.Core.Controllers.Docs20200630Controller.<Get>d__10.MoveNext() in d:\dbs\el\mbas\Source\Search\Product\SearchCore.WebApi.Windows\Controllers\2020-06-30\Docs20200630Controller.cs:line 100 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.WindowsAzure.Search.Core.Controllers.Docs20200630Controller.<Search>d__9.MoveNext() in d:\dbs\el\mbas\Source\Search\Product\SearchCore.WebApi.Windows\Controllers\2020-06-30\Docs20200630Controller.cs:line 86 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Threading.Tasks.TaskHelpersExtensions.<CastToObject>d__31.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.ApiControllerActionInvoker.d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.ExceptionHandlingActionInvoker.d__0.MoveNext() in d:\dbs\el\mbas\Source\Hosting\Product\AzureSearch.Hosting.AspNet.Windows\Http\ExceptionHandlingActionInvoker.cs:line 22

Expected/desired behavior

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)
Azure Apps Service Windows

Versions

Win

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

Images for testing CustomEntityLookup skill show incorrect information

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [x] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

• The images on Workshop 3 under the heading "Testing the Azure Function" seem to make reference to a "custom-search" skill and are confusing

Any log messages given by the failure

Expected/desired behavior

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)

Versions

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

02 - Web UI Template does not return back search results

Please provide us with the following information:

This issue is for a: (mark with an x)

- [x ] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

I cloned the Github repo to a local folder and followed the instructions to deploy the solution via the Portal to my subscription. I tried the Search Explorer and the query returns back search results. Then I set up the appsettings.json outlined in the 02 - Web UI Template and compiled it. The search results do not get displayed now but the entity map returns back the network diagram visual.

Any log messages given by the failure

The console returns an error "entityMap.js:47 Uncaught TypeError: Cannot read property 'style' of null"

Expected/desired behavior

Search Results are returned

OS and Version?

Windows 10, Azure Commercial Subscription

Versions

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

How to remove permanent JSON data from Azure Search Service

How to remove permanent JSON data from Azure Search Service

When I am deleting an uploaded image or document from the blob, and go and search from the search bar the deleted object still shows up in the results with its filename visible too. Still, the image or document is not accessible/missing, or visible, why is this occurring?

For example, the file (gun1.jpg) showing here in the results has been deleted by me but as you can see it's still showing here in the result

Unable to deploy using the UI Template Deployment (need to approve terms but no place to do so)

This issue is for a: (mark with an x)

- [ X] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Click on 'Deploy to Azure' button
fill in parameters, click on 'Review + Create'

Any log messages given by the failure

You will need to Agree to the terms of service below to create this resource successfully.

Expected/desired behavior

Template is validated and offers option to deploy.

OS and Version?

N/A

Versions

N/A

autocomplete does not popup

JS scripts for autocomplete is not invoked , is there additional setup needed?

$(function () {
$('.advancedAutoComplete').autocomplete({
source: "/home/suggest",
minLength: 2,
select: function (event, ui) {
Search();
}
});
})

@LuisCabrer ?

Support macOS for deployments

This issue is for a:

- [ ] bug report -> please search issues before submitting
- [x] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

AzureRM doesn't support macOS.

Expected/desired behavior

MacOS platform supported as option for deployments, which can be achieved by:

• Rewrite Powershell script to use Az module instead of AzureRM module.
• Add a Bash deployment option uzing AZ-CLI.

Search UI Showing Old AZStorage Deleted Documents as results in UI app.

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [X] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

I am writing to ask about how to use a specific feature in the KM accelerator, was able to implement the accelerator, and indexed 25 documents that are searchable however when I removed those initial 25 documents, and then added new batch of documents, reindexed, new docs are showing up in the UI whereas the originally indexed docs are still appearing. What is the best way to clean up those original indexes from UI so that only the active docs in the storage appear in UI app.

Any log messages given by the failure

Expected/desired behavior

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)
Linux

Versions

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.