The Azure Optimization Engine is an extensible solution designed to generate optimization recommendations for your Azure environment. See it like a fully customizable Azure Advisor.
License: MIT License
Hello the weekly SQL ingestion job starts failing after first run
Looks like the job is consuming all the 400 MB memory allocated to it .does a anyone has any work around ?
Job Name:Ingest-RecommendationsToSQLServer
Logs:The runbook job failed due to sandbox running out of memory. Each Azure Automation sandbox is allocated 400 MB of memory. The job was attempted 3 times before it was suspended. See some common ways to resolve this at https://aka.ms/AAMemoryLimit
Costs Growing workbook is not showing any data. Below error is getting. Any suggesstion
Evolution over time (select growing/outlier cost)
This query could not run because some parameters are not set.
Please set: SelectedPerspective, SelectedPerspectiveId
Top contributing instances over time
This query could not run because some parameters are not set.
Please set: SelectedPerspective, SelectedPerspectiveId
Top contributing resource groups over time
This query could not run because some parameters are not set.
Please set: SelectedPerspective, SelectedPerspectiveId
working in azure us government cloud
Attempting to install via azure cloud shell (powershell)
Running Original script gets...
WARNING: Interactive authentication is not supported in this session, please run cmdlet 'Connect-AzAccount -UseDeviceAuthentication'.
Getting Azure subscriptions...
Get-AzSubscription: /home/blah/azureoptimizationengine/DeployAOE.ps1:142
Line |
142 | $subscriptions = Get-AzSubscription | Where-Object { $_.State -eq "En …
| ~~~~~~~~~~~~~~~~~~
| Run Connect-AzAccount to login.
If I make a few Tweaks and add the requested -usedeviceauthentication, it eventually gets to this...
New-AzADServicePrincipal: /home/blah/azureoptimizationengine/Deploy-AzureOptimizationEngine.ps1:54
Line |
54 | … Principal = New-AzADServicePrincipal -ApplicationId $Application.Appl …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| The appId 'd5e41e0f-6273-4b73-a335-fc7e960712b6' of the service principal does not reference a valid application object.
If that happens, the Az Advisor Recommendations Generate API endpoint should be called.
If the deployment script fails after the temporary SQL firewall rule is created, there should be a try/catch/finally mechanism to remove the firewall rule no matter the outcome of the deployment.
Ths requires:
The Details URL field that is populated in each recommendation does not take into account the Azure cloud the recommendation comes from. Instead of using portal.azure.com, use e.g., portal.azure.cn for AzureChinaCloud.
Hello again,
Utilizing azure us government, with Get-AzConsumptionPriceSheet in Recommend-AdvisorCostAugmentedToBlobStorage
noticed that get-azconsumptionpricesheet returns:
Get-AzConsumptionPriceSheet: Operation returned an invalid status code 'NotFound'
Id :
Name :
PriceSheets : {}
Tag :
Type :
hmnm, ran it with -debug
got a snippit
"error": {
"code": "404",
"message": "Cost management data is not supported for subscription(s) xxx in the provided api-version. Please use api-version 2019-10-01 or later. (Request ID: yyy)"
going to do more research on rights to subscription cost management and azure us government
Update: MicrosoftDocs/azure-docs#38660
Changing the API version to the previous stable release (2019-01-01) allows the filter to work.
This is an example of the full URI: https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/providers/Microsoft.Consumption/usageDetails?api-version=2019-05-01&$filter=properties/resourceGroup eq 'dev-rg'
where the -debug of the command reported using
Absolute Uri:
https://management.usgovcloudapi.net/subscriptions/xxx/providers/Microsoft.Consumption
/pricesheets/default?$top=1000&api-version=2018-01-31
If the AzureOptimization_AdvisorFilter variable is set to a single category (other than Cost), then the Cost recommendations are filtered out.
Hi, We have deployed this solution as per the documentation and added multiple Log analytics workspaces for VM/VMSS perf metrics. Runbooks are running as per schedulers, but when we check workspace for these custom tables, we dont see them and workbooks are throwing table not found errors.
AzureOptimizationVMsV1_CL
AzureOptimizationResourceContainersV1_CL
AzureOptimizationConsumptionV1_CL
AzureOptimizationAdvisorV1_CL
As result of which, recommendations exporter runbook throwing tables not found (0 records found) message.
Hi There,
Hoping somebody can assist with this, I have several runbooks failing, these are listed below;
Recommend-VMSSOptimizationsToBlobStorage
Recommend-VMsHighAvailabilityToBlobStorage
Recommend-UnusedLoadBalancersToBlobStorage
Recommend-UnusedAppGWsToBlobStorage
Recommend-SqlDbOptimizationsToBlobStorage
All of these have similar errors, please see two examples below;
Recommend-VMsHighAvailabilityToBlobStorage
Output:
{[AzureChinaCloud, AzureChinaCloud], [AzureCloud, AzureCloud], [AzureGermanCloud, AzureGermanCloud], [AzureUSGovernme...
Finding tables where recommendations will be generated from...
Will run query against tables AzureOptimizationAvailSetsV1_CL, AzureOptimizationVMsV1_CL, AzureOptimizationVMSSV1_CL, AzureOptimizationVhdDisksV1_CL and AzureOptimizationResourceContainersV1_CL
Name : Internal IT (xxxxx) - xxxxx -
MSI@xxxxx
Account : MSI@xxxxx
Environment : AzureCloud
Subscription : xxxxx
Tenant : xxxxx
TokenCache :
VersionProfile :
ExtendedProperties : {}
Looking for Availability Sets with a low fault domain count...
Warnings:
-TenantId ‘xxxxx’ contains more than one active subscription. First one will be selected for further use. To select another subscription, use Set-AzContext.
-Query failed. Debug the following query in the AOE Log Analytics workspace: AzureOptimizationAvailSetsV1_CL | where TimeGenerated > ago(1d) and toint(FaultDomains_s) < 3 and toint(FaultDomains_s) < todouble(VmCount_s)/2 | project TimeGenerated, InstanceId_s, InstanceName_s, ResourceGroupName_s, SubscriptionGuid_g, TenantGuid_g, Cloud_s, Tags_s, FaultDomains_s, VmCount_s | join kind=leftouter ( AzureOptimizationResourceContainersV1_CL | where TimeGenerated > ago(1d) | where ContainerType_s =~ 'microsoft.resources/subscriptions' | project SubscriptionGuid_g, SubscriptionName = ContainerName_s ) on SubscriptionGuid_g
-Operation returned an invalid status code 'BadRequest'
Recommend-VMSSOptimizationsToBlobStorage
Output:
{[AzureChinaCloud, AzureChinaCloud], [AzureCloud, AzureCloud], [AzureGermanCloud, AzureGermanCloud], [AzureUSGovernme...
Finding tables where recommendations will be generated from...
Will run query against tables AzureOptimizationVMSSV1_CL, AzureOptimizationMonitorMetricsV1_CL, AzureOptimizationResourceContainersV1_CL and AzureOptimizationConsumptionV1_CL
Name : Internal IT (xxxxx) - xxxxx -
MSI@xxxxx
Account : MSI@xxxxx
Environment : AzureCloud
Subscription : xxxxx
Tenant : xxxxx
TokenCache :
VersionProfile :
ExtendedProperties : {}
Getting Virtual Machine SKUs for the australiaeast region...
Getting the current Pricesheet...
Consumption pricesheet not available, will estimate savings based in cores count...
Looking for underutilized Scale Sets, with less than 30% CPU and 50% RAM usage...
Warnings:
-TenantId 'xxxxx' contains more than one active subscription. First one will be selected for further use. To select another subscription, use Set-AzContext.
-SubscriptionGuid_g, Cloud_s, TenantGuid_g, VMSSSize_s, NicCount_s, DataDiskCount_s, PMemoryPercentage, PCPUPercentage, Tags_s, Last30DaysCost, Last30DaysQuantity | join kind=leftouter ( AzureOptimizationResourceContainersV1_CL | where TimeGenerated > ago(1d) | where ContainerType_s =~ 'microsoft.resources/subscriptions' | project SubscriptionId = SubscriptionGuid_g, SubscriptionName = ContainerName_s ) on SubscriptionId | where isnotempty(PMemoryPercentage) and isnotempty(PCPUPercentage) and PMemoryPercentage < 50 and PCPUPercentage < 30
-Error getting consumption data: Cost Management not available (Cost management data is not supported for subscription(s) xxxxx in the provided api-version. Please use api-version 2019-10-01 or later. (Request ID: e3cb353c-c42b-4b71-afa1-234e815d4481)). 1 of 3 tries. Waiting 30 seconds...
-Query failed. Debug the following query in the AOE Log Analytics workspace: let billingInterval = 30d; let perfInterval = 7d; let cpuPercentileValue = 99; let memoryPercentileValue = 99; let etime = todatetime(toscalar(AzureOptimizationConsumptionV1_CL | summarize max(UsageDate_t))); let stime = etime-billingInterval; let BilledVMs = AzureOptimizationConsumptionV1_CL | where UsageDate_t between (stime..etime) and InstanceId_s contains 'virtualmachinescalesets' | extend VMConsumedQuantity = iif(InstanceId_s contains 'virtualmachinescalesets' and MeterCategory_s == 'Virtual Machines', todouble(Quantity_s), 0.0) | extend VMPrice = iif(InstanceId_s contains 'virtualmachinescalesets' and MeterCategory_s == 'Virtual Machines', todouble(UnitPrice_s), 0.0) | extend FinalCost = VMPrice * VMConsumedQuantity | summarize Last30DaysCost = sum(FinalCost), Last30DaysQuantity = sum(VMConsumedQuantity) by InstanceId_s; let MemoryPerf = AzureOptimizationMonitorMetricsV1_CL | where TimeGenerated > ago(perfInterval) | where MetricNames_s == "Available Memory Bytes" and AggregationType_s == "Minimum" | extend MemoryAvailableMBs = todouble(MetricValue_s)/1024/1024 | project TimeGenerated, MemoryAvailableMBs, InstanceId_s=ResourceId | join kind=inner ( AzureOptimizationVMSSV1_CL | where TimeGenerated > ago(1d) | distinct InstanceId_s, MemoryMB_s ) on InstanceId_s | extend MemoryPercentage = todouble(toint(MemoryMB_s) - toint(MemoryAvailableMBs)) / todouble(MemoryMB_s) * 100 | summarize PMemoryPercentage = percentile(MemoryPercentage, memoryPercentileValue) by InstanceId_s; let ProcessorPerf = AzureOptimizationMonitorMetricsV1_CL | where TimeGenerated > ago(perfInterval) | where MetricNames_s == "Percentage CPU" and AggregationType_s == 'Maximum' | extend InstanceId_s = ResourceId | summarize PCPUPercentage = percentile(todouble(MetricValue_s), cpuPercentileValue) by InstanceId_s; AzureOptimizationVMSSV1_CL | where TimeGenerated > ago(1d) | distinct InstanceId_s, VMSSName_s, ResourceGroupName_s, SubscriptionGuid_g, Cloud_s, TenantGuid_g, VMSSSize_s, NicCount_s, DataDiskCount_s, Tags_s | join kind=inner ( BilledVMs ) on InstanceId_s | join kind=leftouter ( MemoryPerf ) on InstanceId_s | join kind=leftouter ( ProcessorPerf ) on InstanceId_s | project InstanceId_s, VMSSName = VMSSName_s, ResourceGroup = ResourceGroupName_s, SubscriptionId =
-Operation returned an invalid status code 'BadRequest'
Hi Luke,
Correct me if am wrong, for metrics collection using az monitor command we have option to look back for max 30/90 days to analyse utilisation.
Then why we are exporting daily utilisation to log analytics workspace and analysing average utilisation using kql and variable (look back period).
Besides the current possibility of just choosing a naming prefix at deployment time, I also want to be able to choose the naming for all the Azure resources that are deployed as part of the solution (Log Analytics Workspace, Automation Account, Azure SQL Server, Azure SQL Database and Storage Account). This is required by enterprise scenarios with an existing naming convention.
For Advisor recommendations, there should be a UI solution for invoking the Advisor API to dismiss/snooze a particular recommendation.
For AOE custom recommendations, we should use the same UI solution, but with our own data model and logic.
Should be implemented in conjunction with #13
Originally posted by spike28742 March 22, 2023
Hi
When I try to deploy I am getting an error that the subscription is not supported. I managed to deploy the solution about a month ago. Are there any limitations why the solution will not work on a CSP based subscription?
Also noticed that the cost recommendations never worked, are there any specific requirements for these to work?
Thanks for your support and keep up the great work on this tool.
Based on start time (-1h) and day of week of first schedule
The -AzureEnvironment parameter of the deployment script is not sent to the ARM deployment, which will eventually lead to the AzureOptimization_CloudEnvironment Automation variable to keep the default 'AzureCloud' value.
I have implemented the AOE for one Azure tenant which has multiple subscriptions, and all the VMs are reporting to a single LogAnalytics Workspace. I already have set up permission for the Managed Identity at Management group scope, but when I'm trying to execute queries against the recommendations, I only receive recommendations from the subscriptions where I installed the AOE.
What I'm doing wrong?
Hi All,
Hoping somebody can assist me with this. I am having trouble with no information coming through for the VM right sizing (page shows blank) and in general there are less recommendations than expected. Additionally, the Azure Advisor right-sizing recommendations are not present here either (not sure if that is by design or not). I changed the date range to be 1 month and also refreshed many times over a period of 2 weeks
From what I can see, all the subscriptions appear to be visible to the AOE, because I can see some recommendations for each subscription (i.e. Subnets without any IP usage are a waste of virtual network address space)
During the setup, I noticed our customer already had some VMs attached to log analytics workspaces, I can confirm that I added the AzureOptimization_RightSizeAdditionalPerfWorkspaces variable to the automation account and added all of the workspaces to this 2 weeks ago.
Also, I ensured that the Automation App account has reader access to the tenant root group which has reader access to all subscriptions and log workspaces.
Additional Info;
We have tried to find out why it does not bring the updated information in the workbook: "(Identities and Roles)" of our tenant when we enable it for the first time, however, when detecting that there is no overflow in the database and that the Jobs in the Automation Account -> runbooks are executing correctly and we do not know why it is not updated.
looking over the workbooks in the log analytics workspace, two work, but a third doesn't appear to ..
Is this something I need to set and haven't seen in the documentation because I am silly?
This query could not run because some parameters are not set.
Please set: SelectedPerspective, SelectedPerspectiveId
Based on the Export-ARGUnmanagedDisksPropertiesToBlobStorage.ps1 data collection runbook, generate recommendations for VMs using unmanaged disks to upgrade to Managed Disks.
I really like the concept of this, but I would like to deploy this at resource group level and not at subscription scope.. I have been looking around but can't seem to change it, is there anyone from team who can assist?
"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"ResourceDeploymentFailure","message":"Encountered internal server error. Diagnostic information: timestamp '20220111T025224Z', subscription id 'a5197234-610b-4f87-8f44-a8665e7e8a9d', tracking id '16690699-2899-45e3-9bb0-2b5e2e9bbcd6', request correlation id '0c2d7113-c7a3-4d05-b2aa-e6565501cf3c'."},{"code":"UnsupportedApiVersion","message":"The HTTP resource that matches the request URI 'http://amsweb.amsweb/subscriptions/a5197234-610b-4f87-8f44-a8665e7e8a9d/resourcegroups/arg-syd-platform-mgmt-management/providers/microsoft.operationalinsights/workspaces/law-syd-platform-mgmt-01' does not support the API version '2021-12-01-preview'."}]}
Hello ,
Thankx a lot for the new workbooks however few of the jobs have been failing on query to run .
Manual run of the query shows they fail to map "date_s" and LA worksspace dosent have table with a name "date_s"
would be able to help or point in right direction .
Thankx
Bagel
Investigate usage of integer as Azure Automation variables type in ARM template.
Use the SQL ingestion control table to bring the respective Log Analytics table suffix to be used in queries, instead of having it hard-coded in the runbook code.
Requires improving control table data model to include table abstract name to be used as key to get its suffix.
The runbook Export-PolicyComplianceToBlobStorage of the Automation Account is failing with an error Exception calling "Add" with "2" argument(s): "Item has already been added. Key in dictionary.
This seems to be related to the Get-AzPolicyDefinition.
I checked it locally on my computer and I saw that some $definitions not having the DisplayName-property.
Regarding to this, the if-clause if (-not($policyInitiatives[$initiative.PolicySetDefinitionId])) with the not-parameter will be $true as it has no value and the script try to add the key/value which already exists.
I was using the code with the following command successfully:
$definitions = Get-AzPolicyDefinition
foreach ($definition in $definitions)
{
if (-not($policyDefinitions[$definition.PolicyDefinitionId]))
{
try {
$displayName = $definition.Properties.DisplayName
if([string]::IsNullOrWhiteSpace($displayName)) {
$policyDefinitions.Add($definition.PolicyDefinitionId, 'NotGiven')
}
else {
$policyDefinitions.Add($definition.PolicyDefinitionId, $displayName)
}
}
catch {
Write-Warning "PolicyDefinitionId: $($definition.PolicyDefinitionId), DisplayName: $($definition.Properties.DisplayName)"
}
}
}Periodic remediation for Advisor right-size recommendations:
Azure Advisor not giving VM resizing Recommendations as per the docs, we have created few VMs with size Standard D4as v5 (4 vcpus, 16 GiB memory), which was running without any application ( max CPU utilisation < 6%). Still after 3 weeks no advisor recommendation.
As per the doc https://learn.microsoft.com/en-in/azure/advisor/advisor-cost-recommendations#resize-sku-recommendations , we should get the recommendation.
Can you kindly include VM resizing recommendation as well, instead of augmenting advisor.
Requires:
Let's say Az Advisor recommends downsizing to a SKU a customer does not want to use for whatever reason (Reservations policy, performance requirements, etc.). AOE should allow customers to provide a list of allowed SKU families, from which Az Advisor recommendations would be updated to use a SKU from those families instead.
The URL used for ingesting into Log Analytics only works for Azure Global. Based on the cloud environment variable, it should use the correct URL.
Provide list of Azure AD Service Principal certificates and secrets about to expire
The Azure SQL URL used, for example, when deploying the solution, only works for Azure Global. Needs better support for other clouds (e.g., US Government)
On the 7th Day of a recommendation cycle, prior to the weekly Recommendation runs, the Power BI View filters out the Week Old Data, and does not have new data to replace it with...
This Test Installation within a subscription has a first run on 6/22, second run on 6/29, and todays run has yet to occur
I have verified all the current recommendation data is in the power BI file
Note: I know that I can change the schedule to go after hours or before I start using the views etc, but this introduces a window of time your view will not work if you setup the wrong scheduling.
Hello,
After the deployment the parameter minimum TLS version of SQL server is not set/defined.
It generates TLS alerts if you have policies or CSPM in place.
Are you able to set/configure it to TL1.2 during the deployment?
Thank you!
Should be implemented in conjunction with #11
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.