azure/openshift

Azure Red Hat OpenShift

Azure Red Hat Openshift (ARO)

PLEASE CHECK OFFICIAL AZURE DOCUMENTATION HERE

Overview

This repo is intended for tracking issues with Azure Red Hat Openshift (ARO). It is monitored by the ARO product team. For direct customer support, please take a look at the Azure Support options. Note that the following channels are also monitored and should be used as suggested:

• Updates about the service, including new features and new Azure regions: OpenShift feed in Azure Updates

Code of conduct

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

openshift's Issues

HTTP/2 support

It look like that HTTP/2 is not enabled on the router?

Can't recreate cluster with same resourcegroup

Previously I had created a cluster named kadosaprod (in a RG named the same) using az openshift create. I destroyed it using az openshift delete and also removed the RG (and all traces I could find of resources created by said cluster) but I can't seem to re-create the cluster in a similarly named RG. az openshift create chugs along for a while but I get an error at the end, here's the debug output:

msrest.exceptions : Creating customer service principal failed. Detail: graphrbac.ApplicationsClient#Create: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="Unknown" Message="Unknown service error" Details=[{"odata.error":{"code":"Request_BadRequest","date":"2018-12-03T16:37:03","message":{"lang":"en","value":"Another object with the same value for property identifierUris already exists."},"requestId":"ae79af44-422b-4f2f-966e-59c213209522","values":[{"item":"PropertyName","value":"identifierUris"},{"item":"PropertyErrorCode","value":"ObjectConflict"}]}}]
Deployment failed. Correlation ID: d4df04e6-e072-48e5-a440-e7b335d931a1. Creating customer service principal failed. Detail: graphrbac.ApplicationsClient#Create: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="Unknown" Message="Unknown service error" Details=[{"odata.error":{"code":"Request_BadRequest","date":"2018-12-03T16:37:03","message":{"lang":"en","value":"Another object with the same value for property identifierUris already exists."},"requestId":"ae79af44-422b-4f2f-966e-59c213209522","values":[{"item":"PropertyName","value":"identifierUris"},{"item":"PropertyErrorCode","value":"ObjectConflict"}]}}]

cc @julienstroheker @amanohar

Update documentation with issues around OMS workspace billing restrictions

Customers will need to switch to new April 2018 pricing model (if they are not using it already) using the Azure Monitoring - Usage and estimated costs blade

https://portal.azure.com/?feature.customportal=false%2F#blade/Microsoft_Azure_Monitoring/AzureMonitoringBrowseBlade/usageAndCosts

Document first time OSA creation error because OSA tenant is not allowed to manage subscription

Typically in ARM subs are only registered and managed by one tenant (their home tenant). However, sometimes some aspects of a subscription can be managed by managed application publisher tenant. For this subscription record needs to be updated and this happens the first time an operation happens on sub id + publisher tenant combination.

This operation can fail sometimes with the error below because it takes times for cache to be invalidated.

The access token for this request was issued by the tenant '' which is neither the owner tenant '' nor one of the tenants '' which can manage this subscription ''. The access token must be issued from proper tenant.

Add documentation that customer might need to wait 10-15mins for cache to invalidated and data to be replicated.

Authentication error on Dashboard

Hi all,

i was just creating an OS cluster.
After browsing to the dashboard and logging on with my tenant account, i get the following error message:

"An authentication error occurred."

For test purpose I´ve granted the OC´s ServicePrincipal all Permissions on AAD and MS Graph.

Has anyone an idea, how to solve this problem?

Thanks!

Best regards,

René

az feature register step documentation is unclear

When I run az feature register --namespace Microsoft.ContainerService -n openshiftmanagedcluster I get the message Once the feature 'openshiftmanagedcluster' is registered, invoking 'az provider register -n Microsoft.ContainerService' is required to get the change propagated.

When I then run az provider register -n Microsoft.ContainerService I get the message Registering is still on-going. You can monitor using 'az provider show -n Microsoft.ContainerService'. (note the erroneous trailing quote in the error message).

When I then run az provider show -n Microsoft.ContainerService, it shows "registrationState": "Registered", but if I retry the previous steps I get the same messages as before. How do I know when the process is done?

Also, do I need to run some feature or provider registration process for Microsoft.Solutions?

How do we report OSA installation issues?

Follett is previewing OSA and I have some feedback on installation issues.

Do we just add issues here or on openshift-azure-discuss? This is not clear from the doc.

document registration of openshiftmanagedcluster resource

az feature register --namespace Microsoft.ContainerService -n openshiftmanagedcluster

documentation about number of cores needed

document how many cores are needed and link to how to request more cores from portal

Don't allow changing AAD app id, secret, or tenant during update call

Validate that an update/scale operation with different/changed AAD Auth profile values is not allowed.

Add `LinkedAuthorizationFailed` error to known issues page

First time OSA cluster creator customer might run into the following error. This error should happen the first time a subscription tries to create an OSA cluster.

{
 "code": "LinkedAuthorizationFailed",
 "message": "The client has permission to perform action 'Microsoft.Network/virtualNetworks/subnets/join/action' on scope '/subscriptions/<SUBID>/resourcegroups/<RGNAME>/providers/Microsoft.Compute/virtualMachineScaleSets/ss-compute', however the current tenant '<TENANTID>' is not authorized to access linked subscription '<SUBID>'."
}

Recommended mitigation: Customer should wait 10 mins and try creating a new cluster.

New resource group is created during installation

I created resource group prdusefssocpdemorg
Then create:

CLUSTER_NAME=fssocpdemo
RESOURCE_GROUP_NAME=prdusefssocpdemorg
LOCATION=eastus
az openshift create --resource-group $RESOURCE_GROUP_NAME --name $CLUSTER_NAME -l $LOCATION --fqdn $FQDN 

This created a cluster but in a new resource group called "914a092f-ebca-4ccb-XXa4-XXXXX".

I expected all the new resources to be created in the existing RG prdusefssocpdemorg, but it is empty.

If this is expected behavior, what is the point of supplying the resource group?

fqdn validation - No uppercase

Add steps to accept Marketplace agreement in the documentation

Add steps to accept Marketplace agreement in the documentation for OS and Azure Managed Application images.

Customer will need to run the following commands in Azure Powershell (can be done via CloudShell in Portal) to accept marketplace agreement:

Set-AzureRmContext -SubscriptionId "<SUBSCRIPTION_ID>"

Get-AzureRmMarketplaceTerms -Publisher redhat -Product osa -Name osa_310 | Set-AzureRmMarketplaceTerms -Accept

Get-AzureRmMarketplaceTerms -Publisher osatesting -Product open-shift-azure-proxy-preview -Name byovnet | Set-AzureRmMarketplaceTerms -Accept

Get-AzureRmMarketplaceTerms -Publisher osatesting -Product open-shift-azure-proxy-preview -Name default| Set-AzureRmMarketplaceTerms -Accept

create: FQDN provided is not used; new cluster unavailable

Creating a new cluster

CLUSTER_NAME=fssocpdemo
RESOURCE_GROUP_NAME=prdusefssocpdemorg
LOCATION=eastus
FQDN=$CLUSTER_NAME.$LOCATION.cloudapp.azure.com
# az openshift create --resource-group $RESOURCE_GROUP_NAME --name $CLUSTER_NAME -l $LOCATION --fqdn $FQDN

After creation try to access the new cluster

$ ping prdusefssocpdemorg.eastus.cloudapp.azure.com
ping: prdusefssocpdemorg.eastus.cloudapp.azure.com: Name or service not known

So the dns name is invalid.

FQDN in the return JSON has a valid DNS name.

 "routerProfiles": [
    {
      "fqdn": "osab882XXXXXXX.eastus.cloudapp.azure.com",
      "name": "default",
      "publicSubdomain": "197bff743daXXXXXX.eastus.azmosa.io"
    }

Now use the fqdn value:

curl -k  -v https://osab882XXXXXXX.eastus.cloudapp.azure.com
* About to connect() to osab882XXXXXXX.eastus.cloudapp.azure.com port 443 (#0)
*   Trying 137.135.107.111...
* Connected to osab882XXXXXXX.eastus.cloudapp.azure.com (137.135.107.111) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=197bff743daXXXXXX.eastus.azmosa.io
*       start date: Jan 04 13:33:25 2019 GMT
*       expire date: Jan 04 13:33:25 2021 GMT
*       common name: 197bff743daXXXXXX.eastus.azmosa.io
*       issuer: CN=openshift-signer
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: osab882XXXXXXX.eastus.cloudapp.azure.com
> Accept: */*
> 
* HTTP 1.0, assume close after body
< HTTP/1.0 503 Service Unavailable

The OSA console is not available at this location.

Unable to deploy OSBA to OSA

Following the instructions described here: https://github.com/Azure/open-service-broker-azure to deploy the Open Service Broker for Azure on OpenShift:

oc process -f https://raw.githubusercontent.com/Azure/open-service-broker-azure/master/contrib/openshift/osba-os-template.yaml
-p ENVIRONMENT=AzurePublicCloud
-p AZURE_SUBSCRIPTION_ID=$AZURE_SUBSCRIPTION_ID
-p AZURE_TENANT_ID=$AZURE_TENANT_ID
-p AZURE_CLIENT_ID=$AZURE_CLIENT_ID
-p AZURE_CLIENT_SECRET=$AZURE_CLIENT_SECRET
| oc create -f -

...eventially fails:

secret/osba-redis created
secret/osba-open-service-broker-azure-auth created
secret/osba-open-service-broker-azure created
service/osba-redis created
service/osba-open-service-broker-azure created
persistentvolumeclaim/osba-redis-pv-claim created
deployment.extensions/osba-redis created
deployment.extensions/osba-open-service-broker-azure created
Error from server (Forbidden): clusterservicebrokers.servicecatalog.k8s.io is forbidden: User "" cannot create clusterservicebrokers.servicecatalog.k8s.io at the cluster scope: no RBAC policy matched

Subsequently trying to run the svcat CLI to enumerate services gives a similar error:

svcat get brokers -n osba
Error: unable to list cluster-scoped brokers (clusterservicebrokers.servicecatalog.k8s.io is forbidden: User "" cannot list clusterservicebrokers.servicecatalog.k8s.io at the cluster scope: no RBAC policy matched)

Add a page to list Azure regions OpenShift on Azure is supported in

Add a page to list Azure regions that support OpenShift on Azure.

Doument registration of Microsoft.Solutions RP

Customer need to register for Microsoft.Solutions RP before they can create an OSA cluster.

Known issues should reflect registration on Microsoft.Network Provider

We get an exception if Microsoft.Networks namespace is not registered:

The subscription is not registered to use namespace 'Microsoft.Network'

On https://github.com/Azure/OpenShift/blob/master/docs/known-issues.md#providers-and-features-must-be-registered-manually, we need to add Microsoft.Networks PowerShell commands

workaround: we can do it also from portal

Need admin approval - Authentication

Seems Azure changed their policy to use the following permissions :
Windows Azure Active Directory - Sign in and read user profile

This cause a Need admin approval message at the first authentication into the cluster using Azure AD Authentication method.

--fqdn seems required but is not showing up like that in help

# az openshift create -g kargakisprod -n kargakisprod -l westeurope
Invalid value specified for property 'displayName' of resource 'Application'.
bash-4.4# az openshift create --help

Command
    az openshift create : Create a new managed OpenShift cluster.

Arguments
    --name -n           [Required] : Name of the managed OpenShift cluster.
    --resource-group -g [Required] : Name of resource group. You can configure the default group
                                     using `az configure --defaults group=<name>`.
    --aad-client-app-id            : The ID of an Azure Active Directory client application of type
                                     "Native".
    --aad-client-app-secret        : The secret of an Azure Active Directory client application.
    --aad-tenant-id                : The ID of an Azure Active Directory tenant.
    --fqdn                         : FQDN for OpenShift API server loadbalancer internal hostname.
                                     For example myopenshiftcluster.eastus.cloudapp.azure.com.
bash-4.4# az openshift create -g kargakisprod -n kargakisprod -l westeurope --fqdn kargakisprod.westeurope.cloudapp.azure.com
# Failed with a different error

Microsoft.ContainerService - RegistrationState stays "Pending"

Hi,

I'm super keen to try out the Managed OpenShift on Azure, but when I take the first step as described:

Register-AzProviderFeature -ProviderNamespace Microsoft.ContainerService -FeatureName openshiftmanagedcluster

..I've been checking for about the last 8 hours, but it's still PENDING, and something tells me it isn't going to change any time soon. Any ideas ?? Please let me know if I can provide any further information.

FeatureName             ProviderName               RegistrationState
-----------             ------------               -----------------
openshiftmanagedcluster Microsoft.ContainerService Pending

Thanks and regards,

Jim

Doc to explain which OpenShift features work / don't work on OSA

Can we create a simple doc or section that explains what doesn’t work in OpenShift that the customers may be expecting? i.e. No SSH access, No Privileged user, No customer logging (Kibana), etc?

Update documentation with issues around OMS workspace region restrictions

OMS workspace creation is not available in all regions due to capacity constraint and this might further be limited on a per subscription basis. As a result of that OSA will be using one region per sovereignty to maximize changes to success cluster create.

However, creation might still fail. Update documentation so customers are aware of this restriction.

document only scale is supported for update scenarios

Links to docs/*.md from top level README.md

Need to link to all docs/ documents from top level README.md to make them easy to find

az openshift show not working

# az openshift show --resource-group X --name Y
'NoneType' object has no attribute 'peer_vnet_id'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/knack/cli.py", line 197, in invoke
    cmd_result = self.invocation.execute(args)
  File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 369, in execute
    six.reraise(*sys.exc_info())
  File "/usr/local/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 343, in execute
    result = cmd(params)
  File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 182, in __call__
    return self.handler(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/arm.py", line 701, in handler
    show_exception_handler(ex)
  File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/arm.py", line 711, in show_exception_handler
    raise ex
  File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/arm.py", line 699, in handler
    return getter(**args)
  File "/usr/local/lib/python3.6/site-packages/azure/cli/command_modules/acs/custom.py", line 2368, in openshift_show
    return _remove_osa_nulls([mc])
  File "/usr/local/lib/python3.6/site-packages/azure/cli/command_modules/acs/custom.py", line 2257, in _remove_osa_nulls
    delattr(managed_cluster.network_profile, attr)
AttributeError: 'NoneType' object has no attribute 'peer_vnet_id'

document vnet integration and vnet peering using the CLI

How to use already existing Azure AD App when creating openshift cluster?

Customer got error message below when creating openshift cluster.
az openshift create --resource-group $CLUSTER_NAME --name $CLUSTER_NAME -l $LOCATION --fqdn $FQDN
Directory permission is needed for the current user to register the application. For how to configure, please refer 'https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal'. Original error: Insufficient privileges to complete the operation.

Customer doesn't have privilege to register AD app and have to ask Azure AD admin to do that.
How can customer create openshift cluster with existing AD App?

Thank you!

Document allowed VM sizes and return allowed sizes in an error message to the customer

Document allowed VM sizes and return allowed sizes in an error message to the customer.

T&C creation failure

→ az openshift create --resource-group $CLUSTER_NAME --name $CLUSTER_NAME -l $LOCATION --fqdn $FQDN
Deployment failed. Correlation ID: 095ef6e4-6f39-4bba-94d8-4bc50f925c8c. unable to create managed app. error: managedapplications.ApplicationsClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ResourcePurchaseValidationFailed" Message="User failed validation to purchase resources. Error message: 'Legal terms have not been accepted for this item on this subscription. To accept legal terms using PowerShell, please use Get-AzureRmMarketplaceTerms and Set-AzureRmMarketplaceTerms API(https://go.microsoft.com/fwlink/?linkid=862451) or deploy via the Azure portal to accept the terms'

Delete extraneous branches

Do we need branches amanohar-patch-1 and osadocs on this repo? If not, please delete.

Document create operation idempotency issue

Re putting a previously failed cluster might not work in some cases. Document that customer should delete and recreate cluster if a create operation fails.

Create Process not Accepting Passed in Azure Tenant ID

Passing in app registration credentials from a newly created Azure AD Tenant that is not the default Azure Tenant associated with the Subscription fails.

I created a new Azure AD Tenant so I could control all App Registrations. I then created a New App Registration in the new tenant. I used the newly created App Registration details and new Azure AD Tenant ID in the az openshift create command. After the Cluster was provisioned and tried to sign in to the Open Shift Cluster using an ID from the newly created Azure AD Tenant, it errored saying it could not find the App Registration in Tenant X where Tenant X was the default Azure AD Tenant associated with the Subscription versus the newly created Azure AD Tenant.

Show compute node count in az openshift show table view

It would be very helpful to have that info in openshift show instead of having to read through json/yaml in order to figure out my cluster's capacity.

bash-4.4$ az openshift show -g kargakistesting -n kargakistesting -o table
Fqdn                                       Location    Name             OpenShiftVersion    ProvisioningState    PublicHostname    ResourceGroup
-----------------------------------------  ----------  ---------------  ------------------  -------------------  ----------------  ---------------
kargakistesting.eastus.cloudapp.azure.com  eastus      kargakistesting  v3.11               Succeeded                              kargakistesting

add link to quick walkthrough and known issues from readme

document hidden resource in the portal

AAD required-resource-access

@amanohar @julienstroheker

We are currently setting up our AAD app as follows, the /docs/aad-application-configuration.md (and also CLI?) need to be updated:

az ad app create \
    ...
    --required-resource-accesses @- <<'EOF'
[
    {
      "resourceAppId": "00000003-0000-0000-c000-000000000000",
      "resourceAccess": [
        {
          "id": "5b567255-7703-4780-807c-7be8301ae99b",
          "type": "Role"
        },
        {
          "id": "37f7f235-527c-4136-accd-4a02d197296e",
          "type": "Scope"
        }
      ]
    }
]
EOF

Meaning of the magic numbers:
"00000003-0000-0000-c000-000000000000" -> microsoft graph
"5b567255-7703-4780-807c-7be8301ae99b" -> read all groups # used by aad group sync
"37f7f235-527c-4136-accd-4a02d197296e" -> sign users in
"type": "Role" -> application permission
"type": "Scope" -> delegated permission

best effort support

our README.md mentions "best effort support" - my feeling is that we should say "unsupported". @amanohar your thoughts?

keyboard interrupt stacktrace in the cli

These stacktraces are annoying, we should show more user-friendly output.

bash-4.4$ az openshift create -g counttest -n counttest -c 1 -l westeurope
^CTraceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/site-packages/azure/cli/__main__.py", line 32, in <module>
    az_cli = get_default_cli()
  File "/usr/local/lib/python3.6/site-packages/azure/cli/core/__init__.py", line 497, in get_default_cli
    from azure.cli.core.commands import AzCliCommandInvoker
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 674, in exec_module
  File "<frozen importlib._bootstrap_external>", line 779, in get_code
  File "<frozen importlib._bootstrap_external>", line 487, in _compile_bytecode
KeyboardInterrupt

Update the CLI image name to the official one

mcr.microsoft.com/osa/cli

How to become cluster admin ?

Hi,

After sucessfully deployed a ocp cluster following your guide , how can I log as system:admin ?

Regards.

Another object with the same value for property identifierUris already exists.

If you use az openshift create and let it create your AAD app, then use az openshift delete, then do another az openshift create with the same name, you get the error Another object with the same value for property identifierUris already exists.

I'm not immediately sure what the right route to improve this is, but it definitely needs improvement.

az openshift create "DeploymentNotFound"

Hi,

When running az openshift create we're getting a 404 "DeploymentNotFound" error.

# CLUSTER_NAME=int-az-ocp
# LOCATION=northeurope
# FQDN=$CLUSTER_NAME.$LOCATION.cloudapp.azure.com
$ az group create --name $CLUSTER_NAME --location $LOCATION
$ az openshift create --resource-group $CLUSTER_NAME --name $CLUSTER_NAME -l $LOCATION --fqdn $FQDN
Deployment failed. Correlation ID: _guid_. resources.DeploymentsClient#Get: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="DeploymentNotFound" Message="Deployment '_otherguid_' could not be found."

• Subscription is whitelisted
• Has been tried multiple times
• Tested on westeurope and northeurope

Add Azure File Storage Class to OSA

Please add support for Azure File storage class to OSA.

Only Azure Disk is Available
#oc get storageclass
NAME PROVISIONER AGE
azure (default) kubernetes.io/azure-disk 19d

~Mo

cleanup: Service Principal (app) not deleted when OSA is removed

Deleting OpenShift (OSA) seems to delete all services successfully but Service Principal / App Registration entries are left behind.

When SP is created by the installer it would make sense to delete it when OSA is deleted.

Register Microsoft.Solutions

Got this when running on the e2e sub:

Original error: autorest/azure: Service returned an error. Status=\u003cnil\u003e Code=\"MissingSubscriptionRegistration\" Message=\"The subscription is not registered to use namespace 'Microsoft.Solutions'. See https://aka.ms/rps-not-found for how to register subscriptions.\" Details=[{\"code\":\"MissingSubscriptionRegistration\",\"message\":\"The subscription is not registered to use namespace 'Microsoft.Solutions'. See https://aka.ms/rps-not-found for how to register subscriptions.\",\"target\":\"Microsoft.Solutions\"}]"

Might need to document az provider register -n Microsoft.Solutions for registering AMA provider.

ApplianceManagedResourceGroupMismatch error

I am seeing the following error when trying to create a cluster in a group that has been used in the past for an older prod cluster. I am using the OSA golang sdk to start the cluster and not the az command.

  Code="CreateManagedAppError" Message="unable to create managed app. error: managedapplications.ApplicationsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=400 Code=\"ApplianceManagedResourceGroupMismatch\" Message=\"The given managed resource group '/subscriptions/XXXXXXXX-43d0-43d1-a01a-17e584a4ef69/resourceGroups/aae9344a-883c-4570-a585-4a6882b7851b' did not match that in the existing appliance definition '/subscriptions/225e02bc-43d0-43d1-a01a-17e584a4ef69/resourceGroups/51af21ca-17d5-4d6e-b5dc-81bd55f74078'.\" Persisting configuration into storage account failed."

Unclear whether this is expected or not.

Master nodes have public IP address and they are reachable with SSH

Noticed that master node instances have public IP addresses and they are accessible via SSH. I don't have key or password to login, but due some vulnerability or config failure "wrong" user may get access to machine. In case Openshift vnet is peered with some on-prem transit vnet, this user might get access to on-prem network.

vnet peering doesn't work when target vnet is under different subscription.

Installation failds when vnet to peer is in under different subscription. Tested installation by giving vnet id for vnet that was in under same resource group. Installation worked and vnets were peered. When tried to install with peering to vnet from different subscription, installation failed.

Deployment failed. Correlation ID: xxxxx. Creating virtual network peering failed. Detail: network.VirtualNetworkPeeringsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="LinkedAuthorizationFailed" Message="The client 'xxxxxx' with object id 'xxxxx' has permission to perform action 'Microsoft.Network/virtualNetworks/virtualNetworkPeerings/write' on scope '/subscriptions/xxxxxx/resourceGroups/xxxxx/providers/Microsoft.Network/virtualNetworks/vnet/virtualNetworkPeerings/OSAClusterVNetPeer'; however, it does not have permission to perform action 'peer/action' on the linked scope(s) '/subscriptions/xxxxx/resourceGroups/Customer-Prod-Net-RG/providers/Microsoft.Network/virtualNetworks/Customer-Prod-VNET'."