azure/apim-landing-zone-accelerator

The Azure API Management landing zone accelerator provides an architectural approach and reference implementation to prepare landing zone subscriptions for a secure API Management infrastructure.

Azure API Management Landing Zone Accelerator

Azure API Management Landing Zone Accelerator provides packaged guidance with reference architecture and reference implementation along with design guidance recommendations and considerations on critical design areas for provisioning APIM with a secure baseline. They are aligned with industry proven practices, such as those presented in Azure landing zones guidance in the Cloud Adoption Framework.

Reference Architecture

🔍 Design areas

The enterprise architecture is broken down into six different design areas, where you can find the links to each at:

Design Area	Considerations	Recommendations
Identity and Access Management	Design Considerations	Design Recommendations
Network Topology and Connectivity	Design Considerations	Design Recommendations
Security	Design Considerations	Design Recommendations
Management	Design Considerations	Design Recommendations
Governance	Design Considerations	Design Recommendations
Platform Automation and DevOps	Design Considerations	Design Recommendations

🚀 Deployment scenarios

This repo contains the Azure landing zone accelerator's reference implementations, all with supporting Infrastructure as Code artifacts. The scenarios covered are:

▶️ Scenario 1: Azure API Management - Secure Baseline

Deploys APIM with a secure baseline configuration with no backends and a sample API.

▶️ Scenario 2: Azure API Management - Function Backend

On top of the secure baseline, deploys a private Azure function as a backend and provision APIs in APIM to access the function.

▶️ Scenario 3: Azure API Management - Gen AI Backend

On top of the secure baseline, deploys private Azure OpenAI endpoints (3 endpoints) as backend and provision API that can handle multiple use cases.

More reference implementation scenarios will be added as they become available.

Supported Regions

Some of the new Azure OpenAI policies are not available in al the regions yet. If you see the deployment failures, try chosing a different region. The following regions are more likely to work.

australiacentral, australiaeast, australiasoutheast, brazilsouth, eastasia, francecentral, germanywestcentral, koreacentral, northeurope, southeastasia, southcentralus, uksouth, ukwest, westeurope, westus2, westus3

Got a feedback

Please leverage issues if you have any feedback or request on how we can improve on this repository.

---

Data Collection

The software may collect information about you and your use of the software and send it to Microsoft. Microsoft may use this information to provide services and improve our products and services. You may turn off the telemetry as described in the repository. There are also some features in the software that may enable you and Microsoft to collect data from users of your applications. If you use these features, you must comply with applicable law, including providing appropriate notices to users of your applications together with a copy of Microsoft's privacy statement. Our privacy statement is located at https://go.microsoft.com/fwlink/?LinkId=521839. You can learn more about data collection and use in the help documentation and our privacy statement. Your use of the software operates as your consent to these practices.

Telemetry Configuration

Telemetry collection is on by default.

To opt-out, set the variable ENABLE_TELEMETRY to false in .env file.

---

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

apim-landing-zone-accelerator's Issues

NSG rules to be reviewed and adjusted on each NSG

From Enterprise-Scale-APIM created by seenu433: cykreng/Enterprise-Scale-APIM#32

APIM needs some NSG inbound and outbound rules when internal
RDP rule is not required on all

APIOps Integration

PFX encoding for bicep deployment

From Enterprise-Scale-APIM created by cykreng: cykreng/Enterprise-Scale-APIM#52

null

Update the reference links in user-guide to reflect the new /reference-implementations/ folder

From Enterprise-Scale-APIM created by paromitaroy: cykreng/Enterprise-Scale-APIM#72

Since the deployment folder has been removed and the files have been moved under /reference-implementations/ folder, update the ref . links in the user guide.

Add OpenAI Endpoints for the GenAI Workload

• Provision 3 openAI Endpoints. (1 to be named as PTU, 1 as PAYG1 and 1 as PAYG2)
• Setup Managed Identities between APIM and OpenAI.

backend.bicep has a hardcoded storage name

From Enterprise-Scale-APIM created by ahmedsza: cykreng/Enterprise-Scale-APIM#55

@winmike - Not sure if you created this script. It has a hardcoded storage account name and hence will fail when others run.

reference keyvault instead of PFX in Bicep

Verify with Srini if this is done already

Added appropriate tags on the resources

From Enterprise-Scale-APIM created by seenu433: cykreng/Enterprise-Scale-APIM#31

Tags shoudl be added to all the resources created

Terraform implementation

From Enterprise-Scale-APIM created by cykreng: cykreng/Enterprise-Scale-APIM#63

null

ARM template implementation

From Enterprise-Scale-APIM created by cykreng: cykreng/Enterprise-Scale-APIM#53

null

BCDR Design Recommendation

From Enterprise-Scale-APIM created by cykreng: cykreng/Enterprise-Scale-APIM#71

null

Enable APIM managed identity for named values stored in Key Vault

When creating APIM policies customers often like to reference secrets stored in Key Vault for use in APIM named values. Seeing as though storing secrets in Key Vault is deemed best practice, we should enable a managed identity on APIM and assign get and list permissions to the shared services Key Vault.

Reference

Using key vault secrets is recommended because it helps improve API Management security:

Secrets stored in key vaults can be reused across services
Granular access policies can be applied to secrets
Secrets updated in the key vault are automatically rotated in API Management. After update in the key vault, a named value in API Management is updated within 4 hours. You can also manually refresh the secret using the Azure portal or via the management REST API.

Happy to work on this if agreed.

DevOps pipeline for function deployment

Pipeline warnings and secure parameters

On running the pipeline, the following warnings/errors occur...

Error: WARNING: /home/runner/work/apim-landing-zone-accelerator/apim-landing-zone-accelerator/reference-implementations/AppGW-IAPIM-Func/bicep/shared/createvmwindows.bicep(18,7) : Warning secure-secrets-in-params: Parameter 'password' may represent a secret (according to its name) and must be declared with the '@secure()' attribute. [https://aka.ms/bicep/linter/secure-secrets-in-params]

Then various iterations of...

/home/runner/work/apim-landing-zone-accelerator/apim-landing-zone-accelerator/reference-implementations/AppGW-IAPIM-Func/bicep/gateway/appgw.bicep(183,13) : Warning use-resource-id-functions: If property "id" represents a resource ID, it must use a symbolic resource reference, be a parameter or start with one of these functions: extensionResourceId, guid, if, reference, resourceId, subscription, subscriptionResourceId, tenantResourceId. [https://aka.ms/bicep/linter/use-resource-id-functions]

Issue 1 requires the VMPassword to be set as a secure parameter.

Issue 2 requires a reformat of the resource id references within the app gateway config. For example...

resourceId('Microsoft.Network/applicationGateways/backendHttpSettingsCollection', appGatewayName, 'https')

Potential cost estimate

From Enterprise-Scale-APIM created by cykreng: cykreng/Enterprise-Scale-APIM#8

ADO dependency

From Enterprise-Scale-APIM created by seenu433: cykreng/Enterprise-Scale-APIM#12

Having a dependency on ADO may not be appropriate and ADO self-hosted agent deployment should be optional

Make ADO agent deployment optional using an env variable in the shared.bicep file

zero-trust scenario

From Enterprise-Scale-APIM created by seenu433: cykreng/Enterprise-Scale-APIM#34

Need a scenario where the application gateway routes the traffic to a firewall to align with the zero-trust scenario

https://docs.microsoft.com/en-us/azure/architecture/example-scenario/gateway/application-gateway-before-azure-firewall#:~:text=+Zero-trust+network+for+web+applications+with+Azure,spoke+example.+Typically%2C+a+hub+an...+More+

Extend/New Scenario to include self-hosted Gateways

For hybrid workloads there is often the need to deploy self hosted gateways. The scenario should be expanded to support this including:

1. Networking implications
2. Policies with conditional logic to support different backends in each location
3. OPerational/Monitoring Impacts

Application Gateway Overview

From Enterprise-Scale-APIM created by petemessina: cykreng/Enterprise-Scale-APIM#60

Is it possible in the documentation and user guide to add some high-level reasoning around the use and why app gateway is in the architecture? When working through implementations with customers it is a common question I see and if we can add some guidance here it would clarify the need.

DevOps for TF

Add GenAI related scenarios to the accelerator.

• Add scenarios that are appropriate for GenAI flow
• Add infra automation to provision the policies within APIM
• For Azure OpenAI, use the URLs to beginwith.

Include information for ARM validation

Starter Azure Policy for APIM when deploying the construction set

From Enterprise-Scale-APIM created by cykreng: cykreng/Enterprise-Scale-APIM#27

For instance, customers can use Get-AzPolicyAlias -NamespaceMatch 'ApiManagement' to get a list of Azure Policy aliases supported by APIM to create custom Azure policies for APIM. Ideally, we should create a custom set of policies to recommend to customers when deploying the LZ as a starting point...

Missing files under "Deploy the Backend" section

In the readme file, https://github.com/Azure/apim-landing-zone-accelerator/blob/main/docs/README.md#deploy-the-backend. one yml file seems to be missing, and the other one doesn't have a link to it. Any thoughts?

Thanks in advance,

Investigate implication of user vs system managed identity

Using a user-managed identity decouples the lifecycle of the identity from the resources using it.
Does it simplify the deployment, or improve performance to move everything to a user-managed identity, or should we just keep it system to align the lifecycle.

Todo:
Determine our ultimate approach

Ref:
https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/managed-identity-best-practice-recommendations#choosing-system-or-user-assigned-managed-identities

Create deeper network topology diagram

From Enterprise-Scale-APIM created by winmike: cykreng/Enterprise-Scale-APIM#50

• VNET structure
• network segments/subnets
• sample address spaces
• endpoints
• NSG
• DNS integration
• outbound IP addresses
• gateways
• ...

Issues with backend

From Enterprise-Scale-APIM created by seenu433: cykreng/Enterprise-Scale-APIM#46

The function app with code does not have the storage configured and hence the runtime is unreachable.
Link the storage account with the function app.

The container function app seems to be redundant and consider deleting it.

Do we need pipelines/GitHub/apim-eslz.yml ? if so need to update the repository.

https://github.com/Azure/apim-landing-zone-accelerator/blob/main/reference-implementations/AppGW-IAPIM-Func/pipelines/GitHub/apim-eslz.yml

Line 35. Poiting to the previous repo - repository: cykreng/Enterprise-Scale-APIM

put vm_pw in keyvault and reference the keyvault for anywhere that uses vm_pw

From Enterprise-Scale-APIM created by cykreng: cykreng/Enterprise-Scale-APIM#15

null

User guide

From Enterprise-Scale-APIM created by cykreng: cykreng/Enterprise-Scale-APIM#49

https://github.com/cykreng/Enterprise-Scale-AppService/tree/user-guide/docs

Feel free to use the both as a starting point

create vmwindows fails as a result of refactoring

From Enterprise-Scale-APIM created by ahmedsza: cykreng/Enterprise-Scale-APIM#56

@briandenicola
in createvmwindows this was modified to
module nic './vm-nic.bicep' = {
name: '${vmName}-nic'
params: {
location: location
subnetId: subnetId
nicName: '${vmName}-nic'
}
}

but there is no nicName parameter in vm-nic.bicep. Either need to revert to vmname or variable name in vmnic needs to be changed

Change the git urls in user guide

Parametrize backend bicep and ensure consistent passing with main.bicep

From Enterprise-Scale-APIM created by cykreng: cykreng/Enterprise-Scale-APIM#28

null

Multi-region serverless APIM gateway deployment fronted by Azure Front Door

From Enterprise-Scale-APIM created by kunalbabre: cykreng/Enterprise-Scale-APIM#75

Scenario:
Deploying APIM serverless and exposing it via Azure front door (Both single and multi-region deployments).

e.g., https://www.kunalbabre.com/simple-azure-application-deployment-patterns/

[Private End Points were unsupported by APIM when I blogged above (we can expand it now)]

Advantage:
These can help with much wide-scale adoption of secure API Gateway for many of our customers.

Create automation for recommended custom roles using APIM RBAC operations

From Enterprise-Scale-APIM created by cykreng: cykreng/Enterprise-Scale-APIM#26

https://docs.microsoft.com/en-us/azure/api-management/api-management-role-based-access-control#custom-roles

(From Jonas Norlund)
Is it scope to create automation for this or is it sufficient with link? I have implemented the following scenario:

1. API developer(s) get Reader RBAC role
2. API team get's an SP that has a custom role assignment to the api(s) that they should have access to. This SP is only used in ADO pipelines.

Automation creates, api(s), SP, custom role, assignments.

check for agent pool name that isn't default and that it shows up on ADO

Bicep Deployment Failed after running an hour - System.Management.Automation.ParameterBindingException: Missing an argument for parameter 'certificateName'.

Hey team,

I follow the readme file to configure the deployment yaml file and chose to use auto generated self signed certificate

AZURE_LOCATION: 'westus2'
RESOURCE_NAME_PREFIX: 'apimlz'
ENVIRONMENT_TAG: 'dev'
DEPLOYMENT_NAME: 'ase-demo-deployment'
VM_USERNAME: 'agent'
ACCOUNT_NAME: 'github.com/xuhongl'
CICD_AGENT_TYPE: none
CERT_TYPE: 'selfsigned'

however, after running for ~1h, the Bicep deployment failed with error below

looks like a parameter is missing here - is it possible for us to add this to the parameter file and update our deploy readme?

Thanks in advance

Private DNS zone creates azure-api.net zone, which causes authorizations to break

When deploying API Management in Internal mode, the accelerator creates a private DNS zone named azure-api.net. This causes name lookups to any azure-api.net resource not specified in the zone to fail. The new Authorizations feature calls a public endpoint called "logic-apis-regionname.azure-apim.net" which is an alias under azure-api.net, so this resolution will fail.

The private DNS zone created should be named "apimname.azure-api.net" so it is not authoritative for all azure-api.net endpoints.

create depends on folder with PFX generation

Provide alternative for ADO

From Enterprise-Scale-APIM created by seenu433: cykreng/Enterprise-Scale-APIM#13

Provide an option to deploy Github runner instead of ADO self-hosted agent

Evaluating adherence to APIM security baseline

From Enterprise-Scale-APIM created by cykreng: cykreng/Enterprise-Scale-APIM#20

null

Update deployment doc: networking output incl. bastion

From Enterprise-Scale-APIM created by winmike: cykreng/Enterprise-Scale-APIM#67

the screenshot must be replaced by:

Networking module - Subnet id for agent and jumpbox required

From Enterprise-Scale-APIM created by ahmedsza: cykreng/Enterprise-Scale-APIM#4

@cenkms

Can you return the subnet id for the jumbox and agent. Need these for the jumpbox and agent VM please

Terraform implementation "VMExtensionProvisioningError"

Message="VM has reported a failure when processing extension 'devops_agent'. Error message: "Invalid handler configuration. Exiting. Error Message: Error converting value "https://raw.githubusercontent.com/Azure/apim-landing-zone-accelerator/main/reference-implementations/AppGW-IAPIM-Func/bicep/shared/agentsetup.ps1\" to type 'System.Collections.Generic.IList`1[System.String]'. Path 'runtimeSettings[0].handlerSettings.publicSettings.fileUris', line 5, position 340."\r\n\r\nMore information on troubleshooting is available at https://aka.ms/VMExtensionCSEWindowsTroubleshoot "

private endpoint for functions

From Enterprise-Scale-APIM created by cykreng: cykreng/Enterprise-Scale-APIM#36

add privatelinks to the backend functions

remove cert from source control after testing

Consider including all assets in the repo

From Enterprise-Scale-APIM created by seenu433: cykreng/Enterprise-Scale-APIM#14

The bicep has reference to content from external git repo

Please consider adding them within the repo

Set-up a GitHub workflow for terraform deployment

Network Considerations Lacking

There are lots of gaps in this landing zone accelerator particularly around Network Design. There is no mention of Azure Firewall or VWAN hubs which are the most common seen in Enterprise Scale Network topologies.

reference keyvault instead of PFX in Terraform

appgateway cert type (custom vs. selfsigned)

s_local_certificate = var.certificate_path != null && var.certificate_password != null