lombiq/infrastructure-scripts

PowerShell modules for managing the hosting environment of Orchard (Core) applications in Azure.

Hosting - Infrastructure Scripts readme

Scripts - PowerShell, mostly - for automating processes within a continuous integration and deployment environment of a Hosting Suite-integrated Orchard application.

Overview

The Lombiq Hosting Suite consists of Orchard modules (designed to enhance several parts and features of an Orchard application) and automation scripts to drive the build and deployment processes of an Orchard application.

The scripts are mostly PowerShell modules that can be reused anywhere on the system after they are "installed" by adding the root of the Infrastructure Scripts folder to $PSModulePath. Of course, there's a script for that: See Utilities\AddPathToPSModulePath.ps1.

Infrastructure Scripts is not tied to any specific CI system, but in terms of the hosting environment they rely on Azure features, meaning that most of it depends on the Azure PowerShell SDK. Azure services are usually accessed by defining the name of the Azure subscription, the name of an Azure Web App (and optionally the name of the deployment Slot) and other parameters specific to the operation (in many cases, one of these is the name of a connection string for accessing a resource stored among the settings of the Web App).

The Infrastructure Scripts does not require the Orchard modules of the Hosting Suite to be able to function (and vice versa), although the two together work best for a seamless developer experience.

Folder structure and components

• Utility (submodule): A few useful scripts/modules for everyday use (not closely related to hosting). This component has its own readme.
• PowerShell: The heart and soul of Infrastructure Scripts (see the PowerShell subfolder), divided into these sub-features:
  • Azure: These modules interact directly with Azure services, such as Blob Storages, Databases and Web Apps. This component has its own readme
  • Hosting Suite: These modules interact directly with Orchard modules of the Hosting Suite through WebAPI, e.g. for starting/executing maintenances. This component has its own readme.
  • Utilities: Modules and plain scripts for simple tasks, such as sending a ping request to a URL or extracting a .zip file.

Contributing and support

Bug reports, feature requests, comments, questions, code contributions and love letters are warmly welcome. You can send them to us via GitHub issues and pull requests. Please adhere to our open-source guidelines while doing so.

This project is developed by Lombiq Technologies. Commercial-grade support is available through Lombiq.

infrastructure-scripts's Issues

Make Set-AzureWebAppStorageContentFromStorage faster (INFRA-139)

Set-AzureWebAppStorageContentFromStorage copies blobs one by one. If you have a lot of files (like Orchard Media files), then this will take a lot of time, even if the total size is not that big (like taking >30 minutes to copy 13k blobs with 2,6 GiB total size).

Let's make this faster somehow. E.g., can containers be copied at once too? Or at least not each blob individually? Or can this loop be parallelized? While I don't know, I doubt any of the resources, including network, are maxed out on the CI machine during such a copy, so parallelization may help.

There might be some throttling as well, since the copy process might get stuck on files that are otherwise trivial.

Jira issue

AzureWebAppSql scripts should support not having the DB (server) and App in the same Resource Group (INFRA-141)

The scripts under PowerShell/Azure/Sql are built to work with databases whose connection strings are read from the consumer Web App's configuration. These scripts assume that the DB server lives in the same Resource Group as the App, which is true for our own Azure resources, but it cannot be enforced. Let's make the necessary changes to support this scenario. BTW Blob Storage Accounts don't have the same problem, because those are accessed by their name only, regardless of the Resource Group they are in.

The common entry point for these operations is Get-AzureWebAppSqlDatabase: this module needs to have a non-mandatory parameter added called DatabaseResourceGroupName, which is initialized to the existing ResourceGroupName parameter. Then, DatabaseResourceGroupName should be passed to the Get-AzSqlDatabase command instead of ResourceGroupName.

The same concept has to be applied to every module that calls Get-AzureWebAppSqlDatabase.

Copy-AzureWebAppSqlDatabase is a special case, since the copy operation has a source and a target, so it needs two parameters added instead of just one.

Better solution:
Get-AzureWebAppSqlDatabaseConnection was updated to grab the Server by its name (in contrast to fetching a Database, this one doesn't require a ResourceGroupName parameter), and pass on its ResourceGroupName property in the return object. The scripts calling it (including Get-AzureWebAppSqlDatabase) and/or Get-AzureWebAppSqlDatabase were updated to use and/or pass on this new ResourceGroupName property.

Jira issue

Fix Validate Pull Request configuration (INFRA-154)

...because it didn't add the "Fixes" link to #28. You can copy this: https://github.com/Lombiq/Open-Source-Orchard-Core-Extensions/actions/runs/6488451487/workflow

Jira issue

Code quality and cleanliness changes according to Lombiq's PowerShell conventions (INFRA-128)

Starting from @DAud-IcI's feedback under #5, several code styling patterns in older scripts/modules need to be brought up to par with our conventions, e.g.:

1. HelpMessages should concise and descriptive.
2. Unnecessary parenthesis around the input of throw and Write-Warning/Error commands.
3. Multi-line strings should use the "Binary -join operator" format, see: Lombiq/GitHub-Actions#179 (comment)
4. Parameter splatting instead of backtick and line breaks. Fixed in #10.

Jira issue

Deprecation warning when using Copy-AzureWebAppSqlDatabase (INFRA-129)

When running Copy-AzureWebAppSqlDatabase there's a deprecation warning coming from the Set-AzSqlDatabase cmdlet (in a GitHub Actions workflow) I get the below warning:

WARNING: Upcoming breaking changes in the cmdlet 'Set-AzSqlDatabase' :

- The output type 'Microsoft.Azure.Commands.Sql.Database.Model.AzureSqlDatabaseModel' is changing
- The following properties in the output type are being deprecated : 'BackupStorageRedundancy'
- The following properties are being added to the output type : 'CurrentBackupStorageRedundancy' 'RequestedBackupStorageRedundancy'
- The change is expected to take effect from the version : '3.0.0'

Jira issue

Display exception details when commands fail (INFRA-140)

Currently, commands have a pattern of catching exceptions, displaying an error message, then rethrowing the exception. See e.g. here or here.

When executed from a PS script, this results in outputs like this:

You see the error message, but no details.

I'd be better to also display the exception details in each of these cases, so you have a better chance of debugging the issue.

Jira issue

Testing Jira issue template (INFRA-161)

Jira issue

Use AzCopy for downloading blobs (INFRA-155)

Similarly to how Set-AzureWebAppStorageContentFromStorageWithAzCopy became an alternative to Set-AzureWebAppStorageContentFromStorage in #20.

Jira issue