Easily deploy the Elastic Stack of Elasticsearch, Kibana and Logstash to Azure.
This readme provides an overview of usage and features. For more comprehensive documentation, please refer to the Azure Marketplace and ARM template documentation
This repository consists of:
- src/mainTemplate.json - The main Azure Resource Management (ARM) template. The template itself is composed of many nested linked templates, with the main template acting as the entry point.
- src/createUiDefinition - UI definition file for our Azure Marketplace offering. This file produces an output JSON that the ARM template can accept as input parameters.
After pulling the source, call the following once
npm install
to pull in all devDependencies. You may edit the build/allowedValues.json file, which the build uses to patch the ARM template and Marketplace UI definition. Then, run
npm run build
which will validate EditorConfig settings, lint JSON files, patch the template using build/allowedValues.json
, and create a zip in the dist
folder.
For more details around developing the template, take a look at the Development README
The Azure Marketplace Elastic Stack offering offers a simplified UI and installation experience over the full power of the ARM template.
It will always bootstrap an Elasticsearch cluster complete with a trial license of the Elastic Stack's platinum features.
Deploying through the Marketplace is great and easy way to get your feet wet for the first time with Elasticsearch on Azure, but in the long run, you'll want to deploy the templates directly from GitHub using the Azure CLI or PowerShell SDKs. Check out the CLI examples.
You can view the UI in developer mode by clicking here. If you feel something is cached improperly use this client unoptimized link instead
Have a look at this screenshot to see how you can navigate to the deployment error status message. Please create an issue with that message and in which resource it occured on our github issues
The output from the Azure Marketplace UI is fed directly to the ARM deployment template. You can use the ARM template independently, without going through the Marketplace. In fact, there are many features in the ARM template that are not exposed within the Marketplace UI, such as configuring
- Azure Storage account to use with Azure Repository plugin for Snapshot/Restore
- Application Gateway to use for SSL/TLS and SSL offload
Check out our examples repository for examples of common scenarios and also take a look at the following blog posts for further information
- Spinning up a cluster with Elastic's Azure Marketplace template
- Elasticsearch and Kibana deployments on Azure
- SAML based Single Sign-On with Elasticsearch and Azure Active Directory
Starting with Elasticsearch, Kibana and Logstash 6.3.0, The template deploys with Elastic Stack features bundled as part of the deployment, and
includes the free features under the Basic license level.
The xpackPlugins
parameter determines whether a self-generated trial license is applied,
offering a trial period of 30 days to the Platinum license features. A value of Yes
applies a trial license, a value of No
applies the Basic license.
The license level applied determines the Elastic Stack features activated to use.
For Elasticsearch, Kibana and Logstash prior to 6.3.0, The xpackPlugins
parameter determines whether X-Pack plugins are installed
and a self-generated trial license is applied. In difference to 6.3.0 however, a value of No
for xpackPlugins
means that
X-Pack plugins are not installed, and therefore does not provide the free features under the Basic license level, offering the Open Source features only.
For these versions, you can install X-Pack plugins and register for a free Basic license to apply to the deployment, in
order to use the free features available under the Basic license level.
The ARM template accepts a lot of parameters, but don't fear! Most of them are optional and only used in conjunction with other parameters. Where a parameter value is not explicitly provided, it will take the default value defined in the template.
. When theStandard
load balanacer is selected,
and the loadBalancerType
is internal
, A Network Security Group is also deployed
and a public IP address attached to each VM network interface card in the backend pool, to allow
outbound internet traffic to install the Elastic Stack and dependencies.
Parameter | Type | Description | Default Value |
---|---|---|---|
_artifactsLocation | string | The base URI where artifacts required by this template are located, including a trailing '/'. Use to target a specific branch or release tag | Raw content of the current branch |
_artifactsLocationSasToken | securestring | The sasToken required to access _artifactsLocation . When the template is deployed using
the accompanying scripts, a sasToken will be automatically generated. Use the defaultValue if the staging location is not secured." |
"" |
location | string | The location where to provision all the items in this template. Defaults to inheriting the location from the resource group. Any other value must be a valid Azure region. | [resourceGroup().location] |
vmHostNamePrefix | string | The prefix to use for hostnames when naming virtual machines in the cluster. Hostnames are used for resolution of master nodes on the network, so if you are deploying a cluster into an existing virtual network containing an existing Elasticsearch cluster, be sure to set this to a unique prefix, to differentiate the hostnames of this cluster from an existing cluster. Can be up to 5 characters in length, must begin with an alphanumeric character and can contain alphanumeric and hyphen characters. | "" |
Elasticsearch related settings | |||
esVersion | string | A valid supported Elasticsearch version for the target template version. See this list for supported versions. Required | Latest version supported by target template version |
esClusterName | string | The name of the Elasticsearch cluster. Required | "" |
loadBalancerType | string | The load balancer to set up to access the cluster. Can be internal , external or gateway .
If you are setting up Elasticsearch or Kibana on a publicly available IP address, it is highly recommended to secure access to the cluster with a product like Elastic Stack Security, in addition to configuring SSL/TLS. | internal |
loadBalancerInternalSku | string | The internal load balancer SKU. Can be Basic or Standard . |
Basic |
loadBalancerExternalSku | string | The external load balancer SKU. Can be Basic or Standard .
Only relevant when loadBalancerType is external . When the Standard
load balancer SKU is selected, the public IP address SKU attached to the external load balancer
will also be Standard . A Network Security Group is also deployed, to allow inbound internet traffic
to the load balancer backend pool.
|
Basic |
xpackPlugins | string | Either Yes or No to install a trial license of the Elastic Stack features (formerly X-Pack)
such as Monitoring, Security, Alerting, Graph, Machine Learning (5.5.0+) and SQL. If also installing Kibana, it will have Reporting and Profiler installed.
A value of No for Elasticsearch and Kibana prior to 6.3.0,
will include only the Open Source features.
A value of No for Elasticsearch and Kibana 6.3.0+
will include the free Basic license features.
| Yes |
azureCloudPlugin | string | Either Yes or No to install the Azure Repository plugin for snapshot/restore.
When set to Yes , at least azureCloudStorageAccountName
must be specified to configure the plugin correctly.
| No |
azureCloudStorageAccountName | string | The name of an existing storage account to use for snapshots with Azure Repository plugin. Must be a valid Azure Storage Account name. | "" |
azureCloudStorageAccountResourceGroup | string | The name of an existing resource group containing the storage account azureCloudStorageAccountName
to use for snapshots with Azure Repository plugin. Must be a valid Resource Group name.
| "" |
esAdditionalPlugins | string | Additional Elasticsearch plugins to install. Each plugin must be separated by a semicolon. e.g. analysis-icu;mapper-attachments
| "" |
esAdditionalYaml | string | Additional configuration for Elasticsearch yaml configuration file. Each line must be separated by a newline character \n e.g. "action.auto_create_index: +.*\nindices.queries.cache.size: 5%" . This is an expert level feature - It is recommended that you run your additional yaml through a linter before starting a deployment. | "" |
esHeapSize | integer | The size, in megabytes, of memory to allocate on each Elasticsearch node for the JVM heap. If unspecified, 50% of the available memory will be allocated to Elasticsearch heap, up to a maximum of 31744MB (~32GB).
Take a look at the Elasticsearch documentation for more information. This is an expert level feature - setting a heap size too low, or larger than available memory on the Elasticsearch VM SKU will fail the deployment. | 0 |
esHttpCertBlob | string | A Base-64 encoded form of the PKCS#12 archive (.p12/.pfx) containing the certificate and key to secure communication for HTTP layer to Elasticsearch. xpackPlugins must be Yes , or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above.
| "" |
esHttpCertPassword | securestring | The password for the PKCS#12 archive (.p12/.pfx) containing the certificate and key to secure communication for HTTP layer to Elasticsearch. Optional as the archive may not be protected with a password. If using esHttpCaCertBlob , this password will be used to protect the generated PKCS#12 archive on each node.
xpackPlugins must be Yes , or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above.
| "" |
esHttpCaCertBlob | string | A Base-64 encoded form of a PKCS#12 archive (.p12/.pfx) containing the Certificate Authority (CA) certificate and key to use to generate certificates on each Elasticsearch node, to secure communication for the HTTP layer to Elasticsearch. xpackPlugins must be Yes , or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above.
| "" |
esHttpCaCertPassword | securestring | The password for the PKCS#12 archive (.p12/.pfx) containing the Certificate Authority (CA) certificate and key to secure communication for HTTP layer to Elasticsearch. Optional as the archive may not be be protected with a password. xpackPlugins must be Yes , or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above.
| "" |
esTransportCaCertBlob | string | A Base-64 encoded form of a PKCS#12 archive (.p12/.pfx) containing the Certificate Authority (CA) certificate and key to use to generate certificates on each Elasticsearch node, to secure communication for Transport layer to Elasticsearch. xpackPlugins must be Yes , or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above.
| "" |
esTransportCaCertPassword | securestring | The password for the PKCS#12 archive (.p12/.pfx) containing the Certificate Authority (CA) certificate and key to secure communication for Transport layer to Elasticsearch. Optional as the archive may not be be protected with a password. xpackPlugins must be Yes , or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above.
| "" |
esTransportCertPassword | securestring | The password to protect the generated PKCS#12 archive on each node. xpackPlugins must be Yes , or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above.
| "" |
samlMetadataUri | string | The URI from which the metadata file for the Identity Provider can be retrieved to configure SAML Single-Sign-On. For Azure Active Directory, this can be found in the Single-Sign-On settings of the Enterprise Application, and will look something like https://login.microsoftonline.com/<guid>/federationmetadata/2007-06/federationmetadata.xml?appid=<guid>
| "" |
samlServiceProviderUri | string | The public URI for the Service Provider to configure SAML Single-Sign-On. If samlMetadataUri is provided but no value is provided for samlServiceProviderUri , the public domain name for the deployed Kibana instance will be used.
| "" |
Master node related settings | |||
vmSizeMasterNodes | string | Azure VM size of dedicated master nodes. See this list for supported sizes. By default the template deploys 3 dedicated master nodes, unless dataNodesAreMasterEligible is set to Yes .
Check that the size you choose is available in the region you choose.
| Standard_DS1_v2 |
vmMasterNodeAcceleratedNetworking | string | Whether to enable accelerated networking for Master nodes, which enables single root I/O virtualization (SR-IOV)
to a VM, greatly improving its networking performance. Valid values are
| Default |
Data node related settings | |||
dataNodesAreMasterEligible | string | Either Yes or No to make all data nodes master eligible. This can be useful for small Elasticsearch clusters however, for larger clusters it is recommended to have dedicated master nodes.
When Yes no dedicated master nodes will be provisioned.
| No |
vmSizeDataNodes | string | Azure VM size of the data nodes. See this list for supported sizes. Check that the size you choose is available in the region you choose. | Standard_DS1_v2 |
vmDataNodeAcceleratedNetworking | string | Whether to enable accelerated networking for Data nodes, which enables single root I/O virtualization (SR-IOV)
to a VM, greatly improving its networking performance. Valid values are
| Default |
vmDataNodeCount | int | The number of data nodes you wish to deploy. Must be greater than 0. | 3 |
Data node disk related settings | |||
vmDataDiskCount | int | Number of managed disks to attach to each data node in RAID 0 setup.
Must be equal to or greater than 0 .
If the number of disks selected is more than can be attached to the data node VM (SKU) size,
the maximum number of disks that can be attached for the data node VM (sku) size will be used. Equivalent to
taking
| Maximum number supported disks for data node VM size |
vmDataDiskSize | string | The disk size of each attached disk. Choose 32TiB , 16TiB , 8TiB , 4TiB , 2TiB , 1TiB , 512GiB , 256GiB , 128GiB , 64GiB or 32GiB .
For Premium Storage, disk sizes equate to P80, P70, P60, P50, P40, P30, P20, P15, P10 and P6
storage disk types, respectively.
|
1TiB |
storageAccountType | string | The storage account type of the attached disks. Choose either Default or Standard .
The Default storage account type will be Premium Storage for VMs that
support Premium Storage and Standard Storage for those that do not. Standard will use Standard Storage.
| Default |
Coordinating node related settings | |||
vmClientNodeCount | int | The number of coordinating nodes to provision. Must be a positive integer. By default, the data nodes are added to the backend pool of the loadbalancer but if you provision coordinating nodes, these will be added to the loadbalancer instead. Coordinating nodes can be useful in offloading the gather process from data nodes and are necessary to scale an Elasticsearch cluster deployed with this template beyond 100 data nodes (the maximum number of VMs that can be added to a load balancer backend pool). | 0 |
vmSizeClientNodes | string | Azure VM size of the coordinating nodes see this list for supported sizes. Check that the size you choose is available in the region you choose. | Standard_DS1_v2 |
vmClientNodeAcceleratedNetworking | string | Whether to enable accelerated networking for coordinating nodes, which enables single root I/O virtualization (SR-IOV)
to a VM, greatly improving its networking performance. Valid values are
| Default |
Security related settings | |||
adminUsername | string | Admin username used when provisioning virtual machines. Must be a valid Linux username i.e. avoid any of the following usernames for Ubuntu | "" |
authenticationType | string | The authentication type for the Admin user. Either password or sshPublicKey
| password |
adminPassword | securestring | When authenticationType is password this sets the OS level user's password
| "" |
sshPublicKey | securestring | When authenticationType is sshPublicKey this sets the OS level sshKey that can be used to login.
| "" |
securityBootstrapPassword | securestring | Security password for 6.x bootstrap.password key that is added to the keystore. If no value is supplied, a 13 character password
will be generated using the ARM template uniqueString() function. The bootstrap password is used to seed the built-in
users. Used only in 6.0.0+
| "" |
securityAdminPassword | securestring | Security password Admin user.
This is the built-in elastic user.
should be a minimum of 12 characters, and must be greater than 6 characters. | "" |
securityKibanaPassword | securestring | Security password Kibana.
This is the built-in kibana user.
should be a minimum of 12 characters, and must be greater than 6 characters. | "" |
securityLogstashPassword | securestring | This is the built-in logstash_system user.
should be a minimum of 12 characters, and must be greater than 6 characters. | "" |
securityBeatsPassword | securestring | This is the built-in beats_system user. Valid for Elasticsearch 6.3.0+
should be a minimum of 12 characters, and must be greater than 6 characters. | "" |
securityApmPassword | securestring | This is the built-in apm_system user. Valid for Elasticsearch 6.5.0+
should be a minimum of 12 characters, and must be greater than 6 characters. | "" |
securityRemoteMonitoringPassword | securestring | This is the built-in remote_monitoring_user user. Valid for Elasticsearch 6.5.0+
should be a minimum of 12 characters, and must be greater than 6 characters. | "" |
Kibana related settings | |||
kibana | string | Either Yes or No to provision a machine with Kibana installed and a public IP address to access it.
| Yes |
vmSizeKibana | string | Azure VM size of the Kibana instance. See this list for supported sizes. Check that the size you select is available in the region you choose. | Standard_A2_v2 |
vmKibanaAcceleratedNetworking | string | Whether to enable accelerated networking for Kibana, which enables single root I/O virtualization (SR-IOV)
to a VM, greatly improving its networking performance. Valid values are
| Default |
kibanaCertBlob | string | A Base-64 encoded form of the certificate (.crt) in PEM format to secure HTTPS communication between the browser and Kibana. | "" |
kibanaKeyBlob | securestring | A Base-64 encoded form of the private key (.key) in PEM format to secure HTTPS communication between the browser and Kibana. | "" |
kibanaKeyPassphrase | securestring | The passphrase to decrypt the private key. Optional as the key may not be encrypted. | "" |
kibanaAdditionalYaml | string | Additional configuration for Kibana yaml configuration file. Each line must be separated by a \n newline character e.g. "server.name: \"My server\"\nkibana.defaultAppId: home" . This is an expert level feature - It is recommended that you run your additional yaml through a linter before starting a deployment. | "" |
Logstash related settings | |||
logstash | string | Either Yes or No to provision Logstash VMs.
| No |
vmSizeLogstash | string | Azure VM size of the Logstash instance. See this list for supported sizes. Check that the size you select is available in the region you choose. | Standard_DS1_v2 |
vmLogstashCount | int | The number of Logstash instances | 1 |
vmLogstashAcceleratedNetworking | string | Whether to enable accelerated networking for Logstash, which enables single root I/O virtualization (SR-IOV)
to a VM, greatly improving its networking performance. Valid values are
| Default |
logstashHeapSize | integer | The size, in megabytes, of memory to allocate for the JVM heap for Logstash. If unspecified, Logstash will be configured with the default heap size for the distribution and version.
Take a look at the Logstash documentation on profiling heap size for more information. This is an expert level feature - setting a heap size too low, or larger than available memory on the Logstash VM SKU will fail the deployment. | 0 |
logstashConf | securestring | A Base-64 encoded form of a Logstash config file to deploy. | "" |
logstashKeystorePassword | securestring | The password to protect the Logstash keystore. If no value is supplied, a value will be generated using the ARM template uniqueString() function. Used only in 6.2.0+
| "" |
logstashAdditionalPlugins | string | Additional Logstash plugins to install. Each plugin must be separated by a semicolon. e.g. logstash-input-heartbeat;logstash-input-twitter
| "" |
logstashAdditionalYaml | string | Additional configuration for Logstash yaml configuration file. Each line must be separated by a newline character \n e.g. "pipeline.batch.size: 125\npipeline.batch.delay: 50" . This is an expert level feature - It is recommended that you run your additional yaml through a linter before starting a deployment. | "" |
Jumpbox related settings | |||
jumpbox | string | Either Yes or No to optionally add a virtual machine with a public IP to the deployment, which you can use to connect and manage virtual machines on the internal network.
NOTE: If you are deploying Kibana, the Kibana VM can act
as a jumpbox, so a separate jumpbox VM is not needed.
| No |
Virtual network related settings | |||
vNetNewOrExisting | string | Whether the Virtual Network is new or existing . An existing Virtual Network in
another Resource Group in the same Location can be used.
| new |
vNetName | string | The name of the Virtual Network.
The Virtual Network must already exist when using an existing Virtual Network
| es-net |
vNetExistingResourceGroup | string | The name of the Resource Group in which the Virtual Network resides when using an existing Virtual Network.
Required when using an existing Virtual Network
| "" |
vNetNewAddressPrefix | string | The address prefix when creating a new Virtual Network. Required when creating a new Virtual Network | 10.126.182.0/24 |
vNetLoadBalancerIp | string | The internal static IP address to use when configuring the internal load balancer. Must be an available
IP address on the provided vNetClusterSubnetName .
| 10.126.182.4 |
vNetClusterSubnetName | string | The name of the subnet to which Elasticsearch nodes will be attached.
The subnet must already exist when using an existing Virtual Network
| es-subnet |
vNetNewClusterSubnetAddressPrefix | string | The address space of the subnet.
Required when creating a new Virtual Network
| 10.126.182.0/25 |
vNetAppGatewaySubnetName | string | Subnet name to use for the Application Gateway.
Required when selecting gateway for load balancing.The subnet must already exist when using an existing Virtual Network
| es-gateway-subnet |
vNetNewAppGatewaySubnetAddressPrefix | string | The address space of the Application Gateway subnet.
Required when creating a new Virtual Network and selecting gateway for load balancing.
| 10.126.182.128/28 |
Application Gateway related settings | |||
appGatewayTier | string | The tier of the Application Gateway, either Standard or WAF .
Required when selecting gateway for load balancing.
| Standard |
appGatewaySku | string | The size of the Application Gateway. Choose Small , Medium or Large .
When choosing appGatewayTier WAF , the size must be at least Medium .
Required when selecting gateway for load balancing.
| Medium |
appGatewayCount | int | The number instances of the Application Gateway. Can be a value between 1 and 10 .
A minimum of 2 is recommended for production.
Required when selecting gateway for load balancing.
| 2 |
appGatewayCertBlob | string | A Base-64 encoded form of the PKCS#12 archive (.p12/.pfx) containing the certificate and key for Application Gateway.
This certificate is used to secure HTTPS connections to and from the Application Gateway.
Required when selecting gateway for load balancing.
| "" |
appGatewayCertPassword | securestring | The password for the PKCS#12 archive (.p12/.pfx) containing the certificate and key for Application Gateway.
Required when selecting gateway for load balancing.
| "" |
appGatewayEsHttpCertBlob | securestring | The Base-64 encoded public certificate (.cer) used to secure the HTTP layer of Elasticsearch. Used by the Application Gateway to whitelist certificates used by the backend pool. Required when using esHttpCertBlob to secure the HTTP layer of Elasticsearch and selecting gateway for load balancing. X-Pack plugin must be installed
| "" |
appGatewayWafStatus | string | The firewall status of the Application Gateway, either Enabled or Disabled .
Required when selecting gateway for load balancing and using appGatewayTier WAF
| Enabled |
appGatewayWafMode | string | The firewall mode of the Application Gateway, either Detection or Prevention .
Required when selecting gateway for load balancing and using appGatewayTier WAF
| Detection |
The above button will take you to the autogenerated web based UI based on the parameters from the ARM template.
You can deploy using the template directly from Github using the Azure CLI or Azure PowerShell
Azure CLI 1.0 is no longer supported as the apiVersion
s of resources are newer than those
supported by the last release. It's recommended to update to Azure CLI 2.0.
- Log into Azure
az login
- Create a resource group
<name>
in a<location>
(e.gwesteurope
) where we can deploy too
az group create --name <name> --location <location>
- Use our template directly from GitHub using
--template-uri
az group deployment create \
--resource-group <name> \
--template-uri https://raw.githubusercontent.com/elastic/azure-marketplace/master/src/mainTemplate.json \
--parameters @parameters/password.parameters.json
where <name>
refers to the resource group you just created.
- Log into Azure
Login-AzureRmAccount
- Select a Subscription Id
Select-AzureRmSubscription -SubscriptionId "<subscriptionId>"
- Define the parameters object for your deployment as a PowerShell hashtable. The keys correspond the parameters defined in the Parameters section
$branch = "master"
$esVersion = "7.9.0"
$clusterParameters = @{
"_artifactsLocation" = "https://raw.githubusercontent.com/elastic/azure-marketplace/$branch/src/"
"esVersion" = $esVersion
"esClusterName" = "elasticsearch"
"loadBalancerType" = "internal"
"vmDataDiskCount" = 1
"adminUsername" = "russ"
"adminPassword" = "Password1234"
"securityBootstrapPassword" = "Password1234"
"securityAdminPassword" = "Password1234"
"securityKibanaPassword" = "Password1234"
"securityLogstashPassword" = "Password1234"
"securityBeatsPassword" = "Password1234"
"securityApmPassword" = "Password1234"
"securityRemoteMonitoringPassword" = "Password1234"
}
- Create a resource group
<name>
in a<location>
(e.gwesteurope
) where we can deploy too
New-AzureRmResourceGroup -Name "<name>" -Location "<location>"
- Use our template directly from GitHub
New-AzureRmResourceGroupDeployment -Name "<deployment name>" -ResourceGroupName "<name>" -TemplateUri "https://raw.githubusercontent.com/elastic/azure-marketplace/master/src/mainTemplate.json" -TemplateParameterObject $clusterParameters
You can target a specific version of the template by modifying the URI of the template and
the _artifactsLocation
parameter of the template to point to a specific tagged release.
Targeting a specific template version is recommended for repeatable production deployments.
For example, to target the 7.9.0
tag release with PowerShell
$templateVersion = "7.9.0"
$_artifactsLocation = "https://raw.githubusercontent.com/elastic/azure-marketplace/$templateVersion/src/"
# minimum parameters required to deploy
$clusterParameters = @{
"_artifactsLocation" = $_artifactsLocation
"esVersion" = "7.9.0"
"adminUsername" = "russ"
"adminPassword" = "Password1234"
"securityBootstrapPassword" = "Password1234"
"securityAdminPassword" = "Password1234"
"securityKibanaPassword" = "Password1234"
"securityLogstashPassword" = "Password1234"
"securityBeatsPassword" = "Password1234"
"securityApmPassword" = "Password1234"
"securityRemoteMonitoringPassword" = "Password1234"
}
$resourceGroup = "my-azure-cluster"
$location = "Australia Southeast"
$name = "my-azure-cluster"
New-AzureRmResourceGroup -Name $resourceGroup -Location $location
New-AzureRmResourceGroupDeployment -Name $name -ResourceGroupName $resourceGroup -TemplateUri "$_artifactsLocation/mainTemplate.json" -TemplateParameterObject $clusterParameters
It is strongly recommended that you secure communication using Transport Layer Security when using the template. The Elastic Stack security features can provide Basic Authentication, Role Based Access control, and Transport Layer Security (TLS) for both Elasticsearch and Kibana. For more details, please refer to the Security documentation.
For Elasticsearch versions 6.8.0+ (and less than 7.0.0), and 7.1.0+, the Elastic Stack security features that allow configuring TLS and role based access control are available in the free basic license tier. For all other versions, the Elastic Stack security features require a license level higher than basic; They can be configured with a trial license, which provides access to the Security features for 30 days.
You can secure external access from the browser to Kibana with TLS by supplying
a certificate and private key in PEM format with kibanaCertBlob
and
kibanaKeyBlob
parameters, respectively.
You can secure communication between nodes in the cluster with TLS on the
Transport layer. Configuring TLS for the Transport layer requires
xPackPlugins
be set to Yes
, or an Elasticsearch version 6.8.0+ (and less than 7.0.0) or 7.1.0+.
You must supply a PKCS#12 archive with the esTransportCaCertBlob
parameter (and optional
passphrase with esTransportCaCertPassword
) containing the CA cert which should be used to generate
a certificate for each node within the cluster. An optional
passphrase can be passed with esTransportCertPassword
to encrypt the generated certificate
on each node.
One way to generate a PKCS#12 archive containing a CA certificate and key is using
Elastic's elasticsearch-certutil
command.
The simplest command to generate a CA certificate is
./elasticsearch-certutil ca
and follow the instructions.
You can secure external access to the cluster with TLS with an external
loadbalancer or Application Gateway. Configuring TLS for the HTTP layer requires
xPackPlugins
be set to Yes
, or an Elasticsearch version 6.8.0+ (and less than 7.0.0) or 7.1.0+.
If you choose external
as the value for loadBalancerType
, you must either
- supply a PKCS#12 archive containing the key and certificate with the
esHttpCertBlob
parameter (and optional passphrase withesHttpCertPassword
) containing the certs and private key to secure the HTTP layer. This certificate will be used by all nodes within the cluster, and
or
- supply a PKCS#12 archive containing the key and certificate with the
esHttpCaCertBlob
parameter (and optional passphrase withesHttpCaCertPassword
) containing the CA which should be used to generate a certificate for each node within the cluster to secure the HTTP layer. Kibana will be configured to trust the CA and perform hostname verification for presented certificates. One way to generate a PKCS#12 archive is using Elastic's certutil command.
If you choose gateway
as the value for loadBalancerType
, you must
- supply a PKCS#12 archive containing the key and certificate with the
appGatewayCertBlob
parameter (and optional passphrase withappGatewayCertPassword
) to secure communication to Application Gateway. One way to generate a PKCS#12 archive is using Elastic's certutil command
Application Gateway performs SSL offload, so communication from Application Gateway to Elasticsearch is not encrypted with TLS by default. TLS to Application Gateway may be sufficient for your needs, but if you would like end-to-end encryption by also configuring TLS for Elasticsearch HTTP layer, you can
- supply a PKCS#12 archive containing the key and certificate with the
esHttpCertBlob
parameter (and optional passphrase withesHttpCertPassword
) containing the certs and private key to secure the HTTP layer. This certificate will be used by all nodes within the cluster, and Kibana will be configured to trust the certificate CA (if CA certs are present within the archive). One way to generate a PKCS#12 archive is using Elastic's certutil command, and you must specify a--dns <name>
argument with a name that matches that in the--name <name>
argument.
and
-
supply the public certificate in PEM format from the PKCS#12 archive passed with
esHttpCertBlob
parameter, using theappGatewayEsHttpCertBlob
parameter. Application Gateway whitelists certificates used by VMs in the backend pool. This can be extracted from the PKCS#12 archive of theesHttpCertBlob
parameter usingopenssl pkcs12
openssl pkcs12 -in http_cert.p12 -out http_public_cert.cer -nokeys
and provide the passphrase for the archive when prompted.
IMPORTANT: When configuring end-to-end encryption with Application Gateway,
the certificate to secure the HTTP layer must include a x509v3 Subject Alternative Name
extension with a DNS entry that matches the Subject CN, to work with Application
Gateway's whitelisting mechanism. This can be checked using
openssl x509
openssl x509 -in http_public_cert.cer -text -noout
which will output something similar to
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
// omitted for brevity ...
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=Elastic Certificate Tool Autogenerated CA
Validity
Not Before: Jul 5 02:37:40 2018 GMT
Not After : Jul 4 02:37:40 2021 GMT
Subject: CN=custom
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
// omitted for brevity ...
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Key Identifier:
// omitted for brevity ...
X509v3 Authority Key Identifier:
// omitted for brevity ...
X509v3 Subject Alternative Name:
DNS:custom
X509v3 Basic Constraints:
CA:FALSE
Signature Algorithm: sha256WithRSAEncryption
// omitted for brevity ...
Without this, Application Gateway will return 502 Bad Gateway errors, as the health probe for the backend pool will fail when the whitelisted certificate does not contain this certificate extension. You can typically understand if there is a problem with the key format when
- TLS has been configured on the HTTP layer
- Kibana is able to communicate to the cluster correctly but Application Gateway returns 502 errors.
This may not always be the case, but can be indicative. You should also check the description for Backend Health of the Application Gateway in the Azure portal.
Parameters such as esHttpCertBlob
and kibanaCertBlob
must be provided in Base-64 encoded form. A Base-64 encoded value can be obtained using
-
base64 on Linux, or openssl on Linux and MacOS
base64
httpCert=$(base64 http-cert.p12)
openssl
httpCert=$(openssl base64 -in http-cert.p12)
and including the value assigned to
$httpCert
in the parameters.json file as the value for certificate parameter passed to the Azure CLI command -
PowerShell on Windows
$httpCert = [Convert]::ToBase64String([IO.File]::ReadAllBytes("c:\http-cert.p12"))
and then pass this in the template parameters object passed to the Azure PowerShell command
$clusterParameters = @{ # Other parameters skipped for brevity "esHttpCertBlob"= $httpCert }
This project is MIT Licensed and was originally forked from the Elasticsearch Azure quick start arm template