Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/elastic/azure-marketplace

Elasticsearch Azure Marketplace offering + ARM template
https://github.com/elastic/azure-marketplace

arm arm-templates azure azure-marketplace elasticsearch kibana

Last synced: 7 days ago
JSON representation

Elasticsearch Azure Marketplace offering + ARM template

Host: GitHub
URL: https://github.com/elastic/azure-marketplace
Owner: elastic
License: mit
Archived: true
Created: 2015-11-25T15:29:01.000Z (almost 9 years ago)
Default Branch: master
Last Pushed: 2023-03-30T21:56:22.000Z (over 1 year ago)
Last Synced: 2024-09-22T23:03:25.099Z (11 days ago)
Topics: arm, arm-templates, azure, azure-marketplace, elasticsearch, kibana
Language: Shell
Homepage: https://www.elastic.co/guide/en/elastic-stack-deploy/current/index.html
Size: 79.3 MB
Stars: 120
Watchers: 349
Forks: 162
Open Issues: 45
Metadata Files:
- Readme: README.md
- License: LICENSE.txt

Awesome Lists containing this project

README

# Elastic Stack Azure Marketplace offering

Easily deploy the Elastic Stack of Elasticsearch, Kibana and Logstash to Azure.

**IMPORTANT:** As a prefered solution, we recommend using our newer [Elastic Cloud (Elasticsearch Service)](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/elastic.ec-azure-pp?exp=ubp8&tab=Overview) offering in the Azure Marketplace. To learn more, refer to the [Native Azure integration documentation](https://www.elastic.co/guide/en/cloud/current/ec-azure-marketplace-native.html).

This readme provides an overview of usage and features. For more comprehensive documentation,
please refer to the [**Azure Marketplace and ARM template documentation**](https://www.elastic.co/guide/en/elastic-stack-deploy/current/index.html)

This repository consists of:

* [src/mainTemplate.json](src/mainTemplate.json) - The main Azure Resource Management (ARM) template.
The template itself is composed of many nested linked templates, with the main template acting as the entry point.
* [src/createUiDefinition](src/createUiDefinition.json) - UI definition file for our Azure Marketplace offering.
This file produces an output JSON that the ARM template can accept as input parameters.

## Building

After pulling the source, call the following _once_

```sh
npm install
```

to pull in all devDependencies. You may edit the [build/allowedValues.json](build/allowedValues.json) file, which the build uses to patch the ARM template and Marketplace UI definition. Then, run

```sh
npm run build
```

which will validate EditorConfig settings, lint JSON files, patch the template using `build/allowedValues.json`, and create a zip in the `dist` folder.
For more details around developing the template, take a look at the [Development README](build/README.md)

## Azure Marketplace

The [Azure Marketplace Elastic Stack offering](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/elastic.elasticsearch) offers a simplified UI and installation experience over the full power of the ARM template.

It will always bootstrap an Elasticsearch cluster complete with a trial license of the [Elastic Stack's platinum features](https://www.elastic.co/products/stack).

Deploying through the Marketplace is great and easy way to get your feet wet for the first time with Elasticsearch on Azure, but in the long run, you'll want to deploy the templates directly from GitHub using the Azure CLI or PowerShell SDKs.
Check out the CLI examples.

---

![Example UI Flow](images/ui.gif)

You can view the UI in developer mode by [clicking here](https://portal.azure.com/#blade/Microsoft_Azure_Compute/CreateMultiVmWizardBlade/internal_bladeCallId/anything/internal_bladeCallerParams/{"initialData":{},"providerConfig":{"createUiDefinition":"https%3A%2F%2Fraw.githubusercontent.com%2Felastic%2Fazure-marketplace%2Fmaster%2Fsrc%2FcreateUiDefinition.json"}}). If you feel something is cached improperly use [this client unoptimized link instead](https://portal.azure.com/?clientOptimizations=false#blade/Microsoft_Azure_Compute/CreateMultiVmWizardBlade/internal_bladeCallId/anything/internal_bladeCallerParams/{"initialData":{},"providerConfig":{"createUiDefinition":"https%3A%2F%2Fraw.githubusercontent.com%2Felastic%2Fazure-marketplace%2Fmaster%2Fsrc%2FcreateUiDefinition.json"}})

## Reporting bugs

Have a look at this [screenshot](images/error-output.png) to see how you can
navigate to the deployment error status message.
Please create an issue with that message and in which resource it occured on our
[github issues](https://github.com/elastic/azure-marketplace/issues)

## ARM template

The output from the Azure Marketplace UI is fed directly to the ARM deployment
template. You can use the ARM template independently, without going through the
Marketplace. In fact, there are many features in the ARM template that are
not exposed within the Marketplace UI, such as configuring

* Azure Storage account to use with Azure Repository plugin for Snapshot/Restore
* Application Gateway to use for SSL/TLS and SSL offload

Check out our [**examples repository**](https://github.com/elastic/azure-marketplace-examples)
for examples of common scenarios and also take a look at the following blog
posts for further information

* [Spinning up a cluster with Elastic's Azure Marketplace template](https://www.elastic.co/blog/spinning-up-a-cluster-with-elastics-azure-marketplace-template)
* [Elasticsearch and Kibana deployments on Azure](https://www.elastic.co/blog/elasticsearch-and-kibana-deployments-on-azure)
* [SAML based Single Sign-On with Elasticsearch and Azure Active Directory](https://www.elastic.co/blog/saml-based-single-sign-on-with-elasticsearch-and-azure-active-directory?blade=tw&hulk=social)

### Elastic Stack features (formerly known as X-Pack)

Starting with Elasticsearch, Kibana and Logstash 6.3.0, The template deploys with Elastic Stack features bundled as part of the deployment, and
includes the free features under the [Basic license](https://www.elastic.co/subscriptions) level.
The [`xpackPlugins`](#x-pack) parameter determines whether a self-generated trial license is applied,
offering a trial period of 30 days to the Platinum license features. A value of `Yes` applies a trial license, a value of `No` applies the Basic license.
The license level applied determines the Elastic Stack features activated to use.

For Elasticsearch, Kibana and Logstash prior to 6.3.0, The [`xpackPlugins`](#x-pack) parameter determines whether X-Pack plugins are installed
and a self-generated trial license is applied. In difference to 6.3.0 however, a value of `No` for `xpackPlugins` means that
X-Pack plugins are not installed, and therefore does not provide the free features under the Basic license level, offering the Open Source features only.
For these versions, you can install X-Pack plugins and [**register for a free Basic license** to apply to the deployment](https://register.elastic.co/), in
order to use the free features available under the Basic license level.

## Parameters

The ARM template accepts a _lot_ of parameters, but don't fear! Most of them are **optional** and only used
in conjunction with other parameters. Where a parameter value is not explicitly provided, it will take the default
value defined in the template.

ParameterTypeDescriptionDefault Value

_artifactsLocationstring
The base URI where artifacts required by this template are located, including a trailing '/'.
Use to target a specific branch or release tagRaw content of the current branch

_artifactsLocationSasTokensecurestring
The sasToken required to access _artifactsLocation. When the template is deployed using
the accompanying scripts, a sasToken will be automatically generated. Use the defaultValue if the staging location is not secured."
""

locationstring
The location where to provision all the items in this template. Defaults to inheriting the location
from the resource group. Any other value must be a valid Azure region.
[resourceGroup().location]

vmHostNamePrefixstring
The prefix to use for hostnames when naming virtual machines in the cluster. Hostnames are used for resolution of master nodes on the network, so if you are deploying a cluster into an existing virtual network containing an existing Elasticsearch cluster, be sure to set this to a unique prefix, to differentiate the hostnames of this cluster from an existing cluster. Can be up to 5 characters in length, must begin with an alphanumeric character and can contain alphanumeric and hyphen characters.
""

Elasticsearch related settings

esVersionstring
A valid supported Elasticsearch version for the target template version. See this list for supported versions.
RequiredLatest version supported by target template version

esClusterNamestring
The name of the Elasticsearch cluster. Required
""

loadBalancerTypestring
The load balancer to set up to access the cluster. Can be internal, external or gateway.

By choosing internal, only an internal load balancer is deployed. Useful when connecting to the cluster happens from inside the Virtual Network

By choosing external, both internal and external load balancers will be deployed. Kibana communicates with the cluster through the internal
load balancer.

By choosing gateway, Application Gateway will be deployed for load balancing,
allowing a PKCS#12 archive (.pfx/.p12) containing the certificate and key to be supplied for SSL/TLS to and from Application Gateway, and providing SSL offload.
An internal load balancer will also deployed. Application Gateway and Kibana communicate with the cluster through the internal
load balancer.

If you are setting up Elasticsearch or Kibana on a publicly available IP address, it is highly recommended to secure access to the cluster with a product like
Elastic Stack Security, in addition to configuring SSL/TLS.

internal

loadBalancerInternalSkustring
The internal load balancer SKU. Can be Basic or Standard.
Basic. When the Standard load balanacer is selected,
and the loadBalancerType is internal, A Network Security Group is also deployed
and a public IP address attached to each VM network interface card in the backend pool, to allow
outbound internet traffic to install the Elastic Stack and dependencies.

loadBalancerExternalSkustring
The external load balancer SKU. Can be Basic or Standard.
Only relevant when loadBalancerType is external. When the Standard
load balancer SKU is selected, the public IP address SKU attached to the external load balancer
will also be Standard. A Network Security Group is also deployed, to allow inbound internet traffic
to the load balancer backend pool.

Basic

xpackPluginsstring
Either Yes or No to install a trial license of the Elastic Stack features (formerly X-Pack)
such as Monitoring, Security, Alerting, Graph, Machine Learning (5.5.0+) and SQL. If also installing Kibana, it will have Reporting and Profiler installed.

A value of No for Elasticsearch and Kibana prior to 6.3.0,
will include only the Open Source features.

A value of No for Elasticsearch and Kibana 6.3.0+
will include the free Basic license features.
Yes

azureCloudPluginstring
Either Yes or No to install the Azure Repository plugin for snapshot/restore.
When set to Yes, at least azureCloudStorageAccountName
must be specified to configure the plugin correctly.
No

azureCloudStorageAccountNamestring
The name of an existing storage account to use for snapshots with Azure Repository plugin.
Must be a valid Azure Storage Account name.
""

azureCloudStorageAccountResourceGroupstring
The name of an existing resource group containing the storage account azureCloudStorageAccountName
to use for snapshots with Azure Repository plugin. Must be a valid Resource Group name.
""

esAdditionalPluginsstring
Additional Elasticsearch plugins to install. Each plugin must be separated by a semicolon. e.g. analysis-icu;mapper-attachments
""

esAdditionalYamlstring
Additional configuration for Elasticsearch yaml configuration file. Each line must be separated by a newline character \n e.g. "action.auto_create_index: +.*\nindices.queries.cache.size: 5%".

This is an expert level feature - It is recommended that you run your additional yaml through a linter before starting a deployment.
""

esHeapSizeinteger
The size, in megabytes, of memory to allocate on each Elasticsearch node for the JVM heap. If unspecified, 50% of the available memory will be allocated to Elasticsearch heap, up to a maximum of 31744MB (~32GB).
Take a look at the Elasticsearch documentation for more information.

This is an expert level feature - setting a heap size too low, or larger than available memory on the Elasticsearch VM SKU will fail the deployment.
0

esHttpCertBlobstring
A Base-64 encoded form of the PKCS#12 archive (.p12/.pfx) containing the certificate and key to secure communication for HTTP layer to Elasticsearch. xpackPlugins must be Yes, or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above.
""

esHttpCertPasswordsecurestring
The password for the PKCS#12 archive (.p12/.pfx) containing the certificate and key to secure communication for HTTP layer to Elasticsearch. Optional as the archive may not be protected with a password.

If using esHttpCaCertBlob, this password will be used to protect the generated PKCS#12 archive on each node.
xpackPlugins must be Yes, or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above.
""

esHttpCaCertBlobstring
A Base-64 encoded form of a PKCS#12 archive (.p12/.pfx) containing the Certificate Authority (CA) certificate and key to use to generate certificates on each Elasticsearch node, to secure communication for the HTTP layer to Elasticsearch. xpackPlugins must be Yes, or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above.
""

esHttpCaCertPasswordsecurestring
The password for the PKCS#12 archive (.p12/.pfx) containing the Certificate Authority (CA) certificate and key to secure communication for HTTP layer to Elasticsearch. Optional as the archive may not be be protected with a password. xpackPlugins must be Yes, or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above.
""

esTransportCaCertBlobstring
A Base-64 encoded form of a PKCS#12 archive (.p12/.pfx) containing the Certificate Authority (CA) certificate and key to use to generate certificates on each Elasticsearch node, to secure communication for Transport layer to Elasticsearch. xpackPlugins must be Yes, or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above.
""

esTransportCaCertPasswordsecurestring
The password for the PKCS#12 archive (.p12/.pfx) containing the Certificate Authority (CA) certificate and key to secure communication for Transport layer to Elasticsearch. Optional as the archive may not be be protected with a password. xpackPlugins must be Yes, or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above.
""

esTransportCertPasswordsecurestring
The password to protect the generated PKCS#12 archive on each node. xpackPlugins must be Yes, or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above.
""

samlMetadataUristring
The URI from which the metadata file for the Identity Provider can be retrieved to configure SAML Single-Sign-On. For Azure Active Directory, this can be found in the Single-Sign-On settings of the Enterprise Application, and will look something like https://login.microsoftonline.com/<guid>/federationmetadata/2007-06/federationmetadata.xml?appid=<guid>

Supported only for Elasticsearch 6.2.0+

Kibana must be installed

X-Pack plugin must be installed with a level of license that enables the SAML realm.

SSL/TLS must be configured for HTTP layer of Elasticsearch

""

samlServiceProviderUristring
The public URI for the Service Provider to configure SAML Single-Sign-On. If samlMetadataUri is provided but no value is provided for samlServiceProviderUri, the public domain name for the deployed Kibana instance will be used.

Supported only for Elasticsearch 6.2.0+

Kibana must be installed

SSL/TLS must be configured for HTTP layer of Elasticsearch

""

Master node related settings

vmSizeMasterNodesstring
Azure VM size of dedicated master nodes. See this list for supported sizes. By default the template deploys 3 dedicated master nodes, unless dataNodesAreMasterEligible is set to Yes.
Check that the size you choose is available in the region you choose.
Standard_DS1_v2

vmMasterNodeAcceleratedNetworkingstring
Whether to enable accelerated networking for Master nodes, which enables single root I/O virtualization (SR-IOV)
to a VM, greatly improving its networking performance. Valid values are

Default: enables accelerated networking for VMs known to support it

Yes: enables accelerated networking.

No: does not enable accelerated networking

Default

Data node related settings

dataNodesAreMasterEligiblestring
Either Yes or No to make all data nodes master eligible. This can be useful for small Elasticsearch clusters however, for larger clusters it is recommended to have dedicated master nodes.
When Yes no dedicated master nodes will be provisioned.
No

vmSizeDataNodesstring
Azure VM size of the data nodes. See this list for supported sizes.
Check that the size you choose is available in the region you choose.
Standard_DS1_v2

vmDataNodeAcceleratedNetworkingstring
Whether to enable accelerated networking for Data nodes, which enables single root I/O virtualization (SR-IOV)
to a VM, greatly improving its networking performance. Valid values are

Default: enables accelerated networking for VMs known to support it

Yes: enables accelerated networking.

No: does not enable accelerated networking

Default

vmDataNodeCountint
The number of data nodes you wish to deploy. Must be greater than 0.
3

Data node disk related settings

vmDataDiskCountint
Number of managed disks to attach to each data node in RAID 0 setup.
Must be equal to or greater than 0.

If the number of disks selected is more than can be attached to the data node VM (SKU) size,
the maximum number of disks that can be attached for the data node VM (sku) size will be used. Equivalent to
taking min(vmDataDiskCount, max supported disks for data node VM size)

When 1 disk is selected, the disk is not RAIDed.

When 0 disks are selected, no disks will be attached to each data node. Instead, the temporary disk will be used to store Elasticsearch data.
The temporary disk is ephemeral in nature and not persistent. Consult Microsoft Azure documentation on temporary disks
to understand the trade-offs in using it for storage.

Maximum number supported disks for data node VM size

vmDataDiskSizestring
The disk size of each attached disk. Choose 32TiB, 16TiB, 8TiB, 4TiB, 2TiB, 1TiB, 512GiB, 256GiB, 128GiB, 64GiB or 32GiB.
For Premium Storage, disk sizes equate to P80, P70, P60, P50, P40, P30, P20, P15, P10 and P6
storage disk types, respectively.

1TiB

storageAccountTypestring
The storage account type of the attached disks. Choose either Default or Standard.
The Default storage account type will be Premium Storage for VMs that
support Premium Storage and Standard Storage for those that do not. Standard will use Standard Storage.
Default

Coordinating node related settings

vmClientNodeCountint
The number of coordinating nodes to provision. Must be a positive integer. By default, the data nodes are added to the backend pool of the loadbalancer but
if you provision coordinating nodes, these will be added to the loadbalancer instead. Coordinating nodes can be useful in offloading the gather process from data nodes and are necessary to scale an Elasticsearch cluster deployed with this template beyond 100 data nodes (the maximum number of VMs that can be added to a load balancer backend pool).
0

vmSizeClientNodesstring
Azure VM size of the coordinating nodes see this list for supported sizes.
Check that the size you choose is available in the region you choose.
Standard_DS1_v2

vmClientNodeAcceleratedNetworkingstring
Whether to enable accelerated networking for coordinating nodes, which enables single root I/O virtualization (SR-IOV)
to a VM, greatly improving its networking performance. Valid values are

Default: enables accelerated networking for VMs known to support it

Yes: enables accelerated networking.

No: does not enable accelerated networking

Default

Security related settings

adminUsernamestring
Admin username used when provisioning virtual machines. Must be a valid Linux username i.e. avoid any of the following usernames for Ubuntu
""

authenticationTypestring
The authentication type for the Admin user. Either password or sshPublicKey
password

adminPasswordsecurestring
When authenticationType is password this sets the OS level user's password
""

sshPublicKeysecurestring
When authenticationType is sshPublicKey this sets the OS level sshKey that can be used to login.
""

securityBootstrapPasswordsecurestring
Security password for 6.x bootstrap.password key that is added to the keystore. If no value is supplied, a 13 character password
will be generated using the ARM template uniqueString() function. The bootstrap password is used to seed the built-in
users. Used only in 6.0.0+
""

securityAdminPasswordsecurestring
Security password Admin user.

This is the built-in elastic user.

should be a minimum of 12 characters, and must be greater than 6 characters.
""

securityKibanaPasswordsecurestring
Security password Kibana.

This is the built-in kibana user.

should be a minimum of 12 characters, and must be greater than 6 characters.
""

securityLogstashPasswordsecurestring
This is the built-in logstash_system user.

should be a minimum of 12 characters, and must be greater than 6 characters.
""

securityBeatsPasswordsecurestring
This is the built-in beats_system user. Valid for Elasticsearch 6.3.0+

should be a minimum of 12 characters, and must be greater than 6 characters.
""

securityApmPasswordsecurestring
This is the built-in apm_system user. Valid for Elasticsearch 6.5.0+

should be a minimum of 12 characters, and must be greater than 6 characters.
""

securityRemoteMonitoringPasswordsecurestring
This is the built-in remote_monitoring_user user. Valid for Elasticsearch 6.5.0+

should be a minimum of 12 characters, and must be greater than 6 characters.
""

Kibana related settings

kibanastring
Either Yes or No to provision a machine with Kibana installed and a public IP address to access it.
Yes

vmSizeKibanastring
Azure VM size of the Kibana instance. See this list for supported sizes.
Check that the size you select is available in the region you choose.
Standard_A2_v2

vmKibanaAcceleratedNetworkingstring
Whether to enable accelerated networking for Kibana, which enables single root I/O virtualization (SR-IOV)
to a VM, greatly improving its networking performance. Valid values are

Default: enables accelerated networking for VMs known to support it

Yes: enables accelerated networking.

No: does not enable accelerated networking

Default

kibanaCertBlobstring
A Base-64 encoded form of the certificate (.crt) in PEM format to secure HTTPS communication between the browser and Kibana.""

kibanaKeyBlobsecurestring
A Base-64 encoded form of the private key (.key) in PEM format to secure HTTPS communication between the browser and Kibana.""

kibanaKeyPassphrasesecurestring
The passphrase to decrypt the private key. Optional as the key may not be encrypted.""

kibanaAdditionalYamlstring
Additional configuration for Kibana yaml configuration file. Each line must be separated by a \n newline character e.g. "server.name: \"My server\"\nkibana.defaultAppId: home".

This is an expert level feature - It is recommended that you run your additional yaml through a linter before starting a deployment.""

Logstash related settings

logstashstring
Either Yes or No to provision Logstash VMs.
No

vmSizeLogstashstring
Azure VM size of the Logstash instance. See this list for supported sizes.
Check that the size you select is available in the region you choose.
Standard_DS1_v2

vmLogstashCountint
The number of Logstash instances
1

vmLogstashAcceleratedNetworkingstring
Whether to enable accelerated networking for Logstash, which enables single root I/O virtualization (SR-IOV)
to a VM, greatly improving its networking performance. Valid values are

Default: enables accelerated networking for VMs known to support it

Yes: enables accelerated networking.

No: does not enable accelerated networking

Default

logstashHeapSizeinteger
The size, in megabytes, of memory to allocate for the JVM heap for Logstash. If unspecified, Logstash will be configured with the default heap size for the distribution and version.
Take a look at the Logstash documentation on profiling heap size for more information.

This is an expert level feature - setting a heap size too low, or larger than available memory on the Logstash VM SKU will fail the deployment.
0

logstashConfsecurestring
A Base-64 encoded form of a Logstash config file to deploy.
""

logstashKeystorePasswordsecurestring
The password to protect the Logstash keystore. If no value is supplied, a value will be generated using the ARM template uniqueString() function. Used only in 6.2.0+
""

logstashAdditionalPluginsstring
Additional Logstash plugins to install. Each plugin must be separated by a semicolon. e.g. logstash-input-heartbeat;logstash-input-twitter
""

logstashAdditionalYamlstring
Additional configuration for Logstash yaml configuration file. Each line must be separated by a newline character \n e.g. "pipeline.batch.size: 125\npipeline.batch.delay: 50".

This is an expert level feature - It is recommended that you run your additional yaml through a linter before starting a deployment.
""

Jumpbox related settings

jumpboxstring
Either Yes or No to optionally add a virtual machine with a public IP to the deployment, which you can use to connect and manage virtual machines on the internal network.
NOTE: If you are deploying Kibana, the Kibana VM can act
as a jumpbox, so a separate jumpbox VM is not needed.
No

Virtual network related settings

vNetNewOrExistingstring
Whether the Virtual Network is new or existing. An existing Virtual Network in
another Resource Group in the same Location can be used.
new

vNetNamestring
The name of the Virtual Network.
The Virtual Network must already exist when using an existing Virtual Network
es-net

vNetExistingResourceGroupstring
The name of the Resource Group in which the Virtual Network resides when using an existing Virtual Network.
Required when using an existing Virtual Network
""

vNetNewAddressPrefixstring
The address prefix when creating a new Virtual Network. Required when creating a new Virtual Network
10.0.0.0/24

vNetLoadBalancerIpstring
The internal static IP address to use when configuring the internal load balancer. Must be an available
IP address on the provided vNetClusterSubnetName.
10.0.0.4

vNetClusterSubnetNamestring
The name of the subnet to which Elasticsearch nodes will be attached.
The subnet must already exist when using an existing Virtual Network
es-subnet

vNetNewClusterSubnetAddressPrefixstring
The address space of the subnet.
Required when creating a new Virtual Network
10.0.0.0/25

vNetAppGatewaySubnetNamestring
Subnet name to use for the Application Gateway.
Required when selecting gateway for load balancing.

The subnet must already exist when using an existing Virtual Network
es-gateway-subnet

vNetNewAppGatewaySubnetAddressPrefixstring
The address space of the Application Gateway subnet.
Required when creating a new Virtual Network and selecting gateway for load balancing.
10.0.0.128/28

Application Gateway related settings

appGatewayTierstring
The tier of the Application Gateway, either Standard or WAF.
Required when selecting gateway for load balancing.
Standard

appGatewaySkustring
The size of the Application Gateway. Choose Small, Medium or Large.
When choosing appGatewayTier WAF, the size must be at least Medium.
Required when selecting gateway for load balancing.
Medium

appGatewayCountint
The number instances of the Application Gateway. Can be a value between 1 and 10.
A minimum of 2 is recommended for production.
Required when selecting gateway for load balancing.
2

appGatewayCertBlobstring
A Base-64 encoded form of the PKCS#12 archive (.p12/.pfx) containing the certificate and key for Application Gateway.
This certificate is used to secure HTTPS connections to and from the Application Gateway.
Required when selecting gateway for load balancing.
""

appGatewayCertPasswordsecurestring
The password for the PKCS#12 archive (.p12/.pfx) containing the certificate and key for Application Gateway.
Required when selecting gateway for load balancing.
""

appGatewayEsHttpCertBlobsecurestring
The Base-64 encoded public certificate (.cer) used to secure the HTTP layer of Elasticsearch. Used by the Application Gateway to whitelist certificates used by the backend pool. Required when using esHttpCertBlob to secure the HTTP layer of Elasticsearch and selecting gateway for load balancing. X-Pack plugin must be installed
""

appGatewayWafStatusstring
The firewall status of the Application Gateway, either Enabled or Disabled.
Required when selecting gateway for load balancing and using appGatewayTier WAF.
Enabled

appGatewayWafModestring
The firewall mode of the Application Gateway, either Detection or Prevention.
Required when selecting gateway for load balancing and using appGatewayTier WAF.
Detection

### Web based deploy

The above button will take you to the autogenerated web based UI based on the
parameters from the ARM template.

### Command line deploy

You can deploy using the template directly from Github using the Azure CLI or Azure PowerShell

### Azure CLI 1.0

Azure CLI 1.0 is no longer supported as the `apiVersion`s of resources are newer than those
supported by the last release. It's recommended to update to [Azure CLI 2.0](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest).

### Azure CLI 2.0

1. Log into Azure

```sh
az login
```
2. Create a resource group `` in a `` (e.g `westeurope`) where we can deploy too

```sh
az group create --name --location
```

3. Use our template directly from GitHub using `--template-uri`

```sh
az group deployment create \
--resource-group \
--template-uri https://raw.githubusercontent.com/elastic/azure-marketplace/master/src/mainTemplate.json \
--parameters @parameters/password.parameters.json
```

where `` refers to the resource group you just created.

### Azure PowerShell

1. Log into Azure

```powershell
Login-AzureRmAccount
```

2. Select a Subscription Id

```powershell
Select-AzureRmSubscription -SubscriptionId ""
```

3. Define the parameters object for your deployment as a PowerShell hashtable. The keys
correspond the parameters defined in the [Parameters section](#parameters)

```powershell
$branch = "master"
$esVersion = "7.11.1"

$clusterParameters = @{
"_artifactsLocation" = "https://raw.githubusercontent.com/elastic/azure-marketplace/$branch/src/"
"esVersion" = $esVersion
"esClusterName" = "elasticsearch"
"loadBalancerType" = "internal"
"vmDataDiskCount" = 1
"adminUsername" = "russ"
"adminPassword" = "Password1234"
"securityBootstrapPassword" = "Password1234"
"securityAdminPassword" = "Password1234"
"securityKibanaPassword" = "Password1234"
"securityLogstashPassword" = "Password1234"
"securityBeatsPassword" = "Password1234"
"securityApmPassword" = "Password1234"
"securityRemoteMonitoringPassword" = "Password1234"
}
```

4. Create a resource group `` in a `` (e.g `westeurope`) where we can deploy too

```powershell
New-AzureRmResourceGroup -Name "" -Location ""
```

5. Use our template directly from GitHub

```powershell
New-AzureRmResourceGroupDeployment -Name "" -ResourceGroupName "" -TemplateUri "https://raw.githubusercontent.com/elastic/azure-marketplace/master/src/mainTemplate.json" -TemplateParameterObject $clusterParameters
```

## Targeting a specific template version

You can target a specific version of the template by modifying the URI of the template and
the `_artifactsLocation` parameter of the template to point to a specific tagged release.

**Targeting a specific template version is recommended for repeatable production deployments.**

For example, to target the [`7.11.1` tag release with PowerShell](https://github.com/elastic/azure-marketplace/tree/7.11.1)

```powershell
$templateVersion = "7.11.1"
$_artifactsLocation = "https://raw.githubusercontent.com/elastic/azure-marketplace/$templateVersion/src/"

# minimum parameters required to deploy
$clusterParameters = @{
"_artifactsLocation" = $_artifactsLocation
"esVersion" = "7.11.1"
"adminUsername" = "russ"
"adminPassword" = "Password1234"
"securityBootstrapPassword" = "Password1234"
"securityAdminPassword" = "Password1234"
"securityKibanaPassword" = "Password1234"
"securityLogstashPassword" = "Password1234"
"securityBeatsPassword" = "Password1234"
"securityApmPassword" = "Password1234"
"securityRemoteMonitoringPassword" = "Password1234"
}

$resourceGroup = "my-azure-cluster"
$location = "Australia Southeast"
$name = "my-azure-cluster"

New-AzureRmResourceGroup -Name $resourceGroup -Location $location
New-AzureRmResourceGroupDeployment -Name $name -ResourceGroupName $resourceGroup -TemplateUri "$_artifactsLocation/mainTemplate.json" -TemplateParameterObject $clusterParameters
```

## Configuring TLS

**It is strongly recommended that you secure communication using Transport Layer Security when using the template**.
The Elastic Stack security features can provide Basic Authentication,
Role Based Access control, and Transport Layer Security (TLS)
for both Elasticsearch and Kibana. For more details, please refer to
[the Security documentation](https://www.elastic.co/guide/en/elastic-stack-deploy/current/azure-arm-template-security.html).

For Elasticsearch versions 6.8.0+ (and less than 7.0.0), and 7.1.0+, the Elastic Stack security features
that allow configuring TLS and role based access control are available in the free basic license tier.
For all other versions, the Elastic Stack security features require a license level higher than basic;
They can be configured with a trial license, which provides access to the Security features for 30 days.

### TLS for Kibana

You can secure external access from the browser to Kibana with TLS by supplying
a certificate and private key in PEM format with `kibanaCertBlob` and
`kibanaKeyBlob` parameters, respectively.

### TLS for Elasticsearch Transport layer

You can secure communication between nodes in the cluster with TLS on the
Transport layer. Configuring TLS for the Transport layer requires
`xPackPlugins` be set to `Yes`, or an Elasticsearch version 6.8.0+ (and less than 7.0.0) or 7.1.0+.

You must supply a PKCS#12 archive with the `esTransportCaCertBlob` parameter (and optional
passphrase with `esTransportCaCertPassword`) containing the CA cert which should be used to generate
a certificate for each node within the cluster. An optional
passphrase can be passed with `esTransportCertPassword` to encrypt the generated certificate
on each node.

One way to generate a PKCS#12 archive containing a CA certificate and key is using
[Elastic's `elasticsearch-certutil` command](https://www.elastic.co/guide/en/elasticsearch/reference/current/certutil.html).
The simplest command to generate a CA certificate is

```sh
./elasticsearch-certutil ca
```

and follow the instructions.

### TLS for Elasticsearch HTTP layer

You can secure external access to the cluster with TLS with an external
loadbalancer or Application Gateway. Configuring TLS for the HTTP layer requires
`xPackPlugins` be set to `Yes`, or an Elasticsearch version 6.8.0+ (and less than 7.0.0) or 7.1.0+.

#### External load balancer

If you choose `external` as the value for `loadBalancerType`, you must either

**_or_**

* supply a PKCS#12 archive containing the key and certificate with the `esHttpCaCertBlob` parameter (and optional
passphrase with `esHttpCaCertPassword`) containing the CA which should be used to generate
a certificate for each node within the cluster to secure the HTTP layer.
Kibana will be configured to trust the CA and perform hostname verification for presented
certificates. One way to generate a PKCS#12 archive is using [Elastic's certutil command](https://www.elastic.co/guide/en/elasticsearch/reference/current/certutil.html).

#### Application Gateway

If you choose `gateway` as the value for `loadBalancerType`, you must

* supply a PKCS#12 archive containing the key and certificate with the `appGatewayCertBlob` parameter (and optional
passphrase with `appGatewayCertPassword`) to secure communication to Application Gateway.
One way to generate a PKCS#12 archive is using [Elastic's certutil command](https://www.elastic.co/guide/en/elasticsearch/reference/current/certutil.html)

[Application Gateway](https://azure.microsoft.com/en-au/services/application-gateway/)
performs SSL offload, so communication from Application Gateway to
Elasticsearch is not encrypted with TLS by default. TLS to Application Gateway may be sufficient for your
needs, but if you would like end-to-end encryption by also configuring TLS for Elasticsearch HTTP layer, you can

* supply a PKCS#12 archive containing the key and certificate with the `esHttpCertBlob` parameter (and optional
passphrase with `esHttpCertPassword`) containing the certs and private key to
secure the HTTP layer. This certificate will be used by all nodes within the cluster, and
Kibana will be configured to trust the certificate CA (if CA certs are present within the archive).
One way to generate a PKCS#12 archive is using [Elastic's certutil command](https://www.elastic.co/guide/en/elasticsearch/reference/current/certutil.html), and you must
specify a `--dns ` argument with a name that matches that in the `--name ` argument.

**_and_**

* supply the public certificate in PEM format from the PKCS#12 archive
passed with `esHttpCertBlob` parameter, using the `appGatewayEsHttpCertBlob` parameter.
Application Gateway whitelists certificates used by VMs in the backend pool. This can
be extracted from the PKCS#12 archive of the `esHttpCertBlob` parameter using
[`openssl pkcs12`](https://www.openssl.org/docs/man1.0.2/apps/pkcs12.html)

```sh
openssl pkcs12 -in http_cert.p12 -out http_public_cert.cer -nokeys
```

and provide the passphrase for the archive when prompted.

**IMPORTANT**: When configuring [end-to-end encryption with Application Gateway](https://docs.microsoft.com/en-us/azure/application-gateway/application-gateway-end-to-end-ssl-powershell),
the certificate to secure the HTTP layer **must** include a x509v3 Subject Alternative Name
extension with a DNS entry that matches the Subject CN, to work with Application
Gateway's whitelisting mechanism. This can be checked using
[`openssl x509`](https://www.openssl.org/docs/man1.0.2/apps/x509.html)

```sh
openssl x509 -in http_public_cert.cer -text -noout
```

which will output something similar to

```text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
// omitted for brevity ...
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=Elastic Certificate Tool Autogenerated CA
Validity
Not Before: Jul 5 02:37:40 2018 GMT
Not After : Jul 4 02:37:40 2021 GMT
Subject: CN=custom
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
// omitted for brevity ...
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Key Identifier:
// omitted for brevity ...
X509v3 Authority Key Identifier:
// omitted for brevity ...

X509v3 Subject Alternative Name:
DNS:custom
X509v3 Basic Constraints:
CA:FALSE
Signature Algorithm: sha256WithRSAEncryption
// omitted for brevity ...
```

_Without_ this, Application Gateway will return [502 Bad Gateway errors](https://docs.microsoft.com/en-gb/azure/application-gateway/application-gateway-troubleshooting-502),
as the health probe for the backend pool will fail when the whitelisted certificate does
not contain this certificate extension.
You can typically understand if there is a problem with the key format when

1. TLS has been configured on the HTTP layer
2. Kibana is able to communicate to the cluster correctly _but_ Application Gateway returns 502 errors.

This may not always be the case, but can be indicative. You should also check the description for Backend Health
of the Application Gateway in the Azure portal.

### Passing certificate parameters

Parameters such as esHttpCertBlob and kibanaCertBlob must be provided in Base-64 encoded form. A Base-64 encoded value can be obtained using

1. [base64](https://linux.die.net/man/1/base64) on Linux, or [openssl](https://www.openssl.org/docs/man1.0.2/apps/openssl.html) on Linux and MacOS

base64

```sh
httpCert=$(base64 http-cert.p12)
```

openssl

```sh
httpCert=$(openssl base64 -in http-cert.p12)
```

and including the value assigned to $httpCert in the parameters.json file as the value for certificate parameter passed to the Azure CLI command

2. PowerShell on Windows

```powershell
$httpCert = [Convert]::ToBase64String([IO.File]::ReadAllBytes("c:\http-cert.p12"))
```

and then pass this in the template parameters object passed to the Azure PowerShell command

```powershell
$clusterParameters = @{
# Other parameters skipped for brevity
"esHttpCertBlob"= $httpCert
}
```

## License

This project is [MIT Licensed](https://github.com/elastic/azure-marketplace/blob/master/LICENSE.txt) and was originally forked from
the [Elasticsearch Azure quick start arm template](https://github.com/Azure/azure-quickstart-templates/tree/master/elasticsearch)