https://github.com/logzio/azure-serverless
https://github.com/logzio/azure-serverless
integration
Last synced: 4 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/logzio/azure-serverless
- Owner: logzio
- License: apache-2.0
- Created: 2024-01-17T14:01:49.000Z (over 1 year ago)
- Default Branch: master
- Last Pushed: 2024-12-02T13:52:41.000Z (6 months ago)
- Last Synced: 2025-01-05T08:56:27.720Z (5 months ago)
- Topics: integration
- Language: Python
- Homepage:
- Size: 274 KB
- Stars: 0
- Watchers: 4
- Forks: 0
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
**This project is under development**
---
# azure-serverless
This repository contains the Python code and instructions to ship logs from your Azure services to Logz.io using an Azure Function. After completing the setup, your Azure Function will forward logs from an Azure Event Hub to your Logz.io account.
## Deployment methods:
* [Deploy via Azure ARM Template](#1-deploy-the-logzio-python-template)
* [Deploy via Terraform](#alternative-setup-using-terraform)
## Setting up Log Shipping from Azure
### 1. Deploy the Logz.io Python Template👇
[](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Flogzio%2Fazure-serverless%2Fmaster%2Fdeployments%2Fazuredeploylogs.json)
This deployment will create the following services:
* Serverless Function App (Python-based)
* Event Hubs Namespace
* Event Hubs Instance
* Storage account:
- Function's logs containers
- Failed shipments logs container
* App Service Plan
* Log Analytics Workspace
* Application Insights
### 2. Configure the Template
Use these settings when configuring the template:
| Parameter | Description |
|-----------------|-------------------------------------------------------------------------|
| Resource group* | Create a new resource group or select an existing one. |
| Region* | Select the region closest to your Azure services. |
| LogzioURL* | Use the listener URL specific to your Logz.io account region. |
| LogzioToken* | Your Logz.io logs shipping token. |
| ThreadCount | Number of threads for the Function App (default: 4). |
| bufferSize | Maximum number of messages to accumulate before sending (default: 100). |
| intervalTime | Interval time for sending logs in milliseconds (default: 10000). |
| maxTries | The maximum number of retries for the backoff mechanism (default: 3). |
| logType | The type of the logs being processed (default: eventHub). |
*Required fields.
After setting the parameters, click **Review + Create**, and then **Create** to deploy.
### 3. Stream Azure Service Data
Configure your Azure services to stream logs to the newly created Event Hub. For each service:
1. Create diagnostic settings.
2. Under **Event hub policy name**, select the appropriate policy (e.g., 'RootManageSharedAccessKey').
For more details, see [Microsoft's documentation](https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitor-stream-monitoring-data-event-hubs).
### 4. Verify Data Reception in Logz.io
Allow some time for data to flow from Azure to Logz.io, then check your Logz.io account. You should see logs of the type `eventHub` in Logz.io.
### Backup for Unshipped Logs
The deployment includes a backup mechanism for logs that fail to ship to Logz.io. By default, these logs are stored in the **failedlogbackup** blob container, but this can be customized to a different container name of your choice during the setup process.
### Post-Deployment Configuration
To modify configuration after deployment, visit your Function App's **Configuration** tab. You can adjust settings such as `LogzioURL`, `LogzioToken`, `bufferSize`, and more.
---
## Alternative Setup using Terraform
As an alternative to the Azure Template, you can use Terraform to set up your log shipping environment. The Terraform configuration files are located in the **deployments** folder of this repository. Follow the instructions below to deploy this integration using Terraform.
#### Prerequisites
- Terraform installed on your local machine.
- Azure CLI installed and configured with your Azure account credentials.
#### Steps to Deploy using Terraform
1. **Obtain the Terraform Configuration**: Use curl to download only the azuredeploylogs.tf and variables.tf files from the GitHub repository.
```bash
curl -O https://raw.githubusercontent.com/logzio/azure-serverless/master/deployments/azuredeploylogs.tf \
&&
curl -O https://raw.githubusercontent.com/logzio/azure-serverless/master/deployments/variables.tf
```
2. **Create a `.tfvars` File**: Create a `terraform.tfvars` file in the same folder to specify your configurations, such as your Logz.io token.
```hcl
logzio_url = "https://<>:8071"
logzio_token = "<>"
thread_count = 4
buffer_size = 100
interval_time = 10000
max_tries = 3
log_type = "eventHub"
3. **Initialize Terraform**: Run the Terraform initialization to install the necessary plugins.
```bash
terraform init
```
4. **Apply Terraform Configuration**: Deploy the infrastructure using `terraform apply`. You will be prompted to review the proposed changes before applying the configuration.
```bash
terraform apply
```
Type **yes** when prompted to confirm the deployment.
5. **Verify Deployment**: After successful application of the Terraform configuration, your Azure services will be set up and ready for log shipping.
---
## Changelog
- 0.0.2:
* Added `ENV_FUNCTION_VERSION` parameter for dynamic versioning in ARM template and Terraform.
* FunctionAppVersion now dynamically matches the ARM template's contentVersion.
- 0.0.1:
* Initial release with Python Azure Function.
* Implement log shipping to Logz.io.
* Backup mechanism for failed log shipments.
* Customizable log batching and threading.