https://github.com/resurfaceio/azure-eh
Capture API calls from Azure APIM with Azure Event Hubs
https://github.com/resurfaceio/azure-eh
api-logger azure azure-api-management azure-event-hubs cybersecurity graylog http-logger java logger-java observability
Last synced: 5 months ago
JSON representation
Capture API calls from Azure APIM with Azure Event Hubs
- Host: GitHub
- URL: https://github.com/resurfaceio/azure-eh
- Owner: resurfaceio
- License: apache-2.0
- Created: 2021-05-27T22:55:58.000Z (almost 4 years ago)
- Default Branch: master
- Last Pushed: 2024-08-21T15:55:13.000Z (9 months ago)
- Last Synced: 2024-08-22T18:22:12.110Z (9 months ago)
- Topics: api-logger, azure, azure-api-management, azure-event-hubs, cybersecurity, graylog, http-logger, java, logger-java, observability
- Language: Java
- Homepage:
- Size: 116 KB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# azure-eh
Capture API calls from Azure API Management to your own security data lake using Azure Event Hubs.
[](https://github.com/resurfaceio/azure-eh/blob/master/LICENSE)
[](https://github.com/resurfaceio/azure-eh/blob/master/CONTRIBUTING.md)
## Contents
- [Deployment](#deployment)
- [Configuration](#configuration)
- [Running on AKS](#running-on-aks)
- [Protecting User Privacy](#protecting-user-privacy)
## Deployment
In order to run Resurface for Azure APIM, some previous configuration is needed. Specifically, four resources need to be created and deployed: an [Event Hub](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-about), a [Storage Account](https://docs.microsoft.com/en-us/azure/storage/common/storage-account-overview), a [Logger](https://docs.microsoft.com/en-us/rest/api/apimanagement/current-ga/logger/create-or-update), and a [Policy](https://docs.microsoft.com/en-us/azure/api-management/set-edit-policies) for your APIM instance.
### Automatic deployment
Click the **Deploy to Azure** button below to deploy all necessary resources using an [ARM template](https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/overview):
[](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fresurfaceio%2Fiac-templates%2Fmaster%2Fazure%2Fazuredeployresources.json)
This uses [a custom template](https://github.com/resurfaceio/iac-templates/blob/master/azure/azuredeployresources.json) to create and deploy an [Event Hubs instance](https://github.com/resurfaceio/iac-templates/blob/master/azure/event-hub.json), a [Storage Account](https://github.com/resurfaceio/iac-templates/blob/master/azure/storageaccount.json), and adds a [Logger and Policy](https://github.com/resurfaceio/iac-templates/blob/master/azure/logger-and-policy.json) to your existing APIM instance
> [!WARNING]
> This will overwrite any global policies you have in place in your APIM instance. Make sure you back up existing policies!
Once the automatic deployment finishes, go to the **Outputs** section.
Click to expand
Copy the listed values and update the [required environment variables](#logging-from-azure-event-hubs) accordingly.
Click to expand
### Manual setup
If you would like to configure everything yourself using the Azure console instead, just follow Resurface's [Capturing from APIM get-started guide](https://resurface.io/azure-get-started#manual-setup), where the entire process is documented in a step-by-step manner.
## Configuration
- Set following the environment variables in your `.env` file:
| Variable | Set to |
|:---------|:-------|
|`AZURE_EH_CONNECTION_STRING`|[Connection string for a specific Azure Event Hubs namespace](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string)|
|`EVENT_HUB_NAME` |Name of your Event Hub instance|
|`AZURE_STORAGE_CONNECTION_STRING`|[Connection string for a specific Azure Storage Account]([https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string](https://docs.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string))|
|`STORAGE_CONTAINER_NAME` |Name of your storage container|
|`USAGE_LOGGERS_URL` |DB capture endpoint for your [Resurface instance](https://resurface.io/installation)|
|`USAGE_LOGGERS_RULES` |(**Optional**) Set of [rules](#protecting-user-privacy).
Only necessary if you want to exclude certain API calls from being logged.|
|`EVENT_HUB_CONSUMER_GROUP` |(**Optional**) Name of a consumer group from your Event Hub.
Only necessary if you have created a specific consumer group for your Event Hub instance|
|`PARTITION_NUMBER` |(**Deprecated**) Partition number configured in `policy.xml`. Should be `"0"` by default.|
- (Optional) Build the container image
```bash
docker build -t azure-eh-consumer:1.0.1 .
```
- Run the container
```bash
docker run -d --name azure-eh --env-file .env resurfaceio/azure-eh-consumer:1.0.1
```
Or, if you built the image yourself in the previous step:
```bash
docker run -d --name azure-eh --env-file .env azure-eh-consumer:1.0.1
```
- Use your API as you always do.
- Go to the web UI for your Graylog API Security instance and verify that API Calls are being captured.
## Running on AKS
Using [Helm](https://helm.sh/) you can deploy this listener application to your running cluster
```bash
helm upgrade -i resurface resurfaceio/resurface --namespace resurface \
--set consumer.azure.enabled=true \
--set consumer.azure.ehconnstring=YOUR_AZURE_EH_CONNECTION_STRING \
--set consumer.azure.ehname=YOUR_AZURE_EVENT_HUBS_INSTANCE_NAME \
--set consumer.azure.storageconnstring=YOUR_AZURE_STORAGE_CONNECTION_STRING \
--set consumer.azure.storagecontainer=YOUR_AZURE_STORAGE_CONTAINER_NAME
```
## Protecting User Privacy
Loggers always have an active set of rules that control what data is logged
and how sensitive data is masked. All of the examples above apply a predefined set of rules (`include debug`),
but logging rules are easily customized to meet the needs of any application.
© 2016-2024 Graylog, Inc.