https://github.com/newrelic/nr-entity-tag-sync
The New Relic Entity Tag Sync application is a tool used to map entity metadata from entities in an external system of record to tags on New Relic entities and to keep such tag values synchronized with the external entity metadata values across time.
https://github.com/newrelic/nr-entity-tag-sync
cmdb nrlabs nrlabs-utility servicenow
Last synced: 5 months ago
JSON representation
The New Relic Entity Tag Sync application is a tool used to map entity metadata from entities in an external system of record to tags on New Relic entities and to keep such tag values synchronized with the external entity metadata values across time.
- Host: GitHub
- URL: https://github.com/newrelic/nr-entity-tag-sync
- Owner: newrelic
- License: apache-2.0
- Created: 2023-04-20T13:52:38.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2025-04-16T22:15:42.000Z (about 1 year ago)
- Last Synced: 2025-09-29T15:56:20.884Z (9 months ago)
- Topics: cmdb, nrlabs, nrlabs-utility, servicenow
- Language: Go
- Homepage:
- Size: 144 KB
- Stars: 6
- Watchers: 10
- Forks: 5
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://github.com/newrelic/open-source-office/blob/master/examples/categories/index.md#category-community-project)
# New Relic Entity Tag Sync
## Overview
The New Relic Entity Tag Sync application is a tool used to map entity metadata
from entities in an external system of record to tags on New Relic entities and
to keep such tag values synchronized with the external entity metadata values
across time.
### Concepts
There are several key concepts to be aware of in order to understand how the
entity tag sync application works.
#### New Relic Entities
A New Relic entity is anything that reports data to New Relic or that
contains data that we have access to. More information on New Relic entities
is available [in our documentation](https://docs.newrelic.com/docs/new-relic-solutions/new-relic-one/core-concepts/what-entity-new-relic/).
#### External Entities
An external entity can be generally thought of as any object with metadata in
an external system of record. The specific definition of an external entity is
particular to the external system.
Within the entity tag sync application, external entities are modeled as a set
of key-value pairs and a unique ID. It is these key-value pairs that are used
during the synchronization process to populate and update tags on the New Relic
entities.
#### Providers
A provider is a module capable of retrieving entity metadata from an external
system of record and mapping that data to the internal representation of an
external entity as a set of key-value pairs.
The following providers are supported:
* ServiceNow CMDB
##### ServiceNow CMDB provider
The ServiceNow CMDB provider models CMDB configuration items (CIs) as external
entities, enabling fields on CIs to be mapped to tags on New Relic entities.
The ServiceNow CMDB provider leverages the
[ServiceNow ReST API](https://docs.servicenow.com/bundle/utah-api-reference/page/integrate/inbound-rest/concept/c_RESTAPI.html)
to retrieve CI data and supports both HTTP Basic authentication and OAuth 2.0
authentication. See the [ReST API Security](https://docs.servicenow.com/bundle/utah-api-reference/page/integrate/inbound-rest/concept/c_RESTAPI.html#d773849e666)
documentation for more details on using these authentication methods with the
ServiceNow ReST API. See the [ServiceNow CMDB provider parameters section](#servicenow-cmdb-provider-parameters)
for details on how to configure the ServiceNow CMDB provider to use these
authentication methods.
#### Mappings
Mappings drive the actual synchronization process. Each mapping tells the entity
tag sync application what external entities to select from the provider, what
entities to select from New Relic, how to match external entities with New Relic
entities, and how key-value pairs from the external entity map to tags on the
New Relic entity.
Mappings are specified in the [mappings](#mapping-parameters) section of
[the configuration file](#configuration). During the synchronization process,
the entity tag sync application processes each mapping in order. A mapping is
processed as follows.
1. Fetch external entities
All external entities matching the [external entity query criteria](#external-entity-query-criteria)
are retrieved via the provider. The keys of the [`mapping`](#mapping) node as
well as the value of the `extEntityKey` in the [`match`](#match-strategy)
node are passed to the provider indicating the key-value pairs to retrieve
for each external entity. The [last update timestamp](#delta-synchronization)
is also passed to the provider if one was retrieved.
2. Fetch New Relic entities
All New Relic entities matching the [New Relic entity query critera](#new-relic-entity-query-criteria)
are retrieved via the New Relic GraphQL API
3. Find matching entities
For each selected New Relic entity, the [match strategy](#match-strategy) is
applied to find a matching external entity. If no match is found, processing
proceeds to the next New Relic entity or, if all candidate New Relic entities
have been processed, to the next mapping. If a match is found, processing
continues to the synchronize step.
4. Synchronize tags
If a match is discovered, the synchronization process executes the following
logic for each pair of external entity key, **E**, to New Relic tag name,
**T** in the [`mapping`](#mapping) node.
- If **E** *does not* exist in the external entity metadata and **T**
*does not* exist in the tags of the matching New Relic entity, do nothing
and continue to the next pair.
- If **E** *does not* exist in the external entity metadata and **T** *does*
exist in the tags of the matching New Relic entity, delete the tag **T**
from the New Relic entity.
- If **E** *does* exist in the external entity metadata and **T** *does not*
exist in the tags of the matching New Relic entity, add a tag to thew New
Relic entity with **E** for the tag name and the value of key **E** in the
external entity metadata as the singular tag value.
- If **E** *does* exist in the external entity metadata and **T** *does*
exist in the tags of the matching New Relic entity, scan the values for the
tag **T** in the matching New Relic entity.
- If the value of key **E** in the external entity metadata *is* in the
values of tag **T**, do nothing and continue to the next pair.
- If the value of key **E** in the external entity metadata *is not* in the
values of tag **T**, the values of tag **T** are **replaced** with the
value of key **E** in the external entity metadata.
**NOTE:** Case 2 and case 4, subcase 2 above are destructive. In both cases,
existing tag values removed. In general it should probably be assumed that
the tags being synchronized are managed by the entity tag sync application.
and should be modified via other means.
### Audit Events
The entity tag sync application is capable of producing audit events at various
points during the synchronization process. This feature is disabled by default
but can be enabled by setting the `events.enabled`
[general configuration parameter](#general-parameters). When enabled, the entity
tag sync application will produce events with the event type specified in the
`events.eventType` [general configuration parameter](#general-parameters) or the
event type `EntityTagSync` by default. The following attributes and values are
captured for _every_ event. Additional attributes are [action](#event-actions)
dependent.
| Name | Type | Description |
| --- | --- | --- |
| `id` | string | A canonical RFC-4122 UUID string that uniquely identifies each synchronization cycle |
| `action` | string | A string identifying the [action](#event-actions) that this event describes |
| `error` | bool | Flag indicating if an error occurred during this transaction or not |
| `errorMessage` | string | If an error occurred, a message describing what happened |
#### Event Actions
The following `action`s are produced along with any additional attributes
captured by each action.
**sync_start**
This action is produced at the start of each sync cycle and does not carry any
additional attributes. Note that the `error` attribute for this action will
always be `false` and the `errorMessage` attribute will always be empty.
**sync_end**
This action is produced at the end of each sync cycle and does not carry any
additional attributes. The `error` attribute for this action will be set to
`true` if _any_ error occurred, including the case where the sync cycle finishes
successfully but a specific mapping has update errors. The `errorMessage`
attribute will be set providing more details.
**mapping_complete**
This action is produced each time during a sync cycle that the entity tag sync
application finishes processing a [mapping](#mappings). The set of attributes
captured for this action fall into one of three cases.
1. If an error occurred while processing the mapping, The `error` attribute for
this action will be set to `true`, the `errorMessage` attribute will be set
providing more details, and no additional attributes will be present.
1. If no external entities were returned by the [provider](#providers), not due
to an error, the `error` attribute for this action will be `false`, the
`errorMessage` attribute will be empty, and the `extEntityCount` attribute
will be set to `0`, indicating that no external entities were returned and
the subsequent mapping process was skipped since there was nothing to do.
1. If the overall mapping process completed, even if there were errors updating
_some_ entities, the `error` attribute for this action will be `false`, the
`errorMessage` attribute will be empty and the following attributes will be
set.
* `extEntityCount` - the number of external entities returned by the
[provider](#providers)
* `totalEntityCount` - the total number of New Relic entities that matched
the [New Relic entity criteria](#new-relic-entity-query-criteria)
* `totalEntitiesScanned` - the total number of New Relic entities that were
tested against the external entities using [the match strategy](#match-strategy).
This might be different than the `totalEntityCount` if an error occurred
before all entities could be processed.
* `totalEntitiesMatched` - the total number of New Relic entities that
matched an external entity according to [the match strategy](#match-strategy)
* `totalEntitiesNoMatch` - the total number of New Relic entities that did
not match an external entity according to [the match strategy](#match-strategy)
* `totalEntitiesSkipped` - the total number of New Relic entities that
matched an external entity according to [the match strategy](#match-strategy)
but were up-to-date with the external entity and did not require updates
* `totalEntitiesUpdated` - the total number of New Relic entities that
matched an external entity according to [the match strategy](#match-strategy)
and were updated successfully
* `totalEntitiesWithErrors` - the total number of New Relic entities that
matched an external entity according to [the match strategy](#match-strategy)
but were not updated successfully due to errors
### Delta Synchronization
By default, the synchronization cycle is stateless. As a result, unless
implemented directly by the provider, the provider has no way to determine when
the last synchronization was run which will likely cause the provider to
retrieve all external entities during each synchronization cycle. This may be
expensive in terms of consumed resources, especially as the data set of external
entities and/or the number of key-value pairs being retrieved increase.
To address this issue, the entity tag sync application can be configured to
"maintain" the timestamp of the last synchronization and pass the timestamp to
the provider when retrieving external entities. To enable this feature, the
`provider.useLastUpdate` flag must be set to `true` and [audit events](#audit-events)
must be enabled.
When enabled, the entity tag sync application will query NRDB for the latest
occurence of the [audit event](#audit-events) with the event type specified in
the `events.eventName` configuration parameter for which the value of the
`action` attribute is set to `sync_end`. The resulting timestamp is passed to
the provider implementation as the third parameter of the
[`GetEntities`](https://github.com/newrelic/nr-entity-tag-sync/blob/main/internal/provider/provider.go#L17)
function.
Provider implementations are not required to support this feature but providers
that do support it must honor it when it is passed.
**NOTE:** This functionality is currently implemented in a fairly primitive way.
The timestamp of the last synchronization is determined by querying NRDB for the
latest timestamp of the most recent audit event. This is why
[audit events](#audit-events)must be enabled in order to leverage this feature.
## Getting Started
The New Relic Entity Tag Sync application can be run directly [on a host](#on-host)
or [run as an AWS Lambda function](#aws-lambda-function).
### On-host
The New Relic Entity Tag Sync application provides binaries for the following
host environments.
* [Linux x86](https://github.com/newrelic/nr-entity-tag-sync/releases/latest/download/nr-entity-tag-sync_Linux_i386.tar.gz)
* [Linux amd64](https://github.com/newrelic/nr-entity-tag-sync/releases/latest/download/nr-entity-tag-sync_Linux_x86_64.tar.gz)
* [Windows x86](https://github.com/newrelic/nr-entity-tag-sync/releases/latest/download/nr-entity-tag-sync_Windows_i386.zip)
* [Windows amd64](https://github.com/newrelic/nr-entity-tag-sync/releases/latest/download/nr-entity-tag-sync_Windows_x86_64.zip)
#### Run the application on host
To run the New Relic Entity Tag Sync application as a standalone application on
a host, perform the following steps.
1. Download the binary for your platform from
[the latest release](https://github.com/newrelic/nr-entity-tag-sync/releases/latest)
1. Extract the archive to a new or existing directory.
1. Create a directory named `configs` in the same directory.
1. To use an existing configuration file, copy it to a file named [`config.yml`](#configuration)
in the new `configs` directory. To create a new configuration file, copy the
[`configs/config.sample.yml`](./configs/config.sample.yml).
to `configs/config.yml` and customize it to [configure](#configuration) the
application appropriately for your environment.
1. Optionally, to specify one or more [configuration](#configuration) parameters
using environment variables, set the appropriate environment variables.
1. From the directory where the archive was extracted, execute the integration
binary using the command `./nr-entity-tag-sync` (or
`.\nr-entity-tag-sync.exe` on Windows).
### AWS Lambda Function
To run the New Relic Entity Tag Sync application as an [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html)
function, create an AWS Lambda function using one of the following methods.
* [Use the included deployment scripts](#deploy-using-the-deployment-scripts)
* [Integrate with an existing provisioning process](#deploy-using-an-existing-provisioning-process)
#### Requirements for running the AWS Lambda Function
* The Entity Tag Sync Lambda function requires an execution role that the AWS
Lambda service can assume to run the Lambda function. Either an existing role
can be used or a new role can be created.
When creating a new role, the Entity Tag Sync Lambda function does not need
access to any AWS services other than [CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html)
(to write log events). Attaching the
[AWSLambdaExecute](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AWSLambdaExecute.html)
managed policy or the [AWSLambdaBasicExecutionRole](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AWSLambdaBasicExecutionRole.html)
managed policy or the equivalent thereof, is sufficient.
#### Deploy using the deployment scripts
A [deployment script](./scripts/lambda/deploy.sh) is included that can be run
using the included [`Makefile`](./Makefile) to build the [deployment package](https://docs.aws.amazon.com/lambda/latest/dg/golang-package.html)
and deploy the Entity Tag Sync Lambda function as part of a new
[AWS Stack](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacks.html)
in one step.
[The provided CloudFormation template](./deployments/lambda/cf-template.yaml)
will create a new [AWS Stack](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacks.html)
with four resources.
1. The Entity Tag Sync Lambda function
1. An [EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-what-is.html)
[schedule](https://docs.aws.amazon.com/eventbridge/latest/userguide/using-eventbridge-scheduler.html)
that will invoke the Entity Tag Sync Lambda function every 15 minutes
starting at the beginning of each hour
1. An [EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-what-is.html)
schedule group to contain the schedule
1. An [IAM role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html)
that the [EventBridge scheduler](https://docs.aws.amazon.com/eventbridge/latest/userguide/using-eventbridge-scheduler.html)
can assume to execute the Entity Tag Sync Lambda function
The only _required_ resource in this [AWS Stack](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacks.html)
is the Entity Tag Sync Lambda function. The [EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-what-is.html)
resources and the [IAM role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html)
can be removed from the template if existing resources will be used and/or if
the Entity Tag Sync Lambda function will be invoked as the target of a different
resource.
[The provided CloudFormation template](./deployments/lambda/cf-template.yaml)
leverages [CloudFormation template parameters](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.html)
to customize various properties of the created resources. A [sample parameters file](./deployments/lambda/cf-params.sample.json)
is provided that shows an example of each parameter that can be used with the
CloudFormation template.
**NOTE:** Deploying the Entity Tag Sync Lambda function using the deployment
scripts requires the [AWS CLI](https://aws.amazon.com/cli/) to be installed on
the same system where the repository was cloned. It also requires an S3 bucket
where the [deployment package](https://docs.aws.amazon.com/lambda/latest/dg/golang-package.html)
can be uploaded so that the [CloudFormation template](./deployments/lambda/cf-template.yaml)
can reference it to during deployment. Either an existing bucket can be used or
a new bucket can be created.
To deploy the Entity Tag Sync Lambda function using this method, perform the
following steps.
1. Clone this repository using [`git clone`](https://git-scm.com/docs/git-clone).
1. Navigate to the repository root.
1. To use an existing configuration file, copy it to a file named [`config.yml`](#configuration)
in the `configs` directory. To create a new configuration file, copy the
[`configs/config.sample.yml`](./configs/config.sample.yml) to
`configs/config.yml` and customize it to [configure](#configuration) the
application appropriately for your environment.
1. Copy the [`deployments/lambda/cf-params.sample.json`](./deployments/lambda/cf-params.sample.json)
to `deployments/lambda/cf-params.json`.
1. Use the parameter descriptions in the CloudFormation template at
[`deployments/lambda/cf-template.yaml`](./deployments/lambda/cf-template.yaml)
as a guide to update the template parameters in
`deployments/lambda/cf-params.json`.
1. Ensure that appropriate [authentication and access credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-authentication.html)
are set to allow the AWS CLI to authenticate.
1. Run the command `AWS_S3_BUCKET_NAME=my-s3-bucket make deploy-lambda`, where
`my-s3-bucket` is the same as the value of the `S3BucketName` template
parameter that is set in the `deployments/lambda/cf-params.json` file.
1. Verify that the command ran successfully by looking for the following console
output.
```bash
...
Building lambda zip package...
adding: bootstrap (deflated 51%)
adding: configs/ (stored 0%)
adding: configs/config.yml (deflated 43%)
Uploading lambda zip package...
upload: ../nr-entity-tag-sync.zip to s3://YOUR_BUCKET_NAME/nr-entity-tag-sync.zip
Deploying stack nr-entity-tag-sync...
Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - nr-entity-tag-sync
Done.
```
##### Update or uninstall using the deployment scripts
An [update script](./scripts/lambda/update.sh) is included that can be run using
the included [`Makefile`](./Makefile) to update the Entity Tag Sync Lambda
binary or the [`config.yml`](#configuration) file. This can be useful to update
to the latest version of the Entity Tag Sync application or deploy a new [`config.yml`](#configuration)
file.
To update to the latest version of the Entity Tag Sync application, perform the
following steps.
1. Navigate to the repository root.
1. Update your repository to the latest version using [`git pull`](https://git-scm.com/docs/git-pull),
[`git rebase`](https://git-scm.com/docs/git-rebase), etc.
1. Run the command `AWS_S3_BUCKET_NAME=my-s3-bucket make update-lambda`, where
`my-s3-bucket` is the name of an S3 bucket to upload the
[deployment package](https://docs.aws.amazon.com/lambda/latest/dg/golang-package.html)
to.
1. Verify that the command ran successfully by looking for the following console
output.
```bash
...
Building lambda zip package...
adding: bootstrap (deflated 51%)
adding: configs/ (stored 0%)
adding: configs/config.yml (deflated 43%)
Uploading lambda zip package...
upload: ../nr-entity-tag-sync.zip to s3://YOUR_BUCKET_NAME/nr-entity-tag-sync.zip
Done.
```
To deploy a new [`config.yml`](#configuration), perform the following steps.
1. Navigate to the repository root.
1. Copy the new [`config.yml`](#configuration) file to the `configs`
directory or update the existing [`config.yml`](#configuration) file in the
`configs` directory.
1. Run the command `AWS_S3_BUCKET_NAME=my-s3-bucket make update-lambda`, where
`my-s3-bucket` is the name of an S3 bucket to upload the
[deployment package](https://docs.aws.amazon.com/lambda/latest/dg/golang-package.html)
to.
1. Verify that the command ran successfully by looking for the console output
shown above.
An [uninstall script](./scripts/lambda/delete.sh) is also included that can be
run using the included [`Makefile`](./Makefile) to remove the Entity Tag Sync
[AWS Stack](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacks.html)
that was deployed using the [deploy script](./scripts/lambda/deploy.sh).
To uninstall the Entity Tag Sync [AWS Stack](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacks.html),
perform the following steps.
1. Navigate to the repository root.
1. Run the command `make delete-lambda`.
1. Verify that the command ran successfully by looking for the following console
output.
```bash
...
Deleting stack nr-entity-tag-sync...
Waiting for stack delete to complete...
Done.
```
#### Deploy using an existing provisioning process
To use an existing provisioning process such as [CloudFormation](https://aws.amazon.com/cloudformation/),
[Terraform](https://www.terraform.io/), or [Ansible](https://docs.ansible.com/)
to deploy the Entity Tag Sync Lambda, build the [deployment package](https://docs.aws.amazon.com/lambda/latest/dg/golang-package.html)
and use the appropriate mechanisms of the provisioning tool(s) to create an
[AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html)
function with the properties shown below.
To build the [deployment package](https://docs.aws.amazon.com/lambda/latest/dg/golang-package.html)
perform the following steps.
1. Clone this repository using [`git clone`](https://git-scm.com/docs/git-clone).
1. Navigate to the repository root.
1. To use an existing configuration file, copy it to a file named [`config.yml`](#configuration)
in the `configs` directory. To create a new configuration file, copy the
[`configs/config.sample.yml`](./configs/config.sample.yml) to
`configs/config.yml` and customize it to [configure](#configuration) the
application appropriately for your environment.
1. Run the command `make package-lambda`.
1. Verify that the command ran successfully by looking for the following console
output.
```bash
...
Building lambda zip package...
adding: bootstrap (deflated 51%)
adding: configs/ (stored 0%)
adding: configs/config.yml (deflated 43%)
```
On successful completion of the `make` command, the deployment package will be
located at `dist/nr-entity-tag-sync.zip`. Use this deployment package
along with the following property values to create the AWS Lambda function.
* Lambda package type: `Zip`
* Lambda deployment package: Reference to the location of the generated
deployment package ZIP file
* Lambda entry point/handler name: `bootstrap`
* Lambda runtime identifier: `provided.al2023`
* Lambda environment variables: Specify [configuration](#configuration)
parameters that should be set using environment variables.
#### Run the application as an AWS Lambda Function
The application can be run using any supported [invocation method](https://docs.aws.amazon.com/lambda/latest/dg/lambda-invocation.html).
For example, when deploying the AWS Lambda function using [the provided CloudFormation template](./deployments/lambda/cf-template.yaml),
an [AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-what-is.html)
[schedule](https://docs.aws.amazon.com/eventbridge/latest/userguide/using-eventbridge-scheduler.html)
is also created to invoke the AWS Lambda function on a specified schedule.
## Usage
### Configuration
The Entity Tag Sync application is driven by a YAML configuration file. The
configuration file consists of a set of [general parameters](#general-parameters),
a set of [provider parameters](#provider-parameters), and an array of
[mappings](#mappings). [A sample configuration file](configs/config.sample.yml)
is provided that shows an example of all parameters.
#### General parameters
The following general configuration parameters are supported. Some parameters
can be specified as environment variables as indicated below. Parameters listed
below with dots in their names correspond to nested YAML structures. For example
`log.level` corresponds to the following YAML.
```yaml
log:
level: warn
```
| Name | Environment Variable | Description | Required | Example | Default |
| --- | --- | --- | --- | --- | --- |
| `apiKey` | `NEW_RELIC_API_KEY` | A New Relic User API key | Y | `NRAK-123456` | |
| `licenseKey` | `NEW_RELIC_LICENSE_KEY` | A New Relic Ingest License Key used for the Event API | Y if events enabled | `123456NRAL` | |
| `region` | `NEW_RELIC_REGION` | The New Relic datacenter to access (`US` or `EU`) | N | `US` | `US` |
| `log.level` | | The application log level | N | `debug` | `warn` |
| `log.fileName` | | Log file name | N | `app.log` | Standard output |
| `events.enabled` | | Flag to enable [audit event](#audit-events) | N | `true` | `false` |
| `events.accountId` | `NEW_RELIC_ACCOUNT_ID` | New Relic account where [audit events](#audit-events) are posted | Y if events enabled | `12345` | |
| `events.eventName` | | Name of [audit event](#audit-events) type | N | `MyCustomTagSyncEvent` | `EntityTagSync` |
#### Provider parameters
The `provider` section of the configuration file is used to specify the
parameters for the external entity provider. This section contains common
parameters that are provider indepent and parameters that are specific to the
selected provider. The following common parameters are supported.
| Name | Description | Required | Example | Default |
| --- | --- | --- | --- | --- |
| `type` | The provider implementation to use | Y | `servicenow` | |
| `useLastUpdate` | Flag to enable [delta synchronization](#delta-synchronization) | N | `true` | `false` |
The following values for the `type` parameter are supported.
* [`servicenow`](#servicenow-cmdb-provider-parameters)
##### ServiceNow CMDB provider parameters
The ServiceNow CMDB provider supports the following configuration parmaeters.
| Name | Environment Variable | Description | Required | Example | Default |
| --- | --- | --- | --- | --- | --- |
| `apiUrl` | `NR_CMDB_SNOW_PROVIDER_APIURL` | The ServiceNow ReST API URL | Y | https://my-service-now.service-now.com | |
| `authType` | `NR_CMDB_SNOW_PROVIDER_AUTHTYPE` | The type of authentication to use to authenticate with the ServiceNow instance (`basic` or `oauth`) | Y | `basic` | `basic` |
| `apiUser` | `NR_CMDB_SNOW_PROVIDER_APIUSER` | The ServiceNow username to use when using `basic` authentication | Y if `authType` is `basic` | `admin` | |
| `apiPassword` | `NR_CMDB_SNOW_PROVIDER_APIPASSWORD` | The password to use for the specified ServiceNow username when using `basic` authentication | Y if `authType` is `basic` | `abcd123` | |
| `oauthTokenUrl` | `NR_CMDB_SNOW_PROVIDER_OAUTHTOKENURL` | The token URL to use when using `oauth` authentication | N | `https://myco.apis.com/auth` | `${apiUrl}/oauth_token.do` |
| `oauthGrantType` | `NR_CMDB_SNOW_PROVIDER_GRANTTYPE` | The grant type to use when using `oauth` authentication | N | `client_credentials` | `password` |
| `oauthClientId` | `NR_CMDB_SNOW_PROVIDER_OAUTHCLIENTID` | The client ID to use when using `oauth` authentication | Y if `authType` is `oauth` | `12345` | |
| `oauthClientSecret` | `NR_CMDB_SNOW_PROVIDER_OAUTHCLIENTSECRET` | The client secret to use when using `oauth` authentication | Y if `authType` is `oauth` | `12345` | |
| `oauthClientScopes` | `NR_CMDB_SNOW_PROVIDER_OAUTHCLIENTSCOPES` | The list of OAuth scopes to request when using `oauth` authentication. Separate multiple scopes using whitespace characters. | N | `read_profile` | |
| `pageSize` | `NR_CMDB_SNOW_PROVIDER_PAGESIZE` | A New Relic User API key | N | `10` | `10000` |
#### Mapping parameters
The `mappings` section of the configuration file is used to specify one or more
mapping configurations. Each mapping configuration specifies a set of
configuration parameters that defines the set of criteria for selecting external
entities, the set of criteria for selecting New Relic entities, the criteria
used to match external entities to New Relic entities, and the mapping from
external entity key-values to New Relic entity tags.
##### External entity query criteria
The `extEntityQuery` section of a mapping configuration specifies the query
criteria for selecting a set of external entities. The configuration parameters
in this section are specific to the provider.
###### ServiceNow CMDB entity query criteria
The ServiceNow CMDB provider supports the following configuration parameters for
selecting the set of CIs that are candidates for matching against New Relic
entities.
| Name | Description | Required | Example | Default |
| --- | --- | --- | --- | --- |
| `type` | The ServiceNow CMDB configuration item type name | Y | `cmdb_ci_email_server` | |
| `query` | An [encoded query string](https://docs.servicenow.com/csh?topicname=c_EncodedQueryStrings&version=utah&pubname=utah-platform-user-interface) to use to filter the result using the `sysparm_query` parameter | N | `sys_updated_on>javascript:gs.dateGenerate('{{ .lastUpdateDate }}','{{ .lastUpdateTime }}')^operational_status!=2` | |
| `serverTimezone` | A [location name](https://pkg.go.dev/time#LoadLocation) corresponding to a file in the IANA Time Zone database for the time zone of the local ServiceNow instance | N | `America/New_York` | |
| `urlQueryParams` | Additional URL query parameters to send on the ReST `table` API call specified as a set of key + value pairs | N | (see below) | |
The ServiceNow CMDB query is executed by querying the `table` API using a URL
like the following.
`https://my-service-now.service-now.com/my/api/now/table/CI_TYPE?sysparm_fields=sys_id,FIELD1,...,FIELDN&sysparm_limit=PAGESIZE&sysparm_offset=0`
The value of `CI_TYPE` is the value of the `type` configuration parameter. The
values of `FIELD1,...,FIELDN` are the keys of the `mapping` node as well as the
value of the `extEntityKey` in the `match` node. The value of `PAGESIZE` is the
value of the `pageSize` parameter of the `provider` node.
**Query parameter**
If a `query` is specified in the entity query criteria, the `sysparm_query`
query parameter will be added to the query portion of the URL. The `query`
parameter value will be _automatically_ URL encoded so it should not be
specified in URL encoded format. For example to filter records where the
`active` field is `true` and the `roles` field is `itil`, specify the string
`active=true^roles=itil` and not `active%3Dtrue%5Eroles%3Ditil`.
When using [delta synchronization](#delta-synchronization), the special
character sequences `${lastUpdateDate}` and `${lastUpdateTime}` will be replaced
with the date and time strings, respectively, for the date and time specified by
the last synchronization timestamp. The date and time strings will be in the
format required by the [dateGenerate](https://docs.servicenow.com/bundle/utah-api-reference/page/app-store/dev_portal/API_reference/glideSystemScoped/concept/c_GlideSystemScopedAPI.html#title_r_SGSYS-dateGenerate_S_S)
function and will be converted to strings using the timezone specified in the
`serverTimezone` parameter.
The `query` configuration parameter should not be confused with the
`urlQueryParameters` configuration parameter. The former is used to specify an
[encoded query string](https://docs.servicenow.com/csh?topicname=c_EncodedQueryStrings&version=utah&pubname=utah-platform-user-interface)
used to filter the ServiceNow query result set. The latter is used to specify
additional query parameters to add to the ReST API URL generated to execute the
`table` API query.
**Example**
Consider the following YAML.
```yaml
...
provider:
type: servicenow
pageSize: 50
mappings:
- extEntityQuery:
type: cmdb_ci_email_server
query: 'sys_updated_on>javascript:gs.dateGenerate('${lastUpdateDate}','${lastUpdateTime}')^operational_status!=2'
serverTimezone: America/Los_Angeles
urlQueryParams:
sysparm_display_value: "true"
entityQuery:
...
match:
extEntityKey: name
...
mapping:
sys_class_name: foo
environment: bar
```
Given this YAML, the ServiceNow CMDB provider will access the URL
`https://my-service-now.service-now.com/my/api/now/table/cmdb_ci_email_server?sysparm_fields=sys_id,name,sys_class_name,environment&sysparm_limit=50&sysparm_offset=0&sysparm_display_value=true&sysparm_query=sys_updated_on%3Ejavascript%3Ags.dateGenerate%28%272023-06-01%27%2C%2712%3A00%3A00%27%29%5Eoperational_status%21%3D2`
to retrieve the *display values* for the fields `sys_id`, `name`,
`sys_class_name`, and `environment` for the CI records of type
`cmdb_ci_email_server` that were updated on or after
June 1st, 2023 at 12:00:00 GMT-7 and do not have an `operational_status` of `2`.
##### New Relic entity query criteria
The `entityQuery` section of a mapping configuration specifies the query
criteria for selecting a set of entities in New Relic.
| Name | Description | Required | Example | Default |
| --- | --- | --- | --- | --- |
| `type` | The entity type (`APPLICATION`, `HOST`, etc) | N | `WORKLOAD` | |
| `domain` | The entity domain (`APM`, `BROWSER`, etc) | N | `INFRA` | |
| `name` | The entity name | N | `Billing Service` | |
| `accountId` | The New Relic account ID | N | 123456 | |
| `tags` | A set of tag key + values pairs | N | (see below) | |
| `query` | A raw `entitySearch` query | N | `type IN ('APPLICATION')` | |
The `tags` value is an array of key and values pairs as in the following
example.
```yaml
mappings:
- entityQuery:
...
tags:
foo:
- bar
- baz
...
```
Note that the value part of the pair is an _array_ since New Relic tags can have
multiple values per key.
The query is executed via the Nerdgraph GraphQL API using the `entitySearch`
type of the `actor` type as follows.
- If a value is specified for the `query` key, it will take precedence over the
other values.
- Otherwise, the `entitySearch` query value will be built by `AND`'ing the other
values together. The following examples show the query string that would be
produced for a given `entityQuery` configuration.
**Example 1: YAML**
```yaml
mappings:
- entityQuery:
type: 'HOST'
domain: 'INFRA'
name: 'myinstance'
```
**Example 1: Query**
`type IN ('HOST') AND domain IN ('INFRA') AND name LIKE 'myinstance'`
**Example 2: YAML**
```yaml
mappings:
- entityQuery:
accountId: 12345
tags:
foo:
- bar
beep:
- boop
```
**Example 2: Query**
`tags.accountId = 12345 AND tags.foo IN ('bar') AND tags.beep IN ('boop')`
**Example 3: YAML**
```yaml
mappings:
- entityQuery:
type: 'HOST'
domain: 'INFRA'
query: "name LIKE 'abc123'"
```
**Example 3: Query**
`name LIKE 'abc123'`
##### Match strategy
The `match` section of a mapping configuration is a tuplet that specifies the
name of an external entity key, the name of a New Relic entity attribute/tag,
and an operator used to compare the values of the external entity key to
the New Relic entity attribute/tag.
| Name | Description | Required | Example | Default |
| --- | --- | --- | --- | --- |
| `extEntityKey` | The external entity key to use for comparison | Y | `environment` | |
| `operator` | The type of comparison to use | Y | `equal` | |
| `entityKey` | The New Relic entity attribute/tag to use for comparison | Y | `equal` | |
The `extEntityKey` external entity key will be implicitly added to the list of
keys requested from the provider for each external entity , regardless of whether
the key is referenced in the [mapping](#mapping) section.
The New Relic entity attribute/tag may be any one of the following.
* `name` - the New Relic entity name
* `guid` - the New Relic entity GUID
* `accountId` - the New Relic account ID that the entity belongs to
* Any tag name
The following values for the `operator` attribute are supported.
| Value | Meaning |
| --- | --- |
| `equal` | external entity key value is case-sensitive equivalent to New Relic entity attribute/tag value |
| `equal-ignore-case` | external entity key value is case-insensitive equivalent to New Relic entity attribute/tag value |
| `contains` | New Relic entity attribute/tag value is a case-sensitive sub-string within external entity key value |
| `contains-ignore-case` | New Relic entity attribute/tag value is a case-insensitive sub-string within external entity key value |
| `inverse-contains-ignore-case` | external entity key value is a case-insensitive sub-string within New Relic entity attribute/tag value |
For example, consider the following YAML.
```yaml
match:
extEntityKey: foo
operator: equal-ignore-case
entityKey: bar
```
Given this YAML, New Relic entities will be matched against external entities by
comparing the values of the tag `bar` on the New Relic entities to the values of
the key `foo` on the external entities, case insensitively.
##### Mapping
The `mapping` node of a mapping configuration specifies the mapping from
external entity keys to New Relic entity tags. The keys of the `mapping` node
represent the external entity keys while the values represent the New Relic
entity tag names.
For a given external entity to New Relic entity pair, the value of each of the
external entity keys specified by the keys in the `mapping` node are used as the
values of the New Relic entity tags specified by the values in the `mapping`
node.
For example, consider the following YAML.
```yaml
mapping:
foo: bar
beep: boop
```
Given this YAML, the values of the external entity key `foo` and `beep` of any
external entity that matches a New Relic entity will be set as the values of the
tags `bar` and `boop` on the matching New Relic entity.
#### Full example
This section provides an example configuration and set of entities followed by
a full walk-through of the synchronization process.
##### Sample configuration
```yaml
apiUrl: https://api.newrelic.com
log:
level: warn
provider:
type: servicenow
apiUrl: https://my-service-now.service-now.com
apiUser: admin
mappings:
- extEntityQuery:
type: cmdb_ci_email_server
entityQuery:
type:
- APPLICATION
domain:
- APM
accountId: 1
match:
extEntityKey: name
operator: equal-ignore-case
entityKey: name
mapping:
sys_class_name: SNOW_CI_CLASS
sys_id: SNOW_CMDB_CI
environment: SNOW_ENVIRONMENT
sys_domain.value: SNOW_SYS_DOMAIN
- extEntityQuery:
type: cmdb_ci_app_server
entityQuery:
type:
- HOST
domain:
- INFRA
match:
extEntityKey: name
operator: contains
entityKey: ciMatch
mapping:
sys_class_name: SNOW_CI_CLASS
sys_id: SNOW_CMDB_CI
environment: SNOW_ENVIRONMENT
sys_domain.value: SNOW_SYS_DOMAIN
```
##### ServiceNow CIs
| Type | Name | sys_id | sys_class_name | sys_domain.value | environment |
| --- | --- | --- | --- | --- | --- |
| cmdb_ci_email_server | Microsoft Exchange | abcd123 | cmdb_ci_email_server | global | Production |
| cmdb_ci_app_server | WebSphere Application Server | efgh456 | cmdb_ci_app_server | local | Development |
##### New Relic Entities
| Type | Domain | Name | GUID | tags.ciMatch | tags.SNOW_CI_CLASS | tags.SNOW_CMDB_CI | tags.SNOW_SYS_DOMAIN | tags.SNOW_ENVIRONMENT |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| APPLICATION | APM | microsoft exchange | NR12345 | | cmdb_ci_email_server | abcd123 | | Development |
| HOST | INFRA | WebSphere Application Server | NR45678 | WebSphere | cmdb_ci_app_server | efgh456 | global | |
##### Walkthrough
1. The entity tag sync application starts up and reads in the configuration.
1. A new ServiceNow provider is created with the API base URL
`https://my-service-now.service-now.com` and the API username `admin`. The API
password will be read from the environment variable
`NR_CMDB_SNOW_PROVIDER_APIPASSWORD`. The page size defaults to 10000.
1. The application starts processing the first mapping configuration by
inspecting the `extEntityQuery` node.
1. Using the `type` parameter specified in the `extEntityQuery` node as the
table name to query and using the keys of the `mapping` node as well as the
value of the `extEntityKey` of the `match` node as the CI fields to return,
the ServiceNow CMDB provider will make an HTTP `GET` request for the
following URL.
`https://my-service-now.service-now.com/my/api/now/table/cmdb_ci_email_server?sysparm_fields=sys_id,name,sys_class_name,environment,sys_domain&sysparm_limit=10000&sysparm_offset=0`
This API call will return the requested key-value pairs for all CIs of type
`cmdb_ci_email_server`, including the CI listed above with `sys_id`
`abcd123`.
1. The application next uses the `entityQuery` node to request to make the
following GraphQL query against the New Relic GraphQL API.
```graphql
{
actor {
entitySearch(query: "type IN ('APPLICATION') AND domain IN ('APM') AND tags.accountId` = 1") {
count
results {
entities {
guid
name
accountId
domain
type
alertSeverity
permalink
reporting
tags {
key
values
}
}
}
}
}
}
```
This query will return all APM service entities in account ID 1, including
the APM application listed above with GUID `NR12345`.
1. Next, the application will iterate over the entities returned from the
GraphQL call and, per the `match` node values, will perform a
case-insensitive equals check between each CI name and the current entity
name. In this case, the application will find a match between the CI with
`sys_id` `abcd123` and the New Relic entity with GUID `NR12345` because the
values `Microsoft Exchange` and `microsoft exchange` are case-insensitively
equivalent.
1. The application will update the tags on entity `NR12345` using the values of
the key-value pairs from CI `abcd123` as follows.
| CI Key | CI Value | Entity Tag | Entity `NR12345` before | Entity `NR12345` after|
| --- | --- | --- | --- | --- |
| sys_class_name | cmdb_ci_email_server | SNOW_CI_CLASS | cmdb_ci_email_server | cmdb_ci_email_server |
| sys_id | abcd123 | SNOW_CMDB_CI | abcd123 | abcd123 |
| sys_domain.value | global | SNOW_SYS_DOMAIN | | global |
| environment | Production | SNOW_ENVIRONMENT | Development | Production |
1. The application starts processing the second mapping configuration by
inspecting the `extEntityQuery` node.
1. Using the specified `type` parameter and the keys of the `mapping` node as
well as the value of the `extEntityKey` of the `match` node specified by the
configuration, the ServiceNow CMDB provider will make an HTTP `GET` request
for the following URL.
`https://my-service-now.service-now.com/my/api/now/table/cmdb_ci_app_server?sysparm_fields=sys_id,name,sys_class_name,environment,sys_domain&sysparm_limit=10000&sysparm_offset=0`
This API call will return the requested key-value pairs for all CIs of type
`cmdb_ci_app_server`, including the CI listed above with `sys_id`
`efgh456`.
1. The application next uses the `entityQuery` node to request to make the
following GraphQL query against the New Relic GraphQL API.
```graphql
{
actor {
entitySearch(query: "type IN ('HOST') AND domain IN ('INFRA')") {
count
results {
entities {
guid
name
accountId
domain
type
alertSeverity
permalink
reporting
tags {
key
values
}
}
}
}
}
}
```
This query will return all infrastructure host entities accessible by the
given license key, including the infrastructure host listed above with GUID
`NR45678`.
1. Next, the application will iterate over the entities returned from the
GraphQL call and, per the `match` node values, will perform a
case-insensitive "contains" check between each CI name and the _first_ value
of the `ciMatch` tag of each entity. In this case, the application will find
a match between the CI with `sys_id` `efgh456` and the New Relic entity with
GUID `NR45678` because the value `WebSphere Application Server` includes
the value `WebSphere`.
1. The application will update the tags on entity `NR45678` using the values of
the key-value pairs from CI `efgh456` as follows.
| CI Key | CI Value | Entity Tag | Entity `NR45678` before | Entity `NR45678` after|
| --- | --- | --- | --- | --- |
| sys_class_name | cmdb_ci_app_server | SNOW_CI_CLASS | cmdb_ci_email_server | cmdb_ci_email_server |
| sys_id | efgh456 | SNOW_CMDB_CI | efgh456 | efgh456 |
| sys_domain.value | local | SNOW_SYS_DOMAIN | global | local |
| environment | Production | SNOW_ENVIRONMENT | | Development |
## Building
### Coding Conventions
#### Style Guidelines
While not strictly enforced, the basic preferred editor settings are set in the
[.editorconfig](./.editorconfig). Other than this, no style guidelines are
currently imposed.
#### Static Analysis
This project uses both [`go vet`](https://pkg.go.dev/cmd/vet) and
[`staticcheck`](https://staticcheck.io/) to perform static code analysis. These
checks are run via [`precommit`](https://pre-commit.com) on all commits. Though
this can be bypassed on local commit, both tasks are also run during
[the `validate` workflow](./.github/workflows/validate.yml) and must have no
errors in order to be merged.
#### Commit Messages
Commit messages must follow [the conventional commit format](https://www.conventionalcommits.org/en/v1.0.0/).
Again, while this can be bypassed on local commit, it is strictly enforced in
[the `validate` workflow](./.github/workflows/validate.yml).
The basic commit message structure is as follows.
```
[optional scope][!]:
[optional body]
[optional footer(s)]
```
In addition to providing consistency, the commit message is used by
[svu](https://github.com/caarlos0/svu) during
[the release workflow](./.github/workflows/release.yml). The presence and values
of certain elements within the commit message affect auto-versioning. For
example, the `feat` type will bump the minor version. Therefore, it is important
to use the guidelines below and carefully consider the content of the commit
message.
Please use one of the types below.
- `feat` (bumps minor version)
- `fix` (bumps patch version)
- `chore`
- `build`
- `docs`
- `test`
Please use one of the scopes below.
- `sync` - work related to the synchronization process
- `config` - work related to the config system
- `provider[-optional-name]` - work related to the provider framework or a
specific provider
- `lambda` - work related to the lambda code and/or deployment
- `ci` - work related to continuous integration (GitHub workflow/actions)
- `release` - work related to creating a new release
Any type/scope can be followed by the `!` character to indicate a breaking
change. Additionally, any commit that has the text `BREAKING CHANGE:` in the
footer will indicate a breaking change.
### Local Development
For local development, simply use `go build` and `go run`. For example,
```bash
go build cmd/nr-entity-tag-sync/nr-entity-tag-sync.go
```
Or
```bash
go run cmd/nr-entity-tag-sync/nr-entity-tag-sync.go
```
If you prefer, you can also use [`goreleaser`](https://goreleaser.com/) with
the `--single-target` option to build the binary for the local `GOOS` and
`GOARCH` only.
```bash
goreleaser build --single-target
```
### Releases
Releases are built and packaged using [`goreleaser`](https://goreleaser.com/).
By default, a new release will be built automatically on any push to the `main`
branch. For more details, review the [`.goreleaser.yaml`](./.goreleaser.yaml)
and [the `goreleaser` documentation](https://goreleaser.com/intro/).
The [svu](https://github.com/caarlos0/svu) utility is used to generate the next
tag value [based on commit messages](https://github.com/caarlos0/svu#commit-messages-vs-what-they-do).
### GitHub Workflows
This project utilizes GitHub workflows to perform actions in response to
certain GitHub events.
| Workflow | Events | Description
| --- | --- | --- |
| [validate](./.github/workflows/validate.yml) | `push` | Runs [precommit](https://pre-commit.com) to perform static analysis and runs [commitlint](https://commitlint.js.org/#/) to validate the last commit message |
| [build](./.github/workflows/build.yml) | `pull_request` | Builds and tests code |
| [release](./.github/workflows/release.yml) | `push` to `main` branch | Generates a new tag using [svu](https://github.com/caarlos0/svu) and runs [`goreleaser`](https://goreleaser.com/) |
| [repolinter](./.github/workflows/repolinter.yml) | `pull_request` | Enforces repository content guidelines |
## Testing
TBD
## Support
New Relic has open-sourced this project. This project is provided AS-IS WITHOUT
WARRANTY OR DEDICATED SUPPORT. Issues and contributions should be reported to
the project here on GitHub.
We encourage you to bring your experiences and questions to the
[Explorers Hub](https://discuss.newrelic.com/) where our community members
collaborate on solutions and new ideas.
### Privacy
At New Relic we take your privacy and the security of your information
seriously, and are committed to protecting your information. We must emphasize
the importance of not sharing personal data in public forums, and ask all users
to scrub logs and diagnostic information for sensitive information, whether
personal, proprietary, or otherwise.
We define “Personal Data” as any information relating to an identified or
identifiable individual, including, for example, your name, phone number, post
code or zip code, Device ID, IP address, and email address.
For more information, review [New Relic’s General Data Privacy Notice](https://newrelic.com/termsandconditions/privacy).
### Contribute
We encourage your contributions to improve this project! Keep in mind that
when you submit your pull request, you'll need to sign the CLA via the
click-through using CLA-Assistant. You only have to sign the CLA one time per
project.
If you have any questions, or to execute our corporate CLA (which is required
if your contribution is on behalf of a company), drop us an email at
opensource@newrelic.com.
**A note about vulnerabilities**
As noted in our [security policy](../../security/policy), New Relic is committed
to the privacy and security of our customers and their data. We believe that
providing coordinated disclosure by security researchers and engaging with the
security community are important means to achieve our security goals.
If you believe you have found a security vulnerability in this project or any of
New Relic's products or websites, we welcome and greatly appreciate you
reporting it to New Relic through [HackerOne](https://hackerone.com/newrelic).
If you would like to contribute to this project, review [these guidelines](./CONTRIBUTING.md).
To all contributors, we thank you! Without your contribution, this project
would not be what it is today.
### License
The [New Relic Entity Tag Sync] project is licensed under the
[Apache 2.0](http://apache.org/licenses/LICENSE-2.0.txt) License.