https://github.com/scality/utapi
API for tracking resource utilization and reporting metrics
https://github.com/scality/utapi
Last synced: 2 months ago
JSON representation
API for tracking resource utilization and reporting metrics
- Host: GitHub
- URL: https://github.com/scality/utapi
- Owner: scality
- License: apache-2.0
- Created: 2016-08-09T18:02:14.000Z (almost 10 years ago)
- Default Branch: development/8.2
- Last Pushed: 2026-04-15T17:37:26.000Z (2 months ago)
- Last Synced: 2026-04-15T19:28:53.609Z (2 months ago)
- Language: JavaScript
- Size: 2.5 MB
- Stars: 14
- Watchers: 56
- Forks: 3
- Open Issues: 13
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# utapi
[![Circle CI][badgepub]](https://circleci.com/gh/scality/utapi)
Service Utilization API for tracking resource usage and metrics reporting.
## Design
Please refer to the [design](/DESIGN.md) for more information.
## Server
To run the server:
```
npm start
```
## Client
The module exposes a client, named UtapiClient. Projects can use this client to
push metrics directly to the underlying datastore (Redis) without the need of an
extra HTTP request to Utapi.
```js
const { UtapiClient } = require('utapi');
const config = {
redis: {
host: '127.0.0.1',
port: 6379
},
localCache: {
host: '127.0.0.1',
port: 6379
}
}
const c = new UtapiClient(config);
// The second argument to `pushMetric` is a hexadecimal string Request Unique
// Identifier used for logging.
c.pushMetric('createBucket', '3d534b1511e5630e68f0', { bucket: 'demo' });
c.pushMetric('putObject', '3d534b1511e5630e68f0', {
bucket: 'demo',
newByteLength: 1024,
oldByteLength: null,
});
c.pushMetric('putObject', '3d534b1511e5630e68f0', {
bucket: 'demo',
newByteLength: 1024,
oldByteLength: 256,
});
c.pushMetric('multiObjectDelete', '3d534b1511e5630e68f0', {
bucket: 'demo',
byteLength: 1024,
numberOfObjects: 999,
});
```
If an error occurs during a `pushMetric` call and the client is unable to record
metrics in the underlying datastore, metric data is instead stored in a local
Redis cache. Utapi attempts to push these cached metrics (every five minutes, by
default) using a component named UtapiReplay. If the `pushMetric` call initiated
by UtapiReplay fails, the metric is reinserted into the local Redis cache. The
particularities of this behavior are configurable. For further information, see
[design](/DESIGN.md).
## Listing Metrics with Utapi
To make a successful request to Utapi you would need
1. [IAM user with a policy giving access to Utapi](#iam-user-with-a-policy-giving-access-to-utapi)
2. [Sign request with Auth V4](#signing-request-with-auth-v4)
### IAM user with a policy giving access to Utapi
Note: The examples here use AWS CLI but any AWS SDK is capable of these actions.
**endpoint-url:** This would be `https://:` where your Identity(IAM)
Server is running.
1. Create an IAM user
```
aws iam --endpoint-url create-user --user-name
```
2. Create access key for the user
```
aws iam --endpoint-url create-access-key --user-name
```
3. Define a managed IAM policy
sample utapi policy
```json
cat - > utapipolicy.json < create-policy --policy-name utapipolicy \
--policy-document file://utapipolicy.json
```
A sample output of the above command would look like
```json
{
"Policy": {
"PolicyName": "utapipolicy",
"CreateDate": "2017-06-01T19:31:18.620Z",
"AttachmentCount": 0,
"IsAttachable": true,
"PolicyId": "ZXR6A36LTYANPAI7NJ5UV",
"DefaultVersionId": "v1",
"Path": "/",
"Arn": "arn:aws:iam::0123456789012:policy/utapipolicy",
"UpdateDate": "2017-06-01T19:31:18.620Z"
}
}
```
The arn property of the response, which we call ``, will be used
in the next step to attach the policy to the user.
5. Attach user to the managed policy
```
aws --endpoint-url iam attach-user-policy --user-name
--policy-arn
```
Now the user has access to ListMetrics request in Utapi on all buckets.
### Signing request with Auth V4
There are two options here.
You can generate V4 signature using AWS SDKs or the node module aws4. See the
following urls for reference.
* http://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html
* http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html
* https://github.com/mhart/aws4
You may also view examples making a request with Auth V4 using various languages
and AWS SDKs [here](/examples).
Alternatively, you can use a nifty command line tool available in Scality's
CloudServer.
You can git clone the CloudServer repo from here
https://github.com/scality/cloudserver and follow the instructions in the README
to install the dependencies.
If you have CloudServer running inside a docker container you can docker exec
into the CloudServer container as
```
docker exec -it bash
```
and then run the command
```
node bin/list_metrics
```
It will generate the following output listing available options.
```
Usage: list_metrics [options]
Options:
-h, --help output usage information
-V, --version output the version number
-a, --access-key Access key id
-k, --secret-key Secret access key
-m, --metric Metric type
--buckets Name of bucket(s) with a comma separator if
more than one
--accounts Account ID(s) with a comma separator if more
than one
--users User ID(s) with a comma separator if more than
one
--service Name of service
-s, --start Start of time range
-r, --recent List metrics including the previous and
current 15 minute interval
-e --end End of time range
-h, --host Host of the server
-p, --port Port of the server
--ssl Enable ssl
-v, --verbose
```
An example call to list metrics for a bucket `demo` to Utapi in a https enabled
deployment would be
```
node bin/list_metrics --metric buckets --buckets demo --start 1476231300000
--end 1476233099999 -a myAccessKey -k mySecretKey -h 127.0.0.1 -p 8100 --ssl
```
Both start and end times are time expressed as UNIX epoch timestamps **expressed
in milliseconds**.
Keep in mind, since Utapi metrics are normalized to the nearest 15 min.
interval, start time and end time need to be in the specific format as follows.
#### Start time
Start time needs to be normalized to the nearest 15 minute interval with seconds
and milliseconds set to 0. So valid start timestamps would look something like
`09:00:00:000`, `09:15:00:000`, `09:30:00:000` and `09:45:00:000`.
For example
Date: Tue Oct 11 2016 17:35:25 GMT-0700 (PDT)
Unix timestamp (milliseconds): 1476232525320
Here's an example JS method to get a start timestamp
```javascript
function getStartTimestamp(t) {
const time = new Date(t);
const minutes = time.getMinutes();
const timestamp = time.setMinutes((minutes - minutes % 15), 0, 0);
return timestamp;
}
```
This would format the start time timestamp to `1476231300000`
#### End time
End time needs to be normalized to the nearest 15 minute end interval with
seconds and milliseconds set to 59 and 999 respectively. So valid end timestamps
would look something like `09:14:59:999`, `09:29:59:999`, `09:44:59:999` and
`09:59:59:999`.
Here's an example JS method to get an end timestamp
```javascript
function getEndTimestamp(t) {
const time = new Date(t);
const minutes = time.getMinutes();
const timestamp = time.setMinutes((minutes - minutes % 15) + 15, 0, -1);
return timestamp;
}
```
This would format the end time timestamp to `1476233099999`
## Guidelines
Please read our coding and workflow guidelines at
[scality/Guidelines](https://github.com/scality/Guidelines).
### Contributing
In order to contribute, please follow the
[Contributing Guidelines](
https://github.com/scality/Guidelines/blob/master/CONTRIBUTING.md).
[badgepub]: http://circleci.com/gh/scality/utapi.svg?style=svg