https://github.com/mongodb/ftdc-tools
A FTDC library in python that can decode and stream FTDC data.
https://github.com/mongodb/ftdc-tools
Last synced: 3 months ago
JSON representation
A FTDC library in python that can decode and stream FTDC data.
- Host: GitHub
- URL: https://github.com/mongodb/ftdc-tools
- Owner: mongodb
- Created: 2022-04-21T19:47:53.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2023-10-24T19:46:24.000Z (over 1 year ago)
- Last Synced: 2025-01-29T09:43:41.734Z (3 months ago)
- Language: Python
- Size: 9.3 MB
- Stars: 5
- Watchers: 12
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
# ftdc-tools - Pure Python package for FTDC
This package provides tools required to decode and process FTDC data. So whats unique about this package?
* This a pure python package that relies on pymongo for decoding FTDC data.
* This package provides streaming support. With this feature the whole FTDC files is not loaded in memory and is capable of processing large FTDC files.
## Getting Started - Users
```
pip install ftdc-tools
```
## Usage
### Warning
This package is still in beta. In particular, all bson values are not guaranteed to be supported. This package is used to process Genny/Poplar client-side FTDC files and hasn't been tested with mongo server FTDC.
### Decoding FTDC file from URL - Streaming approach
```
from ftdc_tools.decode import decode_file_iter
import requests
url = "https://genny-metrics.s3.amazonaws.com/performance_linux_wt_repl_genny_scale_InsertRemove_patch_b2098c676bdc64e3194734fa632b133c47496646_61f955933066150fca890e4a_22_02_01_15_58_36_0/canary_InsertRemove.ActorFinished"
# Streaming FTDC data
response = requests.get(url, stream=True)
for ftdc_doc in decode_file_iter(response.raw):
print(ftdc_doc)
```
### Decoding FTDC file from URL - Non-streaming approach
```
from ftdc_tools.decode import decode_iter
import requests
url = "https://genny-metrics.s3.amazonaws.com/performance_linux_wt_repl_genny_scale_InsertRemove_patch_b2098c676bdc64e3194734fa632b133c47496646_61f955933066150fca890e4a_22_02_01_15_58_36_0/canary_InsertRemove.ActorFinished"
response = requests.get(url)
for ftdc_doc in decode_iter(response.content):
print(ftdc_doc)
```
### Rolling up a FTDC file
Rolling up a FTDC file creates a set of summary statistics based on a certain expected schema.
The ClientPerformanceStatistics rollup expects the Genny/Poplar schema described [here](https://github.com/10gen/performance-tooling-docs/blob/main/getting_started/intrarun_data_generation.md#client-side-intra-run-data).
```
from ftdc_tools.decode import decode_file_iter
from ftdc_tools.rollup.client_perf import ClientPerformanceStatistics
import requests
url = "https://genny-metrics.s3.amazonaws.com/performance_linux_wt_repl_genny_scale_InsertRemove_patch_b2098c676bdc64e3194734fa632b133c47496646_61f955933066150fca890e4a_22_02_01_15_58_36_0/canary_InsertRemove.ActorFinished"
# Streaming FTDC data
response = requests.get(url, stream=True)
roll = ClientPerformanceStatistics()
for ftdc_doc in decode_file_iter(response.raw):
roll.add_doc(ftdc_doc)
print("Statistics: ", roll.all_statistics)
```
## Getting Started - Developers
Getting the code:
```
$ git clone [email protected]:mongodb/ftdc-tools.git
$ cd ftdc-tools
```
Installation
```
$ pip install poetry
$ poetry install
```
Testing/linting:
```
$ poetry run pytest
```
### Formatting/Lint Fixing
```
poetry run black src tests
poetry run isort src tests
poetry run autoflake --in-place --remove-all-unused-imports -r src tests
```
### Opening a PR
You MUST bump the package version in pyproject.toml before requesting a review on your PR. Merging this PR will push the package to pypi. We use semver for package versioning.
Read more about semver here: https://semver.org/