https://github.com/gmasse/universal-tsdb
Universal Time-Series Database Python Client (InfluxDB, Warp10, ...)
https://github.com/gmasse/universal-tsdb
influxdb python time-series timeseries-database warp10
Last synced: 10 months ago
JSON representation
Universal Time-Series Database Python Client (InfluxDB, Warp10, ...)
- Host: GitHub
- URL: https://github.com/gmasse/universal-tsdb
- Owner: gmasse
- License: gpl-3.0
- Created: 2020-04-04T16:52:52.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2020-04-13T13:21:21.000Z (about 6 years ago)
- Last Synced: 2025-07-09T19:36:17.655Z (11 months ago)
- Topics: influxdb, python, time-series, timeseries-database, warp10
- Language: Python
- Size: 35.2 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# universal-tsdb
A Universal Time-Series Database Python Client (InfluxDB, Warp10, ...)
## Introduction
This project aims to abstract your Time-Series backend, keeping your code as agnostic as possible.
Some examples:
- proof of concept
- early stages of development (when you are not sure which plateform you should use)
- ETL (Extract-Transform-Load), for the load step
:warning: The current code only offer INGESTING functions (writing points to a backend).
## Quickstart
### Installation
```bash
$ pip install universal-tsdb
```
```python
>>> from universal_tsdb import Client, Ingester
>>> backend = Client('influx', 'http://localhost:8086', database='test')
>>> series = Ingester(backend)
>>> series.append(1585934895000, measurement='data', field1=42.0)
>>> series.payload()
'data field1=42.0 1585934895000000000\n'
>>> series.commit()
```
### InfluxDB
```python
from universal_tsdb import Client, Ingester
backend = Client('influx', 'http://localhost:8086', database='metrics',
backend_username='user', backend_password='passwd')
series = Ingester(backend)
series.append(1585934895000, measurement='mes', field1=42.0)
series.append(1585934896000, measurement='mes', tags={'tag1':'value1'}, field1=43.4, field2='value')
series.commit()
```
The code above will generate a data payload based on InfluxDB line protocol
and send it via a HTTP(S) request.
```
POST /write?db=metrics&u=user&p=passwd HTTP/1.1
Host: localhost:8086
mes field1=42.0 1585934895000000000
mes,tag1=value1 field1=43.4 field2="value" 1585934896000000000
```
### Warp10
```python
from universal_tsdb import Client, Ingester
backend = Client('warp10', 'http://localhost/api/v0', token='WRITING_TOKEN_ABCDEF0123456789')
series = Ingester(backend)
series.append(1585934895000, field1=42.0)
series.append(1585934896000, tags={'tag1':'value1'}, field1=43.4, field2='value')
series.commit()
```
The code above will generate a data payload based on Warp10 GTS format
and send it via a HTTP(S) request.
```
POST /api/v0/update HTTP/1.1
Host: localhost
X-Warp10-Token: WRITING_TOKEN_ABCDEF0123456789
1585934895000000// field1{} 42.0
1585934896000000// field1{tag1=value1} 42.0
1585934896000000// field2{tag1=value1} 'value'
```
## Advanced Usage
### Batch processing
When you have a large volume of data to send, you may want to split in several HTTP requests.
In 'batch'-mode, the library commit (send) the data automatically:
```python
backend = Client('influx', 'http://localhost:8086', database='metrics')
series = Ingester(backend, batch=10)
for i in range(0..26):
series.append(field=i)
series.commit() # final commit to save the last 6 values
```
```
Commit#1 Sent 10 new series (total: 10) in 0.02 s @ 2000.0 series/s (total execution: 0.13 s)
Commit#2 Sent 10 new series (total: 20) in 0.02 s @ 2000.0 series/s (total execution: 0.15 s)
Commit#3 Sent 6 new series (total: 26) in 0.01 s @ 2000.0 series/s (total execution: 0.17 s)
REPORT: 3 commits (3 successes), 26 series, 26 values in 0.17 s @ 2000.0 values/s",
```
### Omitting Timestamp
If you omit timestamp, the library uses the function `time.time()`
to generate a UTC Epoch Time. Precision is system dependent.
### Measurement in Warp10
InfluxDB measurement does not exist in Warp10.
The library emulates measurement by prefixing the Warp10 classname:
```python
backend = Client('warp10', token='WRITING_TOKEN_ABCDEF0123456789')
series = Ingester(backend)
series.append(1585934895000, measurement='mes', field1=42.0)
series.commit()
```
```
1585934896000000// mes.field1{} 42.0
```
## Todo
- [ ] API documentation
- [ ] Examples
- [ ] Data query/fetch functions
- [ ] Refactoring of backend specific code (inherited classes?)
- [ ] Time-Series Line protocol optimization
- [ ] Gzip/deflate HTTP compression
- [ ] Code coverage / additional tests