https://github.com/metrico/influxdb3-community

Community InfluxDB 3.0 "IOx" static builds + containers + Examples for Developers & Integrators. Experiment with low-cost storage, unlimited cardinality and FlightSQL APIs
https://github.com/metrico/influxdb3-community

arrow datafusion flightsql flux influx influxdb influxdb3 iox lineprotocol musl rust

Last synced: 2 months ago
JSON representation

Community InfluxDB 3.0 "IOx" static builds + containers + Examples for Developers & Integrators. Experiment with low-cost storage, unlimited cardinality and FlightSQL APIs

Host: GitHub
URL: https://github.com/metrico/influxdb3-community
Owner: metrico
Created: 2023-05-20T20:58:30.000Z (about 2 years ago)
Default Branch: main
Last Pushed: 2025-03-14T21:12:08.000Z (3 months ago)
Last Synced: 2025-03-26T12:51:09.498Z (3 months ago)
Topics: arrow, datafusion, flightsql, flux, influx, influxdb, influxdb3, iox, lineprotocol, musl, rust
Language: Shell
Homepage: https://metrico.in
Size: 223 KB
Stars: 44
Watchers: 8
Forks: 2
Open Issues: 0
Metadata Files:
- Readme: README.md

Awesome Lists containing this project

README

![image](https://github.com/metrico/iox-community/assets/1423657/52ff076a-4261-48f3-ad43-e3183e0645cd)

# InfluxDB 3.x "IOx" Community
Community Builds and Containers for InfluxDB 3.0 IOx _(eye-ox)_ aka _Edge_

### Motivation
😄 You want to try and experiment with IOx low-cost storage, unlimited cardinality and flight sql

~🥵‍ The IOx project is in _"Cloud Only"_ mode and InfluxDB is not producing builds yet...~

😄 No problem! Meet the _unofficial_ InfluxDB 3.0 _"IOx"_ musl + docker builder for early adopters

#### Nightly Builds
###### amd64/musl
- [x] [docker](https://github.com/metrico/iox-community/pkgs/container/influxdb-edge-musl): `docker pull ghcr.io/metrico/influxdb-edge-musl:latest`
- [x] [binary](https://github.com/metrico/iox-community/releases): `github.com/metrico/iox-community/releases/latest/download/influxdb3`

## Get Started

This guide uses Docker and docker-compose. You can run locally using a [static build](https://github.com/metrico/iox-community/releases).

#### Static
```bash
curl -fsSL github.com/metrico/iox-builder/releases/latest/download/influxdb3 -O \
&& chmod +x influxdb3
```

#### Docker

The default compose uses local filesystem. Use the full recipe for Object Storage and Postgres Catalog.

```
docker-compose up -f docker-compose.yml -d
```

Your local IOx endpoint should be ready on port `8086`

#### Health

Check the instance health:
```
curl http://127.0.0.1:8086/health
```

The expected response is `OK`

IOx Settings

Deploy IOx using different settings

```
INFLUXDB3_MAX_HTTP_REQUEST_SIZE: "10485760"
INFLUXDB3_GEN1_DURATION: "10m"
INFLUXDB3_WAL_FLUSH_INTERVAL: "1s"
INFLUXDB3_WAL_SNAPSHOT_SIZE: "600"
INFLUXDB3_NUM_WAL_FILES_TO_KEEP: "50"
INFLUXDB3_WAL_MAX_WRITE_BUFFER_SIZE: "100000"
INFLUXDB3_BUFFER_MEM_LIMIT_MB: "5000"
INFLUXDB3_PARQUET_MEM_CACHE_SIZE_MB: "1000"
INFLUXDB3_FORCE_SNAPSHOT_MEM_THRESHOLD: "70%"
INFLUXDB3_TELEMETRY_DISABLE_UPLOAD: true
INFLUXDB3_NODE_IDENTIFIER_PREFIX: "bucket-id"
INFLUXDB3_OBJECT_STORE: "file"
INFLUXDB3_DB_DIR: "/data"
```

As write requests come in to the server, they are parsed, validated, and put into an in-memory WAL buffer. This buffer is flushed every second by default (can be changed through configuration), which will create a WAL file. Once the data is flushed to disk, it is put into a queryable in-memory buffer and then a response is sent back to the client that the write was successful. That data will now show up in queries to the server.

InfluxDB periodically snapshots the WAL to persist the oldest data in the queryable buffer, allowing the server to remove old WAL files. By default, the server will keep up to 900 WAL files buffered up (15 minutes of data) and attempt to persist the oldest 10 minutes, keeping the most recent 5 minutes around.

When the data is persisted out of the queryable buffer it is put into the configured object store as Parquet files. Those files are also put into an in-memory cache so that queries against the most recently persisted data do not have to go to object storage.

Each server needs an identifier for writing to object storage and as an identifier that is added to replicated writes, Write Buffer segments and Chunks. Must be unique in a group of connected or semi-connected IOx servers. Must be a number that can be represented by a 32-bit unsigned integer.

```
INFLUXDB3_NODE_IDENTIFIER_PREFIX: 1
```

The demo catalog uses *sqlite* by default. To enable persistent catalog using *postgres*, use the following:

```
- INFLUXDB_IOX_CATALOG_DSN=postgres://postgres@localhost:5432/postgres
```

The demo stores to filesystem. To enable S3/R2/Minio object storage use the following parameters:

```
- INFLUXDB3_OBJECT_STORE=s3
- AWS_ACCESS_KEY_ID=access_key_value
- AWS_SECRET_ACCESS_KEY=secret_access_key_value
- AWS_DEFAULT_REGION=us-east-2
- INFLUXDB3_BUCKET=bucket-name
- AWS_ENDPOINT = http://minio:9000
```

For other storage options refer to [env example](https://github.com/metrico/iox-builder/blob/main/env.example)

### API Proxy

To emulate InfluxDB3.0 Cloud works, an nginx proxy is included to serve all IOx services from a single endpoint.
```
events {}
http {
server {
listen 8086;
http2 on;
location /api {
proxy_pass_request_headers on;
proxy_pass http://iox:8080;
}
location /health {
proxy_pass http://iox:8080;
}
location / {
proxy_pass_request_headers on;
if ($http_content_type = "application/grpc") {
grpc_pass iox:8082;
}
proxy_pass http://iox:8082;
}
}
}
```

IOX Insert & Query

Validate your IOx Setup

Let's start testing and using your brand new IOx instance!

### Line Protocol Examples

Our goal is observability formats ingestion into IOx. Here are some scope examples:

#### Metrics
```
avalanche_metric_mmmmm_0_71 cycle_id="0",gauge=29,host.name="generate-metrics-avalanche",label_key_kkkkk_0="label_val_vvvvv_0",label_key_kkkkk_1="label_val_vvvvv_1",label_key_kkkkk_2="label_val_vvvvv_2",label_key_kkkkk_3="label_val_vvvvv_3",label_key_kkkkk_4="label_val_vvvvv_4",label_key_kkkkk_5="label_val_vvvvv_5",label_key_kkkkk_6="label_val_vvvvv_6",label_key_kkkkk_7="label_val_vvvvv_7",label_key_kkkkk_8="label_val_vvvvv_8",label_key_kkkkk_9="label_val_vvvvv_9",port="9090",scheme="http",series_id="3",service.name="otel-collector" 1613772311130000000
avalanche_metric_mmmmm_0_71 cycle_id="0",gauge=16,host.name="generate-metrics-avalanche",label_key_kkkkk_0="label_val_vvvvv_0",label_key_kkkkk_1="label_val_vvvvv_1",label_key_kkkkk_2="label_val_vvvvv_2",label_key_kkkkk_3="label_val_vvvvv_3",label_key_kkkkk_4="label_val_vvvvv_4",label_key_kkkkk_5="label_val_vvvvv_5",label_key_kkkkk_6="label_val_vvvvv_6",label_key_kkkkk_7="label_val_vvvvv_7",label_key_kkkkk_8="label_val_vvvvv_8",label_key_kkkkk_9="label_val_vvvvv_9",port="9090",scheme="http",series_id="4",service.name="otel-collector" 1613772311130000000
```

##### Insert
```bash
wget -qO- "https://raw.githubusercontent.com/metrico/influxdb_iox/main/test_fixtures/lineproto/metrics.lp" | \
curl -v "http://127.0.0.1:8181/api/v2/write?org=company&bucket=sensors" --data-binary @-
```

#### Logs
```
syslog,appname=myapp,facility=console,host=myhost,hostname=myhost,severity=warning facility_code=14i,message="warning message here",severity_code=4i,procid="12345",timestamp=1613769568895331700,version=1
```

##### Insert
```bash
echo 'syslog,appname=myapp,facility=console,host=myhost,hostname=myhost,severity=warning facility_code=14i,message="warning message here",severity_code=4i,procid="12345",timestamp=1434055562000000000,version=1' | \
curl -v "http://127.0.0.1:8181/api/v2/write?org=company&bucket=logs" --data-binary @-
```

#### Traces
```
spans end_time_unix_nano="2021-02-19 20:50:25.6893952 +0000 UTC",instrumentation_library_name="tracegen",kind="SPAN_KIND_INTERNAL",name="okey-dokey",net.peer.ip="1.2.3.4",parent_span_id="d5270e78d85f570f",peer.service="tracegen-client",service.name="tracegen",span.kind="server",span_id="4c28227be6a010e1",status_code="STATUS_CODE_OK",trace_id="7d4854815225332c9834e6dbf85b9380" 1613767825689169000
spans end_time_unix_nano="2021-02-19 20:50:25.6893952 +0000 UTC",instrumentation_library_name="tracegen",kind="SPAN_KIND_INTERNAL",name="lets-go",net.peer.ip="1.2.3.4",peer.service="tracegen-server",service.name="tracegen",span.kind="client",span_id="d5270e78d85f570f",status_code="STATUS_CODE_OK",trace_id="7d4854815225332c9834e6dbf85b9380" 1613767825689135000
```

##### Insert
```bash
echo 'spans end_time_unix_nano="2025-01-26 20:50:25.6893952 +0000 UTC",instrumentation_library_name="tracegen",kind="SPAN_KIND_INTERNAL",name="okey-dokey",net.peer.ip="1.2.3.4",parent_span_id="d5270e78d85f570f",peer.service="tracegen-client",service.name="tracegen",span.kind="server",span_id="4c28227be6a010e1",status_code="STATUS_CODE_OK",trace_id="7d4854815225332c9834e6dbf85b9380"' | \
curl -v "http://127.0.0.1:8181/api/v2/write?org=company&bucket=traces" --data-binary @-
```

The expected response is `204`

### Query
Let's launch the `sql` client using the `querier` gRPC API on port `8082`

* Using Binary: `./influxdb3 --host http://localhost:8082 sql`
* Using Docker: `docker run -ti --rm ghcr.io/metrico/iox:latest --host http://iox:8082 sql`

#### [Flight SQL](https://github.com/influxdata/influxdb_iox/blob/main/docs/sql.md)

```

We can query our data using the influxdb CLI or the GRPC Flight API

#### Metric Search
```sql
influxdb3 query --database sensors "SELECT * FROM home WHERE temp > 95 LIMIT 4"
+--------+------+-------------------------------+
| room | temp | time |
+--------+------+-------------------------------+
| Garden | 99.0 | 2025-01-25T23:00:01.309084551 |
| Garden | 96.0 | 2025-01-25T16:10:11.745454525 |
| Garden | 98.0 | 2025-01-25T16:10:25.745146130 |
| Garden | 98.0 | 2025-01-25T16:10:37.742997495 |
+--------+------+-------------------------------++---------------------------------+----------------------+-------------+------------------+-------------------+--------------+-----------+------------+---------------+-------------+-------------------+-------------------+
```

IOx Integrations

Integrate your IOx Setup with Go, Rust, Python, etc

> Official IOx FlightSQL clients:

* [influxdb3-go](https://github.com/InfluxCommunity/influxdb3-go)
* [influxdb3-js](https://github.com/InfluxCommunity/influxdb3-js)
* [influxdb3-python](https://github.com/InfluxCommunity/influxdb3-python)
* [influxdb3-java](https://github.com/InfluxCommunity/influxdb3-java)
* [influxdb3-csharp](https://github.com/InfluxCommunity/influxdb3-csharp)

> Generic FlightSQL Drivers

* [iox-community/python](https://github.com/metrico/iox-static-distro/tree/main/examples/python)
* [iox-community/go](https://github.com/metrico/iox-static-distro/tree/main/examples/go)
* [iox-community/rust](https://github.com/metrico/iox-static-distro/tree/main/examples/rust)
* [influxdb-iox-client-go](https://github.com/influxdata/influxdb-iox-client-go)
* [flightsql-dbapi-python](https://github.com/influxdata/flightsql-dbapi)
* [influxdb_iox_client-rust](https://crates.io/crates/influxdb_iox_client)

Grafana

Integrate your IOx Setup with Grafana

### Grafana Client

Your service can be used with the [FlightSQL datasource](https://github.com/influxdata/grafana-flightsql-datasource) in Grafana:

![image](https://user-images.githubusercontent.com/1423657/239708678-8e947ae0-6710-4ae4-85c1-903f4c06b085.png)

Once ready, we can perform queries against our data using the FlightSQL query builder:

![image](https://user-images.githubusercontent.com/1423657/239708634-30b48942-d630-4feb-887d-5b6dc37f54d3.png)

Legal Disclaimers

- All rights reserved by their respective owners. IOx and InfluxDB are a trademark of Influxdata.
- This project is not connected or endorsed by Influxdata or the IOx project. Hopefully one day!
- Original, unstable, nightly. The IOx code is not modified in any way as part of the build process.

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome