{"id":18732443,"url":"https://github.com/metrico/influxdb3-community","last_synced_at":"2025-04-12T18:31:12.453Z","repository":{"id":167742272,"uuid":"643338729","full_name":"metrico/influxdb3-community","owner":"metrico","description":"Community InfluxDB 3.0 \"IOx\" static builds + containers + Examples for Developers \u0026 Integrators. Experiment with low-cost storage, unlimited cardinality and FlightSQL APIs","archived":false,"fork":false,"pushed_at":"2025-03-14T21:12:08.000Z","size":228,"stargazers_count":44,"open_issues_count":0,"forks_count":2,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-03-26T12:51:09.498Z","etag":null,"topics":["arrow","datafusion","flightsql","flux","influx","influxdb","influxdb3","iox","lineprotocol","musl","rust"],"latest_commit_sha":null,"homepage":"https://metrico.in","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/metrico.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-20T20:58:30.000Z","updated_at":"2025-03-14T21:12:11.000Z","dependencies_parsed_at":"2023-11-08T10:42:08.082Z","dependency_job_id":"27358140-8413-417d-ab8a-d803d322da3d","html_url":"https://github.com/metrico/influxdb3-community","commit_stats":null,"previous_names":["metrico/iox-docker","metrico/iox-static-distro","metrico/influxdb3-community","metrico/iox-community"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metrico%2Finfluxdb3-community","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metrico%2Finfluxdb3-community/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metrico%2Finfluxdb3-community/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metrico%2Finfluxdb3-community/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/metrico","download_url":"https://codeload.github.com/metrico/influxdb3-community/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248613316,"owners_count":21133490,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arrow","datafusion","flightsql","flux","influx","influxdb","influxdb3","iox","lineprotocol","musl","rust"],"created_at":"2024-11-07T15:06:07.760Z","updated_at":"2025-04-12T18:31:12.446Z","avatar_url":"https://github.com/metrico.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"![image](https://github.com/metrico/iox-community/assets/1423657/52ff076a-4261-48f3-ad43-e3183e0645cd)\n\n# InfluxDB 3.x \"IOx\" Community\nCommunity Builds and Containers for InfluxDB 3.0 IOx _(eye-ox)_ aka _Edge_\n\n### Motivation\n😄 You want to try and experiment with IOx low-cost storage, unlimited cardinality and flight sql\u003cbr\u003e\n~🥵‍ The IOx project is in _\"Cloud Only\"_ mode and InfluxDB is not producing builds yet...~\u003cbr\u003e\n😄 No problem! Meet the _unofficial_ InfluxDB 3.0 _\"IOx\"_ musl + docker builder for early adopters\n\n#### Nightly Builds\n###### amd64/musl\n  - [x] [docker](https://github.com/metrico/iox-community/pkgs/container/influxdb-edge-musl): `docker pull ghcr.io/metrico/influxdb-edge-musl:latest`\n  - [x] [binary](https://github.com/metrico/iox-community/releases): `github.com/metrico/iox-community/releases/latest/download/influxdb3`\n\n\u003cbr\u003e\n\n## Get Started\n\nThis guide uses Docker and docker-compose. You can run locally using a [static build](https://github.com/metrico/iox-community/releases).\n\n#### Static\n```bash\ncurl -fsSL github.com/metrico/iox-builder/releases/latest/download/influxdb3 -O \\\n\u0026\u0026 chmod +x influxdb3\n```\n\n#### Docker\n\nThe default compose uses local filesystem. Use the full recipe for Object Storage and Postgres Catalog.\n\n```\ndocker-compose up -f docker-compose.yml -d\n```\n\nYour local IOx endpoint should be ready on port `8086`\n\n#### Health\n\nCheck the instance health: \n```\ncurl http://127.0.0.1:8086/health\n```\n\nThe expected response is `OK`\n\n\n\u003cbr\u003e\n  \n\u003cdetails\u003e\n    \u003csummary\u003e\u003ch2\u003eIOx Settings\u003c/h2\u003e Deploy IOx using different settings\u003c/summary\u003e  \n    \n\u003cbr\u003e  \n\n```\n      INFLUXDB3_MAX_HTTP_REQUEST_SIZE: \"10485760\"\n      INFLUXDB3_GEN1_DURATION: \"10m\"\n      INFLUXDB3_WAL_FLUSH_INTERVAL: \"1s\"\n      INFLUXDB3_WAL_SNAPSHOT_SIZE: \"600\"\n      INFLUXDB3_NUM_WAL_FILES_TO_KEEP: \"50\"\n      INFLUXDB3_WAL_MAX_WRITE_BUFFER_SIZE: \"100000\"\n      INFLUXDB3_BUFFER_MEM_LIMIT_MB: \"5000\"\n      INFLUXDB3_PARQUET_MEM_CACHE_SIZE_MB: \"1000\"\n      INFLUXDB3_FORCE_SNAPSHOT_MEM_THRESHOLD: \"70%\"\n      INFLUXDB3_TELEMETRY_DISABLE_UPLOAD: true\n      INFLUXDB3_NODE_IDENTIFIER_PREFIX: \"bucket-id\"\n      INFLUXDB3_OBJECT_STORE: \"file\"\n      INFLUXDB3_DB_DIR: \"/data\"\n```\n\nAs write requests come in to the server, they are parsed, validated, and put into an in-memory WAL buffer. This buffer is flushed every second by default (can be changed through configuration), which will create a WAL file. Once the data is flushed to disk, it is put into a queryable in-memory buffer and then a response is sent back to the client that the write was successful. That data will now show up in queries to the server.\n\nInfluxDB periodically snapshots the WAL to persist the oldest data in the queryable buffer, allowing the server to remove old WAL files. By default, the server will keep up to 900 WAL files buffered up (15 minutes of data) and attempt to persist the oldest 10 minutes, keeping the most recent 5 minutes around.\n\nWhen the data is persisted out of the queryable buffer it is put into the configured object store as Parquet files. Those files are also put into an in-memory cache so that queries against the most recently persisted data do not have to go to object storage.\n\nEach server needs an identifier for writing to object storage and as an identifier that is added to replicated writes, Write Buffer segments and Chunks. Must be unique in a group of connected or semi-connected IOx servers. Must be a number that can be represented by a 32-bit unsigned integer.\n\n```\n      INFLUXDB3_NODE_IDENTIFIER_PREFIX: 1\n```\n\n\nThe demo catalog uses *sqlite* by default. To enable persistent catalog using *postgres*, use the following:\n\n```\n      - INFLUXDB_IOX_CATALOG_DSN=postgres://postgres@localhost:5432/postgres\n```\n\nThe demo stores to filesystem. To enable S3/R2/Minio object storage use the following parameters:\n\n```\n      - INFLUXDB3_OBJECT_STORE=s3\n      - AWS_ACCESS_KEY_ID=access_key_value\n      - AWS_SECRET_ACCESS_KEY=secret_access_key_value\n      - AWS_DEFAULT_REGION=us-east-2\n      - INFLUXDB3_BUCKET=bucket-name\n      - AWS_ENDPOINT = http://minio:9000\n```\n\nFor other storage options refer to [env example](https://github.com/metrico/iox-builder/blob/main/env.example)\n\n### API Proxy\n\nTo emulate InfluxDB3.0 Cloud works, an nginx proxy is included to serve all IOx services from a single endpoint.\n```\nevents {}\nhttp {\n  server {\n    listen 8086;\n    http2 on;\n    location /api {\n       proxy_pass_request_headers on;\n       proxy_pass http://iox:8080;\n    }\n    location /health {\n       proxy_pass http://iox:8080;\n    }\n    location / {\n       proxy_pass_request_headers on;\n       if ($http_content_type = \"application/grpc\") {\n            grpc_pass iox:8082;\n       }\n       proxy_pass http://iox:8082;\n    }\n  }\n}\n```\n\n\n\u003c/details\u003e\n  \n\n\u003cdetails open=true\u003e\n    \u003csummary\u003e\u003ch2\u003eIOX Insert \u0026 Query\u003c/h2\u003e Validate your IOx Setup\u003c/summary\u003e\n\n\u003cbr\u003e\n\nLet's start testing and using your brand new IOx instance!  \n  \n\n### Line Protocol Examples\n\nOur goal is observability formats ingestion into IOx. Here are some scope examples:\n\n#### Metrics\n```\navalanche_metric_mmmmm_0_71 cycle_id=\"0\",gauge=29,host.name=\"generate-metrics-avalanche\",label_key_kkkkk_0=\"label_val_vvvvv_0\",label_key_kkkkk_1=\"label_val_vvvvv_1\",label_key_kkkkk_2=\"label_val_vvvvv_2\",label_key_kkkkk_3=\"label_val_vvvvv_3\",label_key_kkkkk_4=\"label_val_vvvvv_4\",label_key_kkkkk_5=\"label_val_vvvvv_5\",label_key_kkkkk_6=\"label_val_vvvvv_6\",label_key_kkkkk_7=\"label_val_vvvvv_7\",label_key_kkkkk_8=\"label_val_vvvvv_8\",label_key_kkkkk_9=\"label_val_vvvvv_9\",port=\"9090\",scheme=\"http\",series_id=\"3\",service.name=\"otel-collector\" 1613772311130000000\navalanche_metric_mmmmm_0_71 cycle_id=\"0\",gauge=16,host.name=\"generate-metrics-avalanche\",label_key_kkkkk_0=\"label_val_vvvvv_0\",label_key_kkkkk_1=\"label_val_vvvvv_1\",label_key_kkkkk_2=\"label_val_vvvvv_2\",label_key_kkkkk_3=\"label_val_vvvvv_3\",label_key_kkkkk_4=\"label_val_vvvvv_4\",label_key_kkkkk_5=\"label_val_vvvvv_5\",label_key_kkkkk_6=\"label_val_vvvvv_6\",label_key_kkkkk_7=\"label_val_vvvvv_7\",label_key_kkkkk_8=\"label_val_vvvvv_8\",label_key_kkkkk_9=\"label_val_vvvvv_9\",port=\"9090\",scheme=\"http\",series_id=\"4\",service.name=\"otel-collector\" 1613772311130000000\n```\n\n##### Insert\n```bash\nwget -qO- \"https://raw.githubusercontent.com/metrico/influxdb_iox/main/test_fixtures/lineproto/metrics.lp\" | \\\ncurl -v \"http://127.0.0.1:8181/api/v2/write?org=company\u0026bucket=sensors\" --data-binary @-\n```\n\n#### Logs\n```\nsyslog,appname=myapp,facility=console,host=myhost,hostname=myhost,severity=warning facility_code=14i,message=\"warning message here\",severity_code=4i,procid=\"12345\",timestamp=1613769568895331700,version=1\n```\n\n##### Insert\n```bash\necho 'syslog,appname=myapp,facility=console,host=myhost,hostname=myhost,severity=warning facility_code=14i,message=\"warning message here\",severity_code=4i,procid=\"12345\",timestamp=1434055562000000000,version=1' | \\\n curl -v \"http://127.0.0.1:8181/api/v2/write?org=company\u0026bucket=logs\" --data-binary @-\n```\n\n#### Traces\n```\nspans end_time_unix_nano=\"2021-02-19 20:50:25.6893952 +0000 UTC\",instrumentation_library_name=\"tracegen\",kind=\"SPAN_KIND_INTERNAL\",name=\"okey-dokey\",net.peer.ip=\"1.2.3.4\",parent_span_id=\"d5270e78d85f570f\",peer.service=\"tracegen-client\",service.name=\"tracegen\",span.kind=\"server\",span_id=\"4c28227be6a010e1\",status_code=\"STATUS_CODE_OK\",trace_id=\"7d4854815225332c9834e6dbf85b9380\" 1613767825689169000\nspans end_time_unix_nano=\"2021-02-19 20:50:25.6893952 +0000 UTC\",instrumentation_library_name=\"tracegen\",kind=\"SPAN_KIND_INTERNAL\",name=\"lets-go\",net.peer.ip=\"1.2.3.4\",peer.service=\"tracegen-server\",service.name=\"tracegen\",span.kind=\"client\",span_id=\"d5270e78d85f570f\",status_code=\"STATUS_CODE_OK\",trace_id=\"7d4854815225332c9834e6dbf85b9380\" 1613767825689135000\n```\n\n##### Insert\n```bash\necho 'spans end_time_unix_nano=\"2025-01-26 20:50:25.6893952 +0000 UTC\",instrumentation_library_name=\"tracegen\",kind=\"SPAN_KIND_INTERNAL\",name=\"okey-dokey\",net.peer.ip=\"1.2.3.4\",parent_span_id=\"d5270e78d85f570f\",peer.service=\"tracegen-client\",service.name=\"tracegen\",span.kind=\"server\",span_id=\"4c28227be6a010e1\",status_code=\"STATUS_CODE_OK\",trace_id=\"7d4854815225332c9834e6dbf85b9380\"' | \\\n curl -v \"http://127.0.0.1:8181/api/v2/write?org=company\u0026bucket=traces\" --data-binary @-\n```\n\nThe expected response is `204`\n\n### Query\nLet's launch the `sql` client using the `querier` gRPC API on port `8082`\n\n* Using Binary: `./influxdb3 --host http://localhost:8082 sql`\n* Using Docker: `docker run -ti --rm ghcr.io/metrico/iox:latest --host http://iox:8082 sql`\n\n#### [Flight SQL](https://github.com/influxdata/influxdb_iox/blob/main/docs/sql.md)\n\nThe first requirement is to choose a namespace _(or bucket)_ from the available ones:\n```sql\ninfluxdb3 show databases\n+---------------+\n| iox::database |\n+---------------+\n| logs          |\n| mydb          |\n| sensors       |\n| traces        |\n+---------------+\n\n```\n\nWe can query our data using the influxdb CLI or the GRPC Flight API\n\n#### Metric Search\n```sql\ninfluxdb3 query --database sensors \"SELECT * FROM home WHERE temp \u003e 95 LIMIT 4\" \n+--------+------+-------------------------------+\n| room   | temp | time                          |\n+--------+------+-------------------------------+\n| Garden | 99.0 | 2025-01-25T23:00:01.309084551 |\n| Garden | 96.0 | 2025-01-25T16:10:11.745454525 |\n| Garden | 98.0 | 2025-01-25T16:10:25.745146130 |\n| Garden | 98.0 | 2025-01-25T16:10:37.742997495 |\n+--------+------+-------------------------------++---------------------------------+----------------------+-------------+------------------+-------------------+--------------+-----------+------------+---------------+-------------+-------------------+-------------------+\n```\n\n#### Log Search\n##### LIKE\n```sql\ninfluxdb3 query --database logs \"SELECT * FROM syslog WHERE message LIKE '%here%'\" \n+---------+----------+---------------+--------+----------+----------------------+--------+----------+---------------+-------------------------------+----------------+---------+\n| appname | facility | facility_code | host   | hostname | message              | procid | severity | severity_code | time                          | timestamp      | version |\n+---------+----------+---------------+--------+----------+----------------------+--------+----------+---------------+-------------------------------+----------------+---------+\n| myapp   | console  | 14            | myhost | myhost   | warning message here | 12345  | warning  | 4             | 2025-01-25T23:57:02.459118350 | 1.434055562e18 | 1.0     |\n+---------+----------+---------------+--------+----------+----------------------+--------+----------+---------------+-------------------------------+----------------+---------+\n\n```\n##### Regex\n```sql\ninfluxdb3 query --database logs \"SELECT * FROM syslog WHERE message ~ '.+here'\" \n+---------+----------+---------------+--------+----------+----------------------+--------+----------+---------------+-------------------------------+----------------+---------+\n| appname | facility | facility_code | host   | hostname | message              | procid | severity | severity_code | time                          | timestamp      | version |\n+---------+----------+---------------+--------+----------+----------------------+--------+----------+---------------+-------------------------------+----------------+---------+\n| myapp   | console  | 14            | myhost | myhost   | warning message here | 12345  | warning  | 4             | 2025-01-25T23:57:02.459118350 | 1.434055562e18 | 1.0     |\n+---------+----------+---------------+--------+----------+----------------------+--------+----------+---------------+-------------------------------+----------------+---------+\n```\n\n#### Trace Search\n```sql\ninfluxdb3 query --database traces \"SELECT * FROM spans\" \n+---------------------------------------+------------------------------+--------------------+------------+-------------+------------------+-----------------+--------------+-----------+------------------+----------------+-------------------------------+----------------------------------+\n| end_time_unix_nano                    | instrumentation_library_name | kind               | name       | net.peer.ip | parent_span_id   | peer.service    | service.name | span.kind | span_id          | status_code    | time                          | trace_id                         |\n+---------------------------------------+------------------------------+--------------------+------------+-------------+------------------+-----------------+--------------+-----------+------------------+----------------+-------------------------------+----------------------------------+\n| 2025-01-26 20:50:25.6893952 +0000 UTC | tracegen                     | SPAN_KIND_INTERNAL | okey-dokey | 1.2.3.4     | d5270e78d85f570f | tracegen-client | tracegen     | server    | 4c28227be6a010e1 | STATUS_CODE_OK | 2025-01-26T00:01:02.450652384 | 7d4854815225332c9834e6dbf85b9380 |\n| 2025-01-26 20:50:25.6893952 +0000 UTC | tracegen                     | SPAN_KIND_INTERNAL | okey-dokey | 1.2.3.4     | d5270e78d85f570f | tracegen-client | tracegen     | server    | 4c28227be6a010e1 | STATUS_CODE_OK | 2025-01-26T00:01:03.713172859 | 7d4854815225332c9834e6dbf85b9380 |\n+---------------------------------------+------------------------------+--------------------+------------+-------------+------------------+-----------------+--------------+-----------+------------------+----------------+-------------------------------+----------------------------------+\n```\n\n\n\u003c/details\u003e\n                                                        \n\u003cdetails\u003e\n    \u003csummary\u003e\u003ch2\u003eIOx Integrations\u003c/h2\u003e Integrate your IOx Setup with Go, Rust, Python, etc\u003c/summary\u003e  \n  \n\u003e Official IOx FlightSQL clients:\n\n  * [influxdb3-go](https://github.com/InfluxCommunity/influxdb3-go)\n  * [influxdb3-js](https://github.com/InfluxCommunity/influxdb3-js)\n  * [influxdb3-python](https://github.com/InfluxCommunity/influxdb3-python)\n  * [influxdb3-java](https://github.com/InfluxCommunity/influxdb3-java)\n  * [influxdb3-csharp](https://github.com/InfluxCommunity/influxdb3-csharp)\n\n\u003e Generic FlightSQL Drivers\n  \n  * [iox-community/python](https://github.com/metrico/iox-static-distro/tree/main/examples/python)\n  * [iox-community/go](https://github.com/metrico/iox-static-distro/tree/main/examples/go)\n  * [iox-community/rust](https://github.com/metrico/iox-static-distro/tree/main/examples/rust)\n  * [influxdb-iox-client-go](https://github.com/influxdata/influxdb-iox-client-go)\n  * [flightsql-dbapi-python](https://github.com/influxdata/flightsql-dbapi)\n  * [influxdb_iox_client-rust](https://crates.io/crates/influxdb_iox_client)\n\n  \n\u003cdetails\u003e\n    \u003csummary\u003e\u003ch3\u003eGrafana\u003c/h3\u003e Integrate your IOx Setup with Grafana\u003c/summary\u003e  \n\n  ### Grafana Client\n\nYour service can be used with the [FlightSQL datasource](https://github.com/influxdata/grafana-flightsql-datasource) in Grafana:\n\n![image](https://user-images.githubusercontent.com/1423657/239708678-8e947ae0-6710-4ae4-85c1-903f4c06b085.png)\n\nOnce ready, we can perform queries against our data using the FlightSQL query builder:\n\n![image](https://user-images.githubusercontent.com/1423657/239708634-30b48942-d630-4feb-887d-5b6dc37f54d3.png)\n                                                        \n\u003c/details\u003e\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n    \u003csummary\u003e\u003ch2\u003eLegal Disclaimers\u003c/h2\u003e\u003c/summary\u003e  \n  \n- All rights reserved by their respective owners. IOx and InfluxDB are a trademark of Influxdata.   \n- This project is not connected or endorsed by Influxdata or the IOx project. Hopefully one day!\n- Original, unstable, nightly. The IOx code is not modified in any way as part of the build process. \n  \n\u003c/details\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmetrico%2Finfluxdb3-community","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmetrico%2Finfluxdb3-community","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmetrico%2Finfluxdb3-community/lists"}