{"id":17570672,"url":"https://github.com/metrico/fluXpipe","last_synced_at":"2025-03-07T21:30:46.039Z","repository":{"id":41980675,"uuid":"478321630","full_name":"metrico/fluXpipe","owner":"metrico","description":"TLDR: Flux for InfluxDB 3.0. Stand-alone, Serverless Flux API/Pipeline for querying, analyzing and interacting with remote data.","archived":false,"fork":false,"pushed_at":"2023-10-07T18:03:14.000Z","size":16930,"stargazers_count":64,"open_issues_count":1,"forks_count":3,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-02-21T05:33:10.249Z","etag":null,"topics":["analytics","clickhouse","flightsql","flux","fluxlang","fluxlib","go","influx","influxdb","influxdb-client","influxdb3","iox","qryn","query-language","serverless"],"latest_commit_sha":null,"homepage":"https://fluxpipe.fly.dev","language":"FLUX","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/metrico.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null},"funding":{"github":["qxip","gigapipehq"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":["https://qryn.cloud"]}},"created_at":"2022-04-05T22:35:25.000Z","updated_at":"2025-02-20T04:52:10.000Z","dependencies_parsed_at":"2023-02-15T16:31:37.967Z","dependency_job_id":"d6c00769-465c-4615-997f-217cc49f72a3","html_url":"https://github.com/metrico/fluXpipe","commit_stats":null,"previous_names":["lmangani/fluxpipe"],"tags_count":53,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metrico%2FfluXpipe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metrico%2FfluXpipe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metrico%2FfluXpipe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metrico%2FfluXpipe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/metrico","download_url":"https://codeload.github.com/metrico/fluXpipe/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242467161,"owners_count":20133105,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","clickhouse","flightsql","flux","fluxlang","fluxlib","go","influx","influxdb","influxdb-client","influxdb3","iox","qryn","query-language","serverless"],"created_at":"2024-10-21T18:01:16.270Z","updated_at":"2025-03-07T21:30:45.737Z","avatar_url":"https://github.com/metrico.png","language":"FLUX","funding_links":["https://github.com/sponsors/qxip","https://github.com/sponsors/gigapipehq","https://qryn.cloud"],"categories":["Integrations"],"sub_categories":["Data Transfer and Synchronization"],"readme":"# \u003cimg src=\"https://user-images.githubusercontent.com/1423657/231777629-38f9c8f5-7968-4ff3-ae66-2fd2459ffa69.png\" width=200 /\u003e\n\n[![Build-n-Release](https://github.com/metrico/fluXpipe/actions/workflows/go.yml/badge.svg)](https://github.com/metrico/fluXpipe/actions/workflows/go.yml)\n\n**FluxPipe** is an *experimental* stand-alone **Flux API** for *serverless workers* and *embedded datasources*\u003cbr\u003e\nExecute your Flux scripts locally, in serverless functions or anywhere else - _decoupled from the data and database._\u003cbr\u003e\n\n_Fluxpipe runs at 141,6Km/h_[*](https://en.wikipedia.org/wiki/DeLorean_time_machine#:~:text=Various%20proposals%20have%20been%20brought,speedometer%2C%20modified%20for%20the%20movie.) and is compatible with **InfluxDB 3.0 / IOx, ClickHouse, Grafana** and _beyond!_\n\n\u003cbr\u003e\n\n\u003e InfluxDB [Flux](https://github.com/InfluxCommunity/flux) is a lightweight *scripting language* for querying databases and working with data. [^1]\u003cbr\u003e\n\u003e Need a Flux introduction? Check out the [official page](https://www.influxdata.com/products/flux/), [documentation](https://metrico.github.io/fluXpipe/) or [\"3 Minutes to Flux\"](flux.md) guide.\n\n### Demo\nTry our [serverless demo](https://fluxpipe.fly.dev/) or launch your own instance to instantly fall in love with *flux*\n\n\u003ca href=\"https://flyctl.sh/shell?repo=metrico/fluxpipe\" target=\"_blank\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/1423657/236479471-a1cb0484-dfd3-4dc2-8d62-121debd7faa3.png\" width=300\u003e\n\u003c/a\u003e\n\n\u003cbr\u003e\n\n### Instructions\nDownload a [binary release](https://github.com/metrico/fluxpipe/releases/), [docker](https://github.com/metrico/fluXpipe/pkgs/container/fluxpipe) or build from source\n\n#### 📦 Download Binary\n```bash\ncurl -fsSL github.com/metrico/fluxpipe/releases/latest/download/fluxpipe-server -O \\\n\u0026\u0026 chmod +x fluxpipe-server\n```\n##### 🔌 Start Server w/ Options\n```bash\n./fluxpipe-server -port 8086\n```\nRun with `-h` for a full list of parameters\n\n#### 🐋 Using Docker\n```bash\ndocker pull ghcr.io/metrico/fluxpipe:latest\ndocker run -ti --rm -p 8086:8086 ghcr.io/metrico/fluxpipe:latest\n```\n\n\u003cbr\u003e\n\n### 🐛 Usage\n\n💡 _Check out the [scripts](scripts) folder for working examples_\n\n### Playground\nFluxpipe embeds a playground interface to instantly execute queries _(borrowed from ClickHouse [^2])_\n\n\u003ca href=\"https://fluxpipe.fly.dev\"\u003e\u003cimg src=\"https://user-images.githubusercontent.com/1423657/231785057-c1762c65-7c30-483b-a551-57c316caee88.png\"\u003e\u003c/a\u003e\n\n\u003cbr\u003e\n\n#### HTTP API\nFluxpipe serves a simple REST API loosely compatible with existing flux integrations and clients\n\n##### Grafana Flux [^1]\nFluxpipe is compatible with the native **Grafana InfluxDB/Flux datasource** _(url + organization fields are required!)_\n\n\u003cbr\u003e\n\n##### ⭐ FlightSQL\n###### SQL\nYou can query InfluxDB 3.0 IOx with raw SQL using the native `sql.from` handler\n```\nimport \"sql\"\n\nsql.from(\n    driverName: \"influxdb-iox\",\n    dataSourceName: \"iox://iox-server:443/qryn_logs\",\n    query: \"SELECT level, sender, body FROM logs WHERE body LIKE '%DELETE%' limit 10\",\n)\n```\n\n![image](https://github.com/metrico/fluXpipe/assets/1423657/b6c2dcbe-079b-4329-9fee-a8601a8c853c)\n\n\n###### Flux IOx\nYou can query InfluxDB 3.0 IOx with Flux using the `iox.from` handler \n```flux\nimport \"contrib/qxip/iox\"\niox.from(\n     bucket: \"test\",\n     host: \"eu-central-1-1.aws.cloud2.influxdata.com:443\",\n     token: \"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\n     limit: \"10\",\n     columns: \"time, level, sender\",\n     table: \"logs\",\n     start: -100d,\n)\n```\n```\n                    _time:time            level:string           sender:string\n------------------------------  ----------------------  ----------------------\n2023-08-31T00:00:00.091362490Z                    info                 logtest\n2023-08-31T00:00:00.091380034Z                    info                 logtest\n2023-08-31T00:00:00.091381374Z                    info                 logtest\n2023-08-31T00:00:00.091382470Z                    info                 logtest\n2023-08-31T00:00:00.091383354Z                    info                 logtest\n...\n```\n\nYou write data back to InfluxDB 3.0 IOx using the `to` and ` wideTo` functions\n\n```flux\nimport \"contrib/qxip/iox\"\nimport \"influxdata/influxdb\"\niox.from(\n     bucket: \"qxip\",\n     host: \"eu-central-1-1.aws.cloud2.influxdata.com:443\",\n     token: \"\",\n     limit: \"10\",\n     table: \"machine_data\",\n     start: -2d,\n )\n |\u003e range(start: -2d)\n |\u003e aggregateWindow(every: 5s, fn: mean, column: \"load\", createEmpty: false)\n |\u003e set(key: \"_measurement\", value: \"downsampled\")\n |\u003e wideTo(\n     bucket: \"qxip\",\n     host: \"https://eu-central-1-1.aws.cloud2.influxdata.com\",\n     token: \"\",\n     orgID: \"6a841c0c08328fb1\"\n    )\n\n```\n\n\n\n###### ⭐ ClickHouse SQL\n```\nimport \"contrib/qxip/clickhouse\"\n\nclickhouse.query(\n  url: \"https://play@play.clickhouse.com\",\n  query: \"SELECT database, total_rows FROM tables WHERE total_rows \u003e 0\"\n)\n|\u003e rename(columns: {database: \"_value\", total_rows: \"_data\"})\n|\u003e keep(columns: [\"_value\",\"_data\"])\n```\n![image](https://user-images.githubusercontent.com/1423657/162625425-15a92f34-562b-4e27-8832-7bc33a90b185.png)\n\n![image](https://user-images.githubusercontent.com/1423657/162428332-77d869a2-d02b-443d-a3ef-3df1fbf899f6.png)\n\n###### ⭐ LogQL \n```\nimport \"contrib/qxip/logql\"\n\noption logql.defaultURL = \"http://qryn:3100\"\nlogql.query_range(\n     query: \"rate({job=\\\"dummy-server\\\"}[5m])\",\n     start: v.timeRangeStart, \n     end: v.timeRangeStop\n)\n|\u003e map(fn: (r) =\u003e ({r with _time: time(v: uint(v: r.timestamp_ns)), _value: float(v: r.value) }))\n|\u003e drop(columns: [\"timestamp_ns\", \"value\"])\n|\u003e sort(columns: [\"_time\"])\n|\u003e group(columns: [\"labels\"])\n```\n![image](https://user-images.githubusercontent.com/1423657/215287132-dc8e18ca-25f8-40cd-a925-cc9f6c090be5.png)\n\n\n###### ⭐ CURL POST\nUsage with curl\n\n```bash\ncurl -XPOST localhost:8086/api/v2/query -sS \\\n  -H 'Accept:application/csv' \\\n  -H 'Content-type:application/vnd.flux' \\\n  -d 'import g \"generate\" g.from(start: 2022-04-01T00:00:00Z, stop: 2022-04-01T00:03:00Z, count: 3, fn: (n) =\u003e n)'\n```\n\n#### Secrets\nFlux builds using `EnvironmentSecretService` accessing system environment variables from flux scripts.\n```\nimport \"influxdata/influxdb/secrets\"\nkey = secrets.get(key: \"ENV_SECRET\")\n```\n\n------\n\n#### STDIN CMD\nFluxpipe can be used as a command-line tool and stdin pipeline processor\n\n###### Generate CSV\n```bash\necho 'import g \"generate\" g.from(start: 2022-04-01T00:00:00Z, stop: 2022-04-01T00:03:00Z, count: 5, fn: (n) =\u003e 1)' \\\n| ./fluxpipe-server -stdin\n```\n##### Parse CSV\n```bash\ncat scripts/csv.flux | ./fluxpipe-server -stdin\n```\n##### Query SQL\n```bash\ncat scripts/sql.flux | ./fluxpipe-server -stdin\n```\n\n\u003cbr\u003e\n\n## Public Demo\n\n### Grafana Datasource\nConfigure your Grafana instance with our public demo endpoint _(limited resources)_\n![image](https://github.com/metrico/fluXpipe/assets/1423657/c100050a-49a6-4788-86af-15eb7e59c5d1)\n\n\n\n### Documentation\n**Flux**(pipe) is built using the [InfluxCommunity/flux](https://github.com/InfluxCommunity/flux) fork which contains additional features and contributions.\u003cbr\u003e\nAll the standard and additional functions available in Fluxpipe are included in the [Flux community documentation](https://metrico.github.io/fluXpipe/)\n\n\u003cbr\u003e\n\n\n#### Status\n- [x] Fluxlib\n  - [x] parser\n  - [x] executor\n- [x] Contribs\n  - [x] contrib/qxip/clickhouse\n  - [x] contrib/qxip/logql\n  - [x] contrib/qxip/hash\n  - [x] ENV secrets\n- [x] STDIN pipeline\n- [x] HTTP api\n  - [x] plaintext\n  - [x] json support\n  - [x] web playground\n\n\n\n[^1]: Project is not affiliated or endorsed by Influxdata or Grafana Labs. All rights belong to their respective owners.\n[^2]: Used under Apache2 terms. Project is not affiliated or endorsed by ClickHouse Inc. All rights belong to their respective owners.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmetrico%2FfluXpipe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmetrico%2FfluXpipe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmetrico%2FfluXpipe/lists"}