{"id":31648939,"url":"https://github.com/gigapi/gigapi-docs","last_synced_at":"2025-10-07T07:03:33.167Z","repository":{"id":287411067,"uuid":"964538069","full_name":"gigapi/gigapi-docs","owner":"gigapi","description":"GigAPI: DuckDB Powered Parquet Storage Engine \u0026 Data Pond","archived":false,"fork":false,"pushed_at":"2025-04-11T15:53:37.000Z","size":33,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-12T17:27:51.381Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://gigapipe.com","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gigapi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-11T11:22:14.000Z","updated_at":"2025-04-13T04:37:44.000Z","dependencies_parsed_at":"2025-04-18T16:50:34.227Z","dependency_job_id":null,"html_url":"https://github.com/gigapi/gigapi-docs","commit_stats":null,"previous_names":["gigapi/gigapi"],"tags_count":31,"template":false,"template_full_name":null,"purl":"pkg:github/gigapi/gigapi-docs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gigapi%2Fgigapi-docs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gigapi%2Fgigapi-docs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gigapi%2Fgigapi-docs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gigapi%2Fgigapi-docs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gigapi","download_url":"https://codeload.github.com/gigapi/gigapi-docs/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gigapi%2Fgigapi-docs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278734416,"owners_count":26036404,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-07T02:00:06.786Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-07T07:02:01.525Z","updated_at":"2025-10-07T07:03:33.161Z","avatar_url":"https://github.com/gigapi.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# \u003cimg src=\"https://github.com/user-attachments/assets/5b0a4a37-ecab-4ca6-b955-1a2bbccad0b4\" /\u003e\n\n# \u003cimg src=\"https://github.com/user-attachments/assets/74a1fa93-5e7e-476d-93cb-be565eca4a59\" height=25 /\u003e GigAPI Storage Engine\n\nLike a durable parquet floor, GigAPI provides rock-solid data foundation — so you can focus on queries\n\n\u003e GigAPI eliminates classic storage and server limits, unlocking virtually infinite cardinality without compromising query speed and performance. Its DuckDB and Arrow powered engine handles massive parallel ingestion and self-compation of data for heavy aggregations and complex SQL queries, delivering consistent performance as your system scales without storage or cardinality limitations and price tag\n\n\u003cp align=\"center\"\u003e\n  \u003c!-- \u003cimg src=\"https://github.com/user-attachments/assets/1689f098-9773-4bd9-8438-51a93a8777b1\" width=300\u003e --\u003e\n  \u003cimg src=\"https://github.com/user-attachments/assets/573938a0-c2d9-41bf-8cfc-638904d559f0\"\u003e\n\u003c/p\u003e\n\n## \u003cimg src=\"https://github.com/user-attachments/assets/74a1fa93-5e7e-476d-93cb-be565eca4a59\" height=20 /\u003e Write Support\nAs write requests come in to GigAPI they are parsed and progressively appeanded to parquet files alongside their metadata. The ingestion buffer is flushed to disk at configurable intervals using a hive partitioning schema. Generated parquet files and their respective metadata are progressively compacted and sorted over time based on configuration parameters.\n\n### \u003cimg src=\"https://github.com/user-attachments/assets/a9aa3ebd-9164-476d-aedf-97b817078350\" width=18 /\u003e API\nGigAPI provides an HTTP API for clients to write, currently supporting the InfluxDB Line Protocol format \n\n* _more ingestion protocols coming soon_\n\n### \u003cimg src=\"https://github.com/user-attachments/assets/a9aa3ebd-9164-476d-aedf-97b817078350\" width=18 /\u003e Data Schema\nGigAPI is a schema-on-write database managing databases, tables and schemas on the fly. New columns can be added or removed over time, leaving reconciliation up to readers.\n\n```bash\n/data\n  /mydb\n    /weather\n      /date=2025-04-10\n        /hour=14\n          *.parquet\n          metadata.json\n        /hour=15\n          *.parquet\n          metadata.json\n```\n\n## \u003cimg src=\"https://github.com/user-attachments/assets/74a1fa93-5e7e-476d-93cb-be565eca4a59\" height=20 /\u003e Read Support\nAs read requests come in to GigAPI they are parsed and transpiled using the GigAPI Metadata catalog to resolve data location based on database, table and timerange in requests. Series can be used with or without time ranges, ie for calculating averages, etc.\n\n```bash\n$ curl -X POST \"http://localhost:9999/query?db=mydb\" \\\n  -H \"Content-Type: application/json\"  \\\n  -d '{\"query\": \"SELECT count(*), avg(temperature) FROM weather\"}'\n```\n```json\n{\"results\":[{\"avg(temperature)\":87.025,\"count_star()\":\"40\"}]}\n```\n\n\u003e GigAPI readers can be implemented in any language and with any OLAP engine supporting Parquet files.\n\n\u003cbr\u003e\n\n## \u003cimg src=\"https://github.com/user-attachments/assets/74a1fa93-5e7e-476d-93cb-be565eca4a59\" height=20 /\u003e  GigAPI Diagram\n```mermaid\n%%{\n  init: {\n    'theme': 'base',\n    'themeVariables': {\n      'primaryColor': '#6a329f',\n      'primaryTextColor': '#fff',\n      'primaryBorderColor': '#7C0000',\n      'lineColor': '#6f329f',\n      'secondaryColor': '#006100',\n      'tertiaryColor': '#fff'\n    }\n  }\n}%%\n\n  graph TD;\n      GigAPI--\u003eParquetWriter;\n      ParquetWriter--\u003eStorage;\n      ParquetWriter--\u003eMetadata;\n      Storage--\u003eCompactor;\n      Compactor--\u003eStorage;\n      Compactor--\u003eMetadata;\n      Storage-.-\u003eLocalFS;\n      Storage-.-\u003eS3;\n      HTTP-API-- GET/POST --\u003e GigAPI;\n      DuckDB--\u003eStorage;\n      DuckDB--\u003eMetadata;\n\n      subgraph GigAPI[GigAPI Server]\n        ParquetWriter\n        Compactor\n        Metadata;\n        DuckDB;\n      end\n\n```\n\n## \u003cimg src=\"https://github.com/user-attachments/assets/a9aa3ebd-9164-476d-aedf-97b817078350\" width=22 /\u003e License\n\n\u003cimg src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/0/06/AGPLv3_Logo.svg/2560px-AGPLv3_Logo.svg.png\" width=200\u003e\n\n\u003e Gigapipe is released under the GNU Affero General Public License v3.0 ©️ HEPVEST BV, All Rights Reserved.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgigapi%2Fgigapi-docs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgigapi%2Fgigapi-docs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgigapi%2Fgigapi-docs/lists"}