{"id":17254092,"url":"https://github.com/brimdata/super","last_synced_at":"2025-12-14T20:32:42.730Z","repository":{"id":37072365,"uuid":"220333768","full_name":"brimdata/super","owner":"brimdata","description":"An analytics database that puts JSON and relational tables on equal footing","archived":false,"fork":false,"pushed_at":"2025-05-13T18:20:45.000Z","size":36130,"stargazers_count":1463,"open_issues_count":537,"forks_count":68,"subscribers_count":19,"default_branch":"main","last_synced_at":"2025-05-13T18:31:33.967Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://superdb.org","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brimdata.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":"code_of_conduct.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-11-07T21:39:29.000Z","updated_at":"2025-05-13T17:15:57.000Z","dependencies_parsed_at":"2023-10-17T03:09:57.735Z","dependency_job_id":"c54e1550-00c3-4b64-bc62-9b8f392920ea","html_url":"https://github.com/brimdata/super","commit_stats":{"total_commits":3527,"total_committers":21,"mean_commits":"167.95238095238096","dds":0.7059824213212362,"last_synced_commit":"06f8f66940f4a572d36459196d4212beae41b5ff"},"previous_names":["brimsec/zq","brimdata/zq","mccanne/zq","brimdata/super","brimdata/zed"],"tags_count":63,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brimdata%2Fsuper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brimdata%2Fsuper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brimdata%2Fsuper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brimdata%2Fsuper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brimdata","download_url":"https://codeload.github.com/brimdata/super/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254159194,"owners_count":22024558,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-15T07:05:15.260Z","updated_at":"2025-12-14T20:32:42.657Z","avatar_url":"https://github.com/brimdata.png","language":"Go","readme":"# SuperDB [![Tests][tests-img]][tests] [![GoPkg][gopkg-img]][gopkg]\n\n\u003e 🔴 **NOTICE OF PROJECT READINESS** 🔴\n\u003e\n\u003e SuperDB is still under development so there's not yet a GA release.\n\u003e You're welcome to [try it out](#try-it) in its early form (i.e.,\n\u003e tip-of-`main`) and we'd love to hear your feedback. Read on for more info!\n\nSuperDB is a new analytics database that supports relational tables and JSON\non an equal footing.  It shines when it comes to data wrangling where\nyou need to explore or process large eclectic data sets.  It's also pretty\ndecent at analytics and\n[search use cases](https://superdb.org/docs/language/search-expressions).\n\nUnlike other relational systems that do performance-fragile \"schema inference\" of JSON,\nSuperDB won't fall over if you throw a bunch of eclectic JSON at it.\nYou can easily do\n[schema inference if you want](https://superdb.org/docs/language/operators/fuse),\nbut data is ingested by default in its natural form no matter how much heterogeneity\nit might have.  And unlike systems based on the document data model,\nevery value in SuperDB is strongly and dynamically typed thus providing the\nbest of both worlds: the flexibility of the document model and\nthe efficiency and performance of the relational model.\n\nIn SuperDB's SQL dialect, there are no \"JSON columns\" so there isn't a \"relational\nway to do things\" and a different \"JSON way to do things\".  Instead of having\na relational type system for structured data and completely separate JSON type\nsystem for semi-structured data,\nall data handled by SuperDB (e.g., JSON, CSV, Parquet files, Arrow streams, relational tables, etc) is automatically massaged into\n[super-structured data](https://superdb.org/docs/formats/#2-a-super-structured-pattern)\nform.  This super-structured data is then processed by a runtime that simultaneously\nsupports the statically-typed relational model and the dynamically-typed\nJSON data model in a unified compute engine.\n\n## SuperSQL\n\nSuperDB uses SQL as its query language, but it's a SQL that has been extended\nwith [pipe syntax](https://research.google/pubs/sql-has-problems-we-can-fix-them-pipe-syntax-in-sql/)\nand [lots of fun shortcuts](https://superdb.org/docs/language/pipeline-model/#implied-operators).\nThis extended SQL is called SuperSQL.\n\nHere's a SuperSQL query that fetches some data from GitHub Archive,\ncomputes the set of repos touched by each user, ranks them by number of repos,\npicks the top five, and joins each user with their original `created_at` time\nfrom the current GitHub API:\n\n```sql\nFROM 'https://data.gharchive.org/2015-01-01-15.json.gz'\n| SELECT union(repo.name) AS repos, actor.login AS user\n  GROUP BY user\n  ORDER BY len(repos) DESC\n  LIMIT 5\n| FORK (\n  =\u003e FROM eval(f'https://api.github.com/users/{user}')\n   | SELECT VALUE {user:login,created_at:time(created_at)}\n  =\u003e PASS\n  )\n| JOIN USING (user) repos\n```\n\n## Super (SUP) Format\n\nSuper-structured data is strongly typed and \"polymorphic\": any value can take on any type\nand sequences of data need not all conform to a predefined schema.  To this end,\nSuperDB extends the JSON format to support super-structured data in a format called\n[Super (SUP)](https://superdb.org/docs/formats/sup) where all JSON values\nare also SUP.  Similarly,\nthe [Super Binary (BSUP)](https://superdb.org/docs/formats/bsup) format is an efficient\nbinary representation of SUP (a bit like Avro) and the\n[Super Columnar (CSUP)](https://superdb.org/docs/formats/csup) format is a columnar\nrepresentation of SUP (a bit like Parquet).\n\nEven though SuperDB is based on these super-structured data formats, it can read and write\nmost common data formats.\n\n## Project Status\n\nOur long-term goal for SuperSQL is to be Postgres-compatible and interoperate\nwith existing SQL tooling. In the meantime, SuperSQL is a bit of a moving \ntarget and we would love [community engagement](#join-the-community) to evolve and fine tune its\nsyntax and semantics.\n\nOur areas of active development include:\n* the SuperSQL query language,\n* the type-based query compiler and optimizer,\n* fast, vectorized ingest of common file formats,\n* a complete vectorized runtime, and\n* a data lake based on super-structured data.\n\n## Try It\n\nAs SuperDB is still under construction, GA releases are not yet available.\nHowever, you can [install](https://superdb.org/docs/getting_started/install) a build of the\n[`super`](https://superdb.org/docs/commands/super) command-line tool based on\ncode that's under active development to start tinkering. Detailed documentation\nfor the SuperDB system and its piped SQL syntax is available on the\n[SuperDB docs site](https://superdb.org/docs).\n\nAs the code and docs are evolving, we recommend focusing first on what's in the\n[`super` command doc](https://superdb.org/docs/commands/super). Feel free to\nexplore other docs and try things out, but please don't be shocked if you hit\nspeedbumps in the near term, particularly in areas like performance and full\nSQL coverage. We're working on it! :wink:\n\nOnce you've tried it out, we'd love to hear your feedback via our\n[community Slack](https://www.brimdata.io/join-slack/). \n\n\u003e**NOTE:** The SuperDB query engine can run locally without a storage engine by accessing\n\u003efiles, HTTP endpoints, or S3 paths using the `super` command. While\n\u003e[earlier in its development](https://superdb.org/docs/commands/super-db/#status),\n\u003eSuperDB can also run on a\n\u003e[super-structured data lake](https://superdb.org/docs/commands/super-db/#the-lake-model)\n\u003eusing the `super db` sub-commands.\n\n### SuperDB Desktop - Coming Soon\n\n[SuperDB Desktop](https://github.com/brimdata/zui) is an Electron-based\ndesktop app to explore, query, and shape data in a SuperDB data lake.\nIt combines a search experience with a SQL query and has some really slick\ndesign for dealing with complex and large JSON data.\n\nUnlike most JSON browsing tools, it won't slow to a crawl --- or worse crash ---\nif you load it up with ginormous JSON values.\n\n## Contributing\n\nSee the [contributing guide](CONTRIBUTING.md) on how you can help improve SuperDB!\n\n## Join the Community\n\nJoin our [public Slack](https://www.brimdata.io/join-slack/) workspace for announcements, Q\u0026A, and to trade tips!\n\n[tests-img]: https://github.com/brimdata/super/workflows/Tests/badge.svg\n[tests]: https://github.com/brimdata/super/actions?query=workflow%3ATests\n[gopkg-img]: https://pkg.go.dev/badge/github.com/brimdata/super\n[gopkg]: https://pkg.go.dev/github.com/brimdata/super\n\n","funding_links":[],"categories":["Go"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrimdata%2Fsuper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrimdata%2Fsuper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrimdata%2Fsuper/lists"}