{"id":13564720,"url":"https://github.com/sourcenetwork/defradb","last_synced_at":"2026-01-12T12:49:16.451Z","repository":{"id":152947095,"uuid":"192631273","full_name":"sourcenetwork/defradb","owner":"sourcenetwork","description":"DefraDB is a Peer-to-Peer Edge Database. It's the core data storage system for the Source Network Ecosystem, built with IPLD, LibP2P, CRDTs, and Semantic open web properties.","archived":false,"fork":false,"pushed_at":"2025-04-01T19:18:58.000Z","size":68440,"stargazers_count":665,"open_issues_count":402,"forks_count":57,"subscribers_count":15,"default_branch":"develop","last_synced_at":"2025-04-01T19:45:10.141Z","etag":null,"topics":["crdt","database","distributed","documentdb","edge-compute","graphql","linked-data","local-first-software","nosql","peer-to-peer","semantic-web"],"latest_commit_sha":null,"homepage":"https://docs.source.network","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sourcenetwork.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"licenses/APL.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-19T00:41:12.000Z","updated_at":"2025-03-31T21:23:34.000Z","dependencies_parsed_at":null,"dependency_job_id":"b9708863-be08-4a7d-8be0-0de584c6f5b7","html_url":"https://github.com/sourcenetwork/defradb","commit_stats":null,"previous_names":[],"tags_count":23,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sourcenetwork%2Fdefradb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sourcenetwork%2Fdefradb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sourcenetwork%2Fdefradb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sourcenetwork%2Fdefradb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sourcenetwork","download_url":"https://codeload.github.com/sourcenetwork/defradb/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247083374,"owners_count":20880828,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crdt","database","distributed","documentdb","edge-compute","graphql","linked-data","local-first-software","nosql","peer-to-peer","semantic-web"],"created_at":"2024-08-01T13:01:34.967Z","updated_at":"2026-01-12T12:49:16.444Z","avatar_url":"https://github.com/sourcenetwork.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"![Test Coverage Workflow](https://github.com/sourcenetwork/defradb/actions/workflows/test-coverage.yml/badge.svg)\n[![Go Report Card](https://goreportcard.com/badge/github.com/sourcenetwork/defradb)](https://goreportcard.com/report/github.com/sourcenetwork/defradb)\n[![codecov](https://codecov.io/gh/sourcenetwork/defradb/branch/develop/graph/badge.svg?token=RHAORX13PA)](https://codecov.io/gh/sourcenetwork/defradb)\n[![Discord](https://img.shields.io/discord/427944769851752448.svg?color=768AD4\u0026label=discord\u0026logo=https%3A%2F%2Fdiscordapp.com%2Fassets%2F8c9701b98ad4372b58f13fd9f65f966e.svg)](https://discord.gg/w7jYQVJ)\n[![Twitter Follow](https://img.shields.io/twitter/follow/sourcenetwrk.svg?label=\u0026style=social)](https://twitter.com/sourcenetwrk)\n\n\u003cp align=\"center\"\u003e\n  \u003cpicture\u003e\n    \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"docs/DefraDB_White.svg\"\u003e\n    \u003cimg height=\"120px\" width=\"374px\" alt=\"DefraDB\" src=\"docs/DefraDB_Full.svg\"\u003e\n  \u003c/picture\u003e\n\u003c/p\u003e\n\nDefraDB is a user-centric database that prioritizes data ownership, personal privacy, and information security. Its data model, powered by the convergence of [MerkleCRDTs](https://arxiv.org/pdf/2004.00107.pdf) and the content-addressability of [IPLD](https://docs.ipld.io/), enables a multi-write-master architecture. It features [DQL](https://docs.source.network/defradb/references/query-specification/query-language-overview), a query language compatible with GraphQL but providing extra convenience. By leveraging peer-to-peer networking it can be deployed nimbly in novel topologies. Access control is determined by a relationship-based DSL, supporting document or field-level policies, secured by the SourceHub network. DefraDB is a core part of the [Source technologies](https://source.network/) that enable new paradigms of decentralized data and access-control management, user-centric apps, data trustworthiness, and much more.\n\nRead the documentation on [docs.source.network](https://docs.source.network/).\n\n\n## Table of Contents\n\n\u003c!--ts--\u003e\n   * [Install](#install)\n   * [Key Management](#key-management)\n   * [Start](#start)\n   * [Configuration](#configuration)\n   * [External port binding](#external-port-binding)\n   * [Add a schema type](#add-a-schema-type)\n   * [Create a document](#create-a-document)\n   * [Query documents](#query-documents)\n   * [Obtain document commits](#obtain-document-commits)\n   * [DefraDB Query Language (DQL)](#defradb-query-language-dql)\n   * [Peer-to-peer data synchronization](#peer-to-peer-data-synchronization)\n   * [Securing the HTTP API with TLS](#securing-the-http-api-with-tls)\n   * [Access Control System](#access-control-system)\n   * [Supporting CORS](#supporting-cors)\n   * [Backing up and restoring](#backing-up-and-restoring)\n   * [Telemetry](#telemetry)\n   * [Community](#community)\n   * [Playground](#playground)\n   * [Licensing](#licensing)\n   * [Contributors](#contributors)\n\u003c!--te--\u003e\n\nDISCLAIMER: The software is provided \"as is\" and is not guaranteed to be stable, secure, or error-free. We encourage you to experiment with DefraDB and provide feedback, and when you plan to deploy it to production, please thoroughly test your integrations.\n\n## Install\n\nInstall `defradb` by [downloading an executable](https://github.com/sourcenetwork/defradb/releases) or building it locally using the [Go toolchain](https://golang.org/):\n\n```sh\ngit clone https://github.com/sourcenetwork/defradb.git\ncd defradb\nmake install\n```\n\nIn the following sections, we assume that `defradb` is included in your `PATH`. If you installed it with the Go toolchain, use:\n\n```sh\nexport PATH=$PATH:$(go env GOPATH)/bin\n```\n\nWe recommend experimenting with queries using a native GraphQL client. GraphiQL is a popular option - [download and install it](https://altairgraphql.dev/#download).\n\n## Key Management\n\nDefraDB has a built in keyring that can be used to store private keys securely.\n\nThe following keys are loaded from the keyring on start:\n\n- `peer-key` Ed25519 private key (required)\n- `encryption-key` AES-128, AES-192, or AES-256 key (optional)\n- `node-identity-key` Secp256k1 private key (optional). This key is used for node's identity.\n\nA secret to unlock the keyring is required on start and must be provided via the `DEFRA_KEYRING_SECRET` environment variable. If a `.env` file is available in the working directory, the secret can be stored there or via a file at a path defined by the `--secret-file` flag.\n\nThe keys will be randomly generated on the initial start of the node if they are not found.\n\nAlternatively, to randomly generate the required keys, run the following command:\n\nNode identity is an identity assigned to the node. It is used to exchange encryption keys with other nodes. \n\n```\ndefradb keyring generate\n```\n\nTo import externally generated keys, run the following command:\n\n```\ndefradb keyring import \u003cname\u003e \u003cprivate-key-hex\u003e\n```\n\nTo learn more about the available options:\n\n```\ndefradb keyring --help\n```\n\n## Start\n\nStart a node by executing `defradb start`. Keep the node running while going through the following examples.\n\nVerify the local connection to the node works by executing `defradb client collection describe` in another terminal.\n\n## Configuration\n\nIn this document, we use the default configuration, which has the following behavior:\n\n- `~/.defradb/` is DefraDB's configuration and data directory\n- `client` command interacts with the locally running node\n- The GraphQL endpoint is provided at http://localhost:9181/api/v0/graphql\n\nThe GraphQL endpoint can be used with a GraphQL client (e.g., Altair) to conveniently perform requests (`query`, `mutation`) and obtain schema introspection.\n\nRead more about the configuration [here](./docs/config.md).\n\n## External port binding\n\nBy default the HTTP API and P2P network will use localhost. If you want to expose the ports externally you need to specify the addresses in the config or command line parameters.\n\n```\ndefradb start --p2paddr /ip4/0.0.0.0/tcp/9171 --url 0.0.0.0:9181\n```\n\n## Add a schema type\n\nSchemas are used to structure documents using a type system.\n\nIn the following examples, we'll be using a simple `User` schema type.\n\nAdd it to the database with the following command. By doing so, DefraDB generates the typed GraphQL endpoints for querying, mutation, and introspection.\n\n```shell\ndefradb client schema add '\n  type User {\n    name: String \n    age: Int \n    verified: Boolean \n    points: Float\n  }\n'\n```\n\nFind more examples of schema type definitions in the [examples/schema/](examples/schema/) folder.\n\n## Create a document\n\nSubmit a `mutation` request to create a document of the `User` type:\n\n```shell\ndefradb client query '\n  mutation {\n      create_User(input: {age: 31, verified: true, points: 90, name: \"Bob\"}) {\n          _docID\n      }\n  }\n'\n```\n\nExpected response:\n\n```json\n{\n  \"data\": {\n    \"create_User\": [\n      {\n        \"_docID\": \"bae-91171025-ed21-50e3-b0dc-e31bccdfa1ab\",\n      }\n    ]\n  }\n}\n```\n\n`_docID` is the document's unique identifier determined by its schema and initial data.\n\n## Query documents\n\nOnce you have populated your node with data, you can query it:\n\n```shell\ndefradb client query '\n  query {\n    User {\n      _docID\n      age\n      name\n      points\n    }\n  }\n'\n```\n\nThis query obtains *all* users and returns their fields `_docID, age, name, points`. GraphQL queries only return the exact fields requested.\n\nYou can further filter results with the `filter` argument.\n\n```shell\ndefradb client query '\n  query {\n    User(filter: {points: {_ge: 50}}) {\n      _docID\n      age\n      name\n      points\n    }\n  }\n'\n```\n\nThis returns only user documents which have a value for the `points` field *Greater Than or Equal to* (`_ge`) 50.\n\n## Obtain document commits\n\nDefraDB's data model is based on [MerkleCRDTs](https://arxiv.org/pdf/2004.00107.pdf). Each document has a graph of all of its updates, similar to Git. The updates are called `commit`s and are identified by `cid`, a content identifier. Each references its parents by their `cid`s.\n\nTo get the most recent commit in the MerkleDAG for the document identified as `bae-91171025-ed21-50e3-b0dc-e31bccdfa1ab`:\n\n```shell\ndefradb client query '\n  query {\n    _commits(docID: \"bae-91171025-ed21-50e3-b0dc-e31bccdfa1ab\") {\n      cid\n      delta\n      height\n      links {\n        cid\n        name\n      }\n    }\n  }\n'\n```\n\nIt returns a structure similar to the following, which contains the update payload that caused this new commit (`delta`) and any subgraph commits it references.\n\n```json\n{\n  \"data\": {\n    \"_commits\": [\n      {\n        \"cid\": \"bafybeifhtfs6vgu7cwbhkojneh7gghwwinh5xzmf7nqkqqdebw5rqino7u\",\n        \"delta\": \"pGNhZ2UYH2RuYW1lY0JvYmZwb2ludHMYWmh2ZXJpZmllZPU=\",\n        \"height\": 1,\n        \"links\": [\n          {\n            \"cid\": \"bafybeiet6foxcipesjurdqi4zpsgsiok5znqgw4oa5poef6qtiby5hlpzy\",\n            \"name\": \"age\"\n          },\n          {\n            \"cid\": \"bafybeielahxy3r3ulykwoi5qalvkluojta4jlg6eyxvt7lbon3yd6ignby\",\n            \"name\": \"name\"\n          },\n          {\n            \"cid\": \"bafybeia3tkpz52s3nx4uqadbm7t5tir6gagkvjkgipmxs2xcyzlkf4y4dm\",\n            \"name\": \"points\"\n          },\n          {\n            \"cid\": \"bafybeia4off4javopmxcdyvr6fgb5clo7m5bblxic5sqr2vd52s6khyksm\",\n            \"name\": \"verified\"\n          }\n        ]\n      }\n    ]\n  }\n}\n```\n\nObtain a specific commit by its content identifier (`cid`):\n\n```shell\ndefradb client query '\n  query {\n    _commits(cid: \"bafybeifhtfs6vgu7cwbhkojneh7gghwwinh5xzmf7nqkqqdebw5rqino7u\") {\n      cid\n      delta\n      height\n      links {\n        cid\n        name\n      }\n    }\n  }\n'\n```\n\n## DefraDB Query Language (DQL)\n\nDQL is compatible with GraphQL but features various extensions.\n\nRead its documentation at [docs.source.network](https://docs.source.network/defradb/references/query-specification/query-language-overview) to discover its filtering, ordering, limiting, relationships, variables, aggregate functions, and other useful features.\n\n## Peer-to-peer data synchronization\n\nDefraDB leverages peer-to-peer networking for data exchange, synchronization, and replication of documents and commits.\n\nWhen starting a node for the first time, a key pair is generated and stored in its \"root directory\" (`~/.defradb/` by default).\n\nEach node has a unique `PeerID` generated from its public key. This ID allows other nodes to connect to it.\n\nTo view your node's peer info:\n\n```shell\ndefradb client p2p info\n```\n\nThere are two types of peer-to-peer relationships supported: **pubsub** peering and **replicator** peering.\n\nPubsub peering *passively* synchronizes data between nodes by broadcasting *Document Commit* updates to the topic of the commit's document key. Nodes need to be listening on the pubsub channel to receive updates. This is for when two nodes *already* have shared a document and want to keep them in sync.\n\nReplicator peering *actively* pushes changes from a specific collection *to* a target peer.\n\n\u003cdetails\u003e\n\u003csummary\u003ePubsub example\u003c/summary\u003e\n\nPubsub peers can be specified on the command line using the `--peers` flag, which accepts a comma-separated list of peer [multiaddresses](https://docs.libp2p.io/concepts/addressing/). For example, a node at IP `192.168.1.12` listening on 9000 with PeerID `12D3KooWNXm3dmrwCYSxGoRUyZstaKYiHPdt8uZH5vgVaEJyzU8B` would be referred to using the multiaddress `/ip4/192.168.1.12/tcp/9000/p2p/12D3KooWNXm3dmrwCYSxGoRUyZstaKYiHPdt8uZH5vgVaEJyzU8B`.\n\nLet's go through an example of two nodes (*nodeA* and *nodeB*) connecting with each other over pubsub, on the same machine.\n\nStart *nodeA* with a default configuration:\n\n```shell\ndefradb start\n```\n\nObtain the node's peer info:\n\n```shell\ndefradb client p2p info\n```\n\nIn this example, we use `12D3KooWNXm3dmrwCYSxGoRUyZstaKYiHPdt8uZH5vgVaEJyzU8B`, but locally it will be different.\n\nFor *nodeB*, we provide the following configuration:\n\n```shell\ndefradb start --rootdir ~/.defradb-nodeB --url localhost:9182 --p2paddr /ip4/127.0.0.1/tcp/9172 --peers /ip4/127.0.0.1/tcp/9171/p2p/12D3KooWNXm3dmrwCYSxGoRUyZstaKYiHPdt8uZH5vgVaEJyzU8B\n```\n\nAbout the flags:\n\n- `--rootdir` specifies the root dir (config and data) to use\n- `--url` is the address to listen on for the client HTTP and GraphQL API\n- `--p2paddr` is a comma-separated list of multiaddresses to listen on for p2p networking\n- `--peers` is a comma-separated list of peer multiaddresses\n\nThis starts two nodes and connects them via pubsub networking.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eSubscription example\u003c/summary\u003e\n\nIt is possible to subscribe to updates on a given collection by using its ID as the pubsub topic. The ID of a collection is found as the field `collectionID` in one of its documents. Here we use the collection ID of the `User` type we created above. After setting up 2 nodes as shown in the [Pubsub example](#pubsub-example) section, we can subscribe to collections updates on *nodeA* from *nodeB* by using the following command:\n\n```shell\ndefradb client p2p collection add --url localhost:9182 bafkreibpnvkvjqvg4skzlijka5xe63zeu74ivcjwd76q7yi65jdhwqhske\n```\n\nMultiple collection IDs can be added at once.\n\n```shell\ndefradb client p2p collection add --url localhost:9182 \u003ccollection1ID\u003e,\u003ccollection2ID\u003e,\u003ccollection3ID\u003e\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eReplicator example\u003c/summary\u003e\n\nReplicator peering is targeted: it allows a node to actively send updates to another node. Let's go through an example of *nodeA* actively replicating to *nodeB*:\n\nStart *nodeA*:\n\n```shell\ndefradb start\n```\n\nIn another terminal, add this example schema to it:\n\n```shell\ndefradb client schema add '\n  type Article {\n    content: String\n    published: Boolean\n  }\n'\n```\n\nStart (or continue running from above) *nodeB*, that will be receiving updates:\n\n```shell\ndefradb start --rootdir ~/.defradb-nodeB --url localhost:9182 --p2paddr /ip4/0.0.0.0/tcp/9172\n```\n\nHere we *do not* specify `--peers` as we will manually define a replicator after startup via the `rpc` client command.\n\nIn another terminal, add the same schema to *nodeB*:\n\n```shell\ndefradb client schema add --url localhost:9182 '\n  type Article {\n    content: String\n    published: Boolean\n  }\n'\n```\n\nThen copy the peer info from *nodeB*:\n\n```shell\ndefradb client p2p info --url localhost:9182\n```\n\nSet *nodeA* to actively replicate the Article collection to *nodeB*:\n\n```shell\ndefradb client p2p replicator set -c Article \u003cnodeB_peer_info_json\u003e\n```\n\nAs we add or update documents in the Article collection on *nodeA*, they will be actively pushed to *nodeB*. Note that changes to *nodeB* will still be passively published back to *nodeA*, via pubsub.\n\u003c/details\u003e\n\n## Securing the HTTP API with TLS\n\nBy default, DefraDB will expose its HTTP API at `http://localhost:9181/api/v0`. It's also possible to configure the API to use TLS with self-signed certificates or Let's Encrypt.\n\nTo start defradb with self-signed certificates placed under `~/.defradb/certs/` with `server.key`\nbeing the public key and `server.crt` being the private key, just do:\n```shell\ndefradb start --tls\n```\n\nThe keys can be generated with your generator of choice or with `make tls-certs`.\n\nSince the keys should be stored within the DefraDB data and configuration directory, the recommended key generation command is `make tls-certs path=\"~/.defradb/certs\"`.\n\nIf not saved under `~/.defradb/certs` then the public (`pubkeypath`) and private (`privkeypaths`) key paths need to be explicitly defined in addition to the `--tls` flag or `tls` set to `true` in the config.\n\nThen to start the server with TLS, using your generated keys in custom path:\n```shell\ndefradb start --tls --pubkeypath ~/path-to-pubkey.key --privkeypath ~/path-to-privkey.crt\n\n```\n\n## Access Control System\nRead more about the DefraDB ACP System [here](/acp/README.md)\n\n## Supporting CORS\n\nWhen accessing DefraDB through a frontend interface, you may be confronted with a CORS error. That is because, by default, DefraDB will not have any allowed origins set. To specify which origins should be allowed to access your DefraDB endpoint, you can specify them when starting the database:\n```shell\ndefradb start --allowed-origins=https://yourdomain.com\n```\n\nIf running a frontend app locally on localhost, allowed origins must be set with the port of the app:\n```shell\ndefradb start --allowed-origins=http://localhost:3000\n```\n\nThe catch-all `*` is also a valid origin. \n\n## Backing up and restoring\n\nIt is currently not possible to do a full backup of DefraDB that includes the history of changes through the Merkle DAG. However, DefraDB currently supports a simple backup of the current data state in JSON format that can be used to seed a database or help with transitioning from one DefraDB version to another.\n\nTo backup the data, run the following command:\n```shell\ndefradb client backup export path/to/backup.json\n```\n\nTo pretty print the JSON content when exporting, run the following command:\n```shell\ndefradb client backup export --pretty path/to/backup.json\n```\n\nTo restore the data, run the following command:\n```shell\ndefradb client backup import path/to/backup.json\n```\n\n## Telemetry\n\nDefraDB has no telemetry reporting by default. To enable OpenTelemetry in DefraDB you must build with the `telemetry` tag set. To configure the HTTP exporters use the environment variables in the links below.\n\n[Metric exporter documentation](https://pkg.go.dev/go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp)\n\n[Trace exporter documentation](https://pkg.go.dev/go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp)\n\n## Community\n\nDiscuss on [Discord](https://discord.gg/w7jYQVJ) or [Github Discussions](https://github.com/sourcenetwork/defradb/discussions). The Source project is on [Twitter](https://twitter.com/sourcenetwrk).\n\n## Playground\n\nInstructions for the playground can be found [here](./playground/README.md).\n\n## Licensing\n\nDefraDB's code is released under the [Business Source License (BSL)](licenses/BSL.txt). It grants you the right to copy, modify, create derivative works, redistribute, and make non-production use of it. For additional uses, such as deploying in production on a private network, please contact license@source.network for a licensing agreement. Each dated version of the license turns into the more permissive Apache License v2.0 after four years. Please read the complete license before usage.\n\n## Contributors\n\n- John-Alan Simmons ([@jsimnz](https://github.com/jsimnz))\n- Andrew Sisley ([@AndrewSisley](https://github.com/AndrewSisley))\n- Shahzad Lone ([@shahzadlone](https://github.com/shahzadlone))\n- Orpheus Lummis ([@orpheuslummis](https://github.com/orpheuslummis))\n- Fred Carle ([@fredcarle](https://github.com/fredcarle))\n- Islam Aliev ([@islamaliev](https://github.com/islamaliev))\n- Keenan Nemetz ([@nasdf](https://github.com/nasdf))\n- Ivan Vercenco ([@iverc](https://github.com/iverc))\n- Chris Quigley ([@ChrisBQu](https://github.com/ChrisBQu))\n\nYou are invited to contribute to DefraDB. Follow the [Contributing guide](./CONTRIBUTING.md) to get started.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsourcenetwork%2Fdefradb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsourcenetwork%2Fdefradb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsourcenetwork%2Fdefradb/lists"}