{"id":19732354,"url":"https://github.com/terminusdb/research_topics","last_synced_at":"2026-03-19T11:20:59.935Z","repository":{"id":101654931,"uuid":"420011437","full_name":"terminusdb/research_topics","owner":"terminusdb","description":"Running list of research topics","archived":false,"fork":false,"pushed_at":"2021-11-15T15:53:51.000Z","size":7,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-10-08T01:30:22.894Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/terminusdb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-22T07:49:48.000Z","updated_at":"2025-01-06T12:51:56.000Z","dependencies_parsed_at":"2023-05-03T23:30:20.876Z","dependency_job_id":null,"html_url":"https://github.com/terminusdb/research_topics","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/terminusdb/research_topics","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terminusdb%2Fresearch_topics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terminusdb%2Fresearch_topics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terminusdb%2Fresearch_topics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terminusdb%2Fresearch_topics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/terminusdb","download_url":"https://codeload.github.com/terminusdb/research_topics/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terminusdb%2Fresearch_topics/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29508740,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-16T09:05:14.864Z","status":"ssl_error","status_checked_at":"2026-02-16T08:55:59.364Z","response_time":115,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T00:25:58.600Z","updated_at":"2026-02-16T13:03:34.876Z","avatar_url":"https://github.com/terminusdb.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Research Topics\n\nRunning and updated list of research topics regarding efficient\nstorage and search of versioned graphs.\n\n# Data Structures\n\nWhich data-structures support the following operations on versioned\ngraphs with large numbers of commits efficiently.\n\nWe view an edge as a tuple (s,p,o) for s∈S p∈P o∈O for countable\nalphabets S (node), P (edge) , V (value) where O := S⊕V a disjoint\nunion between values and nodes.\n\n1. Search by (s,?,?), (s,p,?), (?,p,o) and (?,?,o)\n2. Check cardinality of (s,p,?), (s,?,?), (?,p,o), (?,?,o)\n3. search additions, deletions at a commit as with 1\n4. add / delete edges\n\n## Some potential approaches\n\n### Delta compression\n\nCurrently in TerminusDB we use an\n[HDT](https://en.wikipedia.org/wiki/HDT_(data_format)) style format\nfor the initial commit, and pairs of HDT style formats for each delta,\nstoring additions and deletions.\n\nLarge numbers of small writes will have different behaviours than\nsmall numbers of large writes. When do we find it is cheaper to\ncompress delta-stacks. If you can do \"short-cut\" queries over a delta\ncompression you get much better behaviour.\n\n```\n                deltas      head\ninitial       ↓   ↓   ↓    ↙\ncommit →  ◯ - ◯ - ◯ - ◯ - ◯\n              |       |\n              \\―――――― ⬤\n                      ↑\n                 delta compression\n\n```\n\nWhere should these delta compressions be introduced to maximise\nperformance with different usage profiles? How should we characterise\nthese profiles?\n\n### Persistent Radix Tree\n\n[Persistent Radix Trees](https://ankurdave.com/dl/part-tr.pdf) give a\nmethod which could work for a commit based data structure. The\ntechnique of extending radix trees to\n[k-dimensions](https://www.cs.umd.edu/~hjs/mkbook/chapter1.pdf) and\nequiping them with range queries may provide a faster and more compact\nrepresentation.\n\nHowever this approach will probably also benefit from delta-compressions.\n\n### Persistent k2-tree\n\n[k2 trees](https://arxiv.org/abs/2002.11622) have been shown to\nrepresent large networks very efficiently and support the types of\nqueries required. However they have not been described with a\npersistent extensions. Other persistent succinct data structures have\nbeen described and it may be possible to adapt these approaches to k-2\ntrees.\n\nAlternatively the chained approach used with HDT above can also be\nused with k2 trees.\n\n## Efficient subgraph search\n\nWhile networks are extremely useful ways of representing data, it is\noften the case that we want to have sub-graphs of the data treated as\na coherent bundle. In TerminusDB we represent sub-graphs as\ninterconvertable to JSON documents. Essentially we require the\nsub-graph to be a DAG and the full network can only exist between\nthese DAG sub-graphs.\n\nIt is convenient to treat these sub-graphs as having links in, and\nlinks out, such that the entire document can be thought of as a\npseudo-node, somewhat like the way we treat hypertext documents on the\nweb (where outgoing links are actually embedded in a structured\ndocument).\n\nEfficient techniques of treating the subgraphs as psuedo-nodes for\nnetwork analysis have not yet been developed and could be an\ninteresting problem.\n\n## Patch manipulation\n\nPatches are important for understanding what updates are taking place,\nespecially in the presence of concurrency. When patches happen at\ndifferent repositories with different cadences, sometimes we must\ncalculate whether a merge is possible without conflict by checking the\ncommutivity of patches.\n\n- What algorithm should be used to check whether an insert / delete\n  patch over a subgraph commutes with another subgraph patch?\n- What types of patches will improve commutivity for tree+list (JSON)\n  structures.\n- What sub-space of JSON style can be equiped with CRDT patch logic to\n  speed up checks of commutivity?\n\n## Shrinking delta dictionaries\nAs described above, we use a delta compression strategy, where a range\nof data layers can be turned into one single data layer for more\nefficient querying. This compression is transparent. we ensure that\nthe resulting compressed layer behaves the same way as the original\nrange. In particular, we ensure that any value that previously existed\n(even if it has since been deleted) will be assigned the same id when\nit appears in a triple, regardless of whether you use the original\nlayer stack or the delta layer as the basis of your operation. This\nensures we can do delta compression at the same time as operations on\nthe graph are occuring, as we can be sure that newly built layers will\nstill work with our delta layer.\n\nOne disadvantage of this approach is that we never throw away a\nvalue. If we did throw away unused values, then when a new layer tries\nto reuse these values, it'd allocate a new ID if built on top of a\ndelta-compressed layer, but reuse the existing ID if built on top of\nthe original layer stack. Therefore, because we wish to retain\nequivalent behavior, we need to keep around old, unused values. This\nmeans that if a particular database has a lot of changes in data\n(rather than a lot of changes in links), over time, this database will\naccrue a lot of unused dictionary entries.\n\nContrast this with a squash operation, where we build a completely new\nlayer without taking care to be compatible. In this case, we can\nactually throw away old values, and build new dictionaries where these\nvalues simply do not occur, resulting in smaller dictionaries. But\nsquashes cannot be done concurrently with graph modifications. They\nhave to stop the world while they're built, preventing new layers from\nbeing constructed on top of the old data until that operation is done.\n\nIt may be worthwhile to see if there's some way to create a delta\nlayer which does omit unused dictionary values, but which can still be\nbuilt concurrently. Possible solutions:\n- have special layers where particular ids are permanently released\n- Use squash and rebuild any new concurrently built layers afterwards\n\nA related problem is that over time IDs get larger, requiring more\nstorage to hold them. It would be good if it was possible to reorder\nIDs such that the most used IDs are small.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fterminusdb%2Fresearch_topics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fterminusdb%2Fresearch_topics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fterminusdb%2Fresearch_topics/lists"}