{"id":37122692,"url":"https://github.com/nuts-foundation/go-leia","last_synced_at":"2026-03-17T09:09:54.720Z","repository":{"id":39996587,"uuid":"333363214","full_name":"nuts-foundation/go-leia","owner":"nuts-foundation","description":"Go Lightweight Embedded Indexed (JSON) Archive","archived":false,"fork":false,"pushed_at":"2026-03-16T18:35:59.000Z","size":261,"stargazers_count":7,"open_issues_count":3,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2026-03-17T01:53:59.397Z","etag":null,"topics":["bbolt","document-database","go","json","key-value"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nuts-foundation.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-01-27T09:09:24.000Z","updated_at":"2025-08-20T08:58:38.000Z","dependencies_parsed_at":"2023-11-30T11:29:53.155Z","dependency_job_id":"294b1be2-a6a7-46ff-9351-d157b7af8f74","html_url":"https://github.com/nuts-foundation/go-leia","commit_stats":null,"previous_names":[],"tags_count":33,"template":false,"template_full_name":null,"purl":"pkg:github/nuts-foundation/go-leia","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nuts-foundation%2Fgo-leia","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nuts-foundation%2Fgo-leia/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nuts-foundation%2Fgo-leia/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nuts-foundation%2Fgo-leia/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nuts-foundation","download_url":"https://codeload.github.com/nuts-foundation/go-leia/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nuts-foundation%2Fgo-leia/sbom","scorecard":{"id":550211,"data":{"date":"2025-08-11","repo":{"name":"github.com/nuts-foundation/go-leia","commit":"29f6c7b1508a3e74d39ab311a7c1489386b1500d"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.6,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Code-Review","score":4,"reason":"Found 12/27 approved changesets -- score normalized to 4","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: COPYING:0","Info: FSF or OSI recognized license: GNU General Public License v3.0: COPYING:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 18 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-20T10:36:54.060Z","repository_id":39996587,"created_at":"2025-08-20T10:36:54.060Z","updated_at":"2025-08-20T10:36:54.060Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30619818,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-17T08:10:05.930Z","status":"ssl_error","status_checked_at":"2026-03-17T08:10:04.972Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bbolt","document-database","go","json","key-value"],"created_at":"2026-01-14T14:10:46.134Z","updated_at":"2026-03-17T09:09:54.714Z","avatar_url":"https://github.com/nuts-foundation.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build](https://circleci.com/gh/nuts-foundation/go-leia.svg?style=svg)](https://circleci.com/gh/nuts-foundation/go-leia)\n[![Code coverage](https://qlty.sh/gh/nuts-foundation/projects/go-leia/coverage.svg)](https://qlty.sh/gh/nuts-foundation/projects/go-leia)\n[![Maintainability](https://qlty.sh/gh/nuts-foundation/projects/go-leia/maintainability.svg)](https://qlty.sh/gh/nuts-foundation/projects/go-leia)\n\n# go-leia\n\nGo Lightweight Embedded Indexed (JSON) Archive\n\ngo-leia is built upon [bbolt](https://github.com/etcd-io/bbolt). \nIt adds indexed based search capabilities for JSON documents to the key-value store.\n\nThe goal is to provide a simple and fast way to find relevant JSON documents using an embedded Go key-value store.\n\n## Table of Contents    \n\n- [Installing](#installing)\n- [Opening a database](#opening-a-database)\n- [Collections](#collections)\n    - [Writing](#writing)\n    - [Reading](#reading)\n    - [Searching](#searching)\n- [Indexing](#indexing)\n    - [Alias option](#alias-option)\n    - [Transform option](#transform-option)\n    - [Tokenizer option](#tokenizer-option)\n- [Query Performance Monitoring](#query-performance-monitoring)\n\n## Installing\n\nInstall Go and run `go get`:\n\n```sh\n$ go get github.com/nuts-foundation/go-leia\n```\n\nWhen using Go \u003e 1.16, Go modules will probably require you to install additional dependencies. \n\n```sh\n$ go get github.com/stretchr/testify\n$ go get github.com/tidwall/gjson\n$ go get go.etcd.io/bbolt\n```\n\n## Opening a database\n\nOpening a database only requires a file location for the bbolt db.\n\n```go\npackage main\n\nimport (\n\t\"log\"\n\t\n\t\"github.com/nuts-foundation/go-leia\"\n)\n\nfunc main() {\n\t// Open the my.db data file in your current directory.\n\t// It will be created if it doesn't exist using filemode 0600 and default bbolt options.\n\tstore, err := leia.NewStore(\"my.db\")\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tdefer store.Close()\n\n\t...\n}\n```\n\n## Collections\n\nLeia adds collections to bbolt. Each collection has its own bucket where documents are stored.\nAn index is also only valid for a single collection.\n\nTo create a collection:\n\n```go\nfunc main() {\n    store, err := leia.NewStore(\"my.db\")\n\t...\n\t\n    // if a collection doesn't exist, it'll be created for you.\n    // the underlying buckets are created when a document is added.\n    collection := store.Collection(\"credentials\")\n}\n```\n\n### Writing\n\nWriting a document to a collection is straightforward:\n\n```go\nfunc main() {\n    store, err := leia.NewStore(\"my.db\")\n    collection := store.Collection(\"credentials\")\n\t...\n\t\n    // leia uses leia.Documents as arguments. Which is basically a []byte\n    documents := make([]leia.Document, 1)\n    documents[1] = leia.DocumentFromString(\"{...some json...}\")\n    \n    // documents are added by slice\n    collection.Add(documents)\n}\n```\n\nDocuments are added by slice. Each operation is done within a single bbolt transaction.\nBBolt is a key-value store, so you've probably noticed the key is missing as an argument.\nLeia computes the sha-1 of the document and uses that as key.\n\nTo get the key when needed:\n\n```go\nfunc main() {\n    store, err := leia.NewStore(\"my.db\")\n    collection := store.Collection(\"credentials\")\n    ...\n    \n    // define your document\n    document := leia.DocumentFromString(\"{...some json...}\")\n    \n    // retrieve a leia.Reference (also a []byte)\n    reference := collection.Reference(document)\n}\n```\n\nDocuments can also be removed:\n\n```go\nfunc main() {\n    store, err := leia.NewStore(\"my.db\")\n    collection := store.Collection(\"credentials\")\n    ...\n    \n    // define your document\n    document := leia.DocumentFromString(\"{...some json...}\")\n    \n    // remove a document using a leia.Document\n    err := collection.Delete(document)\n}\n```\n\n### Reading\n\nA document can be retrieved by reference:\n\n```go\nfunc main() {\n    store, err := leia.NewStore(\"my.db\")\n    collection := store.Collection(\"credentials\")\n    ...\n    \n    // document by reference, it returns nil when not found\n    document, err := collection.Get(reference)\n}\n```\n\n### Searching\n\nThe major benefit of leia is searching.\nThe performance of a search greatly depends on the available indices on a collection.\nIf no index matches the query, a bbolt cursor is used to loop over all documents in the collection.\n\nLeia supports equal, prefix and range queries. \nThe first argument for each matcher is the JSON path using the syntax from [gjson](github.com/tidwall/gjson).\nOnly basic path syntax is used. There is no support for wildcards or comparison operators.\nThe second argument is the value to match against.\nLeia can only combine query terms using **AND** logic.\n\n```go\nfunc main() {\n    ...\n    \n    // define a new query\n    query := leia.New(leia.Eq(\"subject\", \"some_value\")).\n                  And(leia.Range(\"some.path.#.amount\", 1, 100))\n}\n```\n\nGetting results can be done with either `Find` or `Iterate`. \n`Find` will return a slice of documents. `Iterate` will allow you to pass a `DocWalker` which is called for each hit.\n\n```go\nfunc main() {\n    ...\n    \n    // get a slice of documents\n    documents, err := collection.Find(query)\n    \n    // use a DocWalker\n    walker := func(ref []byte, doc []byte) error {\n    \t// do something with the document\n    }\n    err := collection.Iterate(query, walker)\n}\n```\n\n## Indexing\n\nIndexing JSON documents is where the real added value of leia lies.\nFor each collection multiple indices can be added.\nEach added index will slow down write operations.\n\nAn index can be added and removed:\n\n```go\nfunc main() {\n    ...\n    \n    // define the index\n    index := leia.NewIndex(\"compound\",\n                leia.NewFieldIndexer(\"subject\"),\n                leia.NewFieldIndexer(\"some.path.#.amount\"),\n    )\n    \n    // add it to the collection\n    err := collection.AddIndex(index)\n    \n    // remove it from the collection\n    err := collection.DropIndex(\"compound\")\n}\n```\n\nThe argument for `NewFieldIndexer` uses the same notation as the query parameter, also without wildcards or comparison operators.\nAdding an index will trigger a re-index of all documents in the collection.\nAdding an index with a duplicate name will ignore the index.\n\n### Alias option\n\nLeia support indexing JSON paths under an **alias**.\nAn alias can be used to index different documents but use a single query to find both.\n\n```go\nfunc main() {\n    ...\n    \n    // define the index for credentialX\n    indexX := leia.NewIndex(\"credentialX\", leia.NewFieldIndexer(\"credentialSubject.id\", leia.AliasOption{Alias: \"subject\"}))\n    // define the index for credentialY\n    indexY := leia.NewIndex(\"credentialY\", leia.NewFieldIndexer(\"credentialSubject.organization.id\", leia.AliasOption{Alias: \"subject\"}))\n    \n    ...\n\n    // define a new query\n    query := leia.New(leia.Eq(\"subject\", \"some_value\"))\n}\n```\n\nThe example above defines two indices to a collection, each index has a different JSON path to be indexed.\nBoth indices will be used when the given query is executed, resulting in documents that match either index.\n\n### Transform option\n\nA transformer can be defined for a `FieldIndexer`. A transformer will transform the indexed value and query parameter.\nThis can be used to allow case-insensitive search or add a soundex style index.\n\n```go\nfunc main() {\n    ...\n    \n    // This index transforms all values to lowercase\n    index := leia.NewIndex(\"credential\", leia.NewFieldIndexer(\"subject\", leia.TransformOption{Transform: leia.ToLower}))\n    \n    ...\n\n    // these queries will yield the same result\n    query1 := leia.New(leia.Eq(\"subject\", \"VALUE\"))\n    query2 := leia.New(leia.Eq(\"subject\", \"value\"))\n}\n```\n\n### Tokenizer option\n\nSometimes JSON fields contain a whole text. \nLeia has a tokenizer option to split a value at a JSON path into multiple keys to be indexed.\nFor example, the sentence `\"The quick brown fox jumps over the lazy dog\"` could be tokenized so the document can easily be found when the term `fox` is used in a query.\nA more advanced tokenizer could also remove common words like `the`.\n\n```go\nfunc main() {\n    ...\n    \n    // This index transforms all values to lowercase\n    index := leia.NewIndex(\"credential\", leia.NewFieldIndexer(\"text\", leia.TokenizerOption{Tokenizer: leia.WhiteSpaceTokenizer}))\n    \n    ...\n\n    // will match {\"text\": \"The quick brown fox jumps over the lazy dog\"}\n    query := leia.New(leia.Eq(\"subject\", \"fox\"))\n}\n```\n\nAll options can be combined.\n\n## Query Performance Monitoring\n\nLeia can track query performance and identify missing or suboptimal indexes through callbacks.\nThis helps optimize query performance by providing actionable insights about index usage.\n\n```go\nfunc main() {\n    ...\n    \n    // Configure callbacks to monitor query performance\n    callbacks := leia.QueryStatsCallbacks{\n        OnIndexProblem: func(stats leia.IndexStats) {\n            if stats.IndexUsed == \"\" {\n                // Full table scan detected - no index was used\n                log.Printf(\"Missing index! Query: %s, Unindexed fields: %v\", \n                    stats.Query.String(), stats.UnindexedFields)\n            } else if stats.FilterEfficiency \u003c 0.1 {\n                // Suboptimal index - low efficiency\n                log.Printf(\"Inefficient query! Index: %s, Efficiency: %.0f%%, Add fields: %v\", \n                    stats.IndexUsed, stats.FilterEfficiency*100, stats.UnindexedFields)\n            }\n        },\n        SuboptimalIndexThreshold: 3,\n    }\n    \n    store, err := leia.NewStore(\"my.db\", leia.WithQueryStatsCallbacks(callbacks))\n}\n```\n\nThe callback provides detailed statistics including:\n- Number of documents scanned vs matched\n- Size of scanned and matched documents (in bytes)\n- Filter efficiency (for indexed queries)\n- Suggested fields for creating indexes\n- Query string representation (with masked values for security)\n\nThe `SuboptimalIndexThreshold` determines when to report suboptimal indexes. \nThe callback triggers when wasted scans (scanned - matched) exceed this threshold (strictly greater than). \nFor example, with a threshold of 3, the callback triggers when 4 or more documents \nwere scanned but didn't match the query criteria.\n\nFor a complete example, see [examples/query_stats](examples/query_stats).\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnuts-foundation%2Fgo-leia","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnuts-foundation%2Fgo-leia","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnuts-foundation%2Fgo-leia/lists"}