{"id":13564850,"url":"https://github.com/timshannon/bolthold","last_synced_at":"2025-05-15T15:04:18.928Z","repository":{"id":47099675,"uuid":"55197411","full_name":"timshannon/bolthold","owner":"timshannon","description":"BoltHold is an embeddable NoSQL store for Go types built on BoltDB","archived":false,"fork":false,"pushed_at":"2024-03-14T19:40:07.000Z","size":340,"stargazers_count":657,"open_issues_count":9,"forks_count":47,"subscribers_count":23,"default_branch":"master","last_synced_at":"2025-03-31T16:13:12.668Z","etag":null,"topics":["boltdb","bucket","go","golang","nosql","query-criteria"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/timshannon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-04-01T02:22:59.000Z","updated_at":"2025-02-26T21:12:34.000Z","dependencies_parsed_at":"2024-06-18T12:31:11.977Z","dependency_job_id":"174b4b28-9dda-495a-b510-1a3dfcdcd35f","html_url":"https://github.com/timshannon/bolthold","commit_stats":{"total_commits":196,"total_committers":10,"mean_commits":19.6,"dds":0.4336734693877551,"last_synced_commit":"30aac695092830f26fe56b802bfcc1600e09a44c"},"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timshannon%2Fbolthold","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timshannon%2Fbolthold/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timshannon%2Fbolthold/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timshannon%2Fbolthold/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/timshannon","download_url":"https://codeload.github.com/timshannon/bolthold/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247694875,"owners_count":20980733,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["boltdb","bucket","go","golang","nosql","query-criteria"],"created_at":"2024-08-01T13:01:36.982Z","updated_at":"2025-04-07T17:06:44.595Z","avatar_url":"https://github.com/timshannon.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# BoltHold\n\n[![Go](https://github.com/timshannon/bolthold/actions/workflows/go.yml/badge.svg)](https://github.com/timshannon/bolthold/actions/workflows/go.yml)[![GoDoc](https://godoc.org/github.com/timshannon/bolthold?status.svg)](https://pkg.go.dev/github.com/timshannon/bolthold) [![Coverage Status](https://coveralls.io/repos/github/timshannon/bolthold/badge.svg?branch=master)](https://coveralls.io/github/timshannon/bolthold?branch=master) [![Go Report Card](https://goreportcard.com/badge/github.com/timshannon/bolthold)](https://goreportcard.com/report/github.com/timshannon/bolthold)\n\nBoltHold is a simple querying and indexing layer on top of a Bolt DB instance. For a similar library built on [Badger](https://github.com/dgraph-io/badger) see [BadgerHold](https://github.com/timshannon/badgerhold).\n\nThe goal is to create a simple, higher level interface on top of Bolt DB that simplifies dealing with Go Types and finding data, but exposes the underlying Bolt DB for customizing as you wish. By default the encoding used is Gob, so feel free to use the GobEncoder/Decoder interface for faster serialization. Or, alternately, you can use any serialization you want by supplying encode / decode funcs to the `Options` struct on Open.\n\nOne Go Type will have one bucket, and multiple index buckets in a BoltDB file, so you can store multiple Go Types in the same database.\n\n## Why not just use Bolt DB directly?\n\nI love BoltDB, and I've used it in several projects. However, I find myself writing the same code over and over again, for encoding and decoding objects and searching through data. I figure formalizing how I've been using BoltDB and including tests and benchmarks will, at a minimum, be useful to me. Maybe it'll be useful to others as well.\n\n## Indexes\n\nIndexes allow you to skip checking any records that don't meet your index criteria. If you have 1000 records and only 10 of them are of the Division you want to deal with, then you don't need to check to see if the other 990 records match your query criteria if you create an index on the Division field. The downside of an index is added disk reads and writes on every write operation. For read heavy operations datasets, indexes can be very useful.\n\nIn every BoltHold store, there will be a reserved bucket _\\_indexes_ which will be used to hold indexes that point back to another bucket's Key system. Indexes will be defined by setting the `boltholdIndex` struct tag on a field in a type.\n\n```Go\ntype Person struct {\n\tName string\n\tDivision string `boltholdIndex:\"Division\"`\n}\n\n```\n\nThis means that there will be an index created for `Division` that will contain the set of unique divisions, and the main record keys they refer to. More information on how indexes work can be found [here](https://github.com/timshannon/bolthold/issues/36#issuecomment-414720348)\n\nOptionally, you can implement the `Storer` interface, to specify your own indexes, rather than using the `boltholdIndex` struct tag.\n\n### Slice Indexes\n\nWhen you create an index on a slice of items, by default it may not do what you expect. Consider the following records:\n\n| ID  | Name  | Categories       |\n| --- | ----- | ---------------- |\n| 1   | John  | red, green, blue |\n| 2   | Bill  | red, purple      |\n| 3   | Jane  | red, orange      |\n| 4   | Brian | red, purple      |\n\nYou may expect your `Categories` index to look like the following:\n\n| Categories | ID         |\n| ---------- | ---------- |\n| red        | 1, 2, 3, 4 |\n| green      | 1          |\n| blue       | 1          |\n| purple     | 2, 4       |\n| orange     | 3          |\n\nBut they'll actually look like this:\n\n| Categories       | ID   |\n| ---------------- | ---- |\n| red, green, blue | 1    |\n| red, purple      | 2, 4 |\n| red, orange      | 3    |\n\nSo if you did a query like this:\n\n```Go\nbh.Where(\"Categories\").Contains(\"red\").Index(\"Categories\")\n```\n\nIt'll work, but you'll be reading more records than you'd expect. You'd only \"save reads\" on values where the list of categories exactly match.\n\nIf instead you want to index each individual item in the slice, you can use the struct tag `boltholdSliceIndex`. It will then individually index each item in the slice, and potentially give you performance benefits if you have a lot of overlap with individual items in your sliced fields.\n\nBe sure to benchmark both a regular index and a sliced index to see which performs better for your specific dataset.\n\n## Queries\n\nQueries are chain-able constructs that filters out any data that doesn't match it's criteria. An index will be used if the `.Index()` chain is called, otherwise bolthold won't use any index.\n\nQueries will look like this:\n\n```Go\ns.Find(\u0026result, bolthold.Where(\"FieldName\").Eq(value).And(\"AnotherField\").Lt(AnotherValue).Or(bolthold.Where(\"FieldName\").Eq(anotherValue)))\n```\n\nFields must be exported, and thus always need to start with an upper-case letter. Available operators include:\n\n- Equal - `Where(\"field\").Eq(value)`\n- Not Equal - `Where(\"field\").Ne(value)`\n- Greater Than - `Where(\"field\").Gt(value)`\n- Less Than - `Where(\"field\").Lt(value)`\n- Less than or Equal To - `Where(\"field\").Le(value)`\n- Greater Than or Equal To - `Where(\"field\").Ge(value)`\n- In - `Where(\"field\").In(val1, val2, val3)`\n- IsNil - `Where(\"field\").IsNil()`\n- Regular Expression - `Where(\"field\").RegExp(regexp.MustCompile(\"ea\"))`\n- Matches Function\n  - `Where(\"field\").MatchFunc(func(ra *RecordAccess) (bool, error)) // see RecordAccess Type`\n  - `Where(\"field\").MatchFunc(func(m *MyType) (bool, error))`\n  - `Where(\"field\").MatchFunc(func(field string) (bool, error))`\n- Skip - `Where(\"field\").Eq(value).Skip(10)`\n- Limit - `Where(\"field\").Eq(value).Limit(10)`\n- SortBy - `Where(\"field\").Eq(value).SortBy(\"field1\", \"field2\")`\n- Reverse - `Where(\"field\").Eq(value).SortBy(\"field\").Reverse()`\n- Index - `Where(\"field\").Eq(value).Index(\"indexName\")`\n- Not - `Where(\"field\").Not().In(val1, val2, val3)`\n- Contains - `Where(\"field\").Contains(val1)`\n- ContainsAll - `Where(\"field\").Contains(val1, val2, val3)`\n- ContainsAny - `Where(\"field\").Contains(val1, val2, val3)`\n- HasKey - `Where(\"field\").HasKey(val1) // to test if a Map value has a key`\n\nAn empty / zero value query matches against all records, because it has no critiera.  You can then use `Skip` and `Limit` to page through all records in your dataset:\n```Go\nq := \u0026bolthold.Query{}\n\nerr := store.Find(\u0026result, q.Skip(10).Limit(50))\n```\n\nIf you want to run a query's criteria against the Key value, you can use the `bolthold.Key` constant:\n\n```Go\nstore.Find(\u0026result, bolthold.Where(bolthold.Key).Ne(value))\n```\n\nYou can access nested structure fields in queries like this:\n\n```Go\ntype Repo struct {\n  Name string\n  Contact ContactPerson\n}\n\ntype ContactPerson struct {\n  Name string\n}\n\nstore.Find(\u0026repo, bolthold.Where(\"Contact.Name\").Eq(\"some-name\")\n```\n\nInstead of passing in a specific value to compare against in a query, you can compare against another field in the same\nstruct. Consider the following struct:\n\n```Go\ntype Person struct {\n\tName string\n\tBirth time.Time\n\tDeath time.Time\n}\n```\n\nIf you wanted to find any invalid records where a Person's death was before their birth, you could do the following:\n\n```Go\nstore.Find(\u0026result, bolthold.Where(\"Death\").Lt(bolthold.Field(\"Birth\")))\n```\n\nQueries can be used in more than just selecting data. You can delete or update data that matches a query.\n\nUsing the example above, if you wanted to remove all of the invalid records where Death \u003c Birth:\n\n```Go\n// you must pass in a sample type, so BoltHold knows which bucket to use and what indexes to update\nstore.DeleteMatching(\u0026Person{}, bolthold.Where(\"Death\").Lt(bolthold.Field(\"Birth\")))\n```\n\nOr if you wanted to update all the invalid records to flip/flop the Birth and Death dates:\n\n```Go\nstore.UpdateMatching(\u0026Person{}, bolthold.Where(\"Death\").Lt(bolthold.Field(\"Birth\")), func(record interface{}) error {\n\tupdate, ok := record.(*Person) // record will always be a pointer\n\tif !ok {\n\t\treturn fmt.Errorf(\"Record isn't the correct type!  Wanted Person, got %T\", record)\n\t}\n\n\tupdate.Birth, update.Death = update.Death, update.Birth\n\n\treturn nil\n})\n```\n\nIf you simply want to count the number of records returned by a query use the `Count` method:\n\n```Go\n// need to pass in empty datatype so bolthold knows what type to count\ncount, err := store.Count(\u0026Person{}, bolthold.Where(\"Death\").Lt(bolthold.Field(\"Birth\")))\n```\n\n### Keys in Structs\n\nA common scenario is to store the bolthold Key in the same struct that is stored in the boltDB value. You can automatically populate a record's Key in a struct by using the `boltholdKey` struct tag when running `Find` queries.\n\nAnother common scenario is to insert data with an auto-incrementing key assigned by the database. When performing an `Insert`, if the type of the key matches the type of the `boltholdKey` tagged field, the data is passed in by reference, **and** the field's current value is the zero-value for that type, then it is set on the data _before_ insertion.\n\n```Go\ntype Employee struct {\n\tID string `boltholdKey:\"ID\"`  // the tagName isn't required, but some linters will complain without it\n\tFirstName string\n\tLastName string\n\tDivision string\n\tHired time.Time\n}\n```\n\nBolthold assumes only one of such struct tags exists. If a value already exists in the key field, it will be overwritten.\n\nIf you want to insert an auto-incrementing Key you can pass the `bolthold.NextSequence()` func as the Key value.\n\n```Go\nerr := store.Insert(bolthold.NextSequence(), data)\n```\n\nThe key value will be a `uint64`.\n\nIf you want to know the value of the auto-incrementing Key that was generated using `bolthold.NextSequence()`, then make sure to pass your data by reference and that the `boltholdKey` tagged field is of type `uint64`.\n\n```Go\nerr := store.Insert(bolthold.NextSequence(), \u0026data)\n```\n\n### Slices in Structs and Queries\n\nWhen querying slice fields in structs you can use the `Contains`, `ContainsAll` and `ContainsAny` criterion.\n\n```Go\nval := struct {\n    Set []string\n}{\n    Set: []string{\"1\", \"2\", \"3\"},\n}\nbh.Where(\"Set\").Contains(\"1\") // true\nbh.Where(\"Set\").ContainsAll(\"1\", \"3\") // true\nbh.Where(\"Set\").ContainsAll(\"1\", \"3\", \"4\") // false\nbh.Where(\"Set\").ContainsAny(\"1\", \"7\", \"4\") // true\n```\n\nThe `In`, `ContainsAll` and `ContainsAny` critierion accept a slice of `interface{}` values. This means you can build your queries by passing in your values as arguments:\n\n```\nwhere := bolthold.Where(\"Id\").In(\"1\", \"2\", \"3\")\n```\n\nHowever if you have an existing slice of values to test against, you can't pass in that slice because it is not of type\n`[]interface{}`.\n\n```Go\nt := []string{\"1\", \"2\", \"3\", \"4\"}\nwhere := bolthold.Where(\"Id\").In(t...) // compile error\n```\n\nInstead you need to copy your slice into another slice of empty interfaces:\n\n```Go\nt := []string{\"1\", \"2\", \"3\", \"4\"}\ns := make([]interface{}, len(t))\nfor i, v := range t {\n    s[i] = v\n}\nwhere := bolthold.Where(\"Id\").In(s...)\n```\n\nYou can use the helper function `bolthold.Slice` which does exactly that.\n\n```Go\nt := []string{\"1\", \"2\", \"3\", \"4\"}\nwhere := bolthold.Where(\"Id\").In(bolthold.Slice(t)...)\n```\n\n### Unique Constraints\n\nYou can create a unique constraint on a given field by using the `boltholdUnique:\"ConstraintName\"` struct tag:\n\n```Go\ntype User struct {\n  Name string\n  Email string `boltholdUnique:\"UniqueEmail\"` // this field will be indexed with a unique constraint\n}\n```\n\nThe example above will only allow one record of type `User` to exist with a given `Email` field. Any insert, update or upsert that would violate that constraint will fail and return the `bolthold.ErrUniqueExists` error.\n\n\n### ForEach\n\nWhen working with large datasets, you may not want to have to store the entire dataset in memory. It's be much more efficient to work with a single record at a time rather than grab all the records and loop through them, which is what cursors are used for in databases. In BoltHold you can accomplish the same thing by calling ForEach:\n\n```Go\nerr := store.ForEach(boltholdWhere(\"Id\").Gt(4), func(record *Item) error {\n\t// do stuff with record\n\n\t// if you return an error, then the query will stop iterating through records\n\n\treturn nil\n})\n```\n\n### Aggregate Queries\n\nAggregate queries are queries that group results by a field. For example, lets say you had a collection of employees:\n\n```Go\ntype Employee struct {\n\tFirstName string\n\tLastName string\n\tDivision string\n\tHired time.Time\n}\n```\n\nAnd you wanted to find the most senior (first hired) employee in each division:\n\n```Go\nresult, err := store.FindAggregate(\u0026Employee{}, nil, \"Division\") //nil query matches against all records\n```\n\nThis will return a slice of `Aggregate Result` from which you can extract your groups and find Min, Max, Avg, Count, etc.\n\n```Go\nfor i := range result {\n\tvar division string\n\temployee := \u0026Employee{}\n\n\tresult[i].Group(\u0026division)\n\tresult[i].Min(\"Hired\", employee)\n\n\tfmt.Printf(\"The most senior employee in the %s division is %s.\\n\",\n\t\tdivision, employee.FirstName + \" \" + employee.LastName)\n}\n```\n\nAggregate queries become especially powerful when combined with the sub-querying capability of `MatchFunc`.\n\nMany more examples of queries can be found in the [find_test.go](https://github.com/timshannon/bolthold/blob/master/find_test.go) file in this repository.\n\n## Comparing\n\nJust like with Go, types must be the same in order to be compared with each other. You cannot compare an int to a int32. The built-in Go comparable types (ints, floats, strings, etc) will work as expected. Other types from the standard library can also be compared such as `time.Time`, `big.Rat`, `big.Int`, and `big.Float`. If there are other standard library types that I missed, let me know.\n\nYou can compare any custom type either by using the `MatchFunc` criteria, or by satisfying the `Comparer` interface with your type by adding the Compare method: `Compare(other interface{}) (int, error)`.\n\nIf a type doesn't have a predefined comparer, and doesn't satisfy the Comparer interface, then the types value is converted to a string and compared lexicographically.\n\n## Behavior Changes\n\nSince BoltHold is a higher level interface than BoltDB, there are some added helpers. Instead of _Put_, you have the options of:\n\n- _Insert_ - Fails if key already exists.\n- _Update_ - Fails if key doesn't exist `ErrNotFound`.\n- _Upsert_ - If key doesn't exist, it inserts the data, otherwise it updates the existing record.\n\nWhen getting data instead of returning `nil` if a value doesn't exist, BoltHold returns `bolthold.ErrNotFound`, and similarly when deleting data, instead of silently continuing if a value isn't found to delete, BoltHold returns `bolthold.ErrNotFound`. The exception to this is when using query based functions such as `Find` (returns an empty slice), `DeleteMatching` and `UpdateMatching` where no error is returned.\n\n## When should I use BoltHold?\n\nBoltHold will be useful in the same scenarios where BoltDB is useful, with the added benefit of being able to retire some of your data filtering code and possibly improved performance.\n\nYou can also use it instead of SQLite for many scenarios. BoltHold's main benefit over SQLite is its simplicity when working with Go Types. There is no need for an ORM layer to translate records to types, simply put types in, and get types out. You also don't have to deal with database initialization. Usually with SQLite you'll need several scripts to create the database, create the tables you expect, and create any indexes. With BoltHold you simply open a new file and put any type of data you want in it.\n\n```Go\nstore, err := bolthold.Open(filename, 0666, nil)\nif err != nil {\n\t//handle error\n}\nerr = store.Insert(\"key\", \u0026Item{\n\tName:    \"Test Name\",\n\tCreated: time.Now(),\n})\n```\n\nThat's it!\n\nBolthold currently has over 80% coverage in unit tests, and it's backed by BoltDB which is a very solid and well built piece of software, so I encourage you to give it a try.\n\nIf you end up using BoltHold, I'd love to hear about it.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimshannon%2Fbolthold","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftimshannon%2Fbolthold","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimshannon%2Fbolthold/lists"}