Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/gogitdb/gitdb
Distributed Embeddable Database
https://github.com/gogitdb/gitdb
database document-database embedded git gitdb go golang library
Last synced: 3 months ago
JSON representation
Distributed Embeddable Database
- Host: GitHub
- URL: https://github.com/gogitdb/gitdb
- Owner: gogitdb
- License: mit
- Created: 2020-05-04T00:13:25.000Z (over 4 years ago)
- Default Branch: master
- Last Pushed: 2022-07-22T10:28:05.000Z (over 2 years ago)
- Last Synced: 2024-06-28T02:42:15.721Z (5 months ago)
- Topics: database, document-database, embedded, git, gitdb, go, golang, library
- Language: Go
- Size: 309 KB
- Stars: 255
- Watchers: 3
- Forks: 5
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
GitDB
=====
[](https://goreportcard.com/report/github.com/gogitdb/gitdb)
[](https://codecov.io/gh/gogitdb/gitdb)
[](https://travis-ci.com/gogitdb/gitdb)
[](https://godoc.org/github.com/gogitdb/gitdb)
[](https://github.com/gogitdb/gitdb/releases)
[](https://github.com/gogitdb/gitdb/blob/master/LICENSE)
## What is GitDB?
> GitDB is not a binary. It’s a library!
GitDB is a decentralized document database written in Go that uses [Git](https://git-scm.com/) under the hood to provide database-like functionalities via strictly defined interfaces.
GitDB allows developers to create Models of objects in their application which implement a Model Interface that can access it's persistence features. This allows GitDB to work with these objects in database operations.
## Why GitDB - motivation behind project?
- A need for a database that was quick and simple to set up
- A need for a database that was decentralized and each participating client in a system can store their data independent of other clients.
## Features
- Decentralized
- Document store
- Embedded into your go application
- Encryption (encrypt on write, decrypt on read)
- Record locking.
- Simple Indexing System
- Transactions
- Web UI
## Project versioning
GitDB uses [semantic versioning](http://semver.org).
API should not change between patch and minor releases.
New minor versions may add additional features to the API.
## Table of Contents
- [Getting Started](#getting-started)
- [Installing](#installing)
- [Configuration](#configuration)
- [Opening a database](#opening-a-database)
- [Inserting/Updating a record](#insertingupdating-a-record)
- [Fetching a single record](#fetching-a-single-record)
- [Fetching all records in a dataset](#fetching-all-records-in-a-dataset)
- [Deleting a record](#deleting-a-record)
- [Search for records](#search-for-records)
- [Transactions](#transactions)
- [Encryption](#encryption)
- [Resources](#resources)
- [Caveats & Limitations](#caveats--limitations)
- [Reading the Source](#reading-the-source)
## Getting Started
### Installing
To start using GitDB, install Go and run `go get`:
```sh
$ go get github.com/gogitdb/gitdb/v2
```
### Configuration
Below are configuration options provided by GitDB
Name
Description
Type
Required
Default
DbPath
Path on your machine where you want GitDB to create/clone your database
string
Y
N/A
ConnectionName
Unique name for gitdb connection. Use this when opening multiple GitDB connections
string
N
"default"
OnlineRemote
URL for remote git server you want GitDB to sync with e.g [email protected]:user/db.git or https://github.com/user/db.git.
Note: The first time GitDB runs, it will automatically generate ssh keys and will automatically attempt to use this key to sync with the OnlineRemote,
therefore ensure that the generated keys are added to this git server. The ssh keys can be found at Config.DbPath/.gitdb/ssh
string
N
""
SyncInterval
This controls how often you want GitDB to sync with the online remote
time.Duration.
N
5s
EncryptionKey
16,24 or 32 byte string used to provide AES encryption for Models that implement ShouldEncrypt
string
N
""
User
This specifies the user connected to the Gitdb and will be used to commit all changes to the database
gitdb.User
N
ghost <[email protected]>
EnableUI
Use this option to enable GitDB web user interface
bool
N
false
UIPort
Use this option to change the default port which GitDB uses to serve it's web user interface
int
N
4120
Factory
For backward compatibity with v1. In v1 GitDB needed a factory method to be able construct concrete Model for certain database operations.
This has now been dropped in v2
func(dataset string) gitdb.Model
N
nil
Mock
Flag used for testing apps. If true, will return a mock GitDB connection
bool
N
false
You can configure GitDB either using the constructor or constructing it yourself
```go
cfg := gitdb.NewConfig(path)
//or
cfg := gitdb.Config{
DbPath: path
}
```
### Importing GitDB
To use GitDB as an embedded document store, import as:
```go
import "github.com/gogitdb/gitdb/v2"
cfg := gitdb.NewConfig(path)
db, err := gitdb.Open(cfg)
if err != nil {
log.Fatal(err)
}
defer db.Close()
```
### Opening a database
```go
package main
import (
"log"
"github.com/gogitdb/gitdb/v2"
)
func main() {
cfg := gitdb.NewConfig("/tmp/data")
// Open will create or clone down a git repo
// in configured path if it does not exist.
db, err := gitdb.Open(cfg)
if err != nil {
log.Fatal(err)
}
defer db.Close()
...
}
```
### Models
A Model is a struct that represents a record in GitDB. GitDB only works with models that implement the gidb.Model interface
gitdb.TimeStampedModel is a simple struct that allows you to easily add CreatedAt and UpdatedAt to all the Models in your application and will automatically time stamp them before persisting to GitDB. You can write your own base Models to embed common fields across your application Models
```go
type BankAccount struct {
//TimeStampedModel will add CreatedAt and UpdatedAt fields this Model
gitdb.TimeStampedModel
AccountType string
AccountNo string
Currency string
Name string
}
func (b *BankAccount) GetSchema() *gitdb.Schema {
//Dataset Name
name := "Accounts"
//Block ID
block := b.CreatedAt.Format("200601")
//Record ID
record := b.AccountNo
//Indexes speed up searching
indexes := make(map[string]interface{})
indexes["AccountType"] = b.AccountType
return gitdb.NewSchema(name, block, record, indexes)
}
func (b *BankAccount) Validate() error { return nil }
func (b *BankAccount) IsLockable() bool { return false }
func (b *BankAccount) ShouldEncrypt() bool { return false }
func (b *BankAccount) GetLockFileNames() []string { return []string{} }
...
```
### Inserting/Updating a record
```go
package main
import (
"log"
"github.com/gogitdb/gitdb/v2"
)
func main(){
cfg := gitdb.NewConfig("/tmp/data")
db, err := gitdb.Open(cfg)
if err != nil {
log.Fatal(err)
}
defer db.Close()
//populate model
account := &BankAccount{}
account.AccountNo = "0123456789"
account.AccountType = "Savings"
account.Currency = "GBP"
account.Name = "Foo Bar"
err = db.Insert(account)
if err != nil {
log.Println(err)
}
//get account id
log.Println(gitdb.Id(account))
//update account name
account.Name = "Bar Foo"
err = db.Insert(account)
if err != nil {
log.Println(err)
}
}
```
### Fetching a single record
```go
package main
import (
"log"
"github.com/gogitdb/gitdb/v2"
)
func main(){
cfg := gitdb.NewConfig("/tmp/data")
db, err := gitdb.Open(cfg)
if err != nil {
log.Fatal(err)
}
defer db.Close()
//model to passed to Get to store result
var account BankAccount
if err := db.Get("Accounts/202003/0123456789", &account); err != nil {
log.Println(err)
}
}
```
### Fetching all records in a dataset
```go
package main
import (
"fmt"
"log"
"github.com/gogitdb/gitdb/v2"
)
func main(){
cfg := gitdb.NewConfig("/tmp/data")
db, err := gitdb.Open(cfg)
if err != nil {
log.Fatal(err)
}
defer db.Close()
records, err := db.Fetch("Accounts")
if err != nil {
log.Print(err)
return
}
accounts := []*BankAccount{}
for _, r := range records {
b := &BankAccount{}
r.Hydrate(b)
accounts = append(accounts, b)
log.Print(fmt.Sprintf("%s-%s", gitdb.ID(b), b.AccountNo))
}
}
```
### Fetching all records from specific block in a dataset
```go
package main
import (
"fmt"
"log"
"github.com/gogitdb/gitdb/v2"
)
func main(){
cfg := gitdb.NewConfig("/tmp/data")
db, err := gitdb.Open(cfg)
if err != nil {
log.Fatal(err)
}
defer db.Close()
records, err := db.Fetch("Accounts", "b0", "b1")
if err != nil {
log.Print(err)
return
}
var accounts []*BankAccount
for _, r := range records {
b := &BankAccount{}
r.Hydrate(b)
accounts = append(accounts, b)
log.Print(fmt.Sprintf("%s-%s", gitdb.ID(b), b.AccountNo))
}
}
```
### Deleting a record
```go
package main
import (
"log"
"github.com/gogitdb/gitdb/v2"
)
func main(){
cfg := gitdb.NewConfig("/tmp/data")
db, err := gitdb.Open(cfg)
if err != nil {
log.Fatal(err)
}
defer db.Close()
if err := db.Delete("Accounts/202003/0123456789"); err != nil {
log.Print(err)
}
}
```
### Search for records
```go
package main
import (
"fmt"
"log"
"github.com/gogitdb/gitdb/v2"
)
func main(){
cfg := gitdb.NewConfig("/tmp/data")
db, err := gitdb.Open(cfg)
if err != nil {
log.Fatal(err)
}
defer db.Close()
//Find all records that have savings account type
searchParam := &db.SearchParam{Index: "AccountType", Value: "Savings"}
records, err := dbconn.Search("Accounts", []*db.SearchParam{searchParam}, gitdb.SearchEquals)
if err != nil {
log.Println(err.Error())
return
}
accounts := []*BankAccount{}
for _, r := range records {
b := &BankAccount{}
r.Hydrate(b)
accounts = append(accounts, b)
log.Print(fmt.Sprintf("%s-%s", b.ID, b.CreatedAt))
}
}
```
### Transactions
```go
package main
import (
"log"
"github.com/gogitdb/gitdb/v2"
)
func main() {
cfg := gitdb.NewConfig("/tmp/data")
db, err := gitdb.Open(cfg)
if err != nil {
log.Fatal(err)
}
defer db.Close()
func accountUpgradeFuncOne() error { println("accountUpgradeFuncOne..."); return nil }
func accountUpgradeFuncTwo() error { println("accountUpgradeFuncTwo..."); return errors.New("accountUpgradeFuncTwo failed") }
func accountUpgradeFuncThree() error { println("accountUpgradeFuncThree"); return nil }
tx := db.StartTransaction("AccountUpgrade")
tx.AddOperation(accountUpgradeFuncOne)
tx.AddOperation(accountUpgradeFuncTwo)
tx.AddOperation(accountUpgradeFuncThree)
terr := tx.Commit()
if terr != nil {
log.Print(terr)
}
}
```
### Encryption
GitDB suppports AES encryption and is done on a Model level, which means you can have a database with different Models where some are encrypted and others are not. To encrypt your data, your Model must implement `ShouldEncrypt()` to return true and you must set `gitdb.Config.EncryptionKey`. For maximum security set this key to a 32 byte string to select AES-256
```go
package main
import (
"log"
"github.com/gogitdb/gitdb/v2"
)
func main(){
cfg := gitdb.NewConfig("/tmp/data")
cfg.EncryptionKey = "a_32_bytes_string_for_AES-256"
db, err := gitdb.Open(cfg)
if err != nil {
log.Fatal(err)
}
defer db.Close()
//populate model
account := &BankAccount{}
account.AccountNo = "0123456789"
account.AccountType = "Savings"
account.Currency = "GBP"
account.Name = "Foo Bar"
//Insert will encrypt the account
err = db.Insert(account)
if err != nil {
log.Println(err)
}
//Get will automatically decrypt account
var account BankAccount
err = db.Get("Accounts/202003/0123456789", &account)
if err != nil {
log.Println(err)
}
}
```
## Resources
For more information on getting started with Gitdb, check out the following articles:
* [Gtidb - an embedded distributed document database for Go](https://docs.google.com/document/d/1OPalq-7J_Vo_uks35up_4a_V0BsRrpwL1J4JG6ZFoEs/edit?usp=sharing) by Oke Ugwu
## Caveats & Limitations
It's important to pick the right tool for the job and GitDB is no exception.
Here are a few things to note when evaluating and using GitDB:
* GitDB is good for systems where data producers are indpendent.
* GitDB currently depends on the git binary to work
## Reading the Source
GitDB is a relatively small code base (<5KLOC) for an embedded, distributed,
document database so it can be a good starting point for people
interested in how databases work.
The best places to start are the main entry points into GitDB:
- [`Open()`](https://github.com/gogitdb/gitdb/blob/a8088f138b072edd64021591e34adf878f2d0bd5/init.go#L17) - Initializes the reference to the database. It's responsible for
creating the database if it doesn't exist and pulling down existing database
if an online remote is specified.
If you have additional notes that could be helpful for others, please submit
them via pull request.