{"id":48381085,"url":"https://github.com/oliverandrich/den","last_synced_at":"2026-05-03T19:02:44.557Z","repository":{"id":349191314,"uuid":"1201406149","full_name":"oliverandrich/den","owner":"oliverandrich","description":"ODM for Go — SQLite and PostgreSQL backends, same API","archived":false,"fork":false,"pushed_at":"2026-04-26T09:24:42.000Z","size":1626,"stargazers_count":7,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-26T09:29:00.907Z","etag":null,"topics":["database","document-store","go","golang","odm","orm","postgresql","sqlite"],"latest_commit_sha":null,"homepage":"https://den-odm.readthedocs.io","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oliverandrich.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"docs/contributing.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-04T16:28:03.000Z","updated_at":"2026-04-26T09:24:46.000Z","dependencies_parsed_at":null,"dependency_job_id":"4401771a-779b-4ba2-a7bb-841d5346b0a3","html_url":"https://github.com/oliverandrich/den","commit_stats":null,"previous_names":["oliverandrich/den"],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/oliverandrich/den","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oliverandrich%2Fden","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oliverandrich%2Fden/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oliverandrich%2Fden/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oliverandrich%2Fden/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oliverandrich","download_url":"https://codeload.github.com/oliverandrich/den/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oliverandrich%2Fden/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32581021,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-03T06:36:36.687Z","status":"ssl_error","status_checked_at":"2026-05-03T06:36:09.306Z","response_time":103,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","document-store","go","golang","odm","orm","postgresql","sqlite"],"created_at":"2026-04-05T20:03:46.180Z","updated_at":"2026-05-03T19:02:44.551Z","avatar_url":"https://github.com/oliverandrich.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Den\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/assets/cover.jpg\" alt=\"Go gophers organizing documents in their den\" width=\"600\"\u003e\n  \u003cbr\u003e\n  \u003cem\u003e\"Every \u003ca href=\"https://github.com/oliverandrich/burrow\"\u003eburrow\u003c/a\u003e needs a den — a place to store what matters and find it again when you need it.\"\u003c/em\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/oliverandrich/den/actions/workflows/ci.yml\"\u003e\u003cimg src=\"https://img.shields.io/github/actions/workflow/status/oliverandrich/den/ci.yml?branch=main\u0026label=CI\u0026style=for-the-badge\" alt=\"CI\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/oliverandrich/den/releases\"\u003e\u003cimg src=\"https://img.shields.io/github/v/release/oliverandrich/den?style=for-the-badge\" alt=\"Release\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://go.dev/\"\u003e\u003cimg src=\"https://img.shields.io/github/go-mod/go-version/oliverandrich/den?style=for-the-badge\" alt=\"Go Version\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://goreportcard.com/report/github.com/oliverandrich/den\"\u003e\u003cimg src=\"https://goreportcard.com/badge/github.com/oliverandrich/den?style=for-the-badge\" alt=\"Go Report Card\"\u003e\u003c/a\u003e\n  \u003ca href=\"/LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/github/license/oliverandrich/den?style=for-the-badge\" alt=\"License\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://den-odm.readthedocs.io/\"\u003e\u003cimg src=\"https://img.shields.io/badge/docs-den--odm.readthedocs.io-blue?style=for-the-badge\" alt=\"Docs\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\nAn ODM for Go with two storage backends — SQLite and PostgreSQL. Same API, your choice of engine.\n\nEach Go struct you register is a *document*, stored as a JSONB row in a SQL table that Den calls a *collection*. The SQL schema is one table per type with a JSONB `data` column plus a small set of secondary indexes Den maintains for you. You query collections with a fluent builder, relate them with typed links, and run it all in transactions. The SQLite backend compiles into your binary with no external dependencies. The PostgreSQL backend connects to your existing database. Switch between them by changing one line.\n\n\u003e [!NOTE]\n\u003e Den is a document store, not a relational database. It does not support SQL, JOINs, or schema migrations in the traditional sense. If you need relational modeling, use [Bun](https://bun.uptrace.dev/) or [GORM](https://gorm.io/) instead.\n\n## Features\n\n- **Two backends, one API** — SQLite (embedded, pure Go, no CGO) and PostgreSQL (server-based, JSONB + GIN indexes)\n- **Chainable QuerySet** — `NewQuery[T](db).Where(...).Sort(...).Limit(n).All(ctx)` with lazy evaluation\n- **Range iteration** — `Iter()` returns `iter.Seq2[*T, error]` for memory-efficient streaming with Go's `range`\n- **Typed relations** — `Link[T]` for one-to-one, `[]Link[T]` for one-to-many, with cascade write/delete and eager/lazy fetch\n- **Back-references** — `BackLinks[T]` finds all documents referencing a given target\n- **Native aggregation** — `Avg`, `Sum`, `Min`, `Max` pushed down to SQL; `GroupBy` and `Project` for analytics\n- **Full-text search** — FTS5 for SQLite, tsvector for PostgreSQL, same `Search()` API\n- **Lifecycle hooks** — BeforeInsert, AfterUpdate, Validate, and more — interfaces on your struct, no registration\n- **Change tracking** — opt-in via `Tracked`: `IsChanged`, `GetChanges`, `Revert` with byte-level snapshots\n- **Soft delete** — embed `SoftDelete` alongside `Base`, automatic query filtering, `HardDelete` for permanent removal\n- **Attachments \u0026 storage** — embed `Attachment`, install a `den.Storage` backend once, let the hard-delete cascade clean bytes automatically\n- **Optimistic concurrency** — revision-based conflict detection with `ErrRevisionConflict`\n- **Transactions** — `RunInTransaction` with panic-safe rollback\n- **Migrations** — registry-based, each migration runs atomically in a transaction\n- **Struct tag validation** — optional `validate:\"required,email\"` tags via `go-playground/validator`, enabled with `validate.WithValidation()`\n- **Expression indexes** — `den:\"index\"`, `den:\"unique\"`, nullable unique for pointer fields\n\n## Quick Start\n\n```bash\nmkdir myapp \u0026\u0026 cd myapp\ngo mod init myapp\ngo get github.com/oliverandrich/den@latest\n```\n\n```go\npackage main\n\nimport (\n    \"context\"\n    \"fmt\"\n    \"log\"\n\n    \"github.com/oliverandrich/den\"\n    _ \"github.com/oliverandrich/den/backend/sqlite\" // register sqlite:// scheme\n    \"github.com/oliverandrich/den/document\"\n    \"github.com/oliverandrich/den/where\"\n)\n\ntype Product struct {\n    document.Base\n    Name  string  `json:\"name\"  den:\"index\"`\n    Price float64 `json:\"price\" den:\"index\"`\n}\n\nfunc main() {\n    ctx := context.Background()\n\n    // Open a SQLite database\n    db, err := den.OpenURL(ctx, \"sqlite:///products.db\")\n    if err != nil {\n        log.Fatal(err)\n    }\n    defer db.Close()\n\n    // Register document types (creates tables and indexes)\n    if err := den.Register(ctx, db, \u0026Product{}); err != nil {\n        log.Fatal(err)\n    }\n\n    // Insert\n    p := \u0026Product{Name: \"Widget\", Price: 9.99}\n    if err := den.Insert(ctx, db, p); err != nil {\n        log.Fatal(err)\n    }\n    fmt.Printf(\"Inserted: %s (ID: %s)\\n\", p.Name, p.ID)\n\n    // Query\n    products, err := den.NewQuery[Product](db,\n        where.Field(\"price\").Lt(20.0),\n    ).Sort(\"name\", den.Asc).All(ctx)\n    if err != nil {\n        log.Fatal(err)\n    }\n    for _, prod := range products {\n        fmt.Printf(\"  %s — $%.2f\\n\", prod.Name, prod.Price)\n    }\n\n    // Iterate (streaming, memory-efficient)\n    for doc, err := range den.NewQuery[Product](db).Iter(ctx) {\n        if err != nil {\n            log.Fatal(err)\n        }\n        fmt.Printf(\"  %s\\n\", doc.Name)\n    }\n}\n```\n\nTo use PostgreSQL instead, change the DSN and the import:\n\n```go\nimport _ \"github.com/oliverandrich/den/backend/postgres\" // instead of sqlite\n\ndb, err := den.OpenURL(ctx, \"postgres://user:pass@localhost/mydb\")\n```\n\n## Architecture\n\n```\nden/\n├── den.go, crud.go, queryset.go    Core API: Open, CRUD, QuerySet\n├── iter.go                         Iter() — iter.Seq2 for range loops\n├── aggregate.go                    Avg, Sum, Min, Max, GroupBy, Project\n├── link.go, backlinks.go           Link[T] relations, BackLinks\n├── search.go                       Full-text search (FTSProvider)\n├── track.go                        Change tracking: IsChanged, GetChanges\n├── soft_delete.go                  Soft delete, HardDelete\n├── hooks.go                        Lifecycle hook interfaces\n├── revision.go                     Optimistic concurrency\n├── tx.go                           Transactions\n├── storage.go                      Storage interface\n├── storage/                        Storage backend registry + OpenURL\n├── storage/file/                   Local filesystem backend (file:// scheme)\n├── document/                       Base + composable SoftDelete, Tracked, Attachment embeds\n├── where/                          Query condition builders\n├── backend/\n│   ├── sqlite/                     SQLite backend (pure Go, no CGO)\n│   └── postgres/                   PostgreSQL backend (pgx)\n├── validate/                       Optional struct tag validation\n├── migrate/                        Migration framework\n└── dentest/                        Test helpers\n```\n\n### Backend Interface\n\nBoth backends implement the same `Backend` interface. The `ReadWriter` subset is shared between backends and transactions, so CRUD code works identically inside and outside transactions.\n\n```go\ntype ReadWriter interface {\n    Get(ctx, collection, id) ([]byte, error)\n    Put(ctx, collection, id, data) error\n    Delete(ctx, collection, id) error\n    Query(ctx, collection, *Query) (Iterator, error)\n    Count(ctx, collection, *Query) (int64, error)\n    Exists(ctx, collection, *Query) (bool, error)\n    Aggregate(ctx, collection, op, field, *Query) (*float64, error)\n}\n```\n\n### Document Types\n\nEvery document embeds `document.Base` — the required anchor that provides\n`ID`, `CreatedAt`, `UpdatedAt`, `Rev`. Opt-in features are available as\nseparate composable embeds:\n\n| Embed | Purpose |\n|---|---|\n| `document.Base` | Required. Provides `ID`, `CreatedAt`, `UpdatedAt`, `Rev` |\n| `document.SoftDelete` | Adds `DeletedAt` and `IsDeleted()` for non-destructive deletion |\n| `document.Tracked` | Adds byte-snapshot machinery for `IsChanged`, `GetChanges`, `Revert` |\n| `document.Attachment` | Adds `StoragePath`, `Mime`, `Size`, `SHA256` — file reference paired with a `den.Storage` backend |\n\nCompose freely: `struct { document.Base; document.SoftDelete; document.Tracked; document.Attachment; ... }`.\n\n### Query Operators\n\n```go\nwhere.Field(\"price\").Gt(10)           // comparison\nwhere.Field(\"status\").In(\"a\", \"b\")    // set membership\nwhere.Field(\"tags\").Contains(\"go\")    // array contains\nwhere.Field(\"email\").IsNil()          // null check\nwhere.Field(\"name\").RegExp(\"^W\")      // regular expression\nwhere.And(cond1, cond2)               // logical combinators\nwhere.Field(\"addr.city\").Eq(\"Berlin\") // nested fields (dot notation)\n```\n\n## Validation\n\nDen supports automatic struct tag validation via [`go-playground/validator`](https://github.com/go-playground/validator). Enable it as an option when opening the database:\n\n```go\nimport \"github.com/oliverandrich/den/validate\"\n\ndb, err := den.OpenURL(ctx, \"sqlite:///data.db\", validate.WithValidation())\n```\n\nThen add `validate` tags to your document structs:\n\n```go\ntype User struct {\n    document.Base\n    Name  string `json:\"name\"  den:\"unique\" validate:\"required,min=3,max=50\"`\n    Email string `json:\"email\" den:\"unique\" validate:\"required,email\"`\n    Age   int    `json:\"age\"                validate:\"gte=0,lte=130\"`\n}\n```\n\nValidation runs automatically before every insert and update. Errors wrap `den.ErrValidation` and can be inspected for field-level detail:\n\n```go\nerr := den.Insert(ctx, db, \u0026User{Name: \"ab\"})\nif errors.Is(err, den.ErrValidation) {\n    var ve *validate.Errors\n    if errors.As(err, \u0026ve) {\n        for _, fe := range ve.Fields {\n            fmt.Printf(\"%s failed on %s\\n\", fe.Field, fe.Tag)\n        }\n    }\n}\n```\n\nTag validation and the `Validator` interface coexist — tag validation runs first (structural rules), then `Validate()` (business logic). Without `validate.WithValidation()`, no tag validation occurs (fully backward compatible).\n\n## Testing\n\nDen provides a `dentest` package for test setup:\n\n```go\nfunc TestMyFeature(t *testing.T) {\n    db := dentest.MustOpen(t, \u0026Product{}, \u0026Category{})\n    // File-backed SQLite in t.TempDir(), auto-closed via t.Cleanup\n}\n```\n\nFor PostgreSQL tests:\n\n```go\nfunc TestMyFeature(t *testing.T) {\n    db := dentest.MustOpenPostgres(t, \"postgres://localhost/test\", \u0026Product{})\n}\n```\n\n## Benchmarks\n\nMeasured on an Apple M4 Pro (14 cores), Go 1.25, PostgreSQL 17 on localhost. The fixture is a ~1 KB article document (title, body, status, category, tags, price, indexed timestamp, embedded author link, metadata map) — closer to a real blog or catalog entry than a minimal struct.\n\nReproduce locally with `just bench-readme`. Numbers exclude connection-setup overhead (the bench helper opens the DB once and reuses it).\n\n### Serial workloads\n\nSingle-goroutine latency per operation. Lower is better.\n\n\u003c!-- BENCH:SERIAL --\u003e\n| Scenario | SQLite | Postgres | SQLite allocs | Postgres allocs |\n|---|---:|---:|---:|---:|\n| Insert (single) | 148.2 µs | 186.6 µs | 31 | 29 |\n| InsertMany (100) | 9.98 ms | 13.91 ms | 3411 | 2916 |\n| InsertMany (1000) | 91.67 ms | 142.16 ms | 34021 | 29064 |\n| FindByID | 5.1 µs | 37.4 µs | 42 | 31 |\n| FindByIDs (10) | 266.6 µs | 930.5 µs | 343 | 328 |\n| Query + Sort + Limit(10) | 730.5 µs | 2.12 ms | 328 | 291 |\n| Query + Sort + Limit(100) | 1.50 ms | 3.83 ms | 2941 | 2544 |\n| Iter (1000 rows) | 3.07 ms | 2.93 ms | 29050 | 25036 |\n| Count(filter) | 25.5 µs | 805.0 µs | 29 | 31 |\n| Sum(filter) | 177.6 µs | 1.02 ms | 35 | 41 |\n| FTS Search | 885.9 µs | 2.04 ms | 603 | 513 |\n| WithFetchLinks (20 rows) | 78.4 µs | 632.9 µs | 658 | 570 |\n| Update (single) | 119.0 µs | 307.8 µs | 62 | 49 |\n| QuerySet.Update (100) | 9.01 ms | 18.35 ms | 5247 | 4341 |\n| RunInTransaction | 159.8 µs | 299.5 µs | 78 | 55 |\n\n\u003c!-- /BENCH:SERIAL --\u003e\n\n### Concurrent workloads\n\n`b.RunParallel` with Go's default `GOMAXPROCS`. Higher ops/sec is better. SQLite serializes writers by design (BEGIN IMMEDIATE), so write-heavy numbers plateau at single-writer speed; PostgreSQL's MVCC scales writes across connections.\n\n\u003c!-- BENCH:CONCURRENT --\u003e\n| Scenario | SQLite | Postgres |\n|---|---:|---:|\n| FindByID | 70.1k ops/s | 82.9k ops/s |\n| Insert (single) | 6.3k ops/s | 23.4k ops/s |\n| Mixed reads/writes 80/20 | 27.2k ops/s | 63.3k ops/s |\n| Queue consumer (SkipLocked) | 23.9k ops/s | 19.4k ops/s |\n\n\u003c!-- /BENCH:CONCURRENT --\u003e\n\n## Development\n\nDen uses [just](https://github.com/casey/just) as command runner:\n\n```bash\njust setup      # Check that all required dev tools are installed\njust test       # Run all tests (SQLite only)\njust test-all   # Run all tests including PostgreSQL\njust lint       # Run golangci-lint\njust fmt        # Format all Go files\njust coverage   # Run tests with coverage report\njust vuln       # Run vulnerability check\njust tidy       # Tidy module dependencies\njust beans      # List active beans (issue tracker)\n```\n\nRequires Go 1.25+. Run `just setup` to verify your dev environment.\n\n## Dependencies\n\n| Dependency | Purpose |\n|---|---|\n| `github.com/oklog/ulid/v2` | ULID-based document IDs |\n| `github.com/goccy/go-json` | Fast JSON encoding |\n| `modernc.org/sqlite` | SQLite backend (pure Go, no CGO) |\n| `github.com/jackc/pgx/v5` | PostgreSQL backend |\n| `github.com/go-playground/validator/v10` | Struct tag validation (optional, via `den/validate`) |\n\n## License\n\nDen is licensed under the [MIT License](LICENSE).\n\nThe Go Gopher was originally designed by [Renee French](https://reneefrench.blogspot.com/) and is licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foliverandrich%2Fden","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foliverandrich%2Fden","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foliverandrich%2Fden/lists"}