{"id":13396929,"url":"https://github.com/256dpi/lungo","last_synced_at":"2025-05-15T17:03:36.040Z","repository":{"id":52483383,"uuid":"215532341","full_name":"256dpi/lungo","owner":"256dpi","description":"A MongoDB compatible embeddable database and toolkit for Go.","archived":false,"fork":false,"pushed_at":"2024-12-13T23:13:24.000Z","size":625,"stargazers_count":470,"open_issues_count":6,"forks_count":14,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-04-07T22:07:19.092Z","etag":null,"topics":["bson","database","go","golang","mongo","mongodb"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/256dpi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-16T11:32:05.000Z","updated_at":"2025-03-20T04:24:18.000Z","dependencies_parsed_at":"2024-01-15T22:42:52.654Z","dependency_job_id":"e299f958-07b9-4e0c-8b97-1d847120c54a","html_url":"https://github.com/256dpi/lungo","commit_stats":{"total_commits":653,"total_committers":3,"mean_commits":"217.66666666666666","dds":"0.013782542113323082","last_synced_commit":"c3501f262477fe007f91c2f5ef47a9971b36fa2a"},"previous_names":[],"tags_count":33,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/256dpi%2Flungo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/256dpi%2Flungo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/256dpi%2Flungo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/256dpi%2Flungo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/256dpi","download_url":"https://codeload.github.com/256dpi/lungo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254384982,"owners_count":22062422,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bson","database","go","golang","mongo","mongodb"],"created_at":"2024-07-30T18:01:07.919Z","updated_at":"2025-05-15T17:03:35.990Z","avatar_url":"https://github.com/256dpi.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"\u003cimg src=\"http://joel-github-static.s3.amazonaws.com/lungo/logo2.png\" alt=\"Logo\" width=\"400\"/\u003e\n\n# lungo\n\n[![Test](https://github.com/256dpi/lungo/actions/workflows/test.yml/badge.svg)](https://github.com/256dpi/lungo/actions/workflows/test.yml)\n[![GoDoc](https://godoc.org/github.com/256dpi/lungo?status.svg)](https://godoc.org/github.com/256dpi/lungo)\n[![Release](https://img.shields.io/github/release/256dpi/lungo.svg)](https://github.com/256dpi/lungo/releases)\n\n**A MongoDB compatible embeddable database and toolkit for Go.**\n\n- [Installation](#installation)\n- [Example](#example)\n- [Motivation](#motivation)\n- [Architecture](#architecture)\n- [Features](#features)\n- [License](#license)\n\n## Installation\n\nTo get started, install the package using the go tool:\n\n```bash\n$ go get -u github.com/256dpi/lungo\n```\n\n## Example\n\nThis [example](https://github.com/256dpi/lungo/tree/master/example_test.go)\nshows a basic usage of the `mongo` compatible API.\n\n## Motivation\n\nThe document-oriented database MongoDB has become a widely used data store for\nmany applications developed with the Go programming language. Both, the deprecated\n`mgo` and the official `mongo` driver offer a sophisticated interface to connect\nto a deployment and ingest and extract data using various commands. While this\nis enough for most projects, there are situations in which one thinks: \"It would\nbe cool if I could just do that in memory without asking the server.\"\n\nLungo tries to address this need by re-implementing the data handling mechanics\nin Go to be used on the client-side. This allows developers to pre- or\npost-process data in the application relieving the server. For example,\napplications may utilize this functionality to cache documents and query them\nquickly in memory.\n\nBut we do not need to stop there. Many developers coming from the SQL ecosystem\nenjoy working with SQLite as a simple alternative to bigger SQL databases. It\nallows running tests without setting up a database or even small production\napps that write their data to a single backed-up file.\n\nLungo wants to offer a similar experience by implementing a full MongoDB\ncompatible embeddable database that persists data in a single file. The\nproject aims to provide drop-in compatibility with the API exported by the\nofficial Go driver. This way, applications may use lungo for running their\ntests or even low-write production deployments without big code changes.\n\nHowever, one thing this project does not try to do is build another\ndistributed database. MongoDB itself does a pretty good job at that already.\n\n## Architecture\n\nThe codebase is divided into the packages `bsonkit`, `mongokit`, `dbkit` and\nthe main `lungo` package.\n\n- The `bsonkit` package provides building blocks that extend the ones found in\nthe official `bson` package for handling BSON data. Its functions are mainly\nuseful to applications that need to inspect, compare, convert, transform,\nclone, access, and manipulate BSON data directly in memory.\n\n- On top of that, the `mongokit` package provides the MongoDB data handling\nalgorithms and structures. Specifically, it implements the MongoDB querying,\nupdate, and sort algorithms as well as a btree based index for documents. All of\nthat is then bundled as a basic in-memory collection of documents that offers a\nfamiliar CRUD interface.\n\n- The `dbkit` package provides database-centric utilities e.g. atomic file write.\n\n- Finally, the `lungo` package implements the embeddable database and the\n`mongo` compatible driver. The heavy work is done by the engine and transaction\ntypes that manage access to the basic `mongokit.Collection` instances. While both\ncan be used standalone, most users want to use the generic driver interface that\ncan be used with MongoDB deployments and lungo engines.\n\n## Features\n\nOn a high level, lungo provides the following features (unchecked features are\nplanned to be implemented):\n\n- [x] CRUD, Index Management and Namespace Management\n- [x] Single, Compound and Partial Indexes\n- [ ] Index Supported Sorting \u0026 Filtering\n- [x] Sessions \u0026 Multi-Document Transactions\n- [x] Oplog \u0026 Change Streams\n- [ ] Aggregation Pipeline\n- [x] Memory \u0026 Single File Store\n- [x] GridFS\n\nWhile the goal is to implement all MongoDB features in a compatible way, the\narchitectural difference has implications on some features. Furthermore,\nthe goal is to build an open and accessible codebase that favors simplicity.\nCheck out the following sections for details on the implementation.\n\n### CRUD, Index Management and Namespace Management\n\nThe driver supports all standard CRUD, index management and namespace management\nmethods that are also exposed by the official driver. However, to this date, the\ndriver does not yet support any of the MongoDB commands that can be issued using\nthe `Database.RunCommand` method. Most unexported commands are related to query\nplanning, replication, sharding, and user and role management features that we\ndo not plan to support. However, we eventually will support some\nadministrative and diagnostics commands e.g. `renameCollection` and `explain`.\n\nLeveraging the `mongokit.Match` function, lungo supports the following query\noperators:\n\n- `$and`, `$or`, `$nor`, (`$not`)\n- `$eq`, `$gt`, `$lt`, `$gte`, `$lte`, `$ne`\n- (`$in`), (`$nin`), `$exist`, `$type`\n- `$jsonSchema`, `$all`, `$size`, `$elemMatch`\n\nAnd the `mongokit.Apply` function currently supports the following update\noperators:\n\n- `$set`, `$setOnInsert`, `$unset`, `$rename`\n- `$inc`, `$mul`, `$max`, `$min`, (`$push`)\n- `$pop`, `$currentDate`, `$[]`, `$[\u003cidentifier\u003e]`\n\nFinally, the `mongokit.Project` function currently supports the following\nprojection operators:\n\n- `$slice`\n\nOperators in braces are only partially supported, see comments in code.\n\n### Single, Compound and Partial Indexes\n\nThe `mongokit.Index` type supports single field and compound indexes that\noptionally enforce uniqueness or index a subset of documents using a partial\nfilter expression. Single field indexes also support the automated expiry of\ndocuments aka. TTL indexes.\n\nThe more advanced multikey, geospatial, text, and hashed indexes are not yet\nsupported and may be added later, while the deprecated sparse indexes will not.\nThe recently introduced collation feature, as well as wildcard indexes, are also\nsubject to future development.\n\n### Index Supported Sorting \u0026 Filtering\n\nIndexes are currently only used to ensure uniqueness constraints and do not\nsupport filtering and sorting. This will be added in the future together with\nsupport for the `explain` command to debug the generated query plan.\n\n### Sessions \u0026 Multi-Document Transactions\n\nLungo supports multi-document transactions using a basic copy on write mechanism.\nEvery transaction will make a copy of the catalog and clone namespaces before\napplying changes. After the new catalog has been written to disk, the transaction\nis considered successful and the catalog replaced. Read-only transactions are\nallowed to run in parallel as they only serve as snapshots. But write\ntransactions are run sequentially. We assume write transactions to be fast and\ntherefore try to prevent abortions due to conflicts (pessimistic concurrency\ncontrol). The chosen approach might be changed in the future.\n\n### Oplog \u0026 Change Streams\n\nSimilar to MongoDB, every CRUD change is also logged to the `local.oplog`\ncollection in the same format as consumed by change streams in MongoDB. Based on\nthat, change streams can be used in the same way as with MongoDB replica sets.\n\n### Memory \u0026 Single File Store\n\nThe `lungo.Store` interface enables custom adapters that store the catalog to\nvarious mediums. The built-in `MemoryStore` keeps all data in memory while the\n`FileStore` writes all data atomically to a single BSON file. The interface may\nget more sophisticated in the future to allow more efficient storing methods.\n\n### GridFS\n\nThe `lungo.Bucket`, `lungo.UploadStream` and `lungo.DownloadStream` provide a\nGridFS implementation similar to the one found in the `gridfs` package of the\nofficial Go driver. However, some improvements have been made while\nre-implementing the package:\n\n- Support for sessions via the `context.Context` parameter in all `lungo.Bucket`\nmethods.\n- The `lungo.DowloadStream` implements the `io.Seeker` interface for convenient\nrange queries on the file contents.\n- A non-standard \"tracking\" mode in which in-progress uploads and deletions are\ntracked by storing a document in an additional \"markers\" collection. If enabled,\nuploads can be suspended and resumed later and must be explicitly claimed. All\nunclaimed uploads and not fully deleted files can be cleaned up.\n\n## License\n\nThe MIT License (MIT)\n\nCopyright (c) 2019 Joël Gähwiler\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F256dpi%2Flungo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F256dpi%2Flungo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F256dpi%2Flungo/lists"}