{"id":13725118,"url":"https://github.com/davidlatwe/montydb","last_synced_at":"2025-05-14T15:11:34.472Z","repository":{"id":30758759,"uuid":"125165850","full_name":"davidlatwe/montydb","owner":"davidlatwe","description":"Monty, Mongo tinified. MongoDB implemented in Python !","archived":false,"fork":false,"pushed_at":"2025-01-19T21:21:10.000Z","size":1324,"stargazers_count":589,"open_issues_count":27,"forks_count":30,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-04-10T15:48:05.016Z","etag":null,"topics":["lmdb","mocking","mongodb","pymongo","python","tinydb"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/davidlatwe.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-03-14T06:28:11.000Z","updated_at":"2025-03-27T12:53:17.000Z","dependencies_parsed_at":"2023-02-18T11:17:39.860Z","dependency_job_id":"6ddb51c6-0b48-4808-8fc6-e5c86ac16380","html_url":"https://github.com/davidlatwe/montydb","commit_stats":{"total_commits":694,"total_committers":8,"mean_commits":86.75,"dds":0.05475504322766567,"last_synced_commit":"469957a5bdbf06799dbfad835ec9643875c53ef4"},"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidlatwe%2Fmontydb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidlatwe%2Fmontydb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidlatwe%2Fmontydb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidlatwe%2Fmontydb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/davidlatwe","download_url":"https://codeload.github.com/davidlatwe/montydb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248717237,"owners_count":21150389,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["lmdb","mocking","mongodb","pymongo","python","tinydb"],"created_at":"2024-08-03T01:02:13.491Z","updated_at":"2025-04-13T13:13:37.846Z","avatar_url":"https://github.com/davidlatwe.png","language":"Python","funding_links":[],"categories":["Python","Databases"],"sub_categories":[],"readme":"\n\u003cimg src=\"artwork/logo.png\" alt=\"drawing\" width=\"600\"/\u003e\n\n[![Python package](https://github.com/davidlatwe/montydb/actions/workflows/python-package.yml/badge.svg)](https://github.com/davidlatwe/montydb/actions/workflows/python-package.yml)\n[![Version](http://img.shields.io/pypi/v/montydb.svg?style=flat)](https://pypi.python.org/pypi/montydb)\n[![PyPi downloads](https://img.shields.io/pypi/dm/montydb)](https://pypistats.org/packages/montydb)\n\n\u003e Monty, Mongo tinified. MongoDB implemented in Python!\n\n_Inspired by [TinyDB](https://github.com/msiemens/tinydb) and it's extension [TinyMongo](https://github.com/schapman1974/tinymongo)_\n\n\n## What is it?\n\nA pure Python-implemented database that looks and works like [MongoDB](https://www.mongodb.com/).\n\n```python\n\u003e\u003e\u003e from montydb import MontyClient\n\n\u003e\u003e\u003e col = MontyClient(\":memory:\").db.test\n\u003e\u003e\u003e col.insert_many( [{\"stock\": \"A\", \"qty\": 6}, {\"stock\": \"A\", \"qty\": 2}] )\n\u003e\u003e\u003e cur = col.find( {\"stock\": \"A\", \"qty\": {\"$gt\": 4}} )\n\u003e\u003e\u003e next(cur)\n{'_id': ObjectId('5ad34e537e8dd45d9c61a456'), 'stock': 'A', 'qty': 6}\n```\n\nMost of the CRUD operators have been implemented. You can visit [issue #14](https://github.com/davidlatwe/montydb/issues/14) to see the full list.\n\nThis project is tested against:\n\n- MongoDB: 3.6, 4.0, 4.2 (4.4 on the way💦)\n- Python: 3.7, 3.8, 3.9, 3.10, 3.11, 3.12\n\n\n## Install\n\n```sh\npip install montydb\n```\n\n- optional, to use *real* `bson` in operation (`pymongo` will be installed)\n    _For minimum requirements, `montydb` ships with it's own fork of `ObjectId` in `montydb.types`, so you may ignore this option if `ObjectId` is all you need from `bson`_\n\n    ```sh\n    pip install montydb[bson]\n    ```\n- optional, to use lightning memory-mapped db as storage engine\n    ```sh\n    pip install montydb[lmdb]\n    ```\n\n\n## Storage\n\n🦄 Available storage engines:\n\n* in-memory\n* flat-file\n* sqlite\n* lmdb (lightning memory-mapped db)\n\nDepending on which one you use, you may have to configure the storage engine before you start.\n\n\u003e ⚠️\n\u003e\n\u003e The configuration process only required on repository creation or modification. And, one repository (the parent level of databases) can only assign one storage engine.\n\nTo configure a storage, see flat-file storage for example:\n\n```python\nfrom montydb import set_storage, MontyClient\n\n\nset_storage(\n    # general settings\n    \n    repository=\"/db/repo\",  # dir path for database to live on disk, default is {cwd}\n    storage=\"flatfile\",     # storage name, default \"flatfile\"\n    mongo_version=\"4.0\",    # try matching behavior with this mongodb version\n    use_bson=False,         # default None, and will import pymongo's bson if None or True\n\n    # any other kwargs are storage engine settings.\n    \n    cache_modified=10,       # the only setting that flat-file have\n)\n\n# ready to go\n```\n\nOnce that done, there should be a file named `monty.storage.cfg` saved in your db repository path. It would be `/db/repo` for the above examples.\n\n\n## Configuration\n\nNow let's moving on to each storage engine's config settings.\n\n### 🌟 In-Memory\n  \n`memory` storage does not need nor have any configuration, nothing saved to disk.\n\n```python\nfrom montydb import MontyClient\n\n\nclient = MontyClient(\":memory:\")\n\n# ready to go\n```\n\n### 🔰 Flat-File\n  \n`flatfile` is the default on-disk storage engine.\n\n```python\nfrom montydb import set_storage, MontyClient\n\n\nset_storage(\"/db/repo\", cache_modified=5)  # optional step\nclient = MontyClient(\"/db/repo\")  # use current working dir if no path given\n\n# ready to go\n```\n\nFlatFile config:\n\n```ini\n[flatfile]\ncache_modified: 0  # how many document CRUD cached before flush to disk.\n```\n\n### 💎 SQLite\n  \n`sqlite` is NOT the default on-disk storage, need configuration first before getting client.\n\n\u003e Pre-existing sqlite storage file which saved by `montydb\u003c=1.3.0` is not read/writeable after `montydb==2.0.0`.\n\n```python\nfrom montydb import set_storage, MontyClient\n\n\nset_storage(\"/db/repo\", storage=\"sqlite\")  # required, to set sqlite as engine\nclient = MontyClient(\"/db/repo\")\n\n# ready to go\n```\n\nSQLite config:\n\n```ini\n[sqlite]\njournal_mode = WAL\ncheck_same_thread =   # Leave it empty as False, or any value will be True\n```\nOr,\n\n```python\nrepo = \"path_to/repo\"\nset_storage(\n    repository=repo,\n    storage=\"sqlite\",\n    use_bson=True,\n    # sqlite pragma\n    journal_mode=\"WAL\",\n    # sqlite connection option\n    check_same_thread=False,\n)\nclient = MontyClient(repo)\n...\n```\n\nSQLite write concern:\n\n```python\nclient = MontyClient(\"/db/repo\",\n                     synchronous=1,\n                     automatic_index=False,\n                     busy_timeout=5000)\n```\n\n### 🚀 LMDB (Lightning Memory-Mapped Database)\n\n`lightning` is NOT the default on-disk storage, need configuration first before get client.\n\n\u003e Newly implemented.\n\n```python\nfrom montydb import set_storage, MontyClient\n\n\nset_storage(\"/db/repo\", storage=\"lightning\")  # required, to set lightning as engine\nclient = MontyClient(\"/db/repo\")\n\n# ready to go\n```\n\nLMDB config:\n\n```ini\n[lightning]\nmap_size: 10485760  # Maximum size database may grow to.\n```\n\n## URI\n\nOptionally, You could prefix the repository path with montydb URI scheme.\n\n```python\nclient = MontyClient(\"montydb:///db/repo\")\n```\n\n## Utilities\n\n\u003e Pymongo `bson` may required.\n\n* #### `montyimport`\n\n  Imports content from an Extended JSON file into a MontyCollection instance.\n  The JSON file could be generated from `montyexport` or `mongoexport`.\n\n  ```python\n  from montydb import open_repo, utils\n  \n\n  with open_repo(\"foo/bar\"):\n      utils.montyimport(\"db\", \"col\", \"/path/dump.json\")\n  \n  ```\n\n* #### `montyexport`\n\n  Produces a JSON export of data stored in a MontyCollection instance.\n  The JSON file could be loaded by `montyimport` or `mongoimport`.\n\n  ```python\n  from montydb import open_repo, utils\n  \n\n  with open_repo(\"foo/bar\"):\n      utils.montyexport(\"db\", \"col\", \"/data/dump.json\")\n  \n  ```\n\n* #### `montyrestore`\n\n  Loads a binary database dump into a MontyCollection instance.\n  The BSON file could be generated from `montydump` or `mongodump`.\n\n  ```python\n  from montydb import open_repo, utils\n  \n\n  with open_repo(\"foo/bar\"):\n      utils.montyrestore(\"db\", \"col\", \"/path/dump.bson\")\n  \n  ```\n\n* ####  `montydump`\n\n  Creates a binary export from a MontyCollection instance.\n  The BSON file could be loaded by `montyrestore` or `mongorestore`.\n\n  ```python\n  from montydb import open_repo, utils\n  \n\n  with open_repo(\"foo/bar\"):\n      utils.montydump(\"db\", \"col\", \"/data/dump.bson\")\n  \n  ```\n\n* #### `MongoQueryRecorder`\n\n  Record MongoDB query results in a period of time.\n  *Requires to access database profiler.*\n\n  This works via filtering the database profile data and reproduce the queries of `find` and `distinct` commands.\n\n  ```python\n  from pymongo import MongoClient\n  from montydb.utils import MongoQueryRecorder\n  \n  client = MongoClient()\n  recorder = MongoQueryRecorder(client[\"mydb\"])\n  recorder.start()\n  \n  # Make some queries or run the App...\n  recorder.stop()\n  recorder.extract()\n  {\u003ccollection_1\u003e: [\u003cdoc_1\u003e, \u003cdoc_2\u003e, ...], ...}\n  \n  ```\n\n* ####  `MontyList`\n\n  Experimental, a subclass of `list`, combined the common CRUD methods from Mongo's Collection and Cursor.\n\n  ```python\n  from montydb.utils import MontyList\n  \n  mtl = MontyList([1, 2, {\"a\": 1}, {\"a\": 5}, {\"a\": 8}])\n  mtl.find({\"a\": {\"$gt\": 3}})\n  MontyList([{'a': 5}, {'a': 8}])\n  \n  ```\n\n## Development\n\nmontydb uses [Poetry](https://python-poetry.org/) to make it easy manage dependencies and set up the development environment. \n\n### Initial setup\n\nAfter cloning the repository, you need to run the following commands to set up the development environment:\n\n```bash\nmake install\n```\n\nThis will create a virtual environment and download the required dependencies.\n\n### updating dependencies\n\nTo keep dependencies updated after git operations such as local updates or merging changes into local dev branch\n\n```bash\nmake update\n```\n### Makefile\n\nA makefile is used to simplify common operations such as updating, testing, and deploying etc.\n\n```bash\nmake or make help\n\ninstall                        install all dependencies locally\nupdate                         update project dependencies locally (run after git update)\nci                             Run all checks (codespell, lint, bandit, test)\ntest                           Run tests\nlint                           Run linting with flake8\ncodespell                      Find typos with codespell\nbandit                         Run static security analysis with bandit\nbuild                          Build project using poetry\nclean                          Clean project\n```\n\n### Run mongo docker image\nMost of our tests compare montydb CRUD results against real mongodb instance, therefore we must have a running\nmongodb before testing.\n\nFor example, if we want to test against mongo 4.4:\n```shell\ndocker run --name monty-4.4 -p 30044:27017 -d mongo:4.4\n```\n\n### Tests\n```shell\npoetry run pytest --storage {storage engin name} --mongodb {mongo instance url} [--use-bson]\n```\nExample:\n```shell\npoetry run pytest --storage memory --mongodb localhost:30044 --use-bson\n```\n\n## Why did I make this?\n\nMainly for personal skill practicing and fun.\n\nI work in the VFX industry and some of my production needs (mostly edge-case) requires to run in a limited environment (e.g. outsourced render farms), which may have problem to run or connect a MongoDB instance. And I found this project really helps.\n\n---\n\n\u003cp align=center\u003e\n    \u003ca href=\"https://jb.gg/OpenSource\"\u003e\u003ci\u003eThis project is supported by JetBrains\u003c/i\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"artwork/icon.png\" alt=\"drawing\" width=\"100\"/\u003e\n  \u0026nbsp;\u0026nbsp;\n    \u003cimg src=\"artwork/jetbrains.png\" alt=\"drawing\" width=\"100\"/\u003e\n\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidlatwe%2Fmontydb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavidlatwe%2Fmontydb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidlatwe%2Fmontydb/lists"}