{"id":19524419,"url":"https://github.com/sepgh/testudo","last_synced_at":"2025-06-14T02:35:52.939Z","repository":{"id":235181619,"uuid":"790228600","full_name":"sepgh/testudo","owner":"sepgh","description":"Simple embedded java database","archived":false,"fork":false,"pushed_at":"2025-01-16T06:54:28.000Z","size":2730,"stargazers_count":20,"open_issues_count":0,"forks_count":4,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-26T10:35:58.792Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sepgh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-22T13:54:09.000Z","updated_at":"2025-04-14T18:49:48.000Z","dependencies_parsed_at":"2024-05-03T10:57:56.797Z","dependency_job_id":"e988d158-1d5b-44a6-8797-e94c61a72afd","html_url":"https://github.com/sepgh/testudo","commit_stats":null,"previous_names":["sepgh/testudo"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sepgh/testudo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sepgh%2Ftestudo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sepgh%2Ftestudo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sepgh%2Ftestudo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sepgh%2Ftestudo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sepgh","download_url":"https://codeload.github.com/sepgh/testudo/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sepgh%2Ftestudo/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259749562,"owners_count":22905731,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T00:48:35.369Z","updated_at":"2025-06-14T02:35:52.881Z","avatar_url":"https://github.com/sepgh.png","language":"Java","funding_links":[],"categories":["数据库"],"sub_categories":["Spring Cloud框架"],"readme":"# Testudo\n\n\u003cp\u003e\n  \u003cimg width=\"30%\" src=\"https://github.com/sepgh/testudo/blob/main/.docs/assets/Testudo.png\" align=\"left\" /\u003e\n  \n**Simple embedded database for java**\n\n\nThis project is _a practice_ for implementing a minimal database system. The practice includes different indexing mechanisms (B+Tree and Bitmaps for example), \ndealing with data storage on disk (Page and Page buffers), caching (LRU), locking (Reader-Writer Lock), parsing and other topics.\n\n\n\u003cbr clear=\"left\"/\u003e\n\u003c/p\u003e\n\n\n\n## Introduction\n\n\u003e Notice: the idea of the project is still evolving! Anything that you may read here is open for changes in future.\n\nThe library implemented in this project repository empowers a java application to store data in `collection-field` formats and it provides the sensation - and minimal features - of a database system.\n\nThe project is meant to be a practice so that the developer gets involved with some software engineering aspects and gain knowledge, and it may not solve a real world problem that other database implementations (including the embedded ones) don't solve already.\n\nI started studying a tutorial called [\"Let's Build a Simple Database\"](https://cstack.github.io/db_tutorial/) which is about \"writing a sqlite clone from scratch in C\" till I met BTree algorithm section.\nLater, I understood more about BTree and B+Tree from [\"B Trees and B+ Trees. How they are useful in Databases\" on Youtube](https://www.youtube.com/watch?v=aZjYr87r1b8), and eventually found myself working on this mini project.\n\n**The thought process, progress and more details of this project is explained on a [youtube playlist called \"Write a database from scratch\"](https://www.youtube.com/watch?v=HHO2K23XxbM\u0026list=PLWRwj01AnyEtjaw-ZnnAQWnVYPZF5WayV)**. If you are visiting this repository from Youtube, welcome. If not, I suggest you to take a look at the playlist.\n\n\n\n## Development Progress\n\n- [X] Basic implementation of Tree-Based [`Unique Index Management`](https://github.com/sepgh/testudo/blob/main/src/main/java/com/github/sepgh/testudo/index/UniqueTreeIndexManager.java) using B+Tree \n- [X] Cluster implementation of B+Tree\n- [ ] All \"collections\" should have a default cluster B+Tree implementation despite their Primary key. Perhaps the Primary Key should point to the cluster id of a collection object (traditionally known as a row of a table).\n  - [X] Implement UnsignedLong serializer, which would be cluster key type\n  - [ ] Update `SchemeManager` logic\n  - [ ] Update `FieldIndexManagerProvider` logic (name should get updated as well, right?)\n- [X] Storage management of B+Tree\n  - [X] Storage management using `CompactFileIndexStorageManager` and `OrganizedFileIndexStorageManager`\n  - [X] Additional Storage Management implementation using `DatabaseStorageManager` (the `DiskDatabaseStorageManager` uses `PageBuffer`s)\n- [X] Generic decorator pattern for `UniqueTreeIndexManager`\n- [X] LRU Cache implementation for `UniqueTreeIndexManager` decorator. **(More tests required)**\n- [X] Providing solution for differentiating `zero` (number) and `null` in a byte array. **(Flag byte is used, which is not the best solution, bitmaps can help here)**\n- [X] Non-unique Index Manager\n  - [X] B+Tree and `ArrayList` (binary) implementation\n  - [X] Bitmap implementation\n- [ ] Database Storage Manager\n  - [X] Disk Database Storage Manager Basics (Page Buffer)\n  - [X] `RemovedObjectTracer` implementations (default: `InMemoryRemovedObjectTracer`) should support splitting the traced objects if the chosen traced position plus requested size is larger than a threshold.\n  - [ ] Write Queue to make disk writes (page committing) Async (is it even safe/possible?)\n- [X] Query\n  - [X] Implement `QueryableInterface` to support quicker query operations on IndexManagers (Done for `LT`, `LTE`, `GT`, `GTE`, `EQ`)\n  - [X] Implement Query Class\n- [X] Serialization\n  - [X] Defining basic serializers for some types including `int`, `long`, `bool`, `char`\n  - [X] Model to Scheme Collection conversion\n  - [X] Model Serialization\n  - [X] Model DeSerialization\n- [X] Add support for nullable fields and update model serializer logic\n- [ ] Possibly improve `CharArray` serializer to use less space\n- [ ] Support primitive data types (ex: models should be able to have `int` field instead of `Integer`)\n- [X] Insertion (and update) verification:\n  - [X] `primary` index should either have a value or get set as autoincrement\n  - [X] Unique indexes should be verified before any update or insertion is performed (after locking) \n- [X] Database Operations\n  - [X] Reader-Writer Lock support at collection level\n  - [X] Select Operation\n  - [X] Insert Operation\n  - [X] Update Operation\n  - [X] Delete Operation\n- [ ] Either remove IndexIOSession, or improve it to use for `Transaction` support.\n- [X] Cache support for cluster index managers\n- [ ] Cache support for other indexes. Also find proper usage for `CachedIndexStorageManagerDecorator` or remove it!\n- [X] Exception Throwing and Handling\n  - Note: lambdas are going crazy at this point. Use this strategy: https://stackoverflow.com/questions/18198176\n- [X] Shutdown mechanism (gracefully)\n- [ ] Logging\n  - [X] Error logging for ignored or automatically handled exceptions\n  - [ ] Info and Debug logging\n- [ ] Performance and Overall Improvements Ideas\n  - Update process can be distributed into multiple threads for the `databaseStorage.update()` part\n  - There are a bunch of places that we can benefit from Binary Search that have Todos on them.\n  - If index ids have a better way of generation, we can use them in index storage managers such as `DiskPageFileIndexStorageManager` to determine the scheme an object belongs to! This also works for storing bitmaps and array lists in db file for `DuplicateIndexManagers`.\n  - **Bitmap Space**: use compression or sparse bitmaps\n\n\n## Open Problems\n\n### Can't perform `query` operation on fields that are not indexed. \n\nDoing so right now will make us use cluster index, load objects into memory to perform comparisons, and the result would be `Iterator\u003cV\u003e` where V is cluster id.\nThis means that these objects may later get loaded into memory again. We need a solution to avoid loading objects twice (once for query, once for the higher level operation such as read/update/delete)\n\n**Additional Note**: from performance point of view things are not as awful as it seems:\n\n1. We have a pool of DBObjects per each `page` that is loaded in Page Buffer. Pages may already be in LRU cache, and DBObject pool prevents recreation of new objects in memory (more of a way to reduce memory consumption than performance improvement)\n2. We shall use LRU cache for Cluster Index Manager, which means re-reading objects from the cluster index should perform quicker than hitting the disk multiple times.\n\n\n### Bitmap Max Size\n\nWhile cluster IDs are set to support `Unsigned Long`, \ncurrent implementation and usage of Bitmap in indexes can't support any value larger than an integer.\nThis is because a `byte[]` is used and the index passed to an array in java can only be an integer.\n\n### Storing indexes in same file as DB   (DONE | NOT TESTED)\n\nThe current implementation has a problem with this, since two instances of `DiskPageDatabaseStorage` will be created and `synchronized` blocks wouldn't perform validly.\n\nEven though current tests work, when we work with multiple collections things will break.\nThe reason `CollectionSelectInsertOperationMultiThreadedTestCase` can work with Page Buffer is that we lock the collection in `DefaultCollectionInsertOperation`, \nso even though we have multiple instances of `DiskPageDatabaseStorage` (one for DB and one for index), their `.store()` method won't be called from multiple threads.\n\nNote: just using same instance is not enough. The type of the pointer returned by `store` method would be different then. Maybe we should not let the Database Storage Manager handle pointer types? seems totally unnecessary!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsepgh%2Ftestudo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsepgh%2Ftestudo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsepgh%2Ftestudo/lists"}