{"id":18071762,"url":"https://github.com/diverofdark/objectrepository","last_synced_at":"2025-04-12T02:53:07.311Z","repository":{"id":48425175,"uuid":"116201129","full_name":"DiverOfDark/ObjectRepository","owner":"DiverOfDark","description":"EscapeTeams In-Memory Object Database","archived":false,"fork":false,"pushed_at":"2023-02-24T16:35:47.000Z","size":144,"stargazers_count":27,"open_issues_count":15,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-25T22:36:32.556Z","etag":null,"topics":["csharp","in-memory-database","netcore2","netstandard20"],"latest_commit_sha":null,"homepage":null,"language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DiverOfDark.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-01-04T01:46:16.000Z","updated_at":"2024-06-11T15:51:18.000Z","dependencies_parsed_at":"2023-01-25T14:01:16.619Z","dependency_job_id":null,"html_url":"https://github.com/DiverOfDark/ObjectRepository","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiverOfDark%2FObjectRepository","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiverOfDark%2FObjectRepository/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiverOfDark%2FObjectRepository/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiverOfDark%2FObjectRepository/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DiverOfDark","download_url":"https://codeload.github.com/DiverOfDark/ObjectRepository/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248509811,"owners_count":21116124,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csharp","in-memory-database","netcore2","netstandard20"],"created_at":"2024-10-31T09:16:31.062Z","updated_at":"2025-04-12T02:53:07.280Z","avatar_url":"https://github.com/DiverOfDark.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ObjectRepository\nGeneric In-Memory Object Database (Repository pattern)\n\n## Why store anything in memory?\n\nMost people would use SQL-based database for backend.\nBut sometimes SQL just don't fit well - i.e. when you're building a search engine or when you need to query social graph in eloquent way.\n\n**Worst of all is when your teammate doesn't know how to write fast queries. How much time was spent debugging N+1 issues and building additional indexes just for the sake of main page load speed?**\n\nAnother approach would be to use NoSQL. Several years ago there was a big hype about it - every microservice had used MongoDB and everyone was happy getting JSON documents *(btw, how about circular references?)*\n\nWhy not store everything in-memory, sometimes flushing all on the underlying storage (i.e. file, remote database)?\n\nMemory is cheap, and all kind of small and medium-sized projects would take no more than 1 Gb of memory. *(i.e. my favorite home project - [BudgetTracker](https://github.com/DiverOfDark/BudgetTracker), which stores daily stats of all my transcations and balances uses just 45 mb after 1.5 years of history)*\n\nPros:\n\n- Access to data is easier - you don't need to think about writing queries, eager loading or ORM-dependent stuff. You work regular C# objects;\n- No issues due to multithreading;\n- Very fast - no network calls, no generating queries, no (de)serialization;\n- You can store data in any way you like - be it XML file on disk, SQL Server, or Azure Table Storage.\n\nCons:\n\n- You can't scale horizontally, thus no blue-green deployment;\n- If app crashes - you can lost you latest data. *(But YOUR app never crashes, right?)*\n\n\n## How it works?\n\nIn a nutshell:\n\n- On application start connection to data storage is established, and initial load begins;\n- Object model is created, (primary) indexes are calculated;\n- Subscription to model's property changes (INotifyPropertyChanged) and collection changes (INotifyCollectionChanged) is created;\n- When something changes - event is raised and changed object is added to queue for persisting;\n- Persisting occurs by timer in background thread;\n- When application exits - additional save is called.\n\n## Usage:\n\n```cs\n// Required dependencies:\n  \n// Core library\nInstall-Package OutCode.EscapeTeams.ObjectRepository\n    \n// Storage - you one which you need.\nInstall-Package OutCode.EscapeTeams.ObjectRepository.File\nInstall-Package OutCode.EscapeTeams.ObjectRepository.LiteDb\nInstall-Package OutCode.EscapeTeams.ObjectRepository.AzureTableStorage\n    \n// Optional - it is possible to store hangfire data in ObjectRepository\n// Install-Package OutCode.EscapeTeams.ObjectRepository.Hangfire\n```\n\n```cs\n// Data Model - it is how all will be stored.\n  \npublic class ParentEntity : BaseEntity\n{\n    public ParentEntity(Guid id) =\u003e Id = id;\n}\n    \npublic class ChildEntity : BaseEntity\n{\n    public ChildEntity(Guid id) =\u003e Id = id;\n    public Guid ParentId { get; set; }\n    public string Value { get; set; }\n}\n```\n\n```cs\n// Object Model - something your app will work with\n\npublic class ParentModel : ModelBase\n{\n    public ParentModel(ParentEntity entity)\n    {\n        Entity = entity;\n    }\n    \n    public ParentModel()\n    {\n        Entity = new ParentEntity(Guid.NewGuid());\n    }\n    \n    // 1-Many relation\n    public IEnumerable\u003cChildModel\u003e Children =\u003e Multiple\u003cChildModel\u003e(() =\u003e x =\u003e x.ParentId);\n    \n    protected override BaseEntity Entity { get; }\n}\n    \npublic class ChildModel : ModelBase\n{\n    private ChildEntity _childEntity;\n    \n    public ChildModel(ChildEntity entity)\n    {\n        _childEntity = entity;\n    }\n    \n    public ChildModel() \n    {\n        _childEntity = new ChildEntity(Guid.NewGuid());\n    }\n    \n    public Guid ParentId\n    {\n        get =\u003e _childEntity.ParentId;\n        set =\u003e UpdateProperty(_childEntity, () =\u003e x =\u003e x.ParentId, value);\n    }\n    \n    public string Value\n    {\n        get =\u003e _childEntity.Value;\n        set =\u003e UpdateProperty(_childEntity, () =\u003e x =\u003e x.Value, value);\n    }\n    \n    // Indexed access\n    public ParentModel Parent =\u003e Single\u003cParentModel\u003e(ParentId);\n    \n    protected override BaseEntity Entity =\u003e _childEntity;\n}\n```\n\n```cs\n// ObjectRepository itself\n    \npublic class MyObjectRepository : ObjectRepositoryBase\n{\n    public MyObjectRepository(IStorage storage) : base(storage, NullLogger.Instance)\n    {\n        IsReadOnly = true; // For testing purposes. Allows to not save changes to database.\n    \n        AddType((ParentEntity x) =\u003e new ParentModel(x));\n        AddType((ChildEntity x) =\u003e new ChildModel(x));\n    \n        //// If you are using hangfire and want to store it's data in this objectrepo - uncomment this\n        // this.RegisterHangfireScheme(); \n    \n        Initialize();\n    }\n}\n```\n\nCreate ObjectRepository:\n\n```cs\nvar memory = new MemoryStream();\nvar db = new LiteDatabase(memory);\nvar dbStorage = new LiteDbStorage(db);\n    \nvar repository = new MyObjectRepository(dbStorage);\nawait repository.WaitForInitialize();\n\n/* if you need HangFire \npublic void ConfigureServices(IServiceCollection services, ObjectRepository objectRepository)\n{\n    services.AddHangfire(s =\u003e s.UseHangfireStorage(objectRepository));\n}\n*/\n```\n\nInserting new object:\n\n```cs\nvar newParent = new ParentModel()\nrepository.Add(newParent);\n```\n\nAfter this **ParentModel** will be added to both local cache and to the queue to persist. Thus this op is O(1) and you can continue you work right away.\n\nTo check that this object is added and is the same you added:\n\n```cs\nvar parents = repository.Set\u003cParentModel\u003e();\nvar myParent = parents.Find(newParent.Id);\nAssert.IsTrue(ReferenceEquals(myParent, newParent));\n```\n\nWhat happens here? *Set\u0026lt;ParentModel\u0026gt;()* returns *TableDictionary\u0026lt;ParentModel\u0026gt;* which is essentially *ConcurrentDictionary\u0026lt;ParentModel, ParentModel\u0026gt;* and provides additional methods for indexes. This allows to have a Find methods to search by Id (or other fields) without iterating all objects.\n\nWhen you add something to *ObjectRepository* subscription to property changes is created, thus any property change also add object to write queue.\nProperty updating looks just like regular POCO object::\n\n```cs\nmyParent.Children.First().Property = \"Updated value\";\n```\n\nYou can delete object in following ways:\n\n```cs\nrepository.Remove(myParent);\nrepository.RemoveRange(otherParents);\nrepository.Remove\u003cParentModel\u003e(x =\u003e !x.Children.Any());\n```\n\nDeletion also happens via queue in background thread.\n\n## How saving actually works?\n\nWhen any object set tracked by *ObjectRepository* is changed (i.e. added, removed, property update) then event *ModelChanged* is raised.\n*IStorage*, which is used for persistence, is subscribed to this event. All implementations of *IStorage* are enqueueing all *ModelChanged* events to 3 different queues - for addition, update, and removal.\n\nAlso each kind of *IStorage* creates timer which every 5 secs invokes actual saving.\n\n*BTW, there exists separate API for explicit saving: **ObjectRepository.Save()**.*\n\nBefore each saving unneccessary operations are removed from the queue (i.e. multiple property changes, adding and removal of same object). After queue is sanitized actual saving is performed. \n\n*In all cases when object is persisted - it is persisted as a whole. So it is possible a scenario when objects are saving in different order than they were changed, including objects being saved with newer property values than were at the time of adding to queue.*\n\n## What else?\n\n- All libraries are targeted to .NET Standard 2.0. ObjectRepository can be used in any modern .NET app.\n- All API is thread-safe. Inner collections are based on *ConcurrentDictionary* and all handlers are either have locks or don't need them. \n- Only thing you should remember - don't forget to call *ObjectRepository.Save();* when your app is going to shutdown\n- If you need fast search - you can use custom indexes (works only for unique values):\n\n```cs\nrepository.Set\u003cChildModel\u003e().AddIndex(() =\u003e x =\u003e x.Value);\nrepository.Set\u003cChildModel\u003e().Find(() =\u003e x =\u003e x.Value, \"myValue\");\n```\n\n## Who uses this?\n\nI am using this library in all my hobby projects because it is simple and handy. In most cases I don't have to set up SQL Server or use some pricey cloud service - LiteDB/file-based approach is fine.\n\nA while ago, when I was bootstrapping EscapeTeams startup - we used Azure Table Storage as backing storage.\n\n## Plans for future\n\nWe want to solve major pain point of current approach - horizontal scaling. For this to happen we need to either implement distributed transactions(sic!) or to accept the fact that same objects in different instances should be not changed at the same moment of time (or latest who changed wins).\n\nFrom tech point of view this can be solved in following way:\n\n- Store event log and snapshot instead of actual model\n- Find other instances (add endpoints to settings? use udp discovery? master/slave or peers?)\n- Replicate eventlog between instances using any consensus algo, i.e. raft.\n\nOther issue that exists (and worries me) is cascade deletion(and finding cases when you are deleting object that is being references by some other object). It is just not implemented, and currently exceptions may be thrown when such issue happens.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdiverofdark%2Fobjectrepository","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdiverofdark%2Fobjectrepository","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdiverofdark%2Fobjectrepository/lists"}