{"id":13458151,"url":"https://github.com/Coldairarrow/EFCore.Sharding","last_synced_at":"2025-03-24T15:30:51.145Z","repository":{"id":38248724,"uuid":"234742043","full_name":"Coldairarrow/EFCore.Sharding","owner":"Coldairarrow","description":"Database Sharding For EFCore","archived":false,"fork":false,"pushed_at":"2024-04-27T13:35:55.000Z","size":638,"stargazers_count":692,"open_issues_count":23,"forks_count":142,"subscribers_count":24,"default_branch":"master","last_synced_at":"2024-05-15T20:10:08.830Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Coldairarrow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-01-18T13:53:11.000Z","updated_at":"2024-05-31T02:39:38.115Z","dependencies_parsed_at":"2024-02-17T06:31:22.987Z","dependency_job_id":"3d135cef-da50-4986-8e68-bbc048b1f49f","html_url":"https://github.com/Coldairarrow/EFCore.Sharding","commit_stats":{"total_commits":174,"total_committers":6,"mean_commits":29.0,"dds":0.04597701149425293,"last_synced_commit":"30790351ef80f754a2da9c14671c95de2c8db93b"},"previous_names":[],"tags_count":47,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Coldairarrow%2FEFCore.Sharding","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Coldairarrow%2FEFCore.Sharding/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Coldairarrow%2FEFCore.Sharding/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Coldairarrow%2FEFCore.Sharding/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Coldairarrow","download_url":"https://codeload.github.com/Coldairarrow/EFCore.Sharding/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221820672,"owners_count":16886222,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T09:00:45.646Z","updated_at":"2024-10-29T03:30:34.355Z","avatar_url":"https://github.com/Coldairarrow.png","language":"C#","funding_links":[],"categories":["C\\#","C# #"],"sub_categories":[],"readme":"[![Build Status](https://coldairarrow.visualstudio.com/Colder/_apis/build/status/EFCore.Sharding-ci?branchName=master)](https://coldairarrow.visualstudio.com/Colder/_build/latest?definitionId=4\u0026branchName=master)\n- [简介](#简介)\n- [引言](#引言)\n- [开始](#开始)\n  - [准备](#准备)\n  - [配置](#配置)\n  - [使用](#使用)\n  - [按时间自动分表](#按时间自动分表)\n  - [性能测试](#性能测试)\n  - [其它简单操作(非Sharing)](#其它简单操作非sharing)\n- [高级配置](#高级配置)\n  - [多主键等配置](#多主键等配置)\n  - [读写分离](#读写分离)\n- [注意事项](#注意事项)\n- [总结](#总结)\n\n# 简介\n本框架旨在为EF Core提供**Sharding**(即读写分离分库分表)支持,不仅提供了一套强大的普通数据操作接口,并且降低了分表难度,支持按时间自动分表扩容,提供的操作接口简洁统一.\n\n源码地址:[EFCore.SHarding](https://github.com/Coldairarrow/EFCore.Sharding)\n\n# 引言\n读写分离分库分表一直是数据库领域中的重难点,当数据规模达到单库极限的时候,就不得不考虑分表方案。EF Core作为.NET Core中最为主流的ORM,用起来十分方便快捷,但是官方并没有相应的Sharding支持,鄙人不才,经过一番摸索之后终于完成这个框架.\n\n# 开始\n## 准备\n首先根据需要安装对应的Nuget包 \n最新版基于EFCore5，若需要使用EFCore3请使用EFCore3.Sharding\n\n| 包名 | 说明 |\n|--|--|\n| EFCore.Sharding | 必装包 |\n| EFCore.Sharding.MySql | MySql支持 |\n| EFCore.Sharding.PostgreSql | PostgreSql支持 |\n| EFCore.Sharding.SQLite | SQLite支持 |\n| EFCore.Sharding.SqlServer | SqlServer支持 |\n| EFCore.Sharding.Oracle | Oracle支持 |\n| EFCore3.Sharding | 必装包 |\n| EFCore3.Sharding.MySql | MySql支持 |\n| EFCore3.Sharding.PostgreSql | PostgreSql支持 |\n| EFCore3.Sharding.SQLite | SQLite支持 |\n| EFCore3.Sharding.SqlServer | SqlServer支持 |\n| EFCore3.Sharding.Oracle | Oracle支持 |\n\n## 配置\n```c#\n\nServiceCollection services = new ServiceCollection();\n//配置初始化\nservices.AddEFCoreSharding(config =\u003e\n{\n    //添加数据源\n    config.AddDataSource(Config.CONSTRING1, ReadWriteType.Read | ReadWriteType.Write, DatabaseType.SqlServer);\n\n    //对3取模分表\n    config.SetHashModSharding\u003cBase_UnitTest\u003e(nameof(Base_UnitTest.Id), 3);\n});\n```\n上述代码中完成了Sharding配置\n- **AddEFCoreSharding**注入EFCoreSharding\n- **AddDataSource**添加分表数据源\n- **SetHashModSharding**是采用哈希取模的分表规则,分表字段为**Id**,取模值为**3**,会自动生成表Base_UnitTest_0,Base_UnitTest_1,Base_UnitTest_2\n\n## 使用\n配置完成，下面开始使用，使用方式**非常简单**，与平常使用基本一致  \n首先通过注入获取到IShardingDbAccessor\n\n```c#\nvar db=ServiceProvider.GetService\u003cIShardingDbAccessor\u003e();\n```\n\n然后即可进行数据操作：\n\n```c#\nBase_UnitTest _newData  = new Base_UnitTest\n{\n    Id = Guid.NewGuid().ToString(),\n    UserId = \"Admin\",\n    UserName = \"超级管理员\",\n    Age = 22\n};\nList\u003cBase_UnitTest\u003e _insertList = new List\u003cBase_UnitTest\u003e\n{\n    new Base_UnitTest\n    {\n        Id = Guid.NewGuid().ToString(),\n        UserId = \"Admin1\",\n        UserName = \"超级管理员1\",\n        Age = 22\n    },\n    new Base_UnitTest\n    {\n        Id = Guid.NewGuid().ToString(),\n        UserId = \"Admin2\",\n        UserName = \"超级管理员2\",\n        Age = 22\n    }\n};\n//添加单条数据\n_db.Insert(_newData);\n//添加多条数据\n_db.Insert(_insertList);\n//清空表\n_db.DeleteAll\u003cBase_UnitTest\u003e();\n//删除单条数据\n_db.Delete(_newData);\n//删除多条数据\n_db.Delete(_insertList);\n//删除指定数据\n_db.Delete\u003cBase_UnitTest\u003e(x =\u003e x.UserId == \"Admin2\");\n//更新单条数据\n_db.Update(_newData);\n//更新多条数据\n_db.Update(_insertList);\n//更新单条数据指定属性\n_db.Update(_newData, new List\u003cstring\u003e { \"UserName\", \"Age\" });\n//更新多条数据指定属性\n_db.Update(_insertList, new List\u003cstring\u003e { \"UserName\", \"Age\" });\n//更新指定条件数据\n_db.Update\u003cBase_UnitTest\u003e(x =\u003e x.UserId == \"Admin\", x =\u003e\n{\n    x.UserId = \"Admin2\";\n});\n//GetList获取表的所有数据\nvar list=_db.GetList\u003cBase_UnitTest\u003e();\n//Max\nvar max=_db.GetIShardingQueryable\u003cBase_UnitTest\u003e().Max(x =\u003e x.Age);\n//Min\nvar min=_db.GetIShardingQueryable\u003cBase_UnitTest\u003e().Min(x =\u003e x.Age);\n//Average\nvar min=_db.GetIShardingQueryable\u003cBase_UnitTest\u003e().Average(x =\u003e x.Age);\n//Count\nvar min=_db.GetIShardingQueryable\u003cBase_UnitTest\u003e().Count();\n//事务,使用方式与普通事务一致\nbool succcess = _db.RunTransaction(() =\u003e\n{\n    _db.Insert(_newData);\n    var newData2 = _newData.DeepClone();\n    _db.Insert(newData2);\n}).Success;\nAssert.AreEqual(succcess, false);\n```\n上述操作中表面上是操作Base_UnitTest表，实际上却在按照一定规则使用Base_UnitTest_0~2三张表，使分片对业务**操作透明**，极大提高开发效率  \n具体使用方式请参考单元测试源码：[连接](https://github.com/Coldairarrow/EFCore.Sharding/tree/master/src/EFCore.Sharding.Tests/Sharding)\n\n## 按时间自动分表\n上面的哈希取模的方式虽然简单,但是却十分不实用,因为当3张分表到达瓶颈时,将会面临扩容的问题，这种方式扩容需要进行大量的数据迁移，这无疑是十分麻烦的。因此需要一种方式能够系统自动建表扩容，并且无需人工干预，这就是按时间自动分表.\n\n```c#\n\nusing EFCore.Sharding;\nusing EFCore.Sharding.Tests;\nusing Microsoft.Extensions.DependencyInjection;\nusing Microsoft.Extensions.Logging;\nusing System;\nusing System.Threading.Tasks;\n\nnamespace Demo.DateSharding\n{\n    class Program\n    {\n        static async Task Main(string[] args)\n        {\n            DateTime startTime = DateTime.Now.AddMinutes(-5);\n            ServiceCollection services = new ServiceCollection();\n            services.AddLogging(config =\u003e\n            {\n                config.AddConsole();\n            });\n            //配置初始化\n            services.AddEFCoreSharding(config =\u003e\n            {\n                //添加数据源\n                config.AddDataSource(Config.CONSTRING1, ReadWriteType.Read | ReadWriteType.Write, DatabaseType.SqlServer);\n\n                //按分钟分表\n                config.SetDateSharding\u003cBase_UnitTest\u003e(nameof(Base_UnitTest.CreateTime), ExpandByDateMode.PerMinute, startTime);\n            });\n            var serviceProvider = services.BuildServiceProvider();\n\n            using var scop = serviceProvider.CreateScope();\n\n            var db = scop.ServiceProvider.GetService\u003cIShardingDbAccessor\u003e();\n            var logger = scop.ServiceProvider.GetService\u003cILogger\u003cProgram\u003e\u003e();\n\n            while (true)\n            {\n                try\n                {\n                    await db.InsertAsync(new Base_UnitTest\n                    {\n                        Id = Guid.NewGuid().ToString(),\n                        Age = 1,\n                        UserName = Guid.NewGuid().ToString(),\n                        CreateTime = DateTime.Now\n                    });\n\n                    DateTime time = DateTime.Now.AddMinutes(-2);\n                    var count = await db.GetIShardingQueryable\u003cBase_UnitTest\u003e()\n                        .Where(x =\u003e x.CreateTime \u003e= time)\n                        .CountAsync();\n                    logger.LogWarning(\"当前数据量:{Count}\", count);\n                }\n                catch (Exception ex)\n                {\n                    Console.WriteLine(ex.Message);\n                }\n\n                await Task.Delay(1000);\n            }\n        }\n    }\n}\n\n```\n上面Demo都在源码中  \n\n上面的代码实现了将Base_UnitTest表按照时间自动分表，每分钟创建一张表，实际使用中根据业务需求设置ExpandByDateMode参数，常用按天、按月分表\n\n自动分表效果\n![JKxE8K.png](https://s1.ax1x.com/2020/04/19/JKxE8K.png)\n全程无需人工干预，系统会自动定时创建分表，十分简单好用\n\n## 性能测试\n```c#\nusing EFCore.Sharding;\nusing EFCore.Sharding.Tests;\nusing Microsoft.EntityFrameworkCore;\nusing Microsoft.Extensions.DependencyInjection;\nusing System;\nusing System.Diagnostics;\nusing System.Linq;\nusing System.Linq.Expressions;\n\nnamespace Demo.Performance\n{\n    class Program\n    {\n        static void Main(string[] args)\n        {\n            ServiceCollection services = new ServiceCollection();\n            services.AddEFCoreSharding(config =\u003e\n            {\n                config.UseDatabase(Config.CONSTRING1, DatabaseType.SqlServer);\n\n                //添加数据源\n                config.AddDataSource(Config.CONSTRING1, ReadWriteType.Read | ReadWriteType.Write, DatabaseType.SqlServer);\n\n                //对3取模分表\n                config.SetHashModSharding\u003cBase_UnitTest\u003e(nameof(Base_UnitTest.Id), 3);\n            });\n            var serviceProvider = services.BuildServiceProvider();\n\n            var db = serviceProvider.GetService\u003cIDbAccessor\u003e();\n            var shardingDb = serviceProvider.GetService\u003cIShardingDbAccessor\u003e();\n            Stopwatch watch = new Stopwatch();\n\n            Expression\u003cFunc\u003cBase_UnitTest, bool\u003e\u003e where = x =\u003e EF.Functions.Like(x.UserName, $\"%00001C22-8DD2-4D47-B500-407554B099AB%\");\n\n            var q = db.GetIQueryable\u003cBase_UnitTest\u003e()\n                .Where(where)\n                .OrderByDescending(x =\u003e x.Id)\n                .Skip(0)\n                .Take(30);\n            var shardingQ = shardingDb.GetIShardingQueryable\u003cBase_UnitTest\u003e()\n                .Where(where)\n                .OrderByDescending(x =\u003e x.Id)\n                .Skip(0)\n                .Take(30);\n\n            //先执行一次预热\n            q.ToList();\n            shardingQ.ToList();\n\n            watch.Restart();\n            var list1 = q.ToList();\n            watch.Stop();\n            Console.WriteLine($\"未分表耗时:{watch.ElapsedMilliseconds}ms\");\n            watch.Restart();\n            var list2 = shardingQ.ToList();\n            watch.Stop();\n            Console.WriteLine($\"分表后耗时:{watch.ElapsedMilliseconds}ms\");\n\n            Console.WriteLine(\"完成\");\n            Console.ReadLine();\n        }\n    }\n}\n\n\n```\n\n分表Base_UnitTest_0-2各有100万数据,然后将这三张表的数据导入Base_UnitTest中(即Base_UnitTest表的数据与Base_UnitTest_0-2三张表总合数据一致) \n\n分表与不分表测试结果如下\n\n![JMSJBQ.png](https://s1.ax1x.com/2020/04/19/JMSJBQ.png)\n\n这里仅仅分了3张表，其效果立杆见影，若分表几十张，那效果想想就很棒\n\n## 其它简单操作(非Sharing)\n框架不仅支持Sharing,而且封装了常用数据库操作,使用比较简单  \n详细使用方式参考 [链接](https://github.com/Coldairarrow/EFCore.Sharding/blob/master/src/EFCore.Sharding.Tests/DbAccessor/DbAccessorTest.cs)\n\n# 高级配置\n## 多主键等配置\n\n多主键、索引等高级配置请使用**IEntityTypeConfiguration**\n参考[fluentApi](https://www.learnentityframeworkcore.com/configuration/fluent-api)\n\n## 读写分离\n数据库读写分离在大型项目中十分常见,通常在数据库层完成自动读写分离  \n- MySQL可以使用ProxySQL完成全自动读写分离集群  \n- PostgreSQL可以使用Pgool完成全自动读写分离集群  \n- SQLServer可以使用AlwaysOn,但是需要在连接字符串中加上 ApplicationIntent=ReadOnly,因此只是**半自动**的  \n本框架支持将半自动读写分离升级成全自动,即在代码层无需感知读写分离切换,代码层只需跟普通一样使用IDbAccessor即可  \n代码如下([链接](https://github.com/Coldairarrow/EFCore.Sharding/blob/master/examples/Demo.ReadWrite/Program.cs))  \n```c#\nusing EFCore.Sharding;\nusing EFCore.Sharding.Tests;\nusing Microsoft.EntityFrameworkCore;\nusing Microsoft.Extensions.DependencyInjection;\nusing Microsoft.Extensions.Logging;\nusing System;\nusing System.Threading.Tasks;\n\nnamespace Demo.ReadWrite\n{\n    class Program\n    {\n        static async Task Main(string[] args)\n        {\n            ServiceCollection services = new ServiceCollection();\n            services.AddLogging(config =\u003e\n            {\n                config.AddConsole();\n            });\n            services.AddEFCoreSharding(config =\u003e\n            {\n                config.SetEntityAssembly(\"EFCore.Sharding\");\n\n                //SQLITE1作为主库(写库)\n                //SQLITE2作为从库(读库)\n                config.UseDatabase(new (string, ReadWriteType)[]\n                {\n                    (Config.SQLITE1, ReadWriteType.Write),\n                    (Config.SQLITE2, ReadWriteType.Read)\n                }, DatabaseType.SQLite);\n            });\n            var serviceProvider = services.BuildServiceProvider();\n\n            using var scop = serviceProvider.CreateScope();\n            //拿到注入的IDbAccessor即可进行所有数据库操作\n            var db = scop.ServiceProvider.GetService\u003cIDbAccessor\u003e();\n            var logger = scop.ServiceProvider.GetService\u003cILogger\u003cProgram\u003e\u003e();\n            while (true)\n            {\n                await db.InsertAsync(new Base_UnitTest\n                {\n                    Age = 100,\n                    CreateTime = DateTime.Now,\n                    Id = Guid.NewGuid().ToString(),\n                    UserId = Guid.NewGuid().ToString(),\n                    UserName = Guid.NewGuid().ToString()\n                });\n                var count = await db.GetIQueryable\u003cBase_UnitTest\u003e().CountAsync();\n\n                //注意:这里数量始终为0,因为SQLITE1与SQLITE2没有开启主从复制\n                //在实际使用中应在数据库层开启主从复制\n                logger.LogWarning(\"当前数量:{Count}\", count);\n\n                await Task.Delay(1000);\n            }\n        }\n    }\n}\n\n```\n# 注意事项\n\n- 查询尽量使用分表字段进行筛选，避免全表扫描\n\n# 总结\n这个简单实用强大的框架希望能够帮助到大家,力求为.NET生态贡献一份力,大家一起壮大.NET生态\n\n欢迎使用本框架，若觉得不错，请比心\n\n![](https://raw.githubusercontent.com/Coldairarrow/UploadFiles/master/Colder.Fx.Net.AdminLTE/04abaa3d37fa01b4c4058c8163aab6a8.jpg)\n\nGithub欢迎星星:\u003chttps://github.com/Coldairarrow\u003e\n\n博客园欢迎点赞：\u003chttps://www.cnblogs.com/coldairarrow/\u003e\n\nQQ群3:940069478  \n个人QQ:862520575（**欢迎技术支持及商务合作**）    \n\n\u003ccenter\u003e 本人将会对这个快速开发框架不断完善与维护，希望能够帮助到各位\u003ccenter/\u003e\n\u003ccenter\u003e 若遇到任何问题或需要技术支持，请联系我\u003ccenter/\u003e\n\u003ccenter\u003e ------学习永无止境，技术永无上限，代码就是艺术------\u003ccenter/\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FColdairarrow%2FEFCore.Sharding","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FColdairarrow%2FEFCore.Sharding","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FColdairarrow%2FEFCore.Sharding/lists"}