{"id":13431266,"url":"https://github.com/borisdj/EFCore.BulkExtensions","last_synced_at":"2025-03-16T11:31:21.261Z","repository":{"id":37665059,"uuid":"91096213","full_name":"borisdj/EFCore.BulkExtensions","owner":"borisdj","description":"Entity Framework EF Core efcore Bulk Batch Extensions with BulkCopy in .Net for Insert Update Delete Read (CRUD), Truncate and SaveChanges operations on SQL Server, PostgreSQL, MySQL, SQLite, Oracle","archived":false,"fork":false,"pushed_at":"2025-03-11T09:16:04.000Z","size":3769,"stargazers_count":3774,"open_issues_count":90,"forks_count":603,"subscribers_count":85,"default_branch":"master","last_synced_at":"2025-03-11T21:01:05.721Z","etag":null,"topics":["batch","bulk","copy","efcore","entity-framework-core","entityframework","entityframeworkcore","mysql","postgresql","sql","sqlbulkcopy","sqlite","sqlserver"],"latest_commit_sha":null,"homepage":"https://codis.tech/efcorebulk","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/borisdj.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["borisdj"]}},"created_at":"2017-05-12T13:50:07.000Z","updated_at":"2025-03-11T09:16:15.000Z","dependencies_parsed_at":"2022-07-18T16:53:04.780Z","dependency_job_id":"87113cad-d39d-4d64-8f3b-08c07fa4a6a0","html_url":"https://github.com/borisdj/EFCore.BulkExtensions","commit_stats":{"total_commits":1586,"total_committers":121,"mean_commits":"13.107438016528926","dds":0.1702395964691047,"last_synced_commit":"03934e648db21b790f903b008d0deafb0a71cd1e"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/borisdj%2FEFCore.BulkExtensions","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/borisdj%2FEFCore.BulkExtensions/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/borisdj%2FEFCore.BulkExtensions/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/borisdj%2FEFCore.BulkExtensions/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/borisdj","download_url":"https://codeload.github.com/borisdj/EFCore.BulkExtensions/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243862815,"owners_count":20360215,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch","bulk","copy","efcore","entity-framework-core","entityframework","entityframeworkcore","mysql","postgresql","sql","sqlbulkcopy","sqlite","sqlserver"],"created_at":"2024-07-31T02:01:01.810Z","updated_at":"2025-03-16T11:31:21.246Z","avatar_url":"https://github.com/borisdj.png","language":"C#","readme":"# EFCore.BulkExtensions\nEntityFrameworkCore extensions (performance improvement - into overdrive):  \n-Bulk operations (super fast): **Insert, Update, Delete, Read, Upsert, Sync, SaveChanges.**  \n-Batch ops: **Update, Delete** - Deprecated from EF8 since EF7+ has native Execute-Up/Del.  \n-AddOps (additional): **Truncate.**  \nLibrary is Lightweight and very Efficient (warp speed), having all mostly used [CRUD](https://en.wikipedia.org/wiki/Create,_read,_update_and_delete) operation.  \nWas selected in top 20 [EF Core Extensions](https://docs.microsoft.com/en-us/ef/core/extensions/) recommended by Microsoft.  \nLatest version is using EF Core 9.  \nSupports all 5 major sql databases: **SQLServer, PostgreSQL, MySQL, Oracle, SQLite**  \nCheck out [Testimonials](https://docs.google.com/spreadsheets/d/e/2PACX-1vShdv2sTm3oQfowm9kVIx-PLBCk1lGQEa9E6n92-dX3pni7-XQUEp6taVcMSZVi9BaSAizv1YanWTy3/pubhtml?gid=801420190\u0026single=true) from the Community and User Comments.  \nWith thousands of pleased users and many satisfied clients from around the globe.  \nIcon\u003e\u003e and Logo (__):  \n\u003cimg src=\"/EFCore.BulkExtensions/EFCoreBulk.png\" height=60\u003e \u0026 \u003cimg src=\"EFCoreBulkLogo_small.png\" height=60\u003e  \n(f.forward | rocket time)\n\n**Also take a look into others packages:\u003c/br\u003e\nOpen source (MIT or cFOSS) authored [.Net libraries](https://infopedia.io/dot-net-libraries/) and other projects (@[**Infopedia.io**](https://infopedia.io/) personal blog post)\n| №  | Project                  | Type | Description                                              |\n| -  | ------------------------ | ---- | -------------------------------------------------------- |\n| 1* | [EFCore.BulkExtensions](https://github.com/borisdj/EFCore.BulkExtensions) | .Net Lib.(nuget) | EF Core Bulk CRUD Ops (**Flagship** Lib) |\n| 2  | [EFCore.UtilExtensions](https://github.com/borisdj/EFCore.UtilExtensions) | .Net Lib.(nuget) | EF Core Custom Annotations and AuditInfo |\n| 3  | [EFCore.FluentApiToAnnotation](https://github.com/borisdj/EFCore.FluentApiToAnnotation) | .Net Lib.(nuget) | Converting FluentApi configuration to Annotations |\n| 4  | [ExcelIO.FastMapper](https://github.com/borisdj/ExcelIO.FastMapper) | .Net Lib.(nuget) | Excel I/O Mapper to-from Poco \u0026 .xlsx with attribute |\n| 5  | [FixedWidthParserWriter](https://github.com/borisdj/FixedWidthParserWriter) | .Net Lib.(nuget) | Reading \u0026 Writing fixed-width/flat data files |\n| 6  | [CsCodeGenerator](https://github.com/borisdj/CsCodeGenerator) | .Net Lib.(nuget) | C# code generation based on Classes and elements |\n| 7  | [CsCodeExample](https://github.com/borisdj/CsCodeExample) | C# Code | Examples of C# code in form of a simple tutorial |\n\n## License\nBulkExtensions [licensed](https://github.com/borisdj/EFCore.BulkExtensions/blob/master/LICENSE.txt) under [**Dual License v1**](https://codis.tech/efcorebulk) (**cFOSS**: conditionallyFree OSS - [OpenSource Sustainability](https://infopedia.io/solution-to-opensource-sustainability/) \u0026 funding).  \nIf you do not meet criteria for free usage of software with community license then you have to buy commercial one.  \nIf eligible for free usage but still need  active support, consider purchasing Starter Lic.  \n\n## Support\nIf you find this project useful you can mark it by leaving a Github **Star** :star:  \nAnd even with Community license, if you want help Development, you can make a Donation:  \n[\u003cimg src=\"https://www.buymeacoffee.com/assets/img/custom_images/yellow_img.png\" alt=\"Buy Me A Coffee\" height=28\u003e](https://www.buymeacoffee.com/boris.dj) _ or _ \n[![Button](https://img.shields.io/badge/donate-Bitcoin-orange.svg?logo=bitcoin):zap:](https://borisdj.net/donation/donate-btc.html) ([Moneylution](https://infopedia.io/revolution-of-money/))\n\n## Contributing\nPlease read [CONTRIBUTING](https://github.com/borisdj/EFCore.BulkExtensions/blob/master/CONTRIBUTING.md) for details on code of conduct, and the process for submitting pull requests. \u003c!-- valid link short also (CONTRIBUTING.md) --\u003e   \nWhen opening issues do write detailed explanation of the problem or feature with reproducible example.  \nWant to **Contact** for Development \u0026 Consulting: [www.codis.tech](http://www.codis.tech) (*Quality Delivery*)  \n\n## Description\nSupported databases:  \n-**SQLServer** (or AzureSQL) under the hood uses [SqlBulkCopy](https://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlbulkcopy.aspx) for Insert, Update/Delete = BulkInsert + raw Sql [MERGE](https://docs.microsoft.com/en-us/sql/t-sql/statements/merge-transact-sql).  \n-**PostgreSQL** (9.5+) is using [COPY BINARY](https://www.postgresql.org/docs/9.2/sql-copy.html) combined with [ON CONFLICT](https://www.postgresql.org/docs/10/sql-insert.html#SQL-ON-CONFLICT) for Update.  \n-**MySQL** (8+) is using [MySqlBulkCopy](https://mysqlconnector.net/api/mysqlconnector/mysqlbulkcopytype/) combined with [ON DUPLICATE](https://dev.mysql.com/doc/refman/8.0/en/insert-on-duplicate.html) for Update.  \n-**Oracle** (8+) is using [OracleBulkCopy](https://docs.oracle.com/cd/E11882_01/win.112/e23174/OracleBulkCopyClass.htm#ODPNT7446) combined with [MERGE](https://docs.oracle.com/en/database/oracle/oracle-database/21/sqlrf/MERGE.html) for Update.  \n-**SQLite** has no Copy tool, instead library uses [plain SQL](https://learn.microsoft.com/en-us/dotnet/standard/data/sqlite/bulk-insert) combined with [UPSERT](https://www.sqlite.org/lang_UPSERT.html).  \nBulk Tests can not have UseInMemoryDb because InMemoryProvider does not support Relational-specific methods.  \nInstead Test options are  SqlServer(Developer or Express), LocalDb(if alongside [Developer v.](https://stackoverflow.com/questions/42885377/sql-server-2016-developer-version-can-not-connect-to-localdb-mssqllocaldb?noredirect=1\u0026lq=1)), or with  other adapters.\n\n## Installation\nAvailable on [![NuGet](https://img.shields.io/nuget/v/EFCore.BulkExtensions.svg)](https://www.nuget.org/packages/EFCore.BulkExtensions/)  [![Downloads](https://img.shields.io/nuget/dt/EFCore.BulkExtensions.svg)](https://www.nuget.org/packages/EFCore.BulkExtensions/)  \nMain nuget is for all Databases, and specific ones with single provider for those who need small packages.  \nPackage manager console command for installation: *Install-Package EFCore.BulkExtensions*  \nSpecific ones have adapter suffix: MainNuget + *.SqlServer/PostgreSql/MySql/Oracle/Sqlite* \n(\n[![](https://img.shields.io/static/v1?label=\u0026message=MS\u0026color=darkred)](https://www.nuget.org/packages/EFCore.BulkExtensions.SqlServer)\n[![](https://img.shields.io/static/v1?label=\u0026message=PG\u0026color=blue)](https://www.nuget.org/packages/EFCore.BulkExtensions.PostgreSql)\n[![](https://img.shields.io/static/v1?label=\u0026message=MY\u0026color=chocolate)](https://www.nuget.org/packages/EFCore.BulkExtensions.MySql)\n[![](https://img.shields.io/static/v1?label=\u0026message=OR\u0026color=red)](https://www.nuget.org/packages/EFCore.BulkExtensions.Oracle)\n[![](https://img.shields.io/static/v1?label=\u0026message=LT\u0026color=lightgreen)](https://www.nuget.org/packages/EFCore.BulkExtensions.Sqlite)\n)  \nIts assembly is [Strong-Named](https://docs.microsoft.com/en-us/dotnet/standard/library-guidance/strong-naming) and [Signed](https://github.com/borisdj/EFCore.BulkExtensions/issues/161) with a key.\n| Nuget | Target          | Used EF v.| For projects targeting          |\n| ----- | --------------- | --------- | ------------------------------- |\n| 9.x   | Net 9.0         | EF Core 9 | Net 9.0+                        |\n| 8.x   | Net 8.0         | EF Core 8 | Net 8.0+                        |\n| 7.x   | Net 6.0         | EF Core 7 | Net 7.0+ or 6.0+                |\n| 6.x   | Net 6.0         | EF Core 6 | Net 6.0+                        |\n| 5.x   | NetStandard 2.1 | EF Core 5 | Net 5.0+                        |\n| 3.x   | NetStandard 2.0 | EF Core 3 | NetCore(3.0+) or NetFrm(4.6.1+) [info](https://github.com/borisdj/EFCore.BulkExtensions/issues/271#issuecomment-567117488)|\n| 2.x   | NetStandard 2.0 | EF Core 2 | NetCore(2.0+) or NetFrm(4.6.1+) |\n| 1.x   | NetStandard 1.4 | EF Core 1 | NetCore(1.0+)                   |\n\nSupports follows official [.Net lifecycle](https://dotnet.microsoft.com/en-us/platform/support/policy/dotnet-core), currently v.9 as latest and v.8(LTS).  \n**Currently *Pomelo.EntityFrameworkCore.MySql* still does not have full Release for EF9 so its nuget is published as 'rc' and Main package as 9.0.0-rc.1 (mysql adapter is ommited from from main release version 9.0.1)\n\n## Usage\nIt's pretty simple and straightforward.  \n**Bulk** Extensions are made on *DbContext* and are used with entities List (supported both regular and Async methods):\n```C#\ncontext.BulkInsert(entities);                 context.BulkInsertAsync(entities);\ncontext.BulkInsertOrUpdate(entities);         context.BulkInsertOrUpdateAsync(entities);    //Upsert\ncontext.BulkInsertOrUpdateOrDelete(entities); context.BulkInsertOrUpdateOrDeleteAsync(entiti);//Sync\ncontext.BulkUpdate(entities);                 context.BulkUpdateAsync(entities);\ncontext.BulkDelete(entities);                 context.BulkDeleteAsync(entities);\ncontext.BulkRead(entities);                   context.BulkReadAsync(entities);\ncontext.BulkSaveChanges();                    context.BulkSaveChangesAsync();\n```\n\n**-MySQL** when running its Test for the first time execute sql command ([local-data](https://stackoverflow.com/questions/59993844/error-loading-local-data-is-disabled-this-must-be-enabled-on-both-the-client)): `SET GLOBAL local_infile = true;`  \n**-SQLite** requires package: [*SQLitePCLRaw.bundle_e_sqlite3*](https://docs.microsoft.com/en-us/dotnet/standard/data/sqlite/custom-versions?tabs=netcore-cli) with call to `SQLitePCL.Batteries.Init()`  \n\n**Batch** Extensions are made on *IQueryable* DbSet and can be used as in the following code segment.  \nThey are done as pure sql and no check is done whether some are prior loaded in memory and are being Tracked.  \n(*updateColumns* is optional param in which PropertyNames added explicitly when need update to it's default value)  \nInfo about [lock-escalation](https://docs.microsoft.com/en-us/troubleshoot/sql/performance/resolve-blocking-problems-caused-lock-escalation) in SQL Server with Batch iteration example as a solution at the bottom of code segment.\n```C#\n// Delete\ncontext.Items.Where(a =\u003e a.ItemId \u003e  500).BatchDelete();\ncontext.Items.Where(a =\u003e a.ItemId \u003e  500).BatchDeleteAsync();\n\n// Update (using Expression arg.) supports Increment/Decrement \ncontext.Items.Where(a =\u003e a.ItemId \u003c= 500).BatchUpdate(a =\u003e new Item { Quantity = a.Quantity + 100});\ncontext.Items.Where(a =\u003e a.ItemId \u003c= 500).BatchUpdateAsync(a =\u003e new Item {Quantity=a.Quantity+100});\n  // can be as value '+100' or as variable '+incrementStep' (int incrementStep = 100;)\n  \n// Update (via simple object)\ncontext.Items.Where(a =\u003e a.ItemId \u003c= 500).BatchUpdate(new Item { Description = \"Updated\" });\ncontext.Items.Where(a =\u003e a.ItemId \u003c= 500).BatchUpdateAsync(new Item { Description = \"Updated\" });\n// Update (via simple object) - requires additional Argument for setting to Property default value\nvar updateCols = new List\u003cstring\u003e { nameof(Item.Quantity) }; //Update 'Quantity' to default val:'0'\nvar q = context.Items.Where(a =\u003e a.ItemId \u003c= 500);\nint affected = q.BatchUpdate(new Item { Description=\"Updated\" }, updateCols); //result assigned aff.\n\n// Batch iteration (useful in same cases to avoid lock escalation)\ndo {\n    rowsAffected = query.Take(chunkSize).BatchDelete();\n} while (rowsAffected \u003e= chunkSize);\n\n// Truncate\ncontext.Truncate\u003cEntity\u003e();\ncontext.TruncateAsync\u003cEntity\u003e();\n```\n\n## Performances\nFollowing are performances (in seconds)\n* For SQL Server (v. 2019):\n\n| Ops\\Rows | EF 100K | Bulk 100K | EF 1 MIL.| Bulk 1 MIL.|\n| -------- | ------: | --------: | -------: | ---------: |\n| Insert   |  11 s   | 3 s       |   60 s   | 15  s      |\n| Update   |   8 s   | 4 s       |   84 s   | 27  s      |\n| Delete   |  50 s   | 3 s       | 5340 s   | 15  s      |\n\nTestTable has 6 columns (Guid, string x2, int, decimal?, DateTime), all inserted and 2 were updated.  \nTest done locally on configuration: INTEL i7-10510U CPU 2.30GHz, DDR3 16 GB, SSD SAMSUNG 512 GB.  \nFor small data sets there is an overhead since most Bulk ops need to create Temp table and also Drop it after finish.  \nProbably good advice would be to use **Bulk ops for sets greater than 1000** (condition in DbContext or Repository).\n\n## Bulk info\nIf Windows Authentication is used then in ConnectionString there should be *Trusted_Connection=True;* because Sql credentials are required to stay in connection.  \nAnother Conn.Str. config that can be useful for operations with extremely large data sets is [*ConnectionTimeout*](https://learn.microsoft.com/en-us/dotnet/api/system.data.sqlclient.sqlconnection.connectiontimeout?view=netframework-4.8.1\u0026redirectedfrom=MSDN#System_Data_SqlClient_SqlConnection_ConnectionTimeout) that can be increased from default 15 s to 60 or more to avoid '*Execution Timeout*' if it were to occur.\n\nWhen used directly each of these operations are separate transactions and are automatically committed.  \nAnd if we need multiple operations in single procedure then explicit transaction should be used, for example:  \n```C#\nusing (var transaction = context.Database.BeginTransaction())\n{\n    context.BulkInsert(entities1List);\n    context.BulkInsert(entities2List);\n    transaction.Commit();\n}\n\n// or with newer synax as of C# 8.0 (auto disposable objects)\nusing var transaction = context.Database.BeginTransaction();\ncontext.BulkInsert(entities1List);\ncontext.BulkInsert(entities2List);\ntransaction.Commit();\n```\n\n**BulkInsertOrUpdate** method can be used when there is need for both operations but in one connection to database.  \nIt makes Update when PK(PrimaryKey) is matched, otherwise does Insert.  \n\n**BulkInsertOrUpdateOrDelete** effectively [synchronizes](https://www.mssqltips.com/sqlservertip/1704/using-merge-in-sql-server-to-insert-update-and-delete-at-the-same-time/) table rows with input data.  \nThose in Db that are not found in the list will be deleted.  \nPartial Sync can be done on table subset using expression set on config with method:  \n`bulkConfig.SetSynchronizeFilter\u003cItem\u003e(a =\u003e a.Quantity \u003e 0);`  \nNot supported for SQLite (Lite has only UPSERT statement) nor currently for PostgreSQL. Here way to achieve sync functionality is to Select or BulkRead existing data from DB, split list into sublists and call separately Bulk methods for BulkInsertOrUpdate and Delete.\n\n**BulkRead** (SELECT and JOIN done in Sql)  \nUsed when need to Select from big List based on Unique Prop./Columns specified in config `UpdateByProperties`  \n```C#\n// instead of WhereIN which will TimeOut for List with over around 40 K records\nvar entities = context.Items.Where(a=\u003e itemsNames.Contains(a.Name)).AsNoTracking().ToList();//SQL IN\n// or JOIN in Memory that loads entire table\nvar entities = context.Items.Join(itemsNames, a =\u003e a.Name, p =\u003e p,(a,p)=\u003ea).AsNoTracking().ToList();\n\n// USE\nvar items = itemsNames.Select(a =\u003e new Item { Name = a }).ToList(); // Items list with only Name set\nvar bulkConfig = new BulkConfig { UpdateByProperties = new List\u003cstring\u003e { nameof(Item.Name) } };\ncontext.BulkRead(items, bulkConfig); //Items list will be loaded from Db with data(other properties)\n```\nUseful config **ReplaceReadEntities** that works as *Contains/IN* and returns all which match the criteria (not unique).  \n[Example](https://github.com/borisdj/EFCore.BulkExtensions/issues/733) of special use case when need to BulkRead child entities after BulkReading parent list. \n\n**SaveChanges** uses Change Tracker to find all modified(CUD) entities and call proper BulkOperations for each table.  \nBecause it needs tracking it is slower than pure BulkOps but still much faster than regular SaveChanges.  \nWith config *OnSaveChangesSetFK* setting FKs can be controlled depending on whether PKs are generated in Db or in memory.  \nSupport for this method was added in version 6 of the library.  \nBefore calling this method newly created should be added into Range:\n```C#\ncontext.Items.AddRange(newEntities); // if newEntities is parent list it can have child sublists\ncontext.BulkSaveChanges();\n```\nPractical general usage could be made in a way to override regular SaveChanges and if any list of Modified entities entries is greater then say 1000 to redirect to Bulk version.\n\nNote: Bulk ops have optional argument *Type type* that can be set to type of Entity if list has dynamic runtime objects or is inherited from Entity class.\n\n## BulkConfig arguments\n\n**Bulk** methods can have optional argument **BulkConfig** with properties (bool, int, string, object, List\u003cstring\u003e):  \n```C#\nPROPERTY : DEFAULTvalue\n----------------------------------------------------------------------------------------------\n 1 PreserveInsertOrder: true,                   22 PropertiesToInclude: null,\n 2 SetOutputIdentity: false,                    23 PropertiesToIncludeOnCompare: null,\n 3 SetOutputNonIdentityColumns: true,           24 PropertiesToIncludeOnUpdate: null,\n 4 LoadOnlyIncludedColumns: false,              25 PropertiesToExclude: null,\n 5 BatchSize: 2000,                             26 PropertiesToExcludeOnCompare: null,\n 6 NotifyAfter: null,                           27 PropertiesToExcludeOnUpdate: null,\n 7 BulkCopyTimeout: null,                       28 UpdateByProperties: null,\n 8 TrackingEntities: false,                     29 ReplaceReadEntities: false,\n 9 UseTempDB: false,                            30 EnableShadowProperties: false,\n10 UniqueTableNameTempDb: true,                 31 CustomSqlPostProcess: null,\n11 CustomDestinationTableName: null,            32 IncludeGraph: false,\n12 CustomSourceTableName: null,                 33 OmitClauseExistsExcept: false,\n13 CustomSourceDestinationMappingColumns: null, 34 DoNotUpdateIfTimeStampChanged: false,\n14 OnConflictUpdateWhereSql: null,              35 SRID: 4326,\n15 WithHoldlock: true,                          36 DateTime2PrecisionForceRound: false,\n16 CalculateStats: false,                       37 TemporalColumns: { \"PeriodStart\", \"PeriodEnd\" },\n17 SqlBulkCopyOptions: Default,                 38 OnSaveChangesSetFK: true,\n18 SqlBulkCopyColumnOrderHints: null,           39 IgnoreGlobalQueryFilters: false,\n19 DataReader: null,                            40 EnableStreaming: false,\n20 UseOptionLoopJoin:false,                     41 ApplySubqueryLimit: 0\n21 ConflictOption: None\n----------------------------------------------------------------------------------------------\nMETHOD: SetSynchronizeFilter\u003cT\u003e\n        SetSynchronizeSoftDelete\u003cT\u003e\n```\nIf we want to change defaults, BulkConfig should be added explicitly with one or more bool properties set to true, and/or int props like **BatchSize** to different number.   Config also has DelegateFunc for setting *Underlying-Connection/Transaction*, e.g. in UnderlyingTest.  \nWhen doing update we can chose to exclude one or more properties by adding their names into **PropertiesToExclude**, or if we need to update less then half column then **PropertiesToInclude** can be used. Setting both Lists are not allowed.\n\nWhen using the **BulkInsert_/OrUpdate** methods, you may also specify the **PropertiesToIncludeOnCompare** and **PropertiesToExcludeOnCompare** properties (only for SqlServer). By adding a column name to the *PropertiesToExcludeOnCompare*, will allow it to be inserted and updated but will not update the row if any of the other columns in that row did not change. For example, if you are importing bulk data and want to remove from comparison an internal *CreateDate* or *UpdateDate*, you add those columns to the *PropertiesToExcludeOnCompare*.  \nAnother option that may be used in the same scenario are the **PropertiesToIncludeOnUpdate** and **PropertiesToExcludeOnUpdate** properties. These properties will allow you to specify insert-only columns such as *CreateDate* and *CreatedBy*.\n\nIf we want Insert only new and skip existing ones in Db (Insert_if_not_Exist) then use *BulkInsertOrUpdate* with config\n`PropertiesToIncludeOnUpdate = new List\u003cstring\u003e { \"\" }`\n\nAdditionally, there is **UpdateByProperties** for specifying custom properties, by which we want update to be done.  \nWhen setting multiple props in UpdateByProps then match done by columns combined, like unique constrains based on those cols.  \nUsing UpdateByProperties while also having Identity column requires that Id property be [Excluded](https://github.com/borisdj/EFCore.BulkExtensions/issues/131).  \nAlso, with PostgreSQL when matching is done it requires UniqueIndex so for custom UpdateByProperties that do not have Un.Ind., it is temporarily created in which case method can not be in transaction (throws: *current transaction is aborted; CREATE INDEX CONCURRENTLY cannot run inside a transaction block*).  \nSimilar is done with MySQL by temporarily adding UNIQUE CONSTRAINT.  \n\nIf **NotifyAfter** is not set it will have same value as _BatchSize_ while **BulkCopyTimeout** when not set, has SqlBulkCopy default, which is 30 seconds and if set to 0 it indicates no limit.    \n_SetOutputIdentity_ have a purpose only when PK has Identity (usually *int* type with AutoIncrement), while if PK is Guid(sequential) created in Application there is no need for them.  \nAlso, Tables with Composite Keys have no Identity column, so no functionality for them in that case either.\n```C#\nvar bulkConfig = new BulkConfig { SetOutputIdentity = true, BatchSize = 4000 };\ncontext.BulkInsert(entities, bulkConfig);\ncontext.BulkInsertOrUpdate(entities, new BulkConfig { SetOutputIdentity = true }); //e.g.\ncontext.BulkInsertOrUpdate(entities, b =\u003e b.SetOutputIdentity = true); //BulkConfig with Action arg.\n```\n\n**PreserveInsertOrder** is **true** by default and makes sure that entities are inserted to Db as ordered in entitiesList.  \nWhen a table has Identity column (int autoincrement) with 0 values in list, they will temporarily be automatically changed from 0s into range -N:-1.  \nOr it can be manually set with proper values for order (Negative values used to skip conflict with existing ones in Db).  \nHere single Id value itself doesn't matter, db will change it to next in sequence, what matters is their mutual relationship for sorting.  \nInsertion order is implemented with [TOP](https://docs.microsoft.com/en-us/sql/t-sql/queries/top-transact-sql) in conjunction with ORDER BY. [so/merge-into-insertion-order](https://stackoverflow.com/questions/884187/merge-into-insertion-order).  \nThis config should remain true when *SetOutputIdentity* is set to true on Entity containing NotMapped Property. [issues/76](https://github.com/borisdj/EFCore.BulkExtensions/issues/76)  \nWhen using **SetOutputIdentity** Id values will be updated to new ones from database.  \nWith BulkInsertOrUpdate on SQLServer for those that will be updated it has to match with Id column, or other unique column(s) if using UpdateByProperties in which case  [orderBy done with those props](https://github.com/borisdj/EFCore.BulkExtensions/issues/806) instead of ID, due to how Sql MERGE works. To preserve insert order by Id in this case alternative would be first to use BulkRead and find which records already exist, then split the list into 2 lists entitiesForUpdate and entitiesForInsert without configuring UpdateByProps).  \nAlso for SQLite combination of BulkInsertOrUpdate and IdentityId automatic set will not work properly since it does [not have full MERGE](https://github.com/borisdj/EFCore.BulkExtensions/issues/556) capabilities like SqlServer. Instead list can be split into 2 lists, and call separately BulkInsert and BulkUpdate.  \n  \n**SetOutputIdentity** is useful when BulkInsert is done to multiple related tables that have Identity column.  \nAfter Insert is done to the first table, we need Id-s (if using Option 1) that were generated in Db because they are FK(ForeignKey) in second table.  \nIt is implemented with [OUTPUT](https://docs.microsoft.com/en-us/sql/t-sql/queries/output-clause-transact-sql) as part of MERGE Query, so in this case, even the Insert is not done directly to TargetTable but to TempTable and then Merged with TargetTable.  \nWhen used Id-s will be updated on entitiesList, and if *PreserveInsertOrder* is set to *false* then entitiesList will be cleared and reloaded.  \nIf Entity has Json column with null value and we set OutputIdentity then also set OutputNonIdentity to false, because [JsonNull](https://github.com/borisdj/EFCore.BulkExtensions/issues/1572) mapping throws an exception.  \n**SetOutputNonIdentityColumns** used only when *SetOutputIdentity* is set to true, and if this remains True (which is default) all columns are reloaded from Db.  \nWhen changed to false, only the Identity column is loaded to reduce load back from DB for efficiency.  \n  \nExample of *SetOutputIdentity* with parent-child FK related tables:\n```C#\nint numberOfEntites = 1000;\nvar entities = new List\u003cItem\u003e();\nvar subEntities = new List\u003cItemHistory\u003e();\nfor (int i = 1; i \u003c= numberOfEntites; i++)\n{\n    var entity = new Item { Name = $\"Name {i}\" };\n    entity.ItemHistories = new List\u003cItemHistory\u003e()\n    {\n        new ItemHistory { Remark = $\"Info {i}.1\" },\n        new ItemHistory { Remark = $\"Info {i}.2\" }\n    };\n    entities.Add(entity);\n}\n\n// Option 1 (recommended)\nusing (var transaction = context.Database.BeginTransaction())\n{\n    context.BulkInsert(entities, new BulkConfig { SetOutputIdentity = true });\n    foreach (var entity in entities) {\n        foreach (var subEntity in entity.ItemHistories) {\n            subEntity.ItemId = entity.ItemId; // sets FK to match linked PK that was generated in DB\n        }\n        subEntities.AddRange(entity.ItemHistories);\n    }\n    context.BulkInsert(subEntities);\n    transaction.Commit();\n}\n\n// Option 2 using Graph (only for SQL Server and only for simple relationship parent-child)\n// - all entities in relationship with main ones in list are BulkInsertUpdated\ncontext.BulkInsert(entities, b =\u003e b.IncludeGraph = true);\n  \n// Option 3 with BulkSaveChanges() - uses ChangeTracker so little slower then direct Bulk\ncontext.Items.AddRange(entities);\ncontext.BulkSaveChanges();\n```\nWhen **CalculateStats** set to True the result returned in `BulkConfig.StatsInfo` (*StatsNumber-Inserted/Updated/Deleted*).  \nIf used for pure Insert (with Batching) then SetOutputIdentity should also be configured because Merge is required.  \n**TrackingEntities** can be set to True if we want to have tracking of entities from BulkRead or if SetOutputIdentity is set.  \n**WithHoldlock** means [Serializable isolation](https://github.com/borisdj/EFCore.BulkExtensions/issues/41) level that locks the table (can have negative effect on [concurrency](https://www.linkedin.com/posts/milan-jovanovic_ef-core-doesnt-support-pessimistic-locking-activity-7184445256870825984-QSLU/)).  \n_ Setting it False can optionally be used to solve deadlock issue Insert.  \n**UseTempDB** when set then BulkOperation has to be [inside Transaction](https://github.com/borisdj/EFCore.BulkExtensions/issues/49).  \n**UniqueTableNameTempDb** when changed to false, temp table name will be only 'Temp' without random numbers.  \n**CustomDestinationTableName** can be set with 'TableName' only or with 'Schema.TableName'.  \n**CustomSourceTableName** when set enables source data from specified table already in Db, so input list not used and can be empty.  \n**CustomSourceDestinationMappingColumns** dict can be set only if CustomSourceTableName is configured and it is used for specifying Source-Destination column names when they are not the same. Example in test `DestinationAndSourceTableNameTest`.  \n**EnableShadowProperties** to add (normal) Shadow Property and to persist value. Disables automatic discriminator, use manual method.  \n**CustomSqlPostProcess** If used, should be set to valid pure Sql syntax, that would be run after main operation but before deleting temporary tables. One practical use case would be to move data from TempOutput table (set UniqueTableNameTempDb to False know the name) into a some Log table, optionally using FOR JSON PATH (example test: *CustomSqlPostProcessTest*).  \n**IncludeGraph** when set, all entities that have relations with main ones from the list are also merged into theirs tables.  \n**OmitClauseExistsExcept** removes the clause from Merge statement, required when having noncomparable types like XML, and useful when need to activate triggers even for same data.  \n_ Also in some [sql collation](https://github.com/borisdj/EFCore.BulkExtensions/issues/641), small and capital letters are considered same (case-insensitive) so for BulkUpdate set it false.  \n**DoNotUpdateIfTimeStampChanged** if set checks TimeStamp for Concurrency, ones with conflict will [not be updated](https://github.com/borisdj/EFCore.BulkExtensions/issues/469#issuecomment-803662721).  \nReturn info will be in *BulkConfig.**TimeStampInfo*** object within field `NumberOfSkippedForUpdate` and list `EntitiesOutput`.  \n**SRID** Spatial Reference Identifier - for SQL Server with NetTopologySuite.  \n**DateTime2PrecisionForceRound** If dbtype datetime2 has precision less then default 7, example 'datetime2(3)' SqlBulkCopy does Floor instead of Round so when this Property is set then Rounding will be done in memory to make sure inserted values are same as with regular SaveChanges.  \n**TemporalColumns** are shadow columns used for Temporal table. Default elements 'PeriodStart' and 'PeriodEnd' can be changed if those columns have custom names.  \n**OnSaveChangesSetFK** is used only for BulkSaveChanges. When multiply entries have FK relationship which is Db generated, this set proper value after reading parent PK from Db. IF PK are generated in memory like are some Guid then this can be set to false for better efficiency.  \n**ReplaceReadEntities** when set to True result of BulkRead operation will be provided using replace instead of update. Entities list parameter of BulkRead method will be repopulated with obtained data. Enables functionality of Contains/IN which will return all entities matching the criteria (does not have to be by unique columns).  \n**UseOptionLoopJoin** when set it appends 'OPTION (LOOP JOIN)' for SqlServer, to reduce potential deadlocks on tables that have FKs. Use this [sql hint](https://learn.microsoft.com/en-us/sql/t-sql/queries/hints-transact-sql-query?view=sql-server-ver16) as a last resort for experienced devs and db admins.  \n**ConflictOption**: -*None*(as errors), -*Replace*(conflicting with new rows), -*Ignore*(keep old rows)  \n**ApplySubqueryLimit** Default is zero '0'. When set to larger value it appends: LIMIT 'N', to generated query. Used only with PostgreSql.\n\n**DataReader** can be used when DataReader is also configured and when set it is propagated to SqlBulkCopy util object.  \n**EnableStreaming** can be set to True if want to have tracking of entities from BulkRead or when SetOutputIdentity is set, useful for big field like blob, binary column.\n\n**SqlBulkCopyOptions** is Enum (only for SqlServer) with [[Flags]](https://stackoverflow.com/questions/8447/what-does-the-flags-enum-attribute-mean-in-c) attribute which enables specifying one or more options:  \n*Default, KeepIdentity, CheckConstraints, TableLock, KeepNulls, FireTriggers, UseInternalTransaction*  \nIf need to set Identity PK in memory, Not let DB do the autoincrement, then need to use **KeepIdentity**:  \n`var bulkConfig = new BulkConfig { SqlBulkCopyOptions = SqlBulkCopyOptions.KeepIdentity };`  \nUseful for example when copying from one Db to another.\n\n**OnConflictUpdateWhereSql\u003cT\u003e** To define conditional updates on merges, receives (existingTable, insertedTable).  \n--Example: `bc.OnConflictUpdateWhereSql = (ex, in) =\u003e $\"{in}.TimeUpdated \u003e {ex}.TimeUpdated\";`  \n**SetSynchronizeFilter\u003cT\u003e** A method that receives and sets expression filter on entities to delete when using BulkInsertOrUpdateOrDelete. Those that are filtered out will be ignored and not deleted.  \n**SetSynchronizeSoftDelete\u003cT\u003e** A method that receives and sets expresion on entities to update property instead of deleting when using BulkInsertOrUpdateOrDelete.  \n`bulkConfig.SetSynchronizeSoftDelete\u003cSomeObject\u003e(a =\u003e new SomeObject { IsDeleted = true });`  \n\nLast optional argument is **Action progress** (Example in *EfOperationTest.cs* *RunInsert()* with *WriteProgress()*).\n```C#\ncontext.BulkInsert(entitiesList, null, (a) =\u003e WriteProgress(a));\n```\n\nFor **parallelism**, important notes are:  \n-SqlBulk [in Parallel](https://www.adathedev.co.uk/2011/01/sqlbulkcopy-to-sql-server-in-parallel.html)  \n-Concurrent operations not run in the [same Context instance](https://learn.microsoft.com/en-us/ef/core/miscellaneous/async)  \n-Import data to a single unindexed table with [table level lock](https://learn.microsoft.com/en-us/previous-versions/sql/sql-server-2005/ms186341(v=sql.90))  \n\nLibrary supports [Global Query Filters](https://docs.microsoft.com/en-us/ef/core/querying/filters) and [Value Conversions](https://docs.microsoft.com/en-us/ef/core/modeling/value-conversions) as well.  \nAdditionally BatchUpdate and named Property works with [EnumToString Conversion](https://github.com/borisdj/EFCore.BulkExtensions/issues/397)  \nIt can map [OwnedTypes](https://docs.microsoft.com/en-us/ef/core/modeling/owned-entities), also next are links with info how to achieve \n[NestedOwnedTypes](https://github.com/borisdj/EFCore.BulkExtensions/issues/167#issuecomment-476737959) and \n[OwnedInSeparateTable](https://github.com/borisdj/EFCore.BulkExtensions/issues/114#issuecomment-803462928)  \nOn PG when Enum is in OwnedType it needs to have [Converter explicitly](https://github.com/borisdj/EFCore.BulkExtensions/issues/1108) configured in *OnModelCreating*  \n\nTable splitting is somewhat specific but could be configured in the way [Set TableSplit](https://github.com/borisdj/EFCore.BulkExtensions/issues/352#issuecomment-803674404)  \nWith [Computed](https://docs.microsoft.com/en-us/ef/core/modeling/relational/computed-columns) and [Timestamp](https://docs.microsoft.com/en-us/ef/core/modeling/concurrency) Columns, it will work in a way that they are automatically excluded from Insert. And when combined with *SetOutputIdentity* they will be Selected.  \n[Spatial](https://docs.microsoft.com/en-us/sql/relational-databases/spatial/spatial-data-types-overview?view=sql-server-ver15) types, like Geometry, are also supported and if an Entity has one, clause *EXIST ... EXCEPT* is skipped because it's not comparable.  \nPerformance for bulk ops measured with `ActivitySources` named: '*BulkExecute*' (tags: '*operationType*', '*entitiesCount*')  \nBulk Extension methods can be [Overridden](https://github.com/borisdj/EFCore.BulkExtensions/issues/56) if required, for example to set AuditInfo.  \nIf having problems with Deadlock, there is useful info in [issue/46](https://github.com/borisdj/EFCore.BulkExtensions/issues/46).\n\n**TPH** ([Table-Per-Hierarchy](https://docs.microsoft.com/en-us/aspnet/core/data/ef-mvc/inheritance)) inheritance model can be set in 2 ways.  \nFirst is automatically by Convention, in which case the Discriminator column is not directly in the Entity but is [Shadow](https://learn.microsoft.com/en-us/ef/core/modeling/shadow-properties) Property.  \nAnd second is to explicitly define Discriminator property in Entity and configure it with `.HasDiscriminator()`.  \nAn important remark regarding the first case is that since we can not directly set a Discriminator to certain value we need first to add list of entities to DbSet where it will be set and after that we can call Bulk operation. Note that SaveChanges are not called, and we could optionally turn off TrackingChanges for performance. Example:\n```C#\npublic class Student : Person { ... }\ncontext.Students.AddRange(entities); //adding to Context so Shadow property 'Discriminator' gets set\ncontext.BulkInsert(entities);\n```\n**TPT** (Table-Per-Type) way it is [supported](https://github.com/borisdj/EFCore.BulkExtensions/issues/493).\n\n## Structure of SourceCode\n- Action Flow\n```C#\n_CLASSES:                    |DbContextBulk|SqlBulk     |ISqlOperations|SqlOperations  |\nDbContextBulkExtensions:     |-Transaction:|-Operation: |-Adapter:     |-ServerAdapter:|\n_METHODS: Sync/Async========]|[===========]|[==========]|[============]|[=============]|\nC { BulkInsert --------------|             |--Insert ---|--Insert -----|--Insert       |\nU / BulkInsertOrUpdate ------|             | \\                                         |\nU | BulkInsertOrUpdateOrDel.-|             |  \\                                        |\nU \\ BulkUpdate --------------|--Execute ---|--Merge ----|--Merge ------|--Merge        |\nD { BulkDelete --------------|             | /                                         |\nR { BulkRead ----------------|             |--Read -----|--Read -------|--Read         |\n- { Truncate ----------------|             |--Truncate -|--Truncate ---|--Truncate     |\n```\n- Projects Composition:\n\n| Num | Nuget                                  | Reference   | Transitive dep. | Note         | \n| --- | -------------------------------------- | ----------- | --------------- | ------------ | \n| [0] | EFCore.BulkExtensions.Core             |             |                 | shared       |\n| [1] | EFCore.BulkExtensions.***SqlServer***  | [0]         |                 | per provider |\n| [2] | EFCore.BulkExtensions.***PostgreSql*** | [0]         |                 | per provider |\n| [3] | EFCore.BulkExtensions.***MySql***      | [0]         |                 | per provider |\n| [4] | EFCore.BulkExtensions.***Oracle***     | [0]         |                 | per provider |\n| [5] | EFCore.BulkExtensions.***Sqlite***     | [0]         |                 | per provider |\n| [6] | **EFCore.BulkExtensions** - main one   | [1,2,3,4,5] | [0]             | has all      |\n\nEFCore.BulkExtensions is main Project and Nuget that references all other nugets.  \nOther per provider projects have only Core dependency and specific adapter implementation with needed packages.\n\nList of all referenced [Links](https://docs.google.com/spreadsheets/d/e/2PACX-1vRE_rO6799VGMhLAldgDZoHJAdbKI0YZ8gweN7oaBG0KCIMtqZwlaeTyB9IMoThb69Tk0bkjIdNNWl_/pubhtml?gid=2013428247\u0026single=true)\n","funding_links":["https://github.com/sponsors/borisdj","https://www.buymeacoffee.com/boris.dj"],"categories":["Frameworks, Libraries and Tools","C\\#","框架, 库和工具","Data Access","Libraries","SQL Server Web Resources","sql","ORM","Audio","C# #"],"sub_categories":["ORM","对象关系映射ORM","ORM and Micro-ORM","GUI - other"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fborisdj%2FEFCore.BulkExtensions","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fborisdj%2FEFCore.BulkExtensions","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fborisdj%2FEFCore.BulkExtensions/lists"}