{"id":16089644,"url":"https://github.com/jbogard/bulk-writer","last_synced_at":"2025-04-12T09:33:13.622Z","repository":{"id":14612135,"uuid":"17329414","full_name":"jbogard/bulk-writer","owner":"jbogard","description":"Provides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.","archived":false,"fork":false,"pushed_at":"2024-05-29T19:28:25.000Z","size":1661,"stargazers_count":240,"open_issues_count":0,"forks_count":37,"subscribers_count":37,"default_branch":"master","last_synced_at":"2024-10-10T14:05:04.348Z","etag":null,"topics":["bulk-writer","etl","etl-job","pipeline","pipeline-stage","sql","sqlbulkcopy","stream-data"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jbogard.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-03-02T02:37:11.000Z","updated_at":"2024-07-28T20:51:40.000Z","dependencies_parsed_at":"2024-06-18T22:44:42.101Z","dependency_job_id":"5a34e9b6-3eb1-4242-8733-b3d2081df564","html_url":"https://github.com/jbogard/bulk-writer","commit_stats":null,"previous_names":["headspringlabs/bulk-writer"],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jbogard%2Fbulk-writer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jbogard%2Fbulk-writer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jbogard%2Fbulk-writer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jbogard%2Fbulk-writer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jbogard","download_url":"https://codeload.github.com/jbogard/bulk-writer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248545909,"owners_count":21122240,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bulk-writer","etl","etl-job","pipeline","pipeline-stage","sql","sqlbulkcopy","stream-data"],"created_at":"2024-10-09T14:05:03.184Z","updated_at":"2025-04-12T09:33:13.564Z","avatar_url":"https://github.com/jbogard.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Bulk Writer\n\nBulk Writer is a small library which facilitates building fast, pull-based ETL processes in C# using `SqlBulkCopy`. \n\n## Documentation\n\nDocumentation can be found at https://jbogard.github.io/bulk-writer/\n\n## Installation\n\n[Bulk Writer](https://www.nuget.org/packages/BulkWriter/) is available on NuGet and can be installed using the package manager console:\n\n```\nPM\u003e Install-Package BulkWriter\n```\n\n## Usage\n\n```csharp\nvar q =\n   from entity in GetAllEntities()\n   where entity.IsActive \u0026\u0026 SomeOtherPredicate(entity)\n   from zipCode in GetAllZipCodes()\n   where zipCode.IsInContiguousStates \u0026\u0026 SomeOtherPredicate(zipCode)\n   let distance = GetDistance(entity, zipCode)\n   let arbitraryData = CreateSomeArbitraryData(entity, zipCode)\n   where distance \u003e 0\n   select new EntityToZipCodeDistance {\n      EntityId = entity.Id,\n      ZipCode = zipCode.Zip,\n      Distance = distance,\n      ArbitraryData = arbitraryData\n   };\n\nusing (var bulkWriter = new BulkWriter\u003cEntityToZipCodeDistance\u003e(connectionString))\n{\n    bulkWriter.WriteToDatabase(q);\n}\n// or async\n\nusing (var bulkWriter = new BulkWriter\u003cEntityToZipCodeDistance\u003e(connectionString))\n{\n    await bulkWriter.WriteToDatabaseAsync(q);\n}\n\n// or async enumerables with .NET Standard 2.1 or later\nvar u = q.ToAsyncEnumerable(); // \n\nusing (var bulkWriter = new BulkWriter\u003cEntityToZipCodeDistance\u003e(connectionString))\n{\n    await bulkWriter.WriteToDatabaseAsync(u);\n}\n```\n\n## Building Locally\n\nRun the following command once to setup your environment.\n```\nPS\u003e .\\setup.ps1\n```\n\nRun the command below to build and test the project.\n\n```\nPS\u003e .\\psake.cmd\n```\n\n## Contributing\n\nPull Requests are welcome. If you identify a bug or would like to make a feature request feel free to submit a GitHub Issue to start a discussion.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjbogard%2Fbulk-writer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjbogard%2Fbulk-writer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjbogard%2Fbulk-writer/lists"}