https://github.com/jbogard/bulk-writer
Provides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.
https://github.com/jbogard/bulk-writer
bulk-writer etl etl-job pipeline pipeline-stage sql sqlbulkcopy stream-data
Last synced: 7 months ago
JSON representation
Provides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.
- Host: GitHub
- URL: https://github.com/jbogard/bulk-writer
- Owner: jbogard
- License: mit
- Created: 2014-03-02T02:37:11.000Z (over 11 years ago)
- Default Branch: master
- Last Pushed: 2024-05-29T19:28:25.000Z (over 1 year ago)
- Last Synced: 2024-10-10T14:05:04.348Z (about 1 year ago)
- Topics: bulk-writer, etl, etl-job, pipeline, pipeline-stage, sql, sqlbulkcopy, stream-data
- Language: C#
- Homepage:
- Size: 1.58 MB
- Stars: 240
- Watchers: 37
- Forks: 37
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# Bulk Writer
Bulk Writer is a small library which facilitates building fast, pull-based ETL processes in C# using `SqlBulkCopy`.
## Documentation
Documentation can be found at https://jbogard.github.io/bulk-writer/
## Installation
[Bulk Writer](https://www.nuget.org/packages/BulkWriter/) is available on NuGet and can be installed using the package manager console:
```
PM> Install-Package BulkWriter
```
## Usage
```csharp
var q =
from entity in GetAllEntities()
where entity.IsActive && SomeOtherPredicate(entity)
from zipCode in GetAllZipCodes()
where zipCode.IsInContiguousStates && SomeOtherPredicate(zipCode)
let distance = GetDistance(entity, zipCode)
let arbitraryData = CreateSomeArbitraryData(entity, zipCode)
where distance > 0
select new EntityToZipCodeDistance {
EntityId = entity.Id,
ZipCode = zipCode.Zip,
Distance = distance,
ArbitraryData = arbitraryData
};
using (var bulkWriter = new BulkWriter(connectionString))
{
bulkWriter.WriteToDatabase(q);
}
// or async
using (var bulkWriter = new BulkWriter(connectionString))
{
await bulkWriter.WriteToDatabaseAsync(q);
}
// or async enumerables with .NET Standard 2.1 or later
var u = q.ToAsyncEnumerable(); //
using (var bulkWriter = new BulkWriter(connectionString))
{
await bulkWriter.WriteToDatabaseAsync(u);
}
```
## Building Locally
Run the following command once to setup your environment.
```
PS> .\setup.ps1
```
Run the command below to build and test the project.
```
PS> .\psake.cmd
```
## Contributing
Pull Requests are welcome. If you identify a bug or would like to make a feature request feel free to submit a GitHub Issue to start a discussion.