https://github.com/alexalvess/poc-read-dynamic-file
https://github.com/alexalvess/poc-read-dynamic-file
Last synced: 8 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/alexalvess/poc-read-dynamic-file
- Owner: alexalvess
- Created: 2023-03-13T19:14:12.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-05-18T20:23:03.000Z (over 2 years ago)
- Last Synced: 2024-12-31T10:29:11.993Z (10 months ago)
- Language: C#
- Size: 14.8 MB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Read dynamic Files (PoC)
This project has an objective to show how can we build a code to perform when we don't know the files behaviors, that we receive.
## About the Files and Map
First, we have a model class that we use to map file content into this class:
```csharp
public class UserModel
{
public string Name { get; init; }
public string Email { get; init; }
public int ProductCode { get; init; }
public DateTime PaymentDate { get; init; }
public decimal PaymentValue { get; init; }
}
```
And we have these files:
#### File with Position
For this kind of file, we need to know what position starts the field and the length.
#### File with Separator
For this kind of file, we need just map the header to know the fields.
---
## Stack
* .NET 7.0
* Message Broker: RabbitMQ with Masstransit v8.0.15
* Database: Postgres
## Benchmark for Read Performance
Besides showing how we can read a file dynamically, here, we try to show how we can use 2 kinds of read files:
* [Stream Reader](https://learn.microsoft.com/en-us/dotnet/api/system.io.stream?view=net-7.0): traditional way to read files
* [Pipeline Reader](https://learn.microsoft.com/en-us/dotnet/standard/io/pipelines): the way designed to make it easier to do high-performance I/O
_PS: For this example, we show the benchmark references about position strategy_
**About benchmark:**
Steps: Read the file, parse the data, and send it to the database or message broker.
10 Lines - 1 KB
#
Strategy
Mean
Error
Op/s
Send to DB
Stream
275.1 ms
NA
3.635
Pipe
259.6 ms
NA
3.852
Send to Broker
Stream
436.7 ms
NA
2.290
Pipe
419.8 ms
NA
2.382
100 Lines - 7 KB
#
Strategy
Mean
Error
Op/s
Send to DB
Stream
375.8 ms
NA
2.661
Pipe
362.5 ms
NA
2.759
Send to Broker
Stream
457.3 ms
NA
2.187
Pipe
447.9 ms
NA
2.232
1.000 Lines - 66 KB
#
Strategy
Mean
Error
Op/s
Send to DB
Stream
1.575 s
NA
0.6350
Pipe
1.499 s
NA
0.6671
Send to Broker
Stream
960.3 ms
NA
1.041
Pipe
860.9 ms
NA
1.162
10.000 Lines - 655 KB
#
Strategy
Mean
Error
Op/s
Send to DB
Stream
13.01 s
NA
0.0769
Pipe
12.62 s
NA
0.0793
Send to Broker
Stream
3.795 s
NA
0.2635
Pipe
3.819 s
NA
0.2619
100.000 Lines - 6.543 KB
#
Strategy
Mean
Error
Op/s
Send to DB
Stream
125.2 s
NA
0.0080
Pipe
148.7 s
NA
0.0067
Send to Broker
Stream
35.59 s
NA
0.0281
Pipe
40.20 s
NA
0.0249
1.000.000 Lines - 65.430 KB
#
Strategy
Mean
Error
Op/s
Send to DB
Stream
22.48 m
NA
0.0007
Pipe
24.13 m
NA
0.0007
Send to Broker
Stream
421.9 s
NA
0.0024
Pipe
390.1 s
NA
0.0026