{"id":31564256,"url":"https://github.com/gragra33/utf8jsonasyncstreamreader","last_synced_at":"2026-02-17T09:33:40.410Z","repository":{"id":163827064,"uuid":"639258631","full_name":"gragra33/Utf8JsonAsyncStreamReader","owner":"gragra33","description":"High-performance asynchronous JSON streaming parser for .NET that enables memory-efficient processing of large JSON datasets. Built on System.Text.Json.Utf8JsonReader, this library provides forward-only streaming with conditional branch deserialization, keeping memory usage minimal regardless of file size. Perfect for processing massive JSON files","archived":false,"fork":false,"pushed_at":"2024-11-09T12:13:18.000Z","size":23,"stargazers_count":6,"open_issues_count":1,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-09-28T06:58:36.298Z","etag":null,"topics":["json","streaming-api"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gragra33.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-11T05:18:44.000Z","updated_at":"2025-09-16T09:33:16.000Z","dependencies_parsed_at":null,"dependency_job_id":"f4eb66c1-3cb0-43aa-a81e-31663d70dbe7","html_url":"https://github.com/gragra33/Utf8JsonAsyncStreamReader","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/gragra33/Utf8JsonAsyncStreamReader","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gragra33%2FUtf8JsonAsyncStreamReader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gragra33%2FUtf8JsonAsyncStreamReader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gragra33%2FUtf8JsonAsyncStreamReader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gragra33%2FUtf8JsonAsyncStreamReader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gragra33","download_url":"https://codeload.github.com/gragra33/Utf8JsonAsyncStreamReader/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gragra33%2FUtf8JsonAsyncStreamReader/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278411261,"owners_count":25982368,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-05T02:00:06.059Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["json","streaming-api"],"created_at":"2025-10-05T05:03:24.747Z","updated_at":"2026-02-17T09:33:40.405Z","avatar_url":"https://github.com/gragra33.png","language":"C#","readme":"# Utf8JsonAsyncStreamReader\n\n[![NuGet Version](https://img.shields.io/nuget/v/Utf8JsonAsyncStreamReader.svg)](https://www.nuget.org/packages/Utf8JsonAsyncStreamReader)\n[![NuGet Downloads](https://img.shields.io/nuget/dt/Utf8JsonAsyncStreamReader.svg)](https://www.nuget.org/packages/Utf8JsonAsyncStreamReader)\n[![.NET 8+](https://img.shields.io/badge/.NET-8%2B-512BD4)](https://dotnet.microsoft.com/download)\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Getting Started](#getting-started)\n  - [Installation](#installation)\n  - [Requirements](#requirements)\n- [Give a ⭐](#give-a-)\n- [Usage](#usage)\n  - [Basic Stream Reading](#basic-stream-reading)\n  - [Object Deserialization](#object-deserialization)\n  - [Custom JsonSerializerOptions](#custom-jsonserializeroptions)\n  - [Processing Large Collections](#processing-large-collections)\n  - [Exception Handling](#exception-handling)\n- [Value Methods](#value-methods)\n- [How It Works](#how-it-works)\n- [Samples](#samples)\n- [v1.3.0 Performance Graphs](#v130-performance-graphs)\n  - [Speed Improvement by Scenario](#speed-improvement-by-scenario)\n  - [Gen2 GC Reduction](#gen2-gc-reduction)\n- [Real-World Impact](#real-world-impact)\n  - [High-Throughput API Example](#high-throughput-api-example)\n- [Support](#support)\n- [History](#history)\n\n---\n\n## Overview\n\nAn asynchronous forward-only streaming JSON parser and deserializer based on [System.Text.Json.Utf8JsonReader](https://github.com/dotnet/runtime/blob/418aa8ab6bb5cce2be1a8dee292818d2c201f152/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs). This library enables efficient processing of large JSON streams with minimal memory usage by buffering stream reads and supporting conditional branch deserialization. Memory consumption is optimized based on either the buffer size used or the specific JSON property branch being deserialized.\n\nPerfect for scenarios involving:\n- Large JSON file processing\n- Web API streaming responses  \n- Memory-constrained environments\n- Real-time data processing\n\n## Getting Started\n\n### Installation\n\nInstall the package via NuGet Package Manager:\n\n```xml\n\u003cPackageReference Include=\"Utf8JsonAsyncStreamReader\" Version=\"1.2.0\" /\u003e\n```\n\nOr via the Package Manager Console:\n\n```powershell\nInstall-Package Utf8JsonAsyncStreamReader\n```\n\nOr via the .NET CLI:\n\n```bash\ndotnet add package Utf8JsonAsyncStreamReader\n```\n\n### Requirements\n\n- .NET 8.0, .NET 9.0, or .NET 10.0\n- System.IO.Pipelines (automatically included)\n\n## Give a ⭐\n\nIf you like or are using this project to learn or start your solution, please give it a star. Thanks!\n\nAlso, if you find this library useful, and you're feeling really generous, then please consider [buying me a coffee ☕](https://bmc.link/gragra33).\n\n## Usage\n\nThe Utf8JsonAsyncStreamReader provides a forward-only, memory-efficient approach to processing JSON streams of any size. Whether you're reading from files, HTTP responses, or other stream sources, the reader maintains minimal memory footprint by processing data incrementally rather than loading entire documents into memory.\n\n### Basic Stream Reading\n\n```csharp\nusing System.Text.Json.Stream;\n\n// Create a stream (file, HTTP response, etc.)\nusing var fileStream = File.OpenRead(\"large-data.json\");\nusing var reader = new Utf8JsonAsyncStreamReader(fileStream);\n\n// Read JSON tokens asynchronously with cancellation support\nusing var cts = new CancellationTokenSource();\nwhile (await reader.ReadAsync(cts.Token))\n{\n    switch (reader.TokenType)\n    {\n        case JsonTokenType.PropertyName:\n            Console.WriteLine($\"Property: {reader.Value}\");\n            break;\n        case JsonTokenType.String:\n            Console.WriteLine($\"String Value: {reader.Value}\");\n            break;\n        case JsonTokenType.Number:\n            Console.WriteLine($\"Number Value: {reader.Value}\");\n            break;\n        // Handle other token types...\n    }\n}\n```\n\n### Object Deserialization\n\n```csharp\nusing System.Text.Json.Stream;\n\npublic class Person\n{\n    public string Name { get; set; }\n    public int Age { get; set; }\n    public string Email { get; set; }\n}\n\n// Deserialize directly from stream with cancellation support\nusing var stream = GetJsonStream(); // Your JSON stream source\nusing var reader = new Utf8JsonAsyncStreamReader(stream);\nusing var cts = new CancellationTokenSource();\n\nvar person = await reader.DeserializeAsync\u003cPerson\u003e(cancellationToken: cts.Token);\nConsole.WriteLine($\"Name: {person.Name}, Age: {person.Age}\");\n```\n\n### Custom JsonSerializerOptions\n\n```csharp\nvar options = new JsonSerializerOptions\n{\n    PropertyNamingPolicy = JsonNamingPolicy.CamelCase,\n    PropertyNameCaseInsensitive = true\n};\n\nusing var reader = new Utf8JsonAsyncStreamReader(stream);\nusing var cts = new CancellationTokenSource();\nvar result = await reader.DeserializeAsync\u003cMyObject\u003e(options, cts.Token);\n```\n\n### Processing Large Collections\n\n```csharp\n// For large JSON arrays, process items one by one\nusing var reader = new Utf8JsonAsyncStreamReader(stream);\nusing var cts = new CancellationTokenSource();\n\nawait reader.ReadAsync(cts.Token); // StartArray\nwhile (await reader.ReadAsync(cts.Token) \u0026\u0026 reader.TokenType != JsonTokenType.EndArray)\n{\n    var item = await reader.DeserializeAsync\u003cMyItem\u003e(cancellationToken: cts.Token);\n    ProcessItem(item); // Process each item individually\n}\n```\n\n### Exception Handling\n\nThe library provides specific exception types to help you handle different error scenarios:\n\n```csharp\nusing System.Text.Json.Stream;\n\ntry\n{\n    using var reader = new Utf8JsonAsyncStreamReader(stream);\n    using var cts = new CancellationTokenSource();\n    \n    while (await reader.ReadAsync(cts.Token))\n    {\n        // Process tokens...\n    }\n}\ncatch (JsonStreamException ex)\n{\n    // Handle JSON stream-specific errors:\n    // - Invalid JSON structure\n    // - Incomplete tokens\n    // - Buffer undersized for current JSON structure\n    Console.WriteLine($\"JSON Stream Error: {ex.Message}\");\n}\ncatch (JsonException ex)\n{\n    // Handle general JSON serialization errors\n    Console.WriteLine($\"JSON Error: {ex.Message}\");\n}\ncatch (OperationCanceledException)\n{\n    // Handle cancellation\n    Console.WriteLine(\"Operation was cancelled\");\n}\n```\n\n## Value Methods\n\nThe `Utf8JsonHelpers` class provides comprehensive extension methods to extract strongly-typed values from JSON tokens. These methods follow the standard .NET patterns and offer both safe (Try*) and direct (Get*) access patterns.\n\n### String \u0026 Copy Methods\n\n| Method | Returns | Description |\n|--------|---------|-------------|\n| `GetString()` | `string?` | Gets the string value, returns null for JSON null tokens |\n| `CopyString(Span\u003cbyte\u003e)` | `int` | Copies UTF-8 bytes to destination span, throws if buffer too small |\n| `CopyString(Span\u003cchar\u003e)` | `int` | Copies transcoded UTF-16 chars to destination span, throws if buffer too small |\n\n### Boolean Methods\n\n| Method | Returns | Description |\n|--------|---------|-------------|\n| `GetBoolean()` | `bool?` | Gets the boolean value (true/false), returns null for non-boolean tokens |\n\n### Numeric Methods - Byte Types\n\n| Method | Returns | Description |\n|--------|---------|-------------|\n| `GetByte()` | `byte` | Gets byte value (0-255), returns 0 on failure |\n| `TryGetByte(out byte?)` | `bool` | Attempts to get byte value, returns success status |\n| `GetSByte()` | `sbyte` | Gets signed byte value (-128 to 127), returns 0 on failure |\n| `TryGetSByte(out sbyte?)` | `bool` | Attempts to get signed byte value, returns success status |\n| `GetUByte()` | `uint` | Gets unsigned byte value, returns 0 on failure |\n| `TryGetUByte(out uint?)` | `bool` | Attempts to get unsigned byte value, returns success status |\n\n### Numeric Methods - Integer Types\n\n| Method | Returns | Description |\n|--------|---------|-------------|\n| `GetInt16()` | `short` | Gets 16-bit signed integer (-32,768 to 32,767), returns 0 on failure |\n| `TryGetInt16(out short?)` | `bool` | Attempts to get 16-bit signed integer, returns success status |\n| `GetUInt16()` | `uint` | Gets 16-bit unsigned integer (0 to 65,535), returns 0 on failure |\n| `TryGetUInt16(out uint?)` | `bool` | Attempts to get 16-bit unsigned integer, returns success status |\n| `GetInt32()` | `int` | Gets 32-bit signed integer, returns 0 on failure |\n| `TryGetInt32(out int?)` | `bool` | Attempts to get 32-bit signed integer, returns success status |\n| `GetUInt32()` | `uint` | Gets 32-bit unsigned integer, returns 0 on failure |\n| `TryGetUInt32(out uint?)` | `bool` | Attempts to get 32-bit unsigned integer, returns success status |\n| `GetInt64()` | `long` | Gets 64-bit signed integer, returns 0 on failure |\n| `TryGetInt64(out long?)` | `bool` | Attempts to get 64-bit signed integer, returns success status |\n| `GetUInt64()` | `ulong` | Gets 64-bit unsigned integer, returns 0 on failure |\n| `TryGetUInt64(out ulong?)` | `bool` | Attempts to get 64-bit unsigned integer, returns success status |\n\n### Numeric Methods - Floating Point Types\n\n| Method | Returns | Description |\n|--------|---------|-------------|\n| `GetSingle()` | `float` | Gets single-precision floating-point number, returns 0 on failure |\n| `TryGetSingle(out float?)` | `bool` | Attempts to get single-precision floating-point number, returns success status |\n| `GetDouble()` | `double` | Gets double-precision floating-point number, returns 0 on failure |\n| `TryGetDouble(out double?)` | `bool` | Attempts to get double-precision floating-point number, returns success status |\n| `GetDecimal()` | `decimal` | Gets decimal value, returns 0 on failure |\n| `TryGetDecimal(out decimal?)` | `bool` | Attempts to get decimal value, returns success status |\n\n### Date/Time Methods\n\n| Method | Returns | Description |\n|--------|---------|-------------|\n| `GetDateTime()` | `DateTime` | Gets DateTime value, returns default on failure |\n| `TryGetDateTime(out DateTime?)` | `bool` | Attempts to get DateTime value, returns success status |\n| `GetDateTimeOffset()` | `DateTimeOffset` | Gets DateTimeOffset value, returns default on failure |\n| `TryGetDateTimeOffset(out DateTimeOffset?)` | `bool` | Attempts to get DateTimeOffset value, returns success status |\n\n### GUID Methods\n\n| Method | Returns | Description |\n|--------|---------|-------------|\n| `GetGuid()` | `Guid` | Gets Guid value, returns Empty on failure |\n| `TryGetGuid(out Guid?)` | `bool` | Attempts to get Guid value, returns success status |\n\n### Base64 Methods\n\n| Method | Returns | Description |\n|--------|---------|-------------|\n| `GetBytesFromBase64()` | `byte[]?` | Decodes Base64 string to bytes, throws FormatException on failure |\n| `TryGetBytesFromBase64(out byte[]?)` | `bool` | Attempts to decode Base64 string, returns success status |\n\n### Value Extraction Methods\n\n| Method | Returns | Description |\n|--------|---------|-------------|\n| `GetValue()` | `object?` | Gets typed value based on TokenType (string, number, bool, null) |\n| `GetNumber()` | `object?` | Parses number token to best-fit numeric type (Int16, Int32, Int64, Double) |\n\n## How It Works\n\nFor a detailed explanation of the internal architecture, performance optimizations, and streaming mechanisms, see our comprehensive [How It Works](HOW_IT_WORKS.md) documentation, with mermaid diagrams.\n\nThis guide covers:\n\n- Core streaming architecture and `System.IO.Pipelines` integration\n- Memory management and buffer optimization strategies  \n- Asynchronous processing patterns and cancellation handling\n- Performance characteristics and benchmarking methodology\n- Real-world usage patterns and best practices\n\n## Samples\n\nExplore these comprehensive examples and tutorials:\n\n- **[Blazor JSON Streaming Sample](https://github.com/gragra33/BlazorJsonStreamingSample)** - Complete Blazor application demonstrating real-time JSON streaming\n- **[CodeProject Article: Deserializing Json Streams](https://codeproject.com/articles/Deserializing-Json-Streams-using-Newtonsoft-and-Sy)** - Detailed breakdown with C# \u0026 VB.NET examples, benchmarks, and performance comparisons\n\nThese resources include:\n- Full working samples\n- Performance benchmarks  \n- Memory usage comparisons\n- File and web streaming examples\n- Support for compressed (zipped) files\n- Handling very large datasets\n\n## v1.3.0 Performance Graphs\n\nVersion 1.3.0 introduces significant performance optimizations that deliver measurable improvements across all JSON processing scenarios. These enhancements focus on reducing memory allocations and garbage collection pressure while increasing overall throughput.\n\n### Speed Improvement by Scenario\n```\nSmall JSON Token-by-Token:     +7.65% ========·· ✨\nMedium JSON Token-by-Token:    +9.89% =========· ✨✨\nLarge JSON Token-by-Token:    +19.61% ========== ✨✨✨\n```\n\n### Gen2 GC Reduction\n```\nBefore: ========================================  (490 collections)\nAfter:  ===================·····················  (235 collections)\n```\n\nThat is a **52% REDUCTION!** ✨✨✨\n\n## Real-World Impact\n\nThese performance improvements translate directly into cost savings and improved user experience in production environments. Lower memory usage reduces infrastructure requirements, while fewer garbage collections mean more predictable response times and higher system stability.\n\n### High-Throughput API Example\n**Scenario**: API processing 10,000 large JSON documents per second\n\n#### Before (v1.2.0)\n- Processing Time: 56.46 seconds per 10K documents\n- Memory Allocations: ~28.47 GB per 10K documents\n- Gen2 GC: ~4,900 collections per 10K documents\n\n#### After (v1.3.0)\n- Processing Time: **45.39 seconds** per 10K documents\n- Memory Allocations: ~27.09 GB per 10K documents  \n- Gen2 GC: **~2,350 collections** per 10K documents\n\n#### Savings\n- **==· Time**: 11.07 seconds saved per 10K documents (**+19.61% throughput!**)\n- **==· Memory**: 1.38 GB less per 10K documents (**+4.82% efficiency**)\n- **=== Gen2 GC**: 2,550 fewer collections per 10K documents (**-52% reduction!**)\n- **==· Cost**: Reduced infrastructure needs, fewer GC pauses\n- **==· Capacity**: Can handle **20% more concurrent requests** with same resources\n\n## Support\n\nIf you find this library useful, please consider [buying me a coffee ☕](https://bmc.link/gragra33).\n\n## History\n\n### v2.1.0 - 19 November 2025\n\n- Added new Value methods: CopyString (performant UTF8 \u0026 UTF16), GetSByte, TryGetSByte, GetBytesFromBase64, TryGetBytesFromBase64\n- Updated readme with tables of all Value converter methods\n- Added test coverage for new Value methods\n\n### v2.0.0 - 17 November 2025\n\n- Added support for .Net 10\n- Removed support for .Net 7\n\n### v1.3.1 - October 2025\n\n- Added custom `JsonStreamException` for better error handling specificity\n- Added comprehensive [How It Works](HOW_IT_WORKS.md) documentation (uses mermaid diagrams)\n\n### v1.3.0 - October 2025\n\n- Performance improvements\n  - Up to **19.61% faster**\n  - **52% less** Gen2 GC collections\n  - **4-6% less** memory usage\n- Added Benchmarks\n\n### V1.2.0 - October 2025\n\n- Added .NET 9.0 support\n- Added symbols support to NuGet package  \n- Updated readme with improved documentation\n- Enhanced test coverage for all target frameworks\n\n### V1.1.0 - Previous Release\n\n- Updated to support .NET 8.0\n- Fixed missing parameter JsonSerializerOptions in one call\n\n### V1.0.0 - Initial Release\n\n- Initial release with .NET 7.0 support\n- Core streaming JSON functionality\n- Basic deserialization capabilities\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgragra33%2Futf8jsonasyncstreamreader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgragra33%2Futf8jsonasyncstreamreader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgragra33%2Futf8jsonasyncstreamreader/lists"}