{"id":18009915,"url":"https://github.com/kekyo/jsonstreamer","last_synced_at":"2025-07-17T07:03:47.429Z","repository":{"id":195977190,"uuid":"693987921","full_name":"kekyo/JsonStreamer","owner":"kekyo","description":"JSON Lines streaming serializer/deserializer on .NET and ASP.NET Core.","archived":false,"fork":false,"pushed_at":"2023-11-16T01:55:56.000Z","size":205,"stargazers_count":11,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-10-30T03:48:42.709Z","etag":null,"topics":["aspnetcore","dotnet","jsonlines","jsonstreaming","ndjson","streaming"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kekyo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-09-20T05:36:32.000Z","updated_at":"2024-08-16T15:03:52.000Z","dependencies_parsed_at":"2023-11-16T02:46:26.147Z","dependency_job_id":null,"html_url":"https://github.com/kekyo/JsonStreamer","commit_stats":null,"previous_names":["kekyo/aspnetcore.jsonstreamer","kekyo/jsonstreamer"],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kekyo%2FJsonStreamer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kekyo%2FJsonStreamer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kekyo%2FJsonStreamer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kekyo%2FJsonStreamer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kekyo","download_url":"https://codeload.github.com/kekyo/JsonStreamer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245670755,"owners_count":20653413,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aspnetcore","dotnet","jsonlines","jsonstreaming","ndjson","streaming"],"created_at":"2024-10-30T02:11:33.859Z","updated_at":"2025-07-17T07:03:47.423Z","avatar_url":"https://github.com/kekyo.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# JsonStreamer\n\nJSON Lines streaming serializer/deserializer on .NET and ASP.NET Core.\n\n![JsonStreamer](Images/JsonStreamer.100.png)\n\n# Status\n\n[![Project Status: WIP – Initial development is in progress, but there has not yet been a stable, usable release suitable for the public.](https://www.repostatus.org/badges/latest/wip.svg)](https://www.repostatus.org/#wip)\n\n## ASP.NET Core packages\n\n|Target serializer|Pakcage|\n|:----|:----|\n|Newtonsoft.Json|[![NuGet JsonStreamer.NewtonsoftJson](https://img.shields.io/nuget/v/JsonStreamer.NewtonsoftJson.svg?style=flat)](https://www.nuget.org/packages/JsonStreamer.NewtonsoftJson)|\n|System.Text.Json|[![NuGet JsonStreamer.NewtonsoftJson](https://img.shields.io/nuget/v/JsonStreamer.SystemTextJson.svg?style=flat)](https://www.nuget.org/packages/JsonStreamer.SystemTextJson)|\n\n## Client package\n\n|Target serializer|Pakcage|\n|:----|:----|\n|Newtonsoft.Json|[![NuGet JsonStreamer.NewtonsoftJson.Client](https://img.shields.io/nuget/v/JsonStreamer.NewtonsoftJson.Client.svg?style=flat)](https://www.nuget.org/packages/JsonStreamer.NewtonsoftJson.Client)|\n\n----\n\n## What is this?\n\nHas anyone else noticed that ASP.NET Core can send streaming data using asynchronous iterators `IAsyncEnumerable\u003cT\u003e` ?\nIt is code like this:\n\n```csharp\n[HttpGet]\npublic IAsyncEnumerable\u003cWeatherForecast\u003e Get() =\u003e\n    AsyncEnumerable.Range(1, 1000000).\n    Select(index =\u003e new WeatherForecast\n    {\n        Date = DateTime.Now.AddDays(index),\n        TemperatureC = Random.Shared.Next(-20, 55),\n        Summary = Summaries[Random.Shared.Next(Summaries.Length)]\n    });\n```\n\nIn fact, this works as expected on the server side of ASP.NET Core and does not consume memory for buffering.\nWere you aware that this code returns a JSON array?\n\n```json\n[\n    {\n        \"date\": \"2023-09-20T20:23:49.5736146+09:00\",\n        \"temperatureC\": 14,\n        \"temperatureF\": 57,\n        \"summary\": \"Mild\"\n    },\n    {\n        \"date\": \"2023-09-21T20:23:49.5768618+09:00\",\n        \"temperatureC\": 34,\n        \"temperatureF\": 93,\n        \"summary\": \"Freezing\"\n    },\n    {\n        \"date\": \"2023-09-22T20:23:49.5768924+09:00\",\n        \"temperatureC\": 1,\n        \"temperatureF\": 33,\n        \"summary\": \"Mild\"\n    },\n\n    // (continues a lot of JObject)\n]\n```\n\nWhat about on the browser side receiving this?\n\nIn .NET, JavaScript and other platforms, there are few libraries that can streaming deserialize huge JSON arrays.\nStreaming sending is easy, but streaming receiving is very difficult.\n\nOn the other hand, there is a variant of the JSON format designed for this purpose.\nThese are [JSON Lines (NDJSON) format](https://jsonlines.org/).\n\nIt is a variant, but very simple and easy to understand:\n\n```json\n{\"date\":\"2023-09-20T20:23:49.5736146+09:00\",\"temperatureC\":14,\"temperatureF\":57,\"summary\":\"Mild\"}\n{\"date\":\"2023-09-21T20:23:49.5768618+09:00\",\"temperatureC\":34,\"temperatureF\":93,\"summary\":\"Freezing\"}\n{\"date\":\"2023-09-22T20:23:49.5768924+09:00\",\"temperatureC\":1,\"temperatureF\":33,\"summary\":\"Mild\"}\n\n// (continues a lot of JObject)\n```\n\nThat is instead of JSON array, JObjects are sent only separated by LF (Newline delimited).\nThis means that deserializer implementation is easier.\n\nFor example, in the case of JavaScript, the package [can-ndjson-stream](https://github.com/canjs/can-ndjson-stream) exists and can be used immediately.\n\nJsonStreamer overrides the serializer so that when returning asynchronous iterators, they are automatically sent in JSON Lines.\n(Other types use the default serializer)\nAnd now supports on client-side .NET package!\n\n----\n\n## Target .NET platforms (ASP.NET Core)\n\n|Serializer|ASP.NET Core versions|\n|:----|:----|\n|Newtonsoft.Json|ASP.NET Core 9 to 3|\n|System.Text.Json|ASP.NET Core 9 to 5|\n\n* We tested on only ASP.NET Core 9, 8 and 6.\n\n## Target .NET platforms (Client)\n\n|Serializer|.NET versions|\n|:----|:----|\n|Newtonsoft.Json|.NET 9 to 5, .NET Core 3.1, 2.2, .NET Standard 2.1, 2.0, .NET Framework 4.8.1 to 4.6.1|\n\n* Newtonsoft.Json version is 13.0.1 or higher.\n\n----\n\n## How to use (ASP.NET Core)\n\nIt is very easy to use, just install this package and set it up in the builder as follows.\n\n|Serializer|Application|\n|:----|:----|\n|Newtonsoft.Json|Install package [JsonStreamer.NewtonsoftJson](https://www.nuget.org/packages/JsonStreamer.NewtonsoftJson) and call `AddNewtonsoftJsonStreamer()` instead of `AddNewtonsoftJson()`.|\n|System.Text.Json|Install package [JsonStreamer.SystemTextJson](https://www.nuget.org/packages/JsonStreamer.SystemTextJson) and call `AddJsonStreamer()`.|\n\nFor builder configuration example:\n\n```csharp\npublic static void Main(string[] args)\n{\n    // ...\n\n    // Add services to the container.\n    builder.Services.AddControllers().\n        AddNewtonsoftJsonStreamer();    // Enable JsonStreamer.\n\n    // ...\n\n    var app = builder.Build();\n\n    app.MapControllers();\n    app.Run();\n}\n```\n\nAnd you have to use with asynchronous iterator `IAsyncEnumerable\u003cT\u003e` on the webapi result type:\n\n```csharp\n// Use IAsyncEnumerable\u003cT\u003e type on entry point result.\n[HttpGet]\npublic IAsyncEnumerable\u003cWeatherForecast\u003e Get()\n{\n    // (Asynchronous iteration implementation)\n}\n```\n\nThe sample fragment use with `HttpGet`, but this serializer can use any other http methods.\n\nComplete ASP.NET Core example projects are located in the [samples directory](samples/).\n\n----\n\n## How to use (Client)\n\nIt is very easy to use, just install this package.\n\n|Serializer|Application|\n|:----|:----|\n|Newtonsoft.Json|Install package [JsonStreamer.NewtonsoftJson.Client](https://www.nuget.org/packages/JsonStreamer.NewtonsoftJson.Client).|\n|System.Text.Json|TODO:|\n\n### For using with HttpClient\n\nAvailable varies of many overloads.\n\n```csharp\nusing System.Net.Http;\nusing JsonStreamer;\n\nvar httpClient = new HttpClient();\n\n// ...\n\n// HTTP GET\nawait foreach (var order in\n    await httpClient.GetStreamingAsync\u003cOrder\u003e(\n        \"https://example.com/api/orders\"))\n{\n    // Already deserialized the \"Order\" object.\n}\n\n// HTTP POST\nawait foreach (var order in\n    await httpClient.PostStreamingAsync\u003cOrder\u003e(\n        \"https://example.com/api/orders\",\n        new StringContent(\"ORDER DATA\")))\n{\n    // Already deserialized the \"Order\" object.\n}\n```\n\n```csharp\nvar httpClient = new HttpClient();\n\n// ...\n\n// Get from the response interface\nawait using var response = await httpClient.GetAsync(...);\n\nawait foreach (var order in\n    await response.Content.ReadStreamingAsync\u003cOrder\u003e())\n{\n    // Already deserialized the \"Order\" object.\n}\n```\n\n### For using with Stream\n\nYou can use JsonStreamer outsite HTTP access.\n\n```csharp\nusing System.IO;\nusing JsonStreamer;\n\nusing var fs = new FileStream(\n    \"huge_data.jsonl\",\n    FileMode.Open, FileAccess.Read, FileShare.Read,\n    65536, true);\n\n// ...\n\nawait foreach (var order in\n    await fs.ReadStreamingAsync\u003cOrder\u003e())\n{\n    // Already deserialized the \"Order\" object.\n}\n```\n\n----\n\n## Tips and Notes\n\n### Newtonsoft.Json version is NOT buffering\n\nASP.NET Core official package [Microsoft.AspNetCore.Mvc.NewtonsoftJson](https://www.nuget.org/packages/Microsoft.AspNetCore.Mvc.NewtonsoftJson/) has a problem for serializing `IAsyncEnumerable\u003cT\u003e` as a JSON array actually buffers the data entirely.\n\n* See also: [ASP.NET Core `AsyncEnumerableReader.ReadInternal\u003cT\u003e()`](https://github.com/dotnet/aspnetcore/blob/e6c7c01bce4fce79bf5bc84098ea8d347ef358cc/src/Mvc/Mvc.Core/src/Infrastructure/AsyncEnumerableReader.cs#L79)\n\nHowever, the JsonStreamer implementation avoids this limitation because the operation of serializing JObject elements is performed in each iterative asynchronous iteration.\n\n### JavaScript deserializer\n\nWe introduced [can-ndjson-stream](https://github.com/canjs/can-ndjson-stream) for deserialization in JavaScript,\nwhich can be used to configure the following asynchronous iterator (in TypeScript):\n\n```typescript\nimport ndjsonStream from 'can-ndjson-stream';\n\nasync function* streamingFetch\u003cT\u003e(url: string) {\n    const response = await fetch(url);\n    const reader =\n        ndjsonStream(response.body).\n        getReader();\n    while (true) {\n        const result = await reader.read();\n        if (result.done) {\n            break;\n        }\n        yield result.value as T;\n    }\n}\n```\n\nThe `streamingFetch()` function can be used to drive an asynchronous iterator as follows:\n\n```typescript\ninterface WeatherForecast {\n    date: string;\n    temperatureC: number;\n    temperatureF: number;\n    summary: string;\n}\n\nasync function fetchWeatherForeastItems() {\n    const url = \"http://localhost:4242/weatherforecast\";\n\n    for await (const item of streamingFetch\u003cWeatherForecast\u003e(url)) {\n\n        // ...\n    }\n}\n```\n\n\n----\n\n## TODO\n\n* Enabling attribute-based control.\n\n\n----\n\n## License\n\nApache-v2\n\n## History\n\n* 1.1.0:\n  * Supported .NET 9.0.\n  * Supported ASP.NET Core 9.0.\n* 1.0.0:\n  * Supported ASP.NET Core 8.0 and 3.0 (netcoreapp3.0).\n  * Downgraded ASP.NET Core Newtonsoft.Json package requirements.\n* 0.4.0:\n  * (Breaking change) Renamed package name `JsonStreamer.*` instead of `AspNetCore.JsonStreamer.*`.\n  * Added JsonStreamer client package.\n* 0.3.0:\n  * Supported System.Text.Json based serializing.\n  * Improved cancellation.\n* 0.2.0:\n  * Refactored.\n* 0.1.0:\n  * Initial release for Newtonsoft.Json serializer.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkekyo%2Fjsonstreamer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkekyo%2Fjsonstreamer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkekyo%2Fjsonstreamer/lists"}