{"id":18009866,"url":"https://github.com/kekyo/massivepoints","last_synced_at":"2025-06-14T11:40:06.275Z","repository":{"id":245701482,"uuid":"818941669","full_name":"kekyo/MassivePoints","owner":"kekyo","description":".NET implementation of modified QuadTree, perform faster range searches from very large number of multi-dimensional coordinates in the portable way.","archived":false,"fork":false,"pushed_at":"2024-07-14T12:38:06.000Z","size":408,"stargazers_count":40,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-09T04:59:14.740Z","etag":null,"topics":["2d","3d","4d","adonet","dotnet","gis","in-memory","lookup","octatree","quadtree"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kekyo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-23T10:10:35.000Z","updated_at":"2025-03-17T12:52:44.000Z","dependencies_parsed_at":"2024-07-13T09:25:29.135Z","dependency_job_id":"c3dc9465-c62c-4a67-b7c6-69d86d42ad18","html_url":"https://github.com/kekyo/MassivePoints","commit_stats":null,"previous_names":["kekyo/massivepoints"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/kekyo/MassivePoints","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kekyo%2FMassivePoints","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kekyo%2FMassivePoints/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kekyo%2FMassivePoints/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kekyo%2FMassivePoints/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kekyo","download_url":"https://codeload.github.com/kekyo/MassivePoints/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kekyo%2FMassivePoints/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259810848,"owners_count":22915133,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["2d","3d","4d","adonet","dotnet","gis","in-memory","lookup","octatree","quadtree"],"created_at":"2024-10-30T02:11:24.875Z","updated_at":"2025-06-14T11:40:06.214Z","avatar_url":"https://github.com/kekyo.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MassivePoints\n\n.NET implementation of modified QuadTree, perform faster range searches from very large number of multi-dimensional coordinates, with in-memory and database offloading.\n\n![MassivePoints](Images/MassivePoints.200.png)\n\n# Status\n\n[![Project Status: WIP – Initial development is in progress, but there has not yet been a stable, usable release suitable for the public.](https://www.repostatus.org/badges/latest/wip.svg)](https://www.repostatus.org/#wip)\n\n|Target|Pakcage|\n|:----|:----|\n|Any|[![NuGet MassivePoints](https://img.shields.io/nuget/v/MassivePoints.svg?style=flat)](https://www.nuget.org/packages/MassivePoints)|\n\n----\n\n## What is this?\n\nHave you ever tried to store a large amount of 2D/3D coordinate points and extract these from any given coordinate range?\nNormally for such requests, we would use a GIS-compatible database system or service with complex management.\n\nThis library provides the ability to store and filter ranges of multi-dimensional coordinate points in the portable way.\n\nIt's very easy to use:\n\n```csharp\nusing MassivePoints;\n\n// Create QuadTree dictionary with 2D coordinate bound\n// and pair of value type (string) on the memory.\ndouble width = 100000.0;\ndouble height = 100000.0;\nQuadTree\u003cstring\u003e quadTree =\n    QuadTree.Factory.Create\u003cstring\u003e(width, height);\n\n// Begin a session for update.\nawait using QuadTreeSession\u003cstring\u003e session =\n    await quadTree.BeginSessionAsync(true);\n\n// Insert a lot of random 2D coordinates.\nvar count = 1000000;\nvar r = new Random();\nfor (var index = 0; index \u003c count; index++)\n{\n    double x = r.Next(0, width - 1);\n    double y = r.Next(0, height - 1);\n    await session.InsertPointAsync(\n        new Point(x, y),    // x, y\n        $\"Point{index}\");   // value\n}\n\n// Extract values by specifying 2D coordinate range.\ndouble x0 = 30000.0;\ndouble y0 = 40000.0;\ndouble x1 = x0 + 35000.0;\ndouble y1 = y0 + 23000.0;\nforeach (PointItem\u003cstring\u003e entry in\n    await session.LookupBoundAsync(new Bound(x0, y0, x1, y1)))\n{\n    Console.WriteLine($\"{entry.Point}: {entry.Value}\");\n}\n```\n\nIt has the following features:\n\n* Implements modified QuadTree coordinate search algorithm.\n  * This algorithm stores multiple coordinate points as a linear list for each node in QuadTree.\n  * A pure QuadTree has the problem of deeper node nesting, but with this method the nodes do not become deeply nested until they are filled.\n  * Searching within a node is `O(n)`, but the CPU cache works absolutely faster in the in-memory data provider,\n    and it is possible to speed up the process by using clever indexing on the database backend (of course, there are trade-offs).\n* Included add a coordinate point, lookup and remove features.\n* Supported multi-dimensional coordinate points.\n  * By extending it to N-dimensions, you will be using a naturally extended algorithm,\n    such as BinaryTree, QuadTree, OctaTree and more.\n* Completely separates between QuadTree controller and data provider.\n  * Builtin data providers: In-memory and ADO.NET.\n  * Using the SQLite ADO.NET provider (System.Data.SQLite),\n    it is possible to perform bulk inserts of 170,000 2D coordinate points per second in my environment.\n* Fully asynchronous operation.\n* Supported asynchronous streaming lookup (`IAsyncEnumerable\u003cT\u003e`).\n\n### Target .NET platforms\n\n* .NET 8.0 to 5.0\n* .NET Core 3.1 to 2.0\n* .NET Standard 2.1 and 2.0\n* .NET Framework 4.8.1 to 4.6.1\n\n----\n\n## How to use\n\nInstall [MassivePoints](https://www.nuget.org/packages/MassivePoints) from NuGet.\n\n### Create in-memory QuadTree\n\nYou can use the factory to easy to use QuadTree:\n\n```csharp\nusing MassivePoints;\n\n// Create QuadTree dictionary with 2D coordinate bound\n// and pair of value type (string) on the memory.\ndouble width = 100000.0;\ndouble height = 100000.0;\nQuadTree\u003cstring\u003e quadTree = QuadTree.Factory.\n    Create\u003cstring\u003e(width, height);          // TValue = string\n\n// Create QuadTree (OctaTree) dictionary with 3D coordinate bound\n// and pair of value type (string) on the memory.\ndouble width = 100000.0;\ndouble height = 100000.0;\ndouble depth = 100000.0;\nQuadTree\u003cstring\u003e octaTree = QuadTree.Factory.\n    Create\u003cstring\u003e(width, height, depth);   // TValue = string\n```\n\nIf you want to use a dimension that goes over three dimensions, use `new Bound(...)` to specify it.\n\nIn all the cases discussed below, 2D coordinate points are used as examples.\nHowever, please remember that you can use various overloads to express N-dimensional coordinates.\n\n### Create QuadTree with ADO.NET provider\n\nThis sample code uses SQLite ADO.NET provider: [System.Data.SQLite](https://www.nuget.org/packages/System.Data.SQLite/)\n\n```csharp\nusing MassivePoints;\nusing MassivePoints.Data;\nusing System.Data.SQLite;\n\n// Open SQLite database.\nvar connectionString = new SQLiteConnectionStringBuilder()\n{\n    DataSource = \"points.db\",\n}.ToString();\n\n// Create QuadTree provider using SQLite database.\ndouble width = 100000.0;   // 2D coordinate bound.\ndouble height = 100000.0;\nvar provider = QuadTree.Factory.CreateProvider\u003cstring\u003e(\n    async () =\u003e   // ADO.NET connection factory\n    {\n        var connection = new SQLiteConnection(connectionString);\n        await connection.OpenAsync();\n        return connection;\n    },\n    new DbDataProviderConfiguration(),\n    new Bound(width, height));\n\n// Setup the SQLite tables to be used with QuadTree.\nawait provider.CreateSQLiteTablesAsync(false);\n\n// Create QuadTree dictionary.\nQuadTree\u003cstring\u003e quadTree = QuadTree.Factory.Create(provider);\n```\n\nEven when using ADO.NET, it is possible to handle dimensions greater than three dimensions.\nIn this case, the database schema definition will also use a definition that supports N-dimensions.\n\nWhen using SQLite, you can use `provider.CreateSQLiteTablesAsync(...)`\nto automatically generate N-dimensional tables can be generated automatically.\n\nThe `DbDataProviderConfiguration` class provides customization points for use with various database systems.\n\n### Begin a session\n\nYou have to use `BeginSessionAsync()` to start QuadTree manipulation:\n\n```csharp\n// Begin a reading session.\nawait using (QuadTreeSession\u003cstring\u003e session =\n    await quadTree.BeginSessionAsync())\n{\n    // (Do lookup manipulation)\n    await session.LookupPointAsync(...);\n}\n```\n\nOr, use `BeginUpdateSessionAsync()` to start updating QuadTree manipulation:\n\n```csharp\n// Begin a update session.\nawait using (QuadTreeUpdateSession\u003cstring\u003e session =\n    await quadTree.BeginUpdateSessionAsync())\n{\n    // (Do insert, lookup and remove manipulation)\n    await session.InsertPointAsync(...);\n\n    // Finish a session.\n    await session.FinishAsync();\n}\n```\n\nThese methods can be categorized according to whether or not they perform an update process.\nIf use `BeginUpdateSessionAsync()`, the concurrent operation may fail.\nConversely if `BeginSessionAsync()`, concurrent operation is possible and multiple lookup operations may be performed simultaneously.\n\n|Method|Lookup|Update: Insert,Remove|Concurrency|\n|:----|:----|:----|:----|\n|`BeginUpdateSessionAsync()`|Yes|Yes|No|\n|`BeginSessionAsync()`|Yes|No|Yes|\n\nAlso, be sure to call `FinishAsync()` after any updates.\nDepending on the backend data provider, the updates may be undone.\n\n### Insert coordinate points\n\nYou can insert a coordinate point and a value to associate with it using `InsertPointAsync()`:\n\n```csharp\n// Insert a random 2D coordinate point.\nvar r = new Random();\nPoint point = new Point(r.Next(0, width - 1), r.Next(0, height - 1));\nawait session.InsertPointAsync(\n    point,              // x, y\n    $\"Point{index}\");   // value\n```\n\nIf you want to insert a N-dimensional coordinate point,\nyou can use the overload of `new Point(x, y, z)` or others.\n\nYou can also perform other bulk inserts,\ninserting a large number of coordinate points to faster with `InsertPointsAsync()`.\n\n```csharp\n// Insert a lot of random coordinates.\nvar count = 1000000;\nvar r = new Random();\n\nawait session.InsertPointsAsync(\n    Enumerable.Range(0, count).\n    Select(_ =\u003e PointItem.Create(   // Makes pair of a point and a value.\n        r.Next(0, width - 1),    // x\n        r.Next(0, height - 1),   // y\n        $\"Point{index}\")));      // value\n```\n\nWhen performing bulk insertion,\nuse `PointItem\u003cTValue\u003e` to represent the pairs of values associated with the coordinate points.\nAs shown above, you can omit the generic type parameter by using `PointItem.Create(...)` method.\n\n### Lookup coordinate points\n\nWith exact coordinate point by `LookupPointAsync()`:\n\n```csharp\n// Extract values by specifying a 2D coordinate point.\n// There is a possibility that multiple values\n// with the same coordinates will be extracted.\n\nPoint targetPoint = new Point(31234.0, 45678.0);\n\nforeach (PointItem\u003cstring\u003e entry in\n    await session.LookupPointAsync(targetPoint))\n{\n    Console.WriteLine($\"{entry.Point}: {entry.Value}\");\n}\n```\n\nWith coordinate range by `LookupBoundAsync()`:\n\n```csharp\n// Extract values by specifying coordinate range.\nBound targetBound = new Bound(\n    30000.0, 40000.0,                       // x0, y0\n    30000.0 + 35000.0, 40000.0 + 23000.0);  // x1, y1 (exclusive, right-opened)\n\nforeach (PointItem\u003cstring\u003e entry in\n    await session.LookupBoundAsync(targetBound))\n{\n    Console.WriteLine($\"{entry.Point}: {entry.Value}\");\n}\n```\n\nNote that the coordinate range is right-open interval.\n\n* ex: `[30000.0,40000.0 - 65000.0,63000.0)`\n\n### Streaming lookup\n\nMassivePoints supported `IAsyncEnumerable\u003cT\u003e` asynchronous streaming.\nUse `EnumerateBoundAsync()`:\n\n```csharp\n// Extract values on asynchronous iterator.\nBound targetBound = new Bound(\n    30000.0, 40000.0,                       // x0, y0\n    30000.0 + 35000.0, 40000.0 + 23000.0);  // x1, y1 (exclusive, right-opened)\n\nawait foreach (PointItem\u003cstring\u003e entry in\n    session.EnumerateBoundAsync(targetBound))\n{\n    Console.WriteLine($\"{entry.Point}: {entry.Value}\");\n}\n```\n\nBecause of the streaming process, `EnumerateBoundAsync()` can enumerate even a huge set of coordinate points in the result without any problem.\nHowever, be aware that its performance is not as good as that of `LookupBoundAsync()`.\n\n### Remove coordinate points\n\nWith exact coordinate point by `RemovePointAsync()`:\n\n```csharp\n// Remove exact coordinate point.\nPoint targetPoint = new Point(31234.0, 45678.0);\n\nlong removed = await session.RemovePointAsync(targetPoint);\n```\n\nWith coordinate range by `RemoveBoundAsync()`:\n\n```csharp\n// Remove coordinate range.\nBound targetBound = new Bound(\n    30000.0, 40000.0,                       // x0, y0\n    30000.0 + 35000.0, 40000.0 + 23000.0);  // x1, y1 (exclusive, right-opened)\n\nlong removed = await session.RemoveBoundAsync(targetBound);\n```\n\n----\n\n## Advanced topics\n\n### Node and node depth\n\nTODO:\n\n```csharp\n// Insert a coordinate point to get node depth.\nvar nodeDepth = await session.InsertPointAsync(point, $\"Point{index}\");\n```\n\nTODO:\n\n### Perform index shrinking\n\nIn the default configuration of MassivePoints,\neven if you remove a coordinate point, the index will not be shrinked.\n\nThe index shrinking is a slow process because it is expensive to determine when shrinking is necessary.\nWhile understanding this drawback, set the `performShrinking` argument to `true` as follows:\n\n```csharp\n// Remove coordinate range with index shrinking.\nBound targetBound = new Bound(\n    30000.0, 40000.0,                       // x0, y0\n    30000.0 + 35000.0, 40000.0 + 23000.0);  // x1, y1 (exclusive, right-opened)\n\nlong removed = await session.RemoveBoundAsync(\n    targetBound, performShrinking: true);\n```\n\n### ADO.NET database schema\n\nTODO:\n\nMassivePoints provides a `provider.CreateSQLiteTablesAsync()` method that makes it easy to create tables for SQLite.\n\n* [`CreateSQLiteTablesAsync()`](MassivePoints/Data/DbDataProviderExtension.cs)\n\nUnfortunately, there is no way to define table definitions in a platform-neutral way in .NET, so this method is only for SQLite.\nHowever, it can be used with different database systems by manually defining tables similar to the following schema.\n\nOnly the following two tables are required:\n\n#### Node table\n\nTODO:\n\n![](Images/dbschema1.png)\n\n#### Node-point table\n\nTODO:\n\n![](Images/dbschema2.png)\n\n\n----\n\n## TODO\n\n* Additional xml comment and documents.\n* Supports F# friendly interfaces.\n* Added more useful helper methods.\n\n## License\n\nApache-v2\n\n## History\n\n* 0.13.0:\n  * Splitted core library (`MassivePoints.Core`).\n    * As before, there is no problem if you use the `MassivePoints` package. Rebuilding is required.\n  * Interface types is removed.\n    * ex: `IQuadTree\u003cT\u003e` --\u003e `QuadTree\u003cT\u003e`\n  * Supported asynchronous factory for database connection.\n* 0.12.0:\n  * Fixed boundary coordinate precision on calculation for splitting.\n  * Fixed the globe bound.\n  * Added coordinate points validation process on sample code.\n  * Added more xml comments.\n* 0.11.0:\n  * Added flush method.\n  * Improved bulk insertion when the node already dense points.\n  * Improved parallel distribution works.\n  * Fixed infinite recursive calls when bulk insert count exceeds ceiling.\n  * Distribution is not performed when coordinate points are inserted in excess of MaxNodePoints.\n  * Added OpenStreetMap pbf insertion sample ([in the samples directory](samples/ImportOsmNode))\n* 0.10.0:\n  * Split session interface between updatable and readable.\n  * Supported multi-dimensional coordinate points.\n* 0.9.0:\n  * Added bulk insert features.\n  * Improved concurrency.\n  * Implemented index shrinking.\n* 0.8.0:\n  * Initial release.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkekyo%2Fmassivepoints","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkekyo%2Fmassivepoints","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkekyo%2Fmassivepoints/lists"}