{"id":17008776,"url":"https://github.com/i-e-b/promantle","last_synced_at":"2026-04-27T12:39:27.069Z","repository":{"id":139826056,"uuid":"595522508","full_name":"i-e-b/Promantle","owner":"i-e-b","description":"Experimental ways of storing data for high speed queries","archived":false,"fork":false,"pushed_at":"2023-02-21T15:02:08.000Z","size":161,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-22T12:30:29.966Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/i-e-b.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-31T08:57:20.000Z","updated_at":"2023-01-31T15:50:13.000Z","dependencies_parsed_at":null,"dependency_job_id":"a78538ed-7a67-49d2-8333-669e771bcbda","html_url":"https://github.com/i-e-b/Promantle","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/i-e-b/Promantle","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/i-e-b%2FPromantle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/i-e-b%2FPromantle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/i-e-b%2FPromantle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/i-e-b%2FPromantle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/i-e-b","download_url":"https://codeload.github.com/i-e-b/Promantle/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/i-e-b%2FPromantle/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32337274,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-26T23:26:28.701Z","status":"online","status_checked_at":"2026-04-27T02:00:06.769Z","response_time":128,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-14T05:29:14.893Z","updated_at":"2026-04-27T12:39:27.035Z","avatar_url":"https://github.com/i-e-b.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Promantle\nExperimental ways of storing data for high speed queries\n\n* [x] Custom range triangular data\n* [ ] Multi-page data with baked-in sorting\n\n## Requirements\n\nUses Cockroach DB: https://www.cockroachlabs.com/docs/releases/index.html\nor Postgresql DB: https://www.postgresql.org/\n\n## Triangular data\n\nA pre-aggregated data set for keeping a log of data, and being able to query arbitrary\nranges of data at different levels of detail.\n\nYou need to set-up a triangular list, with a list of the data-points to aggregate\nand the different levels of detail to keep. This needs to be specified up-front.\n\n```csharp\n     // Create a Triangular list with a given 'key' and source data type\n     // Here we are keying on date (so we can aggregate across hours, days, months, etc)\n     // And we are going to feed the list with 'MySourceType' objects\nvar subject = TriangularList\u003cDateTime, MySourceType\u003e\n     // Supply database connection\n    .Create.UsingStorage(storage)\n     // Set the 'key' used for ranges and levels-of-detail. This is: Database-type, Function to read from source data items, Function used to read ranges\n    .KeyOn(\"TIMESTAMP\", DateFromSourceType, DateMinMax)\n     // Add an aggregation. This names a value extracted from source data\n     // Arguments are: Name for storing and querying, Function to read from source data, Function that aggregates multiple items, Database-type\n    .Aggregate\u003cdecimal\u003e(\"Spent\", item =\u003e item.SpentAmount, (a,b) =\u003e a+b, \"DECIMAL\")        // \u003c-- functions can be lambdas\n     // We can add as many aggregations as we want\n    .Aggregate\u003cdecimal\u003e(\"Earned\", EarnedFromSourceType, DecimalSumAggregate, \"DECIMAL\")    // \u003c-- or methods from a class\n     // And have complex aggregations that calculate a value from the source data (rather than just extracting a stored value)\n    .Aggregate\u003cdecimal\u003e(\"MaxTransaction\", item =\u003e Math.Max(item.EarnedAmount, item.SpentAmount), (a,b) =\u003e Math.Max(a,b), \"DECIMAL\")\n    \n     // We then add the ranges over which the data will be aggregated.\n     // The ranks should have the smallest key range as the lowest rank,\n     // and continue growing in range to the highest rank.\n     // It's up to the programmer to ensure the ranges are strictly increasing.\n     \n     // Arguments are: Rank order, Rank name used for querying, Function that returns the key split into these ranges\n     // The smallest rank is the smallest range you can query for.\n    .Rank(1, \"PerMinute\", DateTimeMinutes) // multiple source data aggregated into per-minute buckets\n    .Rank(2, \"PerHour\",   DateTimeHours)   // multiple per-minute buckets aggregated into per-hour buckets\n    .Rank(3, \"PerDay\",    DateTimeDays)    // hours into days\n    .Rank(4, \"PerWeek\",   DateTimeWeeks)   // days into weeks\n    \n    // Finally, build to get the TriangularList instance.\n    // This will throw exceptions if it finds basic problems.\n    // This will also create the required tables if not already present.\n    .Build();\n```\n\nNote: Only the stated `Aggregate\u003c\u003e`s are stored in the database. If the source data items have extra data, it is ignored.\n\nThen you can feed the list with new data. If you have already populated the database, you can query the existing data\nand add to it.\n\n```csharp\nsubject.WriteItem(new MySourceType{\n    EarnedAmount = 2.5m,\n    SpentAmount = 5.1m,\n    RecordedDate = new DateTime(2020,5,5, 10,11,12, DateTimeKind.Utc)\n});\n```\n\nWhen querying for aggregate data, it will be returned with a count of source items that have been combined at that point,\nand the upper and lower bounds of the 'key' value in the data items combined at that point (which is different from the range that\nthe aggregation covers).\n\nBe careful with aggregations; make sure they make sense. Do not aggregate averages, as you will get invalid value. Use a sum (adding)\naggregation and divide the outcome by value count instead:\n\n```csharp\nvar spentOnDay = subject.ReadDataAtPoint\u003cdecimal\u003e(\"Spent\", \"PerDay\", new DateTime(2020,5,5,  0,0,1, DateTimeKind.Utc));\nvar averageSpend = spentOnDay.Value / spentOnDay.Count;\n```\n\nWhen making rank-range functions, it usually makes sense to transform the key value to a scalar, and then divide it by some amount.\nIt's best that the boundaries for each rank line-up, but it is not required.\n\n```csharp\n    private static readonly DateTime _baseDate = new(2000, 1, 1, 0, 0, 0, DateTimeKind.Utc);\n    private static long DateTimeMinutes(DateTime item) =\u003e (long)Math.Floor((item - _baseDate).TotalMinutes);\n    private static long DateTimeHours(DateTime item) =\u003e DateTimeMinutes(item) / 60;\n    private static long DateTimeDays(DateTime item)  =\u003e DateTimeHours(item)   / 24;\n    private static long DateTimeWeeks(DateTime item) =\u003e DateTimeDays(item)    / 7;\n```\n\nItem keys and rank-range functions are allowed to be entirely arbitrary: (see `PromantleTests.TriangularListTests.can_use_arbitrary_values_for_keys_and_ranks` for a worked example)\n\n```csharp\npublic enum Geolocation: long\n{\n    China, India, USA, ...\n}\n\n. . .\n\nvar subject = TriangularList\u003cGeolocation, SaleWithLocation\u003e\n    .Create.UsingStorage(storage)\n    .KeyOn(\"INT\", s=\u003es.SalesLocation, Regions.MinMax)\n    .Aggregate\u003cdecimal\u003e(\"Cost\", s=\u003es.Cost, DecimalSumAggregate, \"DECIMAL\")\n    .Aggregate\u003cdecimal\u003e(\"Price\", s=\u003es.SoldPrice, DecimalSumAggregate, \"DECIMAL\")\n    .Rank(1, \"Country\",   r=\u003e(long)r)\n    .Rank(2, \"Landmass\",   r=\u003e(long)Regions.LocationToLandmass(r))\n    .Rank(3, \"Zone\",    r=\u003e(long)Regions.LocationToGeoZone(r))\n    .Rank(4, \"Worldwide\",    Regions.LocationToWorldwide)\n    .Build();\n\nsubject.WriteItem(new SaleWithLocation(DateTime.Now, 10.00m, 35.00m, Geolocation.Angola));\n// and so on...\n\n// Get sales across Europe (that is, all in same 'Landmass' as Germany)\nvar sales = subject.ReadDataAtPoint\u003cdecimal\u003e(\"Price\", \"Landmass\", Geolocation.Germany);\nvar costs = subject.ReadDataAtPoint\u003cdecimal\u003e(\"Cost\", \"Landmass\", Geolocation.Germany);\nvar profit = sales.Value - costs.Value;\nif (profit \u003c 0) RunNewAdCampaign();\n```\n\nData layout concept \n\n```\n\nRank Zero\n(original\ndata pts)    Rank 1      Rank 2    . . .   Rank n\n            :  ___      :\nA     \\     :     \\     :\nB      |    :      |    :\nC      +---  AE    |    :\nD      |    :      |    :\nE     /     :      |___  AJ               ↑\nF  \\        :      |    :                 :\nG   |       :      |    :                 :\nH   +------  FJ    |    :                 : AN\nI   |       :      |    :                 :\nJ  /        :  ___/     :                 :\nK    \\      :       \\   :                 ↓\nL     |____  KN      |__ KN\nM     |     :        |  :\nN    /      :       /   :\n\n```\n\n## Multi-page data pre sorted\n\n### Idea\n\nFor cached queries, each column has a matching 'page-column', that gives the page is should be on *if sorted by that column*\n\nThat way, we can have a single 'paged' data table, but still correctly sort and page by any column.\n\ne.g. (using sample data below)\n\nTo get page 3 when ordering by `Type`\n```\nSELECT * FROM AssetTable WHERE Type_page = 3 ORDER BY Type_data;\n```\n\nTo get page 1 when ordering by `AssetId`\n```\nSELECT * FROM AssetTable WHERE AssetId_page = 1 ORDER BY AssetId_data;\n```\n\nTo get \"2nd\" page when sorting by location in reverse order\n```\nSELECT * FROM AssetTable WHERE Location_page = (MAX(Location_page)-1) ORDER BY Location_data DESC;\n```\n\n### Example\n\nPage size of two to make the example short. Should really be 10 or so.\n\nAsset Table\n\n```\nAssetId_data    AssetId_page    Name_data    Name_page    Location_data    Location_page    Type_data        Type_page    AcquiredDate_data    AcquiredDate_page\n1               1               Chair        2            London           2                OfficeFurniture  3            2021-01-05           2\n2               1               Cabinet      1            London           2                OfficeFurniture  3            2022-06-04           5\n3               2               Desk         4            London           3                OfficeFurniture  4            2020-08-11           1\n4               2               Computer     3            London           3                IT               2            2021-07-05           3\n5               3               Chair        2            Paris            4                OfficeFurniture  5            2022-05-04           4\n6               3               Cabinet      1            Paris            4                OfficeFurniture  4            2021-05-05           3\n7               4               Desk         4            Paris            5                OfficeFurniture  5            2020-06-08           1\n8               4               Computer     3            Paris            5                IT               2            2021-04-04           2\n9               5               Server       5            Berlin           1                IT               1            2022-01-05           4\n10              5               Server       5            Berlin           1                IT               1            2023-01-11           5\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fi-e-b%2Fpromantle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fi-e-b%2Fpromantle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fi-e-b%2Fpromantle/lists"}