{"id":22713597,"url":"https://github.com/program--/flatpoints","last_synced_at":"2025-03-29T22:44:00.826Z","repository":{"id":107794091,"uuid":"543725343","full_name":"program--/flatpoints","owner":"program--","description":"Flattened Points of Interest Data Format","archived":false,"fork":false,"pushed_at":"2022-11-14T00:10:19.000Z","size":464,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-05T00:27:47.636Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/program--.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-30T17:58:11.000Z","updated_at":"2022-10-16T00:32:33.000Z","dependencies_parsed_at":"2024-06-14T00:17:37.231Z","dependency_job_id":null,"html_url":"https://github.com/program--/flatpoints","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/program--%2Fflatpoints","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/program--%2Fflatpoints/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/program--%2Fflatpoints/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/program--%2Fflatpoints/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/program--","download_url":"https://codeload.github.com/program--/flatpoints/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246254097,"owners_count":20747948,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-10T14:09:38.030Z","updated_at":"2025-03-29T22:44:00.815Z","avatar_url":"https://github.com/program--.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# flatpoints\n\n**flatpoints** is an in-development data format for representing coordinates, particularly [*points of interest*](https://en.wikipedia.org/wiki/Point_of_interest), as a compressed list of integers (i.e. an [inverted list](https://en.wikipedia.org/wiki/Inverted_index)).\n\n\u003e **Warning**: this library is currently in active development, so its API and the underlying data format are subject to change at any moment.\n\n## Format\n\nSection | Description          | Bytes\n------- | -------------------- | ------\nHeader  | Magic Number         | 3 Bytes\nHeader  | Start of Data (S)    | 8 Bytes\nHeader  | Coordinate Count (N) | 8 Bytes\nHeader  | Properties Count     | 8 Bytes\nHeader  | Property Types       | N Bytes\nHeader  | Property Names       | ((8 * (N + 3) + 3) - S) Bytes\nHeader  | Property Offsets     | 8 * N Bytes\nData    | Coordinate Data      |\nData    | Property 1 Data      |\nData    | Property 2 Data      |\nData    | ...                  |\nData    | Property N Data      |\n\n## Braindumping\n\n- Coordinate Encoding Codec: Sorted Hilbert Delta Indexing\n- Integer Compression Codec: LEB128\n- Properties Compression: Gzip\n\n\u003e **Note**: These codecs are defaults for now, the plan is to allow for other encodings in later implementations.\n\nThe general idea for the *flatpoints* data format is to store point geometry in a small footprint, with cloud-native capabilities. The offset approach allows users to take advantage of [HTTP Range Requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests).\n\nCoordinates are first projected to 1D by gridding the coordinates using the world extent, then filling the space with a [Hilbert Curve](https://en.wikipedia.org/wiki/Hilbert_curve). Point positions are then associated along the curve to get an unsigned 64-bit integer representation. These integers (and the associated properties) are then sorted in ascending order, and [delta encoded](https://en.wikipedia.org/wiki/Delta_encoding). The resulting differences are then compressed using a codec of choice. In this case, they are compressed into [LEB128](https://en.wikipedia.org/wiki/LEB128) form. This encoding scheme is *slightly* lossy, but can be mitigated to be practically lossless by modifying the extent of the coordinate grid, and the number of iterations (exponent) for the Hilbert Curve. The defaults used (world extent with `N = 31`) are the maximums, and result in a (at most) ±1m margin of error in accuracy (see: [Decimal Degrees](http://wiki.gis.com/wiki/index.php/Decimal_degrees)). This can be reduced to \u003c±1cm margin of error in accuracy by reducing the extent to the coordinate's extent, but would require storing the extent to disk/in-memory.\n\n\u003e See [spress](https://github.com/UFOKN/spress) for the basis of this idea.\n\nReading the flatpoints format starts with a pre-flight request of reading the first 11 bytes, to ensure the file is a flatpoints file by the magic number, and to get the header size. The start of data (SOD) explicitly states the first (inclusive) position where the *coordinate data* starts, and implicitly gives the header size `(SOD - 1)`.\n\nA header request can then be made to get the entirety of the header.\n\nCoordinate data and the feature properties can be read independently of one another. So, if a user only requires the coordinates and property 1, then they can read only those byte ranges via the offsets \u0026 SOD.\n\n## References\n- Giulio Ermanno Pibiri, Rossano Venturini (2019). \"Techniques for Inverted Index Compression.\" [arXiv:1908.10598v2](https://arxiv.org/abs/1908.10598v2).\n- Vicente Romera Calero (2022). \"VTEnc\". [GitHub Repository](https://github.com/vteromero/VTEnc).\n- Vukovic, Tibor (2016). \"Hilbert-Geohash - Hashing Geographical Point Data Using the Hilbert Space-Filling Curve.\"\n\n## License\n\nThe reference implementation of flatpoints is licensed under APLv2.\nThe format specification for the flatpoints data format is public domain (or under a CC0 license as applicable).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprogram--%2Fflatpoints","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprogram--%2Fflatpoints","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprogram--%2Fflatpoints/lists"}