{"id":16386350,"url":"https://github.com/tmcw/memory-geojson","last_synced_at":"2025-03-23T04:31:27.860Z","repository":{"id":50624276,"uuid":"519656921","full_name":"tmcw/memory-geojson","owner":"tmcw","description":"experimental memory-efficient geojson representation","archived":false,"fork":false,"pushed_at":"2022-08-01T13:07:18.000Z","size":26,"stargazers_count":19,"open_issues_count":5,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-16T01:55:50.304Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tmcw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-07-31T01:46:21.000Z","updated_at":"2022-11-29T22:49:58.000Z","dependencies_parsed_at":"2022-09-06T01:31:29.090Z","dependency_job_id":null,"html_url":"https://github.com/tmcw/memory-geojson","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tmcw%2Fmemory-geojson","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tmcw%2Fmemory-geojson/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tmcw%2Fmemory-geojson/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tmcw%2Fmemory-geojson/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tmcw","download_url":"https://codeload.github.com/tmcw/memory-geojson/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245056889,"owners_count":20553855,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T04:17:02.884Z","updated_at":"2025-03-23T04:31:27.524Z","avatar_url":"https://github.com/tmcw.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# memory-geojson (experimental 🧪)\n\nA memory-efficient GeoJSON representation.\n\nThis is not a new format. It's not meant to be serialized, and it doesn't\nadd any features on top of the GeoJSON format.\n\nWhat it attempts to do is provide an in-memory representation of GeoJSON\nthat uses TypedArrays to store flattened coordinates. The main benefits\nand goals are:\n\n- Reduce memory requirements of GeoJSON data.\n- Support [transferrable](https://developer.mozilla.org/en-US/docs/Glossary/Transferable_objects) or\n  [shared](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer)\n  array buffers which make communication with WebWorkers much\n  faster.\n\nThe GeoJSON format is almost perfect, but the way it represents\ncoordinates with nested arrays can be a performance issue. This\nis an experiment to support flattened arrays.\n\nHeavy inspiration taken from [mapshaper](https://github.com/mbloch/mapshaper)\nperhaps the only tool that I know of that has a strategy of\nflattening those arrays.\n\n## Storage scheme\n\n_This might change, I'm just braindumping what's in the code right now_.\n\nThis takes GeoJSON as input and stores it in three objects:\n\n1. An array of objects called \"featureProperties\". This is where the \"properties\"\n   data goes, as well as any data like bounding boxes or arbitrary data\n   attached to the Feature object.\n2. A Uint32Array of indexes, which informs the reader of the types of geometries\n   and lengths of coordinate rings.\n3. A Float64Array of coordinates.\n3. A Uint32Array of lookup offsets into the indexes \u0026 coordinates arrays.\n\nBasically it's an offset based system. You read the file from\nthe start, and let's say the first feature has a Point geometry.\nPoint has a geometry code of 0, which informs the reader to\nread the first 3 numbers from the coordinate array and move on.\n\nIf the next geometry is a LineString (code 2), then the reader\nreads the next number from the indexes array, which contains\nthe number of coordinates in the linestring. Given that information,\nit reads that number of coordinates and produces a LineString geometry.\n\n## Discussion\n\nThis schema has tradeoffs.\n\n- ~~Seeking is difficult, there is currently no affordance for\n  randomly jumping to and extracting a geometry. It's not entirely\n  clear that this is necessary - skipping would be simple to implement.\n  However, something like an index-of-indexes could be constructed.~~\n  Seeking to a specific feature is implemented!\n- ~~It's not clear yet how to encode the z index, the 3rd item in a\n  GeoJSON Position. Right now this defaults that 3rd item to 0, but\n  that is not ideal: a coordinate with z=0 is not the same as a coordinate\n  with no z value. The latter implies that the z value is unknown, not 0.~~\n  z indexes are encoded as NaN.\n- Could it be done with just one array, instead of separate\n  arrays for indexes and coordinates?\n- Should coordinates be Float32? I suspect that, while this would make\n  encoding lossy (JavaScript floats are 52-bit), it would easily satisfy\n  geospatial accuracy needs while halving the space requirement.\n- Some data updates in this format would be very expensive, and also\n  updates would require some fairly custom operations. For example,\n  adding a new coordinate in the middle of a line, in the a feature\n  in the middle of the dataset would require, probably,\n  intelligently updating the LineString length, slicing the dataset\n  into two TypedArrays, and plopping the new coordinate\n  in the middle. It's doable, but makes updates much less\n  obvious.\n- Is it useful, or beneficial, to get fancy with properties? GeoJSON\n  files certainly tend to share property names and often values, so\n  it's conceivable that a bunch of features with a value for a property\n  like \"x\" could have their values of \"x\" encoded as a flat array,\n  hence saving valuable object space. But doing this well and not\n  accidentally _increasing_ the memory requirements of some datasets\n  seems like it would require compression-like logic.\n\n## Running it\n\nThis repo is using Node's built-in test framework (as of Node v18).\nSo, have Node v18 and run `node --test`. No deps are required so far.\n\n## Future\n\nThe future of this would be to use it in [Placemark](https://www.placemark.io/),\nwhich would benefit from a more efficient memory encoding of features.\n\n## Prior art\n\n- [mapshaper](https://github.com/mbloch/mapshaper)\n- [visgl geojsonToBinary](https://github.com/visgl/loaders.gl/blob/master/modules/gis/docs/api-reference/geojson-to-binary.md)\n- [geo-arrow-spec](https://github.com/geopandas/geo-arrow-spec/blob/main/format.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftmcw%2Fmemory-geojson","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftmcw%2Fmemory-geojson","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftmcw%2Fmemory-geojson/lists"}