{"id":16599765,"url":"https://github.com/pomadchin/vlm-performance","last_synced_at":"2025-07-05T13:34:28.483Z","repository":{"id":75108717,"uuid":"184303084","full_name":"pomadchin/vlm-performance","owner":"pomadchin","description":"GeoTrellis RasterSources Ingest benchmark","archived":false,"fork":false,"pushed_at":"2021-10-12T16:26:53.000Z","size":3746,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-17T06:45:40.398Z","etag":null,"topics":["aws","emr","geotrellis","gis","raster","spark"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pomadchin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-04-30T17:20:56.000Z","updated_at":"2022-05-04T09:48:38.000Z","dependencies_parsed_at":"2023-06-05T09:15:31.171Z","dependency_job_id":null,"html_url":"https://github.com/pomadchin/vlm-performance","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pomadchin%2Fvlm-performance","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pomadchin%2Fvlm-performance/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pomadchin%2Fvlm-performance/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pomadchin%2Fvlm-performance/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pomadchin","download_url":"https://codeload.github.com/pomadchin/vlm-performance/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242538984,"owners_count":20145882,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","emr","geotrellis","gis","raster","spark"],"created_at":"2024-10-12T00:12:32.157Z","updated_at":"2025-03-08T10:58:29.950Z","avatar_url":"https://github.com/pomadchin.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# VLM Performance project\n\nThis project is created to track down RasterSources API regressions.\n\nNOTE: at this point, this project depends on [GeoTrellis Contrib 3.14.0-SNAPSHOT](https://github.com/geotrellis/geotrellis-contrib/tree/cc6b022d5f4ac1266b23962181d00cc9cce79e40),\nit requires GeoTrellis Contrib local publish.\n\n## Notes\n\nWorking on a cluster take into account the fact that GDAL requires a different strategy with the resources allocation.\nIt is not possible to use `maximizeResourceAllocation` flag with using JNI bindings. \n\nAs the result of this work, was also figured out that `maximizeResourceAllocation` in general is not the best solution \nfor GeoTrellis ingests.\n\n### GDAL Tips\n\nPay attention to GDAL proper configuration:\n\n```conf\ngdal.options {\n  GDAL_DISABLE_READDIR_ON_OPEN     = \"TRUE\" # we don't usually want to read the entire dir with tiff metadata\n  CPL_VSIL_CURL_ALLOWED_EXTENSIONS = \".tif\" # filter files read by extension to speed up reads\n  GDAL_MAX_DATASET_POOL_SIZE       = \"256\" # number of allocated GDAL datasets\n  GDAL_CACHEMAX                    = \"1000\" # number in megabyes to limit GDAL apetite\n  # CPL_DEBUG                        = \"ON\" # to eanble GDAL logging on all nodes\n}\n```\n\nFor `50` `i3.xlarge` nodes it turned out that `GDAL_CACHEMAX = 1000` and `200` single core executors \nlooks like a good option. For `25` `i3.xlarge` nodes `GDAL_CACHEMAX = 500` and `70` single core executors, etc.\n\n### Ingest Results\n\nThe test dataset: [s3://azavea-datahub/raw/ned-13arcsec-geotiff](s3://azavea-datahub/raw/ned-13arcsec-geotiff)\nThe test dataset size: `1115 Objects - 210.7 GB`\n\n#### 20 i3.xlarge nodes\n\nLegacy GeoTrellis Ingest: `1 core per executor`, `1500M` RAM per executor\n![Ingest](img/20/ingest-i3-200exec.png)\n\nGeoTiff RasterSources Ingest: `1 core per executor`, `1500M` RAM per executor\n![GeoTiffRaterSource](img/20/ingest-rs-gdal-i3-70.png)\n\nGDAL RasterSources Ingest: `70 executors`, `1 core per executor`, `1500M` RAM per executor, `GDAL_CACHEMAX = 500`\n![GDALRasterSource](img/20/ingest-rs-gdal-i3-70.png)\n\n#### 50 i3.xlarge nodes\n\nLegacy GeoTrellis Ingest: `1 core per executor`, `1500M` RAM per executor\n![Ingest](img/50/ingest-i3-50-dynalloc.png)\n\nGeoTiff RasterSources Ingest: `1 core per executor`, `1500M` RAM per executor\n![GeoTiffRasterSource](img/50/geotiff-i3-50-dynalloc.png)\n\nGDAL RasterSources Ingest: `200 executors`, `1 core per executor`, `1500M` RAM per executor, `GDAL_CACHEMAX = 1000`\n![GDALRasterSource](img/50/gdal-i3-50-1000size-200.png)\n\n#### 50 i3.xlarge nodes (max resources allocation)\n\nLegacy GeoTrellis Ingest: `max resources allocation`, `200 executors`, `1 core per executor`, `4200M` RAM per executor.\nWith less RAM job is failing, maxmizing resources usage kills job as well.\n![Ingest](img/50/maxResourcesAllocation/ingest-i3-50-4200M.png)\n\nGeoTiff RasterSources Ingest: `max resources allocation`, `200 executors`, `1 core per executor`, `4200M` RAM per executor\nWith less RAM job is failing, maxmizing resources usage kills job as well.\n![GeoTiffRasterSource](img/50/maxResourcesAllocation/geotiff-i3-50-4200M.png)\n\nGDAL RasterSources Ingest: `max resources allocation`, `200 executors`, `1 core per executor`, `1500M` RAM per executor, `GDAL_CACHEMAX = 1000`\n![GDALRasterSource](img/50/maxResourcesAllocation/gdal-i3-50-1000size.png)\n\n\n### Conclusion\n\n~~(OLD Version, is deprecated; it was written because of cluster misconfiguration (see the next section)) The new API completely replaces the old one. The two ingests are a bit different. GDAL Ingest requires a bit\nmore complicated settings tuning, however, the new API is not slower and sometimes even faster.~~\n\nGDALRasterSources are much more complicated in tuning and give no significant performance improvements,\nhowever, it is probably because of an old GDAL 2.3.x version that was used on EMR cluster \nthat doesn't take into account CGroups. GDAL tests would be relaunched once we'll have GDAL 2.4 RPMs. \n\n### EMR maximizeResourceAllocation flag usage tip \n\nIn terms of this benchmark, we figured out that [maximizeResolurceAllocation](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-configure.html#emr-spark-maximizeresourceallocation) flag\ncan behave _not_ like everybody expects it to behave. The main danger here that it sets \n`spark.default.parallelism` to `2X number of CPU cores available to YARN containers`. It is a pretty\nsmall number usually and in fact forces spark to use `spark.default.parallelism` in all \n`reduce` operations and to reshuffle data into this particular number of partitions. \n\nBy default `Spark` tries to _preserve_ partitioning scheme. But with this option enabled it will force shuffle \nif the `partitioner` option was not _explicitly_ passed into all operations that potentially may cause shuffle.\n\n`./sbt ingest-ned` on the cluster without `maximizeResourceAllocation` flag usage (`20 i3.xlarge nodes`):\n![Ingest](img/20/ingest-i3-200exec.png)\n\n`./sbt ingest-ned` with `maximizeResourceAllocation` flag usage:\n![Ingest](img/20/ingest-i3-200exec-max-res-alloc.png)\n\nYou can notice that in the first picture we can see the _partitioning scheme preserving_.\nIn the second picture we see that _exactly the same application_ behaves differently after \nthe `CutTiles` step and the data is repartitioned into `160` partitions \n(in this case `spark.default.parallelism` was set to `160`):\n\n![Ingest](img/20/env-20-max-res-alloc-setting.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpomadchin%2Fvlm-performance","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpomadchin%2Fvlm-performance","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpomadchin%2Fvlm-performance/lists"}