{"id":47102701,"url":"https://github.com/mchaynes/geodatadownloader","last_synced_at":"2026-03-12T15:35:00.777Z","repository":{"id":38399544,"uuid":"433622313","full_name":"mchaynes/geodatadownloader","owner":"mchaynes","description":"A downloader of geodata","archived":false,"fork":false,"pushed_at":"2026-03-06T23:44:39.000Z","size":10367,"stargazers_count":99,"open_issues_count":2,"forks_count":18,"subscribers_count":2,"default_branch":"master","last_synced_at":"2026-03-07T04:38:36.788Z","etag":null,"topics":["arcgis","geojson","javascript"],"latest_commit_sha":null,"homepage":"https://geodatadownloader.com","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mchaynes.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-11-30T23:45:45.000Z","updated_at":"2026-03-06T23:44:42.000Z","dependencies_parsed_at":"2024-01-01T12:20:51.060Z","dependency_job_id":"f3c87835-2bed-4b9a-aff5-555325651397","html_url":"https://github.com/mchaynes/geodatadownloader","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mchaynes/geodatadownloader","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchaynes%2Fgeodatadownloader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchaynes%2Fgeodatadownloader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchaynes%2Fgeodatadownloader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchaynes%2Fgeodatadownloader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mchaynes","download_url":"https://codeload.github.com/mchaynes/geodatadownloader/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchaynes%2Fgeodatadownloader/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30430748,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-12T14:34:45.044Z","status":"ssl_error","status_checked_at":"2026-03-12T14:09:33.793Z","response_time":114,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arcgis","geojson","javascript"],"created_at":"2026-03-12T15:35:00.153Z","updated_at":"2026-03-12T15:35:00.760Z","avatar_url":"https://github.com/mchaynes.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# geodatadownloader.com (GDD)\n\nThis repo contains _all_ of the code for \u003chttps://geodatadownloader.com\u003e\n\n[![Playwright Tests](https://github.com/mchaynes/geodatadownloader/actions/workflows/playwright.yml/badge.svg)](https://github.com/mchaynes/geodatadownloader/actions/workflows/playwright.yml)\n\n\n## What is geodatadownloader?\n\nGDD is client-side browser application that will download all the data contained\nin an ArcGIS feature layer onto your computer.\nIt is not limited by max query size, and can download any size dataset\n(yes that includes those huge parcel layers from your local county).\nYou can choose a custom extent for your download, and pick the output columns you want to use.\n\nGDD runs entirely in your browser and stores nothing besides what your\nbrowser caches locally. There is no backend to the application,\nbesides the CDN used to serve up the html/javascript.\nIn order to draw an extent the map uses ESRI's javascript library\n(and therefore ESRI's servers to serve up the data for the map).\nConversion to from arcgis json to geojson is done browser side as well.\n\n## What formats does this support?\n\n- GeoJSON\n- CSV\n- SHP (ESRI Shapefile)\n- GPKG\n- KML\n- GPX\n- PGDUMP\n- DXF\n- SQLite\n\n## How does geodatadownloader download all of the data for a layer?\n\nIt executes a query on the arcgis feature service that says `where: 1=1`.\nOr, in other words, return everything. When it executes this query, it specifies the parameter `returnOnlyObjectIds`.\nThis returns all objectIds in a list.\nThen, GDD paginates those objectIds into chunks of 500.\nIt then constructs a `where: OBJECTID IN (...objectIds...)` which returns those 500 features.\nIt then proceeds to do this until every chunk has been requested and written into the downloaded dataset.\n\nYou may be asking yourself \"Why even ask for the objectIds and instead just grab all the features in that original `where: 1=1`?\".\nIf GDD could, it would. ArcGIS REST services are typically are limited by a specific number of features they can return (usually around 1000, but it depends).\nSome endpoints are \"paginated\", meaning that you can fetch features one page at a time. Not all services support this, though.\nSo instead, we use the objectIds method because it works a lot more consistently.\n\n## Developer Setup\n\n### Prerequisites\n\n- **Node.js**: Version specified in `.nvmrc` (v20.19.5)\n  - We recommend using [nvm](https://github.com/nvm-sh/nvm) to manage Node.js versions\n- **npm**: Comes with Node.js\n\n### Installation\n\n1. Clone the repository:\n   ```bash\n   git clone https://github.com/mchaynes/geodatadownloader.git\n   cd geodatadownloader\n   ```\n\n2. Install Node.js (if using nvm):\n   ```bash\n   nvm install\n   nvm use\n   ```\n\n3. Install dependencies:\n   ```bash\n   npm install\n   ```\n\n### Running the Development Server\n\nStart the development server on port 3000:\n\n```bash\nnpm start\n```\n\nThe application will be available at `http://localhost:3000`\n\n### Building for Production\n\nBuild the application for production:\n\n```bash\nnpm run build\n```\n\nThe built files will be in the `dist/` directory.\n\n### Testing\n\n#### Unit Tests\n\nRun Jest unit tests:\n\n```bash\nnpm test\n```\n\n#### End-to-End Tests\n\nRun Playwright E2E tests:\n\n```bash\nnpm run test:e2e\n```\n\nFor interactive E2E testing:\n\n```bash\nnpm run test:e2e:ui\n```\n\nFor headed browser mode:\n\n```bash\nnpm run test:e2e:headed\n```\n\n### Linting\n\nRun ESLint to check code quality:\n\n```bash\nnpm run lint\n```\n\n### Project Structure\n\n- `app/` - Main application code\n- `src/` - Legacy test setup files\n- `public/` - Static assets\n- `tests/` - Test files\n\n### Technology Stack\n\n- **Framework**: React 18 with TypeScript\n- **Build Tool**: Vite\n- **Styling**: Tailwind CSS, Material-UI, Flowbite\n- **Maps**: ArcGIS API for JavaScript\n- **Data Processing**: GDAL WebAssembly (gdal3.js)\n- **Testing**: Jest (unit), Playwright (E2E)\n\n### Contributing\n\nPlease ensure your code passes linting and all tests before submitting a pull request.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmchaynes%2Fgeodatadownloader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmchaynes%2Fgeodatadownloader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmchaynes%2Fgeodatadownloader/lists"}