{"id":30619110,"url":"https://github.com/oxcone1/tet-offer-map","last_synced_at":"2026-04-13T06:02:12.117Z","repository":{"id":311767294,"uuid":"1039653386","full_name":"OxCone1/tet-offer-map","owner":"OxCone1","description":"Interactive React+Vite map visualizing Tet (Latvia) internet availability from scraped OSM addresses (NDJSON)","archived":false,"fork":false,"pushed_at":"2025-08-26T12:23:37.000Z","size":711,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-26T16:18:42.311Z","etag":null,"topics":["fusejs","geojson","latvia","leaflet","localforage","mapping","ndjson","overpass-turbo","puppeteer","react","shadcn","tailwindcss","tet","vite"],"latest_commit_sha":null,"homepage":"https://oxcone.com/tet-map/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OxCone1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-17T17:40:59.000Z","updated_at":"2025-08-26T12:23:40.000Z","dependencies_parsed_at":"2025-08-26T16:24:32.356Z","dependency_job_id":"b95721b1-4b0a-4c02-b2e2-bb6cc02ce0c5","html_url":"https://github.com/OxCone1/tet-offer-map","commit_stats":null,"previous_names":["oxcone1/tet-offer-map"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/OxCone1/tet-offer-map","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OxCone1%2Ftet-offer-map","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OxCone1%2Ftet-offer-map/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OxCone1%2Ftet-offer-map/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OxCone1%2Ftet-offer-map/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OxCone1","download_url":"https://codeload.github.com/OxCone1/tet-offer-map/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OxCone1%2Ftet-offer-map/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272847520,"owners_count":25003216,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-30T02:00:09.474Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fusejs","geojson","latvia","leaflet","localforage","mapping","ndjson","overpass-turbo","puppeteer","react","shadcn","tailwindcss","tet","vite"],"created_at":"2025-08-30T12:17:16.550Z","updated_at":"2025-10-18T14:34:08.040Z","avatar_url":"https://github.com/OxCone1.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tet Offer Map – Monorepo\n\nLive Demo: https://oxcone.com/tet-map/\n\nMulti‑stage pipeline \u0026 web UI to extract Latvian Tet internet availability data, normalize it against OpenStreetMap (OSM) address geometry, and visualize per‑area internet technologies on an interactive map. Supports user‑provided connectivity overlays and fine‑grained per‑neighborhood slicing (Riga included).\n\n## Motivation\nI built this to give everyday users a clear, map‑based view of the real connectivity landscape around them: what kinds of internet technologies their neighbours already have, and where gaps still exist. Having that visibility helps people argue for upgrades, choose a better provider/technology, and generally raise local awareness about broadband availability (or the lack of it) in their area. The goal is empowerment through transparency – turning scattered, opaque availability checks into an explorable shared layer.\n\n## Contents\n| Folder | Purpose |\n|--------|---------|\n| [`data-extractor/`](data-extractor/README.md) | Scrapers (API-first + Puppeteer fallback) enriching OSM addresses with Tet offers → NDJSON. |\n| [`frontend/`](frontend/README.md) | React + Vite application (Leaflet map, search, filters, sectors, uploads). |\n\n## High‑Level Flow (Deployed Usage)\nThe production site fetches all authoritative datasets (pointer index + per‑area NDJSON partitions) directly from the external data repository:\n\nhttps://github.com/OxCone1/data-tet-map\n\nNo local bundling of pointer / NDJSON files is required in the frontend build. End users can optionally add their own local connectivity information only via the in‑browser upload interface (user data layer). Local filesystem copies of exported NDJSON files are not used by the running app.\n\n## 1. Prerequisites\n- Node.js \u003e= 18 (frontend uses ES modules; scraper uses Puppeteer which benefits from modern Node).\n- Git, PowerShell (Windows) or any shell.\n\n## 2. Generate Input GeoJSON (Overpass Turbo)\nOpen https://overpass-turbo.eu/ and zoom to your target bounding box. Paste this query:\n\n```\n[out:json][timeout:60];\n(\n  way[\"highway\"][\"name\"]({{bbox}});\n  node[\"addr:housenumber\"]({{bbox}});\n  way[\"addr:housenumber\"]({{bbox}});\n  relation[\"addr:housenumber\"]({{bbox}});\n);\nout body;\n\u003e;\nout skel qt;\n```\n\nExport → Download → GeoJSON → save as `data-extractor/export.geojson`.\n\n## 3. Data Consumption in Frontend\nOn load the frontend downloads the remote pointer index, then selectively fetches per‑area slices on demand. There is no need to place dataset files in `frontend/public/`.\n\n## 4. Local Development Frontend\nFor development you also rely on the remote dataset. Only user‑generated uploads (GeoJSON / NDJSON) are stored locally (IndexedDB) after using the upload UI. No manual copying of pointer or NDJSON export files is necessary.\n\nBuild production bundle:\n```powershell\nnpm run build\nnpm run preview   # optional local preview\n```\n\n## 5. Deployment (Example: Heroku / Any Static Host)\nFrontend build produces `frontend/dist/`. Serve it with any static server (Express example):\n```js\napp.use(express.static(path.join(__dirname, 'dist')));\napp.get('*', (_,res)=\u003eres.sendFile(path.join(__dirname,'dist','index.html')));\n```\nThe build does not bundle dataset lines; network fetch is performed at runtime.\n\n## 6. Technology Stack (Selected)\n| Area | Tech | Notes |\n|------|------|-------|\n| Scraper | Node.js, Axios (API) + Puppeteer (fallback) | API first; browser only if needed. |\n| Frontend Core | React 19, Vite, TypeScript (partial) | Fast dev + modern JSX runtime. |\n| Styling | Tailwind CSS 4, shadcn/ui patterns (Radix primitives + class-variance-authority + tailwind-merge + lucide-react icons) | Utility-first styling with composable accessible components. |\n| Map | Leaflet, custom clustering (DBSCAN + convex hull) | No heavyweight server; client clustering. |\n| Search | Fuse.js fuzzy search | Boosted house number relevance. |\n| Storage | localforage (IndexedDB wrapper) | Persistent user uploaded datasets. |\n| Intl | Simple context + translation hook | Lightweight language switching. |\n| Server (deploy) | Express 5 (optional) | Static file + SPA fallback; not required for static hosts. |\n\n## 7. Data Contracts\nPer-area NDJSON line shape:\n```json\n{\n  \"id\": \"123456\",\n  \"address\": \"Street 12, Riga\",\n  \"geometry\": { \"type\": \"Point\", \"coordinates\": [24.123, 56.95] },\n  \"offers\": [ { \"connectionType\": \"Fiber\", \"speed\": { \"minMbit\": 300, \"maxMbit\": 1000 }, \"pricePerMonth\": \"19.99\" } ],\n  \"scrapedAt\": \"2025-08-17T12:34:56.000Z\",\n  \"source\": \"tet\"\n}\n```\nUser uploaded Feature requirement:\n```json\n{\n  \"type\": \"Feature\",\n  \"geometry\": { \"type\": \"Point\", \"coordinates\": [24.12, 56.95] },\n  \"properties\": { \"address\": \"User St 5\", \"connection_type\": \"fiber\" }\n}\n```\n\n## 8. Common Tasks (Maintainers Only)\n| Goal | Notes |\n|------|-------|\n| Scrape \u0026 publish new dataset | Run extractor (private), push updated pointer + partitions to data repo. |\n| Retry failed IDs | Use extractor `--validate` or recovery script, republish. |\n| Adjust batch size | Configure extractor run; frontend unaffected. |\n| Add new geographic coverage | Add additional import GeoJSON(s), scrape, publish. |\n\n## 9. Troubleshooting (Frontend)\n| Symptom | Fix |\n|---------|-----|\n| Dataset not loading | Check network access to data repo raw URLs (CORS / connectivity). |\n| Blank map | Ensure container height \u0026 tile network access. |\n| Modal empty | Underlying record has empty offers; verify upstream scrape. |\n| User upload not appearing | Confirm file is valid GeoJSON Feature(s) with required fields. |\n\n\n\n## 10. Future Enhancements\n- Pointer‑driven lazy loading in frontend (if not yet merged)\n- Streaming parser for very large NDJSON (progressive rendering)\n- Configurable data endpoint / CDN base (ENV)\n- Geometry simplification pipeline\n- Scheduled CI scrape \u0026 publish to dataset repo\n- Optional concave hull for outlines\n\n## 11. Data Source Repository\nAll datasets (pointer + per‑area NDJSON partitions) are fetched dynamically from:\n\nhttps://github.com/OxCone1/data-tet-map\n\nLocal substitution is intentionally not supported; use the web UI upload for custom local layers.\n\n## 12. License \u0026 Attribution\nOpenStreetMap data © OpenStreetMap contributors. Tet brand/data belong to Tet. This project is for informational / analytical purposes.\n\n---\nPull requests welcome. Keep per‑area files lean (drop unused props) for faster load and smaller pointer outlines.\n\n❤️ Created by OxCone\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxcone1%2Ftet-offer-map","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxcone1%2Ftet-offer-map","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxcone1%2Ftet-offer-map/lists"}