{"id":13585514,"url":"https://github.com/Jimut123/jimutmap","last_synced_at":"2025-04-07T10:31:03.342Z","repository":{"id":39636422,"uuid":"169246557","full_name":"Jimut123/jimutmap","owner":"Jimut123","description":"API to get enormous amount of high resolution satellite images from satellites.pro quickly through multi-threading! create map your own map dataset. Bringing data to Humans.","archived":true,"fork":false,"pushed_at":"2024-07-08T17:12:57.000Z","size":4077,"stargazers_count":152,"open_issues_count":0,"forks_count":17,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-03-08T12:17:06.989Z","etag":null,"topics":["api","areal-image","beautifulsoup4","dataset","deep-learning-dataset","enormous","fake-header","geo","high","image","images","jimutmap","ml","multithreading","resolution","satellite","satellite-data","scrape","scraping","segmentation-mask"],"latest_commit_sha":null,"homepage":"https://jimutmap.readthedocs.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Jimut123.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"COPYING","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-05T13:35:49.000Z","updated_at":"2025-03-06T13:24:05.000Z","dependencies_parsed_at":"2023-02-17T18:16:11.277Z","dependency_job_id":"7cba38d9-9016-4918-89dc-b11426844b7c","html_url":"https://github.com/Jimut123/jimutmap","commit_stats":{"total_commits":256,"total_committers":9,"mean_commits":"28.444444444444443","dds":0.0703125,"last_synced_commit":"b80ec1f180016eeb9a35dcc349365ba7ba1ff191"},"previous_names":[],"tags_count":20,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jimut123%2Fjimutmap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jimut123%2Fjimutmap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jimut123%2Fjimutmap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jimut123%2Fjimutmap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Jimut123","download_url":"https://codeload.github.com/Jimut123/jimutmap/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247636072,"owners_count":20970856,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","areal-image","beautifulsoup4","dataset","deep-learning-dataset","enormous","fake-header","geo","high","image","images","jimutmap","ml","multithreading","resolution","satellite","satellite-data","scrape","scraping","segmentation-mask"],"created_at":"2024-08-01T15:04:59.297Z","updated_at":"2025-04-07T10:31:03.090Z","avatar_url":"https://github.com/Jimut123.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"--------------------------------------------------------------------\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/wiki/jimut123/jimutmap/satellite_data/logo.png\" width=\"40%\" height=\"40%\"\u003e\n  ...Bringing Data to Humans\n\u003c/p\u003e\n\n--------------------------------------------------------------------\n\n\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://pepy.tech/project/jimutmap\"\u003e\u003cimg src=\"https://pepy.tech/badge/jimutmap\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://pypi.org/project/jimutmap/\"\u003e\u003cimg src=\"https://d25lcipzij17d.cloudfront.net/badge.svg?id=py\u0026type=6\u0026v=1.4.2\"\u003e\u003c/a\u003e\n\u003ca href='https://jimutmap.readthedocs.io/?badge=latest'\u003e\n    \u003cimg src='https://readthedocs.org/projects/jimutmap/badge/?version=latest' alt='Documentation Status' /\u003e\n\u003c/a\u003e  \u003ca href=\"https://www.gnu.org/licenses/gpl-3.0\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-GPL%20v3-blue.svg\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://pypistats.org/packages/jimutmap\"\u003e\n    \u003cimg src=\"https://img.shields.io/pypi/dm/jimutmap.svg\" alt=\"Downloads per month\"/\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://zenodo.org/badge/latestdoi/169246557\"\u003e\n    \u003cimg src=\"https://zenodo.org/badge/169246557.svg\" alt=\"zenodo\"/\u003e\n  \u003c/a\u003e\n\n\n\u003c/div\u003e\n\n***\n\n\u003e [!CAUTION]\n\u003e **I am actively looking for project maintainers who can volunteer to fix bugs/issues and work on [TODOs](https://github.com/Jimut123/jimutmap/blob/master/TODO.md), due to my limited time in maintaining this project. If you want to be a maintainer, either solve a bug (by making this software work for newer versions of apple maps) or successfully complete a TODO, then email me for the role (this process is for selecting valid maintainers).**\n\n## 📋 Contents \u003ca name=\"contents\"\u003e\u003c/a\u003e\n\n* [Purpose](#purpose)\n* [Need for scraping satellite data](#need-for-scraping-satellite-data)\n* [Installation and Usages](#installation-and-usages)\n* [Some of the example images downloaded at different scales](#some-of-the-example-images-downloaded-at-different-scales)\n* [Datasets](#datasets)\n* [YouTube video](#youtube-video)\n* [Sample of the images downloaded](#sample-of-the-images-downloaded)\n* [Perks](#perks)\n* [Additional Note](#additional-note)\n* [TODOs](#todos)\n* [Contribution](#contribution)\n* [LICENSE](#license)\n* [BibTeX and citations](#bibtex-and-citations)\n\n\n## 🔁 Purpose \u003ca name=\"purpose\"\u003e\u003c/a\u003e\n\nThis package collects data from [satellites.pro](https://satellites.pro/#32.916485,62.578125,4). It fetches all the tiles (image and road mask pair) as given by the parameters provided by the user. This uses an API-key generated at the time of browsing the map. **There are some future plans for this project, check [TODO](https://github.com/Jimut123/jimutmap/blob/master/TODO.md) to see what this will support in the future.**\n\nThe api `accessKey` token is automatically fetched if you have Google Chrome or Chromium installed using `chromedriver-autoinstaller`. Otherwise, you'll have to fetch it manually and set the `ac_key` parameter (which can be found out by selecting one tile from Apple Map, through chrome/firefox by going Developer-\u003eNetwork, looking at the assets, and finding the part of the link beginning with `\u0026accessKey=` until the next `\u0026`) every 10-15 mins. \n\n[[Back to Top](#contents)]\n\n\n## 💡 Need for scraping satellite data \u003ca name=\"need-for-scraping-satellite-data\"\u003e\u003c/a\u003e\n\nWell it's good (best in the world) satellite images, we just need to give the coordinates (Lat,Lon, and zoom) to get your dataset\nof high resolution satellite images! Create your own dataset and apply ML algorithms :')\n\n\nThe scraping API is present, call it and download it.\n\n[[Back to Top](#contents)]\n\n\n## 🛠 Installation and Usages \u003ca name=\"installation-and-usages\"\u003e\u003c/a\u003e\n\n```\nsudo pip3 install jimutmap\n\n# Install google chrome for chrome driver\nwget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb\nsudo apt install ./google-chrome-stable_current_amd64.deb\n\n# optional for viewing the temporary files generated by internal databases\nsudo apt-get install sqlite sqlitebrowser\n```\n\nNeeds to have google chrome web browser in the system.\n\nFor example usage, check [test.py](https://github.com/Jimut123/jimutmap/blob/master/test.py)\n\n```python3jimut@jimut:~/Desktop/GIT/jimutmap$ python3 test.py \nSorry, 5 -- threads unavailable, using maximum CPU threads : 4\nInitializing jimutmap ... Please wait...\nSorry, 50 -- threads unavailable, using maximum CPU threads : 4\nInitializing jimutmap ... Please wait...\n100%|██████████████████████████████████████████████| 20/20 [00:00\u003c00:00, 113.67it/s]\nSorry, 50 -- threads unavailable, using maximum CPU threads : 4\nInitializing jimutmap ... Please wait...\n100%|██████████████████████████████████████████████| 20/20 [00:00\u003c00:00, 722.10it/s]\nTotal satellite images to be downloaded =  210\nTotal roads tiles to be downloaded =  210\nApprox. estimated disk space required = 4.1015625 MB\nTotal number of satellite images needed to be downloaded =  210\nTotal number of satellite images needed to be downloaded =  210\nBatch =============================================================================  1\n===================================================================================\nSorry, 50 -- threads unavailable, using maximum CPU threads : 4\nDownloading all the satellite tiles: \nUpdating sanity db ...\n100%|████████████████████████████████████████████| 27/27 [00:00\u003c00:00, 13291.81it/s]\nTotal number of satellite images needed to be downloaded =  197\nTotal number of satellite images needed to be downloaded =  196\nDownloading speed == 0.09333877563476563 MiB/s \nWaiting for 15 seconds... Busy downloading\nDownloading speed == 0.11976458231608073 MiB/s \nWaiting for 15 seconds... Busy downloading\nDownloading speed == 0.01717344919840495 MiB/s \nWaiting for 15 seconds... Busy downloading\nBatch =============================================================================  2\n===================================================================================\nDownloading all the satellite tiles: \nUpdating sanity db ...\n100%|██████████████████████████████████████████| 420/420 [00:00\u003c00:00, 99921.03it/s]\nTotal number of satellite images needed to be downloaded =  0\nTotal number of satellite images needed to be downloaded =  0\n************************* Download Sucessful *************************\nCleaning up... hold on\nUpdating sticher db ...\n100%|██████████████████████████████████████████| 420/420 [00:00\u003c00:00, 24357.17it/s]\nTotal number of satellite images needed to be downloaded =  0\nTotal number of satellite images needed to be downloaded =  0\nCalculating bounding boxes for tiles :: \nTotal number of rows present in the database=  210\n100%|█████████████████████████████████████████| 210/210 [00:00\u003c00:00, 528693.78it/s]\nMin lat tile = 390842, Max lat tile = 390855, Min lon tile = 228264, Max lon tile = 228278\nNo. of tiles in latitude = 13, and longitude = 14\nCreating an image of size : 3328x3584 pixels ...\n100%|███████████████████████████████████████████████| 13/13 [00:00\u003c00:00, 28.89it/s]\n100%|███████████████████████████████████████████████| 13/13 [00:00\u003c00:00, 42.02it/s]\nTemporary sqlite files to be deleted = ['temp_sanity.sqlite', 'sticher.sqlite'] ? \n(y/N) : y\nTemporary chromedriver folders to be deleted = ['100'] ? \n(y/N) : y\n```\n\n[[Back to Top](#contents)]\n\n\n## 📚 Some of the example images downloaded at different scales \u003ca name=\"some-of-the-example-images-downloaded-at-different-scales\"\u003e\u003c/a\u003e\n\n| | | | |\n|:-------------------------:|:-------------------------:|:-------------------------:|:-------------------------:|\n| \u003cimg width=\"1604\" src=\"https://raw.githubusercontent.com/wiki/jimut123/jimutmap/satellite_data/1_urban_map_sat.jpeg\"\u003e | \u003cimg width=\"1604\" src=\"https://raw.githubusercontent.com/wiki/jimut123/jimutmap/satellite_data/1_urban_map_mask.png\"\u003e | \u003cimg width=\"1604\" src=\"https://raw.githubusercontent.com/wiki/jimut123/jimutmap/satellite_data/different_zoom_map.jpeg\"\u003e|\u003cimg width=\"1604\" src=\"https://raw.githubusercontent.com/wiki/jimut123/jimutmap/satellite_data/different_zoom_mask.png\"\u003e|\n|\u003cimg width=\"1604\" src=\"https://raw.githubusercontent.com/wiki/jimut123/jimutmap/satellite_data/higher_scale_map.jpeg\"\u003e  |  \u003cimg width=\"1604\" src=\"https://raw.githubusercontent.com/wiki/jimut123/jimutmap/satellite_data/higher_scale_mask.png\"\u003e|\u003cimg width=\"1604\" src=\"https://raw.githubusercontent.com/wiki/jimut123/jimutmap/satellite_data/map_us_1.jpeg\"\u003e|\u003cimg width=\"1604\" src=\"https://raw.githubusercontent.com/wiki/jimut123/jimutmap/satellite_data/mask_us_1.png\"\u003e|\n|\u003cimg width=\"1604\" src=\"https://raw.githubusercontent.com/wiki/jimut123/jimutmap/satellite_data/raj_map_1.jpeg\"\u003e  |  \u003cimg width=\"1604\" src=\"https://raw.githubusercontent.com/wiki/jimut123/jimutmap/satellite_data/raj_mask_1.png\"\u003e|\u003cimg width=\"1604\" src=\"https://raw.githubusercontent.com/wiki/jimut123/jimutmap/satellite_data/us_1_map.jpeg\"\u003e|\u003cimg width=\"1604\" src=\"https://raw.githubusercontent.com/wiki/jimut123/jimutmap/satellite_data/us_1_mask.png\"\u003e|\n\n[[Back to Top](#contents)]\n\n# 📘 Datasets \u003ca name=\"datasets\"\u003e\u003c/a\u003e\n\nJimutmap might behave weirdly in some cases. Please check the list of [datasets here](https://github.com/Jimut123/jimutmap/blob/master/DATASETS.md).\n\n## 📚 Stitched tiles for Kolkata \n\n| | | \n|:-------------------------:|:-------------------------:|\n| \u003cimg width=\"1604\" src=\"https://raw.githubusercontent.com/wiki/jimut123/jimutmap/stiched_kolkata/Kolkata_sat.png\"\u003e | \u003cimg width=\"1604\" src=\"https://raw.githubusercontent.com/wiki/jimut123/jimutmap/stiched_kolkata/Kolkata_road.png\"\u003e |\n\n[[Back to Top](#contents)]\n\n\n\n## 📹 YouTube video \u003ca name=\"youtube-video\"\u003e\u003c/a\u003e\n\nIf you are confused with the documentation, please see this video, to see the scraping in action [Apple Maps API to get enormous amount of satellite data for free using Python3](https://www.youtube.com/watch?v=voH0qhGXfsU).\n\n[[Back to Top](#contents)]\n\n\n## 📚 Sample of the images downloaded \u003ca name=\"sample-of-the-images-downloaded\"\u003e\u003c/a\u003e\n\n\u003ccenter\u003e\n\u003ca href=\"https://www.youtube.com/watch?v=wCbZhtWe72w\" alt=\"yt video\" target=\"_blank\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/wiki/jimut123/jimutmap/satellite_data/scrn.png\" alt=\"img of sat dat\" width=85% height=85%\u003e\u003c/a\u003e\n\u003c/center\u003e\n\n[[Back to Top](#contents)]\n\n\n#### :feelsgood: Perks \u003ca name=\"perks\"\u003e\u003c/a\u003e\n\nThis is done through parallel proccessing, so this will take maximum threads available in your CPU, change the \ncode to your own requirements! \n\nIf you want to re-fetch tiles, remember to delete/move tiles after every fetch request done! Else you won't get the updated information (tiles) of satellite data after that tile. It is calculated automatically so that all the progress remains saved!\n\n[[Back to Top](#contents)]\n\n\n## 📓 Additional Note \u003ca name=\"additional-note\"\u003e\u003c/a\u003e\n\nThis is created for educational and research purposes only! The [authors](https://github.com/Jimut123/jimutmap/blob/master/CONTRIBUTORS.md) are not liable for any damage to private property.\n\n[[Back to Top](#contents)]\n\n\n## :atom: TODOs \u003ca name=\"todos\"\u003e\u003c/a\u003e\n\nPlease check [TODOs](https://github.com/Jimut123/jimutmap/blob/master/TODO.md), since this project needs collaborators.\n\n[[Back to Top](#contents)]\n\n\n## ❓ Questions or want to discuss about something ? \u003ca name=\"\"\u003e\u003c/a\u003e\n\nSubmit an issue. \n\n[[Back to Top](#contents)]\n\n\n## 🤝 Contribution \u003ca name=\"contribution\"\u003e\u003c/a\u003e\n\nPlease see [Contributing.md](https://github.com/Jimut123/jimutmap/blob/master/CONTRIBUTING.md)\n\n[[Back to Top](#contents)]\n\n\n## 🛡️ [LICENSE](https://github.com/Jimut123/jimutmap/blob/master/LICENSE) \u003ca name=\"license\"\u003e\u003c/a\u003e\n```\n GNU GENERAL PUBLIC LICENSE\n                       Version 3, 29 June 2007\n\n Copyright (C) 2019-20 Jimut Bahan Pal, \u003chttps://jimut123.github.io/\u003e\n Everyone is permitted to copy and distribute verbatim copies\n of this license document, but changing it is not allowed.\n```\n\n[[Back to Top](#contents)]\n\n\n# 📝 BibTeX and citations \u003ca name=\"bibtex-and-citations\"\u003e\u003c/a\u003e\n\n```\n@misc{jimutmap_2019,\n  author = {Jimut Bahan Pal},\n  title = {jimutmap},\n  year = {2019},\n  publisher = {GitHub},\n  journal = {GitHub repository},\n  howpublished = {\\url{https://github.com/Jimut123/jimutmap}}\n}\n```\n\n[[Back to Top](#contents)]\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJimut123%2Fjimutmap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FJimut123%2Fjimutmap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJimut123%2Fjimutmap/lists"}