Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/Jimut123/jimutmap

API to get enormous amount of high resolution satellite images from satellites.pro quickly through multi-threading! create map your own map dataset. Bringing data to Humans.
https://github.com/Jimut123/jimutmap

api areal-image beautifulsoup4 dataset deep-learning-dataset enormous fake-header geo high image images jimutmap ml multithreading resolution satellite satellite-data scrape scraping segmentation-mask

Last synced: 3 months ago
JSON representation

API to get enormous amount of high resolution satellite images from satellites.pro quickly through multi-threading! create map your own map dataset. Bringing data to Humans.

Awesome Lists containing this project

README 

        
--------------------------------------------------------------------





  

  ...Bringing Data to Humans




--------------------------------------------------------------------





  

  



    Documentation Status

  

  

    Downloads per month

  

  

    zenodo

  






***



> [!CAUTION]

> **I am actively looking for project maintainers who can volunteer to fix bugs/issues and work on [TODOs](https://github.com/Jimut123/jimutmap/blob/master/TODO.md), due to my limited time in maintaining this project. If you want to be a maintainer, either solve a bug (by making this software work for newer versions of apple maps) or successfully complete a TODO, then email me for the role (this process is for selecting valid maintainers).**



## 📋 Contents 



* [Purpose](#purpose)

* [Need for scraping satellite data](#need-for-scraping-satellite-data)

* [Installation and Usages](#installation-and-usages)

* [Some of the example images downloaded at different scales](#some-of-the-example-images-downloaded-at-different-scales)

* [Datasets](#datasets)

* [YouTube video](#youtube-video)

* [Sample of the images downloaded](#sample-of-the-images-downloaded)

* [Perks](#perks)

* [Additional Note](#additional-note)

* [TODOs](#todos)

* [Contribution](#contribution)

* [LICENSE](#license)

* [BibTeX and citations](#bibtex-and-citations)



## 🔁 Purpose 



This package collects data from [satellites.pro](https://satellites.pro/#32.916485,62.578125,4). It fetches all the tiles (image and road mask pair) as given by the parameters provided by the user. This uses an API-key generated at the time of browsing the map. **There are some future plans for this project, check [TODO](https://github.com/Jimut123/jimutmap/blob/master/TODO.md) to see what this will support in the future.**



The api `accessKey` token is automatically fetched if you have Google Chrome or Chromium installed using `chromedriver-autoinstaller`. Otherwise, you'll have to fetch it manually and set the `ac_key` parameter (which can be found out by selecting one tile from Apple Map, through chrome/firefox by going Developer->Network, looking at the assets, and finding the part of the link beginning with `&accessKey=` until the next `&`) every 10-15 mins. 



[[Back to Top](#contents)]



## 💡 Need for scraping satellite data 



Well it's good (best in the world) satellite images, we just need to give the coordinates (Lat,Lon, and zoom) to get your dataset

of high resolution satellite images! Create your own dataset and apply ML algorithms :')



The scraping API is present, call it and download it.



[[Back to Top](#contents)]



## 🛠 Installation and Usages 



```

sudo pip3 install jimutmap



# Install google chrome for chrome driver

wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb

sudo apt install ./google-chrome-stable_current_amd64.deb



# optional for viewing the temporary files generated by internal databases

sudo apt-get install sqlite sqlitebrowser

```



Needs to have google chrome web browser in the system.



For example usage, check [test.py](https://github.com/Jimut123/jimutmap/blob/master/test.py)



```python3jimut@jimut:~/Desktop/GIT/jimutmap$ python3 test.py 

Sorry, 5 -- threads unavailable, using maximum CPU threads : 4

Initializing jimutmap ... Please wait...

Sorry, 50 -- threads unavailable, using maximum CPU threads : 4

Initializing jimutmap ... Please wait...

100%|██████████████████████████████████████████████| 20/20 [00:00<00:00, 113.67it/s]

Sorry, 50 -- threads unavailable, using maximum CPU threads : 4

Initializing jimutmap ... Please wait...

100%|██████████████████████████████████████████████| 20/20 [00:00<00:00, 722.10it/s]

Total satellite images to be downloaded =  210

Total roads tiles to be downloaded =  210

Approx. estimated disk space required = 4.1015625 MB

Total number of satellite images needed to be downloaded =  210

Total number of satellite images needed to be downloaded =  210

Batch =============================================================================  1

===================================================================================

Sorry, 50 -- threads unavailable, using maximum CPU threads : 4

Downloading all the satellite tiles: 

Updating sanity db ...

100%|████████████████████████████████████████████| 27/27 [00:00<00:00, 13291.81it/s]

Total number of satellite images needed to be downloaded =  197

Total number of satellite images needed to be downloaded =  196

Downloading speed == 0.09333877563476563 MiB/s 

Waiting for 15 seconds... Busy downloading

Downloading speed == 0.11976458231608073 MiB/s 

Waiting for 15 seconds... Busy downloading

Downloading speed == 0.01717344919840495 MiB/s 

Waiting for 15 seconds... Busy downloading

Batch =============================================================================  2

===================================================================================

Downloading all the satellite tiles: 

Updating sanity db ...

100%|██████████████████████████████████████████| 420/420 [00:00<00:00, 99921.03it/s]

Total number of satellite images needed to be downloaded =  0

Total number of satellite images needed to be downloaded =  0

************************* Download Sucessful *************************

Cleaning up... hold on

Updating sticher db ...

100%|██████████████████████████████████████████| 420/420 [00:00<00:00, 24357.17it/s]

Total number of satellite images needed to be downloaded =  0

Total number of satellite images needed to be downloaded =  0

Calculating bounding boxes for tiles :: 

Total number of rows present in the database=  210

100%|█████████████████████████████████████████| 210/210 [00:00<00:00, 528693.78it/s]

Min lat tile = 390842, Max lat tile = 390855, Min lon tile = 228264, Max lon tile = 228278

No. of tiles in latitude = 13, and longitude = 14

Creating an image of size : 3328x3584 pixels ...

100%|███████████████████████████████████████████████| 13/13 [00:00<00:00, 28.89it/s]

100%|███████████████████████████████████████████████| 13/13 [00:00<00:00, 42.02it/s]

Temporary sqlite files to be deleted = ['temp_sanity.sqlite', 'sticher.sqlite'] ? 

(y/N) : y

Temporary chromedriver folders to be deleted = ['100'] ? 

(y/N) : y

```



[[Back to Top](#contents)]



## 📚 Some of the example images downloaded at different scales 



| | | | |

|:-------------------------:|:-------------------------:|:-------------------------:|:-------------------------:|

|  |  | ||

|  |  |||

|  |  |||



[[Back to Top](#contents)]



# 📘 Datasets 



Jimutmap might behave weirdly in some cases. Please check the list of [datasets here](https://github.com/Jimut123/jimutmap/blob/master/DATASETS.md).



## 📚 Stitched tiles for Kolkata 



| | | 

|:-------------------------:|:-------------------------:|

|  |  |



[[Back to Top](#contents)]



## 📹 YouTube video 



If you are confused with the documentation, please see this video, to see the scraping in action [Apple Maps API to get enormous amount of satellite data for free using Python3](https://www.youtube.com/watch?v=voH0qhGXfsU).



[[Back to Top](#contents)]



## 📚 Sample of the images downloaded 



img of sat dat



[[Back to Top](#contents)]



#### :feelsgood: Perks 



This is done through parallel proccessing, so this will take maximum threads available in your CPU, change the 

code to your own requirements! 



If you want to re-fetch tiles, remember to delete/move tiles after every fetch request done! Else you won't get the updated information (tiles) of satellite data after that tile. It is calculated automatically so that all the progress remains saved!



[[Back to Top](#contents)]



## 📓 Additional Note 



This is created for educational and research purposes only! The [authors](https://github.com/Jimut123/jimutmap/blob/master/CONTRIBUTORS.md) are not liable for any damage to private property.



[[Back to Top](#contents)]



## :atom: TODOs 



Please check [TODOs](https://github.com/Jimut123/jimutmap/blob/master/TODO.md), since this project needs collaborators.



[[Back to Top](#contents)]



## ❓ Questions or want to discuss about something ? 



Submit an issue. 



[[Back to Top](#contents)]



## 🤝 Contribution 



Please see [Contributing.md](https://github.com/Jimut123/jimutmap/blob/master/CONTRIBUTING.md)



[[Back to Top](#contents)]



## 🛡️ [LICENSE](https://github.com/Jimut123/jimutmap/blob/master/LICENSE) 

```

 GNU GENERAL PUBLIC LICENSE

                       Version 3, 29 June 2007



 Copyright (C) 2019-20 Jimut Bahan Pal, 

 Everyone is permitted to copy and distribute verbatim copies

 of this license document, but changing it is not allowed.

```



[[Back to Top](#contents)]



# 📝 BibTeX and citations 



```

@misc{jimutmap_2019,

  author = {Jimut Bahan Pal},

  title = {jimutmap},

  year = {2019},

  publisher = {GitHub},

  journal = {GitHub repository},

  howpublished = {\url{https://github.com/Jimut123/jimutmap}}

}

```



[[Back to Top](#contents)]