{"id":22763156,"url":"https://github.com/rachhshruti/py-scrape-flickr","last_synced_at":"2025-03-30T09:41:26.294Z","repository":{"id":50208743,"uuid":"118210088","full_name":"rachhshruti/py-scrape-flickr","owner":"rachhshruti","description":"Image scraper for Flickr using Multiprocessing in Python","archived":false,"fork":false,"pushed_at":"2024-07-20T18:47:26.000Z","size":3976,"stargazers_count":1,"open_issues_count":4,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-05T11:44:31.895Z","etag":null,"topics":["flickrapi","googlemaps-api","multiprocessing","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rachhshruti.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-01-20T04:48:41.000Z","updated_at":"2024-11-29T04:06:05.000Z","dependencies_parsed_at":"2025-02-05T11:40:54.731Z","dependency_job_id":"b771d3b7-1a17-4f55-9f71-69563b2f8e13","html_url":"https://github.com/rachhshruti/py-scrape-flickr","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rachhshruti%2Fpy-scrape-flickr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rachhshruti%2Fpy-scrape-flickr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rachhshruti%2Fpy-scrape-flickr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rachhshruti%2Fpy-scrape-flickr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rachhshruti","download_url":"https://codeload.github.com/rachhshruti/py-scrape-flickr/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246301965,"owners_count":20755512,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["flickrapi","googlemaps-api","multiprocessing","python3"],"created_at":"2024-12-11T11:07:21.212Z","updated_at":"2025-03-30T09:41:26.267Z","avatar_url":"https://github.com/rachhshruti.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Image scraper for Flickr using Multiprocessing in Python\nPython library to scrape images in parallel from Flickr based on given list of locations like rome, paris and so on.\nIt extracts the filename and geo information about the images and inserts into SQLite database. In case of missing geo\ninformation, it uses Bing Maps API to extract this information based on the generic location (example, paris) that was\nsearched.\n\n# Requirements\n1. [Python3](https://www.python.org/downloads/release/python-364/)\n2. [Pip3: python3 get-pip.py](https://bootstrap.pypa.io/get-pip.py)\n3. API keys: Get the Flickr and Bing Maps API keys from below links and insert it into scrape-flickr/config.py\n    * [Flickr API keys](https://www.flickr.com/services/api/misc.api_keys.html)\n    * [Bing Maps API key](https://docs.microsoft.com/en-us/bingmaps/getting-started/bing-maps-dev-center-help/getting-a-bing-maps-key)\n\n\n# SQLite database\nThe following tables get created in this code:\n1. __image_metadata:__ used to store image information such as filename and geo information and consists of following fields:\n    * id: unique image id\n    * filename: title of the image\n    * latitude: latitude of the location in the image\n    * longitude: longitude of the location in the image\n2. __default_geo_info:__ used to store missing geo information of images using Bing Maps API\n    * search_text: location that was searched on Flickr\n    * latitude: latitude of the location\n    * longitude: longitude of the location\n\n# Running the code (Note: Please run all of these commands from project directory py-scrape-flickr)\nThis code is tested on Mac and Windows 10.\n1. Run the shell script which creates a virtual environment named scraper and installs the needed python packages\n\n        sh setup.sh\n\n2. Activate virtualenv, if not activated already\n\n        . scraper/bin/activate\n\n3. Run the code from the project directory py-scrape-flickr\n\n        python scrape-flickr/scrape_flickr.py paris rome \"new york\" [--photos_per_page] [-h]\n\n   It takes the following arguments:\n    * list of locations each separated by space and put double quotes around locations containing space\n    * optional --photos_per_page: number of photos to be retrieved at same time (max=500)\n    * optional -h: check usage\n\n   The database scraper.db gets created in the project folder (py-scrape-flickr) when running it for the first time.\n\n4. Check results\n\n        sqlite3 scraper.db\n        select * from image_metadata;\n\n5. Time in minutes for various input sizes on a 4 processors system\n    * 3 locations: 16 mins\n    * 6 locations: 60 mins\n    * 10 locations: 104 mins\n\n   This time will vary depending on what locations were searched and how many images they have and also on the number of\n   processors on the system and how strong is the internet connection.\n\n6. Run unit tests\n\n        python -m unittest discover scrape-flickr/\n\n# References\n[Multiprocessing](https://docs.python.org/3/library/multiprocessing.html)\n\n[Sub-processes in multiprocessing](https://stackoverflow.com/a/8963618)\n\n[Flickr Photos Search](https://www.flickr.com/services/api/flickr.photos.search.html)\n\n[Bing Maps Geocoding](https://geocoder.readthedocs.io/providers/Bing.html)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frachhshruti%2Fpy-scrape-flickr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frachhshruti%2Fpy-scrape-flickr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frachhshruti%2Fpy-scrape-flickr/lists"}