{"id":13557338,"url":"https://github.com/elvisyjlin/media-scraper","last_synced_at":"2025-04-03T11:31:48.067Z","repository":{"id":51331533,"uuid":"123089731","full_name":"elvisyjlin/media-scraper","owner":"elvisyjlin","description":"Scrapes all photos and videos in a web page / Instagram / Twitter / Tumblr / Reddit / pixiv / TikTok","archived":false,"fork":false,"pushed_at":"2020-08-02T08:58:17.000Z","size":88,"stargazers_count":383,"open_issues_count":20,"forks_count":49,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-11-04T07:35:56.449Z","etag":null,"topics":["crawler","instagram","pixiv","reddit","scraper","tiktok","tumblr","twitter"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/elvisyjlin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-02-27T07:22:12.000Z","updated_at":"2024-11-02T08:12:34.000Z","dependencies_parsed_at":"2022-08-22T07:00:33.662Z","dependency_job_id":null,"html_url":"https://github.com/elvisyjlin/media-scraper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elvisyjlin%2Fmedia-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elvisyjlin%2Fmedia-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elvisyjlin%2Fmedia-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elvisyjlin%2Fmedia-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/elvisyjlin","download_url":"https://codeload.github.com/elvisyjlin/media-scraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246993094,"owners_count":20865934,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","instagram","pixiv","reddit","scraper","tiktok","tumblr","twitter"],"created_at":"2024-08-01T12:04:17.393Z","updated_at":"2025-04-03T11:31:43.054Z","avatar_url":"https://github.com/elvisyjlin.png","language":"Python","funding_links":[],"categories":["Python","twitter"],"sub_categories":[],"readme":"# Media Scraper\n\n`media-scraper` scrapes all photos and videos in a web page. \nIt supports general-purpose scraping as well as SNS-specific scraping. \n\n`media-scraper` utilizes the web driver to simulate a user browsing web pages. \nWith the web driver, sessions and cookies easily can be handled easily but it works slightly slowly.\nOn the other hand, I'm currently working on the migration of another [repository](https://github.com/elvisyjlin/tumblrer), \nwhich crawls media only by HTTP requests, to this repository. See [here](https://github.com/elvisyjlin/media-scraper/tree/master/tumblrer).\n\n\n##### General-purpose Scraping\n\nThe general media scraper scrapes and downloads all photos and videos \nin all links `\u003ca/\u003e`, images `\u003cimg/\u003e` and videos `\u003cvideo/\u003e` of a web page. \n\n\n##### SNS-specific\n\nCurrently there are **Instagram** scraper and **Twitter** scraper, \nwhich crawl all posts of a given user and download media in a proper way for each SNS. \n\n\n## Updates\n\nCurrent, `media-scraper` is merged to contain two methods of scraping: _by request_ and _by browser_.\n\n\n### Usage\n\n```bash\npython3 m-scraper.py rq instagram [USERNAME1 USERNAME2 ...] [-e] [-s SAVE_PATH] [-c CRED_FILE]\npython3 m-scraper.py rq tumblr [SITE1 SITE2 ...] [-e] [-s SAVE_PATH] [-c CRED_FILE]\npython3 m-scraper.py rq reddit [SUBREDDIT1 SUBREDDIT2 ...] [-e] [-s SAVE_PATH] [-c CRED_FILE]\npython3 m-scraper.py rq pixiv [USERID1 USERID2 ...] [-e] [-s SAVE_PATH] [-c CRED_FILE]\npython3 m-scraper.py rq tiktok [USERID1 USERID2 ...] [-e] [-s SAVE_PATH] [-c CRED_FILE]\n```\n\nIf you'd like to download with your own credentials, i.e. logging in your account, please put your username and password in `credentials.json` and run `m-scraper.py` with `-c credentials.json`.\n\n```bash\nmv ./credentials.json.example ./credentials.json\nvim ./credentials.json\n```\n\nNote that pixiv requires a user login to view all illustrations and mangas. If you scrape pixiv without logging in, you'll get only some of them.\n\nFor scraping TikTok videos, you'll need to get the user id first. Go to the user page in your TikTok App, share it via link, paste it in a browser, and you'll see the user id in the url bar. E.g. user id of _TikTok_ is 107955. However, `m-scraper` here fetch video list via TikTok shared content, it contains most of the videos but not all of them. I'll dig into the mobile App API in the future.\n\n\n## Installation\n\nClone the `media-scraper` git repository.\n\n```bash\ngit clone https://github.com/elvisyjlin/media-scraper.git\ncd media-scraper\n```\n\nInstall Python 3 (at least 3.5) and get all dependencies.\n\n```bash\npip3 install -r requirements.txt\n```\n\n\n#### Web Driver\n\n`media-scraper` loads the content of a web page by web driver (PhantomJS). \nThe needed web driver will be downloaded automatically when it is used.\n\nIf you meet permission error, for example,  \n\n```python\nselenium.common.exceptions.WebDriverException: Message: 'phantomjs' executable may have wrong permissions.\n```\n\nPlease set the web driver to `777` for convenience.\n\n```bash\nchmod 777 webdriver/phantomjsdriver_2.1.1_win32/phantomjs.exe\nchmod 777 webdriver/phantomjsdriver_2.1.1_mac64/phantomjs\nchmod 777 webdriver/phantomjsdriver_2.1.1_linux64/phantomjs\n```\n\n\n## To Scrape\n\n```bash\npython3 -m mediascraper.general [WEB PAGE 1] [WEB PAGE 2] ...\n```\n\nThe media will be stored in the folder `download/general`.\n\n```bash\npython3 -m mediascraper.instagram [USER NAME 1] [USER NAME 2] ...\n```\n\nThe media will be stored in the folder `download/instagram`.\n\n```bash\npython3 -m mediascraper.twitter [USER NAME 1] [USER NAME 2] ...\n```\n\nThe media will be stored in the folder `download/twitter`.\n\n\nFor example, to scrape [`Twitter`](https://twitter.com/Twitter)\n\n```bash\npython3 -m mediascraper.twitter Twitter\n```\n\n\n### Login with Credentials\n\nIf you want to scrape a user's media with your account, \njust rename `credentials.json.example` to `credentials.json` and fill in your username and password. \n\n\n## To Import\n\nIt is easy to import `media-scraper` into your scripts and make use of it.\nPlease refer to the [example code](https://github.com/elvisyjlin/media-scraper/tree/master/mediascraper) for more details.\n\n\n### Media Scraper\n\n```\npython3 -m mediascraper.general [URL1 URL2 ...]\npython3 -m mediascraper.instagram [USERNAME1 USERNAME2 ...]\npython3 -m mediascraper.twitter [USERNAME1 USERNAME2 ...]\n```\n\n\n### Parameters of Scraper\n\nParameter | Description | Default Value\n--- | --- | ---\nscroll_pause | the pause interval when scrolling| `0.5` (seconds)\nmode | `'silent'`, `normal'` or `'verbose'` | `'normal'`\ndebug | prints debugging messages if `True` | `False`\n\n\n### Connect Methods of Scraper\n\nScraper | Methods\n--- | ---\nMediaScraper | `connect(URL)`\nInstagramScraper | `username(USERNAME)`\nTwitterScraper | `username(USERNAME)`\n\n\n## Note\n\n\n### Instagram\n\nInstagram changed API 3 times this year (2018), so the query API in `media-scraper` is out-of-date. \nPlease see [`instagramer.py`](https://github.com/elvisyjlin/media-scraper/blob/master/m_scraper/rq/instagramer.py), which works well for downloading all media form Instagram.\nThe instruction is [here](https://github.com/elvisyjlin/media-scraper#usage).\n\n\n### Twitter [Solved]\n\nFor some reasons, Twitter utilizes blob url for videos, which is not supported by `media-scraper` currently. \nI'm still working on this problem.\n\n06/07/2018 Update: It supports downloading videos in Twitter now!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felvisyjlin%2Fmedia-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felvisyjlin%2Fmedia-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felvisyjlin%2Fmedia-scraper/lists"}