{"id":21481943,"url":"https://github.com/methyldragon/fb_embedded_comment_scraper","last_synced_at":"2025-03-17T09:15:55.662Z","repository":{"id":98740051,"uuid":"120712006","full_name":"methylDragon/fb_embedded_comment_scraper","owner":"methylDragon","description":"A scraper for gathering data from Facebook's embedded comment widgets for all pages on any number of URLs! It bypasses the Facebook graph API (you don't need an access token) so there's little risk of throttling.","archived":false,"fork":false,"pushed_at":"2018-04-10T21:43:47.000Z","size":22848,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-23T18:50:13.470Z","etag":null,"topics":["dataset-generation","python3","scraping","script","website-scraper"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/methylDragon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-02-08T04:44:34.000Z","updated_at":"2024-12-28T07:28:30.000Z","dependencies_parsed_at":"2023-03-03T15:45:30.332Z","dependency_job_id":null,"html_url":"https://github.com/methylDragon/fb_embedded_comment_scraper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/methylDragon%2Ffb_embedded_comment_scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/methylDragon%2Ffb_embedded_comment_scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/methylDragon%2Ffb_embedded_comment_scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/methylDragon%2Ffb_embedded_comment_scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/methylDragon","download_url":"https://codeload.github.com/methylDragon/fb_embedded_comment_scraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244006395,"owners_count":20382443,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataset-generation","python3","scraping","script","website-scraper"],"created_at":"2024-11-23T12:29:05.097Z","updated_at":"2025-03-17T09:15:55.630Z","avatar_url":"https://github.com/methylDragon.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fb_embedded_comment_scraper\nNow updated to be able to pull every comment in every page of a domain (if you input it correctly!)\n\nA scraper for gathering data from Facebook's embedded comment widgets on any number of URLs! It bypasses the Facebook graph API (you don't need an access token) so there's little risk of throttling.\n\nThis script extracts JSON data that Facebook uses to power its embedded comment\nplatform, recursively picking out comments. (TO A MAXIMUM OF 100 TOP LEVEL COMMENTS PER LINK!)\n\nIt can display nested replies, all in order too, so the .csv reads naturally like the Facebook comment feed!\n\nIt writes to a specified .csv file the parsed data with columns:    \nReply Depth | Link | Description | Total Comments | Username | City | Date-Time | Comment | Likes | Replies\n\n### Usage\n\nJust change the relevant parameters!\n\n```python\n# =============================================================================\n# SET PARAMETERS\n# =============================================================================\n\nif __name__ == \"__main__\":\n    \n    \"\"\"\n    NOTE: Make sure there's no http:// or https:// in the source_URL list\n    \n    If you want to crawl something like\n    \"troublesome_site.com/page=\u003c??\u003e/other_stuff\"\n    \n    Write it as a tuple! (\"troublesome_site.com/page=\",\"/other_stuff)\n    \"\"\"\n    \n    # [[INSERT YOUR DOMAINS HERE!]]\n    source_URLs = [\"somesite.com/page/\", \"methylDragon.com\"]\n\n    # If you want to pull all pages from a site, ensure the site's page\n    # structure is numeric (Eg. somesite.com/page/\u003cNUMBER\u003e )\n\n    start_page = 0\n```\n\n\n\n### How it works\n\nI noticed that you can access the source of the iframes that Facebook dynamically generates via embedded JS in a webpage. Said source has JSON data hidden in a mess of text that can be parsed accordingly!\n\n\n\n### Applications\n\nI wrote this as part of a bigger project where we wanted to do natural language processing on a dataset of comments. This can help build that dataset, 100k+ comments strong!","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmethyldragon%2Ffb_embedded_comment_scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmethyldragon%2Ffb_embedded_comment_scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmethyldragon%2Ffb_embedded_comment_scraper/lists"}