{"id":15484803,"url":"https://github.com/kevincolemaninc/mm-crawler","last_synced_at":"2026-05-07T09:31:32.192Z","repository":{"id":148799077,"uuid":"115432427","full_name":"KevinColemanInc/mm-crawler","owner":"KevinColemanInc","description":"Scrapes meetme user profiles","archived":false,"fork":false,"pushed_at":"2017-12-31T17:30:13.000Z","size":5,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-01T23:42:27.518Z","etag":null,"topics":["crawler","docker","fake-data","meetme","ruby","scraper","sidekiq"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KevinColemanInc.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-12-26T15:13:05.000Z","updated_at":"2017-12-31T17:31:11.000Z","dependencies_parsed_at":"2023-05-29T00:30:40.681Z","dependency_job_id":null,"html_url":"https://github.com/KevinColemanInc/mm-crawler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KevinColemanInc%2Fmm-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KevinColemanInc%2Fmm-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KevinColemanInc%2Fmm-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KevinColemanInc%2Fmm-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KevinColemanInc","download_url":"https://codeload.github.com/KevinColemanInc/mm-crawler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240063926,"owners_count":19742227,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","docker","fake-data","meetme","ruby","scraper","sidekiq"],"created_at":"2024-10-02T05:51:03.034Z","updated_at":"2025-11-11T09:31:16.176Z","avatar_url":"https://github.com/KevinColemanInc.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MM Crawler\n\nThis project scrapes profiles from meetme.com using docker.  Follow the quick start and it will create an API for accessing all of the created profiles to be hosted on netlify.\n\nScrapes profiles between ages 19 and 35.\n\n## Quick start\n\n### Required ENV variables\n\n- MM_EMAIL=\n- MM_PASSWORD=\n\n### Steps\n\nNote the locations in `app.rb`.\n\n1. `docker-compose up --build`\n1. ssh into docker host\n1. run `ruby app.rb`\n1. wait for sidekiq jobs to finish (takes about 1-2 hours)\n1. run `ruby consolidate.rb` - creates `/results/profiles.json` with all crawled profile info and relative photo paths\n\n### Netlify deploy\n\nIn the `~/Desktop/results`, run `netlify deploy`.  Make note of the domain.\n\n## Workers\n\n1. `nearby_crawler` - fetches the profiles nearby and stores them. Queues fetching photo json\n1. `get_photo_jsons(member_id)` - returns json of photos\n1. `get_photo(url)` - persists photos\n\nOutput folder structure (in Docker):\n\n`/results/`\n\n- {member_id}.json\n- {member_id}_{photo_id}.jpg\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkevincolemaninc%2Fmm-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkevincolemaninc%2Fmm-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkevincolemaninc%2Fmm-crawler/lists"}