{"id":13910802,"url":"https://github.com/soxoj/socid-extractor","last_synced_at":"2026-10-19T16:41:42.808Z","repository":{"id":40572570,"uuid":"222293272","full_name":"soxoj/socid-extractor","owner":"soxoj","description":"⛏️ The extraction engine behind Maigret: turn any profile URL into a structured OSINT record  across 150+ sites","archived":false,"fork":false,"pushed_at":"2026-10-04T15:19:51.000Z","size":798,"stargazers_count":1115,"open_issues_count":13,"forks_count":124,"subscribers_count":26,"default_branch":"master","last_synced_at":"2026-10-12T00:41:18.610Z","etag":null,"topics":["account-extraction","data-extraction","html-parsing","osint","osint-framework","osint-python","osint-tool","scraping","socid","socmint"],"latest_commit_sha":null,"homepage":"https://socid-extractor.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/soxoj.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"claude":null,"gemini":null,"cursor":null,"copilot":null,"dco":null,"cla":null,"disclosure":null},"funding":{"patreon":"soxoj","github":"soxoj","buy_me_a_coffee":"soxoj"}},"created_at":"2019-11-17T18:30:06.000Z","updated_at":"2026-10-10T11:34:12.000Z","dependencies_parsed_at":"2026-10-04T14:44:47.987Z","dependency_job_id":"8dc647b9-8f80-4c94-8723-e3a5a5b11e8d","html_url":"https://github.com/soxoj/socid-extractor","commit_stats":{"total_commits":210,"total_committers":5,"mean_commits":42.0,"dds":0.2761904761904762,"last_synced_commit":"076178aaafb943e662f71241c6aa02228a320d64"},"previous_names":["soxoj/socid_extractor"],"tags_count":31,"template":false,"template_full_name":null,"purl":"pkg:github/soxoj/socid-extractor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soxoj%2Fsocid-extractor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soxoj%2Fsocid-extractor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soxoj%2Fsocid-extractor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soxoj%2Fsocid-extractor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/soxoj","download_url":"https://codeload.github.com/soxoj/socid-extractor/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soxoj%2Fsocid-extractor/sbom","scorecard":{"id":839680,"data":{"date":"2025-08-11","repo":{"name":"github.com/soxoj/socid-extractor","commit":"940b55983fe61522c7913177ba9d18e9982a12c7"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.9,"checks":[{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Code-Review","score":0,"reason":"Found 1/16 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/python-package.yml:1","Warn: no topLevel permission defined: .github/workflows/python-publish.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/python-package.yml:18: update your workflow using https://app.stepsecurity.io/secureworkflow/soxoj/socid-extractor/python-package.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/python-package.yml:20: update your workflow using https://app.stepsecurity.io/secureworkflow/soxoj/socid-extractor/python-package.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/python-publish.yml:17: update your workflow using https://app.stepsecurity.io/secureworkflow/soxoj/socid-extractor/python-publish.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/python-publish.yml:19: update your workflow using https://app.stepsecurity.io/secureworkflow/soxoj/socid-extractor/python-publish.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/python-publish.yml:30: update your workflow using https://app.stepsecurity.io/secureworkflow/soxoj/socid-extractor/python-publish.yml/master?enable=pin","Warn: pipCommand not pinned by hash: .github/workflows/python-package.yml:25","Warn: pipCommand not pinned by hash: .github/workflows/python-package.yml:26","Warn: pipCommand not pinned by hash: .github/workflows/python-package.yml:29","Warn: pipCommand not pinned by hash: .github/workflows/python-package.yml:36","Warn: pipCommand not pinned by hash: .github/workflows/python-publish.yml:24","Warn: pipCommand not pinned by hash: .github/workflows/python-publish.yml:25","Info:   0 out of   4 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   1 third-party GitHubAction dependencies pinned","Info:   0 out of   6 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: GNU General Public License v3.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Packaging","score":10,"reason":"packaging workflow detected","details":["Info: Project packages its releases by way of GitHub Actions.: .github/workflows/python-publish.yml:8"],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":7,"reason":"3 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-9hjg-9r4m-mvj7","Warn: Project is vulnerable to: GHSA-9wx4-h78v-vm56","Warn: Project is vulnerable to: PYSEC-2023-74 / GHSA-j8r2-6x86-q33q"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 27 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-23T20:09:17.858Z","repository_id":40572570,"created_at":"2025-08-23T20:09:17.859Z","updated_at":"2025-08-23T20:09:17.859Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":344872684,"owners_count":38362009,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-10-03T21:59:58.778Z","status":"online","status_checked_at":"2026-10-19T02:00:06.414Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["account-extraction","data-extraction","html-parsing","osint","osint-framework","osint-python","osint-tool","scraping","socid","socmint"],"created_at":"2024-08-07T00:01:46.221Z","updated_at":"2026-10-19T16:41:42.800Z","avatar_url":"https://github.com/soxoj.png","language":"Python","funding_links":["https://patreon.com/soxoj","https://github.com/sponsors/soxoj","https://buymeacoffee.com/soxoj","https://www.patreon.com/musemercier'","https://www.patreon.com/annetlovart'"],"categories":["🕵️ Username \u0026 People Search","Python"],"sub_categories":[],"readme":"# socid_extractor\n\n[![PyPI version](https://img.shields.io/pypi/v/socid-extractor.svg)](https://pypi.org/project/socid-extractor/)\n[![Downloads/month](https://static.pepy.tech/badge/socid-extractor/month)](https://pepy.tech/project/socid-extractor)\n[![Total downloads](https://static.pepy.tech/badge/socid-extractor)](https://pepy.tech/project/socid-extractor)\n[![License](https://img.shields.io/pypi/l/socid-extractor.svg)](https://github.com/soxoj/socid-extractor/blob/master/LICENSE)\n\n[![CI](https://img.shields.io/github/actions/workflow/status/soxoj/socid-extractor/python-package.yml?branch=master\u0026label=tests)](https://github.com/soxoj/socid-extractor/actions/workflows/python-package.yml)\n[![GitHub stars](https://img.shields.io/github/stars/soxoj/socid-extractor.svg?style=social)](https://github.com/soxoj/socid-extractor/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/soxoj/socid-extractor.svg?style=social)](https://github.com/soxoj/socid-extractor/network/members)\n\n\u003e Turn any public profile page into a structured account record — usernames, display names, bios, avatars, locations, joined-at dates, follower counts, external links, and the **stable internal identifiers** that uniquely pin an account across renames, redesigns, and deletions.\n\n`socid_extractor` parses HTML pages and API responses from 250+ schemes and returns a flat, machine-readable dictionary of account fields. No API keys required, no headless browser — just a single function call on response text.\n\n**Why it's useful**\n\n- **Stable cross-service IDs.** Get GAIA ID (Google), Facebook UID, Yandex Public ID, Instagram pk, and dozens more — values that survive username changes and let you correlate accounts across leaks, archives, and search-engine indices.\n- **One uniform interface.** Same `extract()` call for Instagram, GitHub, VK, Reddit, Substack, Bluesky, TikTok — no per-platform glue code on your side.\n- **Field ontology.** [Normalized field names](https://github.com/soxoj/socid-extractor/blob/master/FIELDS.md) across platforms (`username`, `fullname`, `created_at`, `is_verified`, …) so downstream pipelines don't need 130 mappings.\n- **Battle-tested.** Powers [Maigret](https://github.com/soxoj/maigret) and a number of other OSINT tools.\n\n## Installation\n\nPython: 3.10+.\n\n```sh\npip install socid-extractor\n```\n\nFor a clean CLI install on a workstation:\n\n```sh\npipx install socid-extractor\n```\n\nThe latest development version:\n\n```sh\npip install -U git+https://github.com/soxoj/socid-extractor.git\n```\n\n## Quick start\n\n**As a CLI:**\n\n```sh\n$ socid_extractor --url https://www.deviantart.com/muse1908\ncountry: France\ncreated_at: 2005-06-16 18:17:41\ngender: female\nusername: Muse1908\nwebsite: www.patreon.com/musemercier\nlinks: ['https://www.facebook.com/musemercier', 'https://www.instagram.com/muse.mercier/', 'https://www.patreon.com/musemercier']\ntagline: Nothing worth having is easy...\n```\n\n**As a Python library:**\n\n```python\nimport requests\nimport socid_extractor\n\nr = requests.get('https://www.patreon.com/annetlovart')\nprint(socid_extractor.extract(r.text))\n# {'patreon_id': '33913189', 'patreon_username': 'annetlovart',\n#  'fullname': 'Annet Lovart',\n#  'links': \"['https://www.facebook.com/322598031832479', ...]\"}\n```\n\n**Tip — batch runs:** pass `--skip-fetch-if-no-url-hint` to skip the HTTP request when the URL doesn't match any known site hint (faster, but may skip generic engines such as forum templates):\n\n```sh\n$ socid_extractor --url https://example.com/foo --skip-fetch-if-no-url-hint\n```\n\n## Supported sites\n\n[**250+ schemes**, covering roughly 2,000 of the sites Maigret checks — see METHODS.md for the full list.](https://github.com/soxoj/socid-extractor/blob/master/METHODS.md)\n\nA non-exhaustive sample:\n\n- **Major networks:** Facebook (user \u0026 group pages), Instagram, VK.com, OK.ru, Reddit, TikTok, Bluesky, Tumblr, Flickr\n- **Google ecosystem:** Google docs/maps contributions (cookies required), Google Play, YouTube\n- **Mail.ru:** my.mail.ru user mainpage, photo, video\n- **Dev / writing platforms:** GitHub, Stack Overflow (HTML + API), LeetCode, Hashnode, Medium, Substack, Paragraph, WordPress.org, Virgool\n- **Forums (universal detectors):** Discourse, MediaWiki / Fandom wikis, Mastodon\n- **Niche / vertical:** Chess.com, Roblox, MyAnimeList, Scratch, Wikipedia, DailyMotion, SlideShare, Weebly, Calendly, Amazon Author, Boosty, Warpcast (Farcaster), Fragment (TON/Telegram), Rarible, CSSBattle, lnk.bio, Spatial, TwitchTracker, Max (max.ru)\n\n…and many others.\n\nFor data examples, see [`tests/test_e2e.py`](https://github.com/soxoj/socid-extractor/blob/master/tests/test_e2e.py); for the parsing logic, see [`socid_extractor/schemes.py`](https://github.com/soxoj/socid-extractor/blob/master/socid_extractor/schemes.py); for the field ontology, see [FIELDS.md](https://github.com/soxoj/socid-extractor/blob/master/FIELDS.md).\n\n## Use cases\n\n- **Pivot from a profile to everything you can see.** One call returns the visible info plus the hidden internal IDs the platform uses behind the scenes. Background reading: [Week in OSINT — Getting a grasp on Google IDs](https://medium.com/week-in-osint/getting-a-grasp-on-googleids-77a8ab707e43).\n- **Track accounts across renames, redesigns, and deletions.** Stable IDs (GAIA, FB UID, Yandex Public ID, Instagram pk, …) let you re-identify the same person even when every visible field has changed. Background: [Aware Online — User IDs in social-media investigations](https://www.aware-online.com/en/importance-of-user-ids-in-social-media-investigations/).\n- **Search by cross-service UID.** Once you have a stable identifier you can pivot into:\n  - SQL / leaked databases (forum dumps, breach data) where the UID is the join key,\n  - Google / Yandex / archive.org indices that captured URLs containing the UID.\n- **Feed downstream OSINT tooling.** A normalized record is much easier to ingest than per-site scrapers — used by [Maigret](https://github.com/soxoj/maigret) and similar tools for enrichment.\n\n## Commercial Use\n\nThe open-source `socid_extractor` is MIT-licensed and free for commercial use without restriction — but page parsers break over time as platforms change their HTML and APIs, and they need active maintenance.\n\nFor serious commercial use — a hosted extraction API, or maintenance of the parsers your product depends on — reach out: 📧 socid@soxoj.com\n\n- Extraction API — integrate `socid_extractor` into your product\n\n## SOWEL classification\n\nMaps to the following [SOWEL](https://sowel.soxoj.com/) techniques:\n- [SOTL-1.4. Analyze Internal Identifiers](https://sowel.soxoj.com/internal-identifiers)\n- [SOTL-11.1. Check Outdated And Unused Functionality](https://sowel.soxoj.com/outdated-unused-functionality)\n\n## Tools using socid_extractor\n\n- [**Maigret**](https://github.com/soxoj/maigret) — powerful namechecker that generates a report with all available info from accounts found across 3000+ sites.\n- [**TheScrapper**](https://github.com/champmq/TheScrapper) — scrape emails, phone numbers, and social-media accounts from a website.\n- [**InfoHunter**](https://github.com/sweetnight19/InfoHunter) — open-source OSINT tool to search, collect, and analyze information online.\n- [**YaSeeker**](https://github.com/HowToFind-bot/YaSeeker) — gather all available information about a Yandex account by login/email.\n- [**Marple**](https://github.com/soxoj/marple) — scrape search-engine results for a given username.\n\n## See also\n\n- [**ShareTrace**](https://github.com/soxoj/sharetrace) — the mirror image of this project: `socid_extractor` reads a profile page, ShareTrace takes a *share link* (TikTok, Instagram, ChatGPT, Discord, Telegram and more) and recovers who created or forwarded it, decoding some share tokens offline. Nothing to install if you just want to try it: the community-run [Telegram bot](https://t.me/sharetrace_bot?start=c29jaWQtZXh0cmFjdG9yOnJlYWRtZQ) or [web UI](https://share.whopostedwhat.com/).\n\n## Testing\n\nInstall the test extras from [`pyproject.toml`](pyproject.toml), then run pytest:\n\n```sh\npip install '.[test]'   # pytest, pytest-rerunfailures, pytest-xdist\npython3 -m pytest tests/test_e2e.py -n 10 -k 'not cookies' -m 'not github_failed and not rate_limited'\n```\n\nUse `pip install '.[dev]'` instead if you also want flake8 / mypy / black (the full set used by CI).\n\n**Every new scheme must have an e2e test** in `tests/test_e2e.py` hitting a real URL/API. Unit tests with inline fixtures (`tests/test_socid_improvements.py`) are also required but do not replace e2e coverage. See [docs/testing-and-ci.md](docs/testing-and-ci.md) for details.\n\nDeveloper documentation (architecture, modules, CI) lives in [docs/](docs/).\n\n## Contributing\n\nSee the [contributing guide](https://github.com/soxoj/socid-extractor/blob/master/CONTRIBUTING.md) if you want to add a new scheme or fix anything.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoxoj%2Fsocid-extractor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsoxoj%2Fsocid-extractor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoxoj%2Fsocid-extractor/lists"}