{"id":51503665,"url":"https://github.com/3kh0/slack-datamine","last_synced_at":"2026-07-07T22:03:00.867Z","repository":{"id":362298485,"uuid":"1255320517","full_name":"3kh0/slack-datamine","owner":"3kh0","description":"Datamining Slack changes from the webpack manifest","archived":false,"fork":false,"pushed_at":"2026-07-02T18:50:27.000Z","size":4276,"stargazers_count":9,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-07-02T19:12:06.841Z","etag":null,"topics":["slack"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/3kh0.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-31T17:25:48.000Z","updated_at":"2026-07-02T18:50:32.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/3kh0/slack-datamine","commit_stats":null,"previous_names":["3kh0/slack-datamine"],"tags_count":703,"template":false,"template_full_name":null,"purl":"pkg:github/3kh0/slack-datamine","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3kh0%2Fslack-datamine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3kh0%2Fslack-datamine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3kh0%2Fslack-datamine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3kh0%2Fslack-datamine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/3kh0","download_url":"https://codeload.github.com/3kh0/slack-datamine/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3kh0%2Fslack-datamine/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35243957,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-07T02:00:07.222Z","response_time":90,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["slack"],"created_at":"2026-07-07T22:03:00.345Z","updated_at":"2026-07-07T22:03:00.862Z","avatar_url":"https://github.com/3kh0.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# slack-datamine\n\nTracks Slack webpack bundle changes.\n\nThe miner reads Slack's authed client HTML, extracts the current build metadata and webpack chunk list, downloads the public CDN chunks, and writes the results into `build/`. Each mined build becomes a commit and tag for your convenience.\n\n## What It Writes\n\n`datamine.js` dumps to `build/` on each run and dumps the following:\n\n- API methods and method args\n- feature flags and experiments\n- error codes, slash commands, and OAuth scopes\n- analytics events\n- workflow functions and events\n- notification types\n- AI, MCP, and Block Kit enum values\n- trace spans, icon names, shortcuts, dev commands, and UI strings\n- `chunks.txt` and `meta.json`\n\n## The Setup\n\nThis is an archaic ass setup, but the intended strategy is to have a logged-in Slack desktop client exposing the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/) on port 9222.\n\n- `bin/watch-cdp.sh` continuously peeks at the client HTML into\n  `queue/html/`.\n- `bin/poll.sh` mines queued snapshots in capture order, commits `build/`, tags\n  the build, and optionally pushes.\n- `bin/snapshot-cdp.js` captures one Slack client snapshot through CDP.\n- `bin/reload-cdp.js` reloads or navigates the Slack renderer and reports the\n  ready build.\n\n## Local Commands\n\nCreate local config:\n\n```sh\ncp .env.example .env\n```\n\nMine from a saved Slack client HTML file:\n\n```sh\nnode datamine.js --html page.html\n```\n\nMine from a live Slack CDP instance:\n\n```sh\nnode datamine.js --port 9222\n```\n\nQueue one snapshot and mine it through the normal poller path:\n\n```sh\nnode bin/snapshot-cdp.js --port 9222 --dir queue/html --fetch-client\nbash bin/poll.sh\n```\n\nFor quick extractor checks:\n\n```sh\nnode datamine.js --html page.html --limit 40\nnode datamine.js --port 9222 --loaded-only\n```\n\n## systemd\n\nSystemd can help automate things, the checked-in units run Slack under Xvfb, keep CDP available, watch for new client snapshots, and mine the queue every 30 seconds. Adjust as needed, then install and start:\n\n```sh\ninstall -m 0644 systemd/slack-xvfb.service /etc/systemd/system/slack-xvfb.service\ninstall -m 0644 systemd/slack-cdp.service /etc/systemd/system/slack-cdp.service\ninstall -m 0644 systemd/slack-datamine-watch.service /etc/systemd/system/slack-datamine-watch.service\ninstall -m 0644 systemd/slack-datamine.service /etc/systemd/system/slack-datamine.service\ninstall -m 0644 systemd/slack-datamine.timer /etc/systemd/system/slack-datamine.timer\n\nsystemctl daemon-reload\nsystemctl enable --now slack-xvfb.service slack-cdp.service\nsystemctl enable --now slack-datamine-watch.service slack-datamine.timer\n```\n\nUseful commands:\n\n```sh\nsystemctl status slack-datamine-watch.service --no-pager\nsystemctl status slack-datamine.timer --no-pager\njournalctl -u slack-datamine-watch.service -u slack-datamine.service -f\n```\n\nSet `GIT_PUSH=1` in da `.env` if you want to push commits and tags.\n\n## Notes\n\n- Slack can skip public build numbers sometimes (because fuck logic). The miner logs the numeric gaps, but it cannot backfill a build after Slack stops serving that exact client HTML.\n- If Slack serves a lower build number again, the poller commits it as an observed rollback.\n- Some manifest chunks can 404 on every run. They are recorded in `meta.json` and skipped after retries.\n- Don't use any of the datamines as proof a feature is coming.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F3kh0%2Fslack-datamine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F3kh0%2Fslack-datamine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F3kh0%2Fslack-datamine/lists"}