{"id":49675508,"url":"https://github.com/openclaw/crawlkit","last_synced_at":"2026-05-08T03:01:17.523Z","repository":{"id":355057253,"uuid":"1226596301","full_name":"openclaw/crawlkit","owner":"openclaw","description":null,"archived":false,"fork":false,"pushed_at":"2026-05-06T02:17:34.000Z","size":354,"stargazers_count":18,"open_issues_count":3,"forks_count":3,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-06T02:34:26.694Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openclaw.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null},"funding":{"github":["moltbot"]}},"created_at":"2026-05-01T15:47:28.000Z","updated_at":"2026-05-06T02:17:38.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/openclaw/crawlkit","commit_stats":null,"previous_names":["vincentkoc/crawlkit","openclaw/crawlkit"],"tags_count":21,"template":false,"template_full_name":null,"purl":"pkg:github/openclaw/crawlkit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclaw%2Fcrawlkit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclaw%2Fcrawlkit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclaw%2Fcrawlkit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclaw%2Fcrawlkit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openclaw","download_url":"https://codeload.github.com/openclaw/crawlkit/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclaw%2Fcrawlkit/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32719572,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-07T00:29:05.620Z","status":"online","status_checked_at":"2026-05-07T02:00:07.170Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-07T02:00:54.054Z","updated_at":"2026-05-08T03:01:17.444Z","avatar_url":"https://github.com/openclaw.png","language":"Go","funding_links":["https://github.com/sponsors/moltbot"],"categories":[],"sub_categories":[],"readme":"# crawlkit\n\nShared Go infrastructure for local-first crawler archives.\n\n`crawlkit` is not a universal Slack, Discord, Notion, or GitHub crawler. It is\nthe reusable foundation beneath those tools: SQLite hygiene, TOML config\ndefaults, portable JSONL/Gzip packing, git-backed snapshot sharing, sync state,\nCLI output helpers, control/status metadata, a shared terminal explorer, and\nsafe desktop-cache snapshot utilities.\n\n## Install\n\n```bash\ngo get github.com/vincentkoc/crawlkit@latest\n```\n\nGo packages are published by tagging this repository. There is no separate\npackage registry step. See `docs/publishing.md` for the release commands.\nSee `docs/boundary.md` for the crawlkit-versus-app ownership boundary.\n\n## Packages\n\n- `config`: standard TOML config paths, runtime dirs, and token diagnostics.\n- `store`: SQLite open/read-only/transaction/query helpers.\n- `snapshot`: `manifest.json` plus JSONL/Gzip table snapshot export and import.\n- `mirror`: clone/init/pull/commit/push helpers for private snapshot repos.\n- `state`: generic crawler cursor and freshness records.\n- `output`: text/json/log output helpers.\n- `control`: crawl app metadata, command manifests, status payloads, and\n  database inventory for launchers and automation.\n- `tui`: shared terminal archive explorer with gitcrawl-style responsive panes, entity/member/detail lanes, compact sortable headers, mouse selection, floating right-click actions, sorting/filtering, and local/remote source status.\n- `cache`: safe read-only local cache snapshot helpers.\n\n## Downstream apps\n\n- `gitcrawl` and `discrawl` consume `crawlkit` on `main`.\n- `slacrawl` and `notcrawl` consume `crawlkit` on their `feat/use-crawlkit`\n  integration branches until those app rewires are merged.\n- The apps keep provider schemas, auth, desktop/API parsing, privacy filters,\n  and user-facing CLI contracts. `crawlkit` owns only the reusable mechanics.\n\n## Safety\n\nLibrary tests use temporary directories. They do not touch app runtime stores\nsuch as `~/.config/gitcrawl`, `~/.slacrawl`, `~/.discrawl`, or `~/.notcrawl`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclaw%2Fcrawlkit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenclaw%2Fcrawlkit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclaw%2Fcrawlkit/lists"}