{"id":19361249,"url":"https://github.com/kazu/vfs-index","last_synced_at":"2025-04-23T12:32:35.827Z","repository":{"id":46278404,"uuid":"275287400","full_name":"kazu/vfs-index","owner":"kazu","description":"no process, indexer like a DB on VFS(virtual filesystem) ","archived":false,"fork":false,"pushed_at":"2023-02-25T09:30:57.000Z","size":3156,"stargazers_count":5,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-02T15:11:12.244Z","etag":null,"topics":["csv","flatbuffers","go","golang","json","jsonl","lz4","search-engine"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kazu.png","metadata":{"files":{"readme":"README.ja.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"kazu"}},"created_at":"2020-06-27T02:46:43.000Z","updated_at":"2024-12-11T21:51:41.000Z","dependencies_parsed_at":"2024-11-10T07:32:43.646Z","dependency_job_id":null,"html_url":"https://github.com/kazu/vfs-index","commit_stats":null,"previous_names":[],"tags_count":31,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kazu%2Fvfs-index","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kazu%2Fvfs-index/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kazu%2Fvfs-index/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kazu%2Fvfs-index/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kazu","download_url":"https://codeload.github.com/kazu/vfs-index/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250435140,"owners_count":21430226,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv","flatbuffers","go","golang","json","jsonl","lz4","search-engine"],"created_at":"2024-11-10T07:21:28.686Z","updated_at":"2025-04-23T12:32:34.004Z","avatar_url":"https://github.com/kazu.png","language":"Go","funding_links":["https://github.com/sponsors/kazu"],"categories":[],"sub_categories":[],"readme":"vfs-index\n===================\n\nvfs-index は簡単なVFS上の json/csv のようなデータの検索インデクサです。 db のようにindex を作成して検索をしますが、index には\nロックはなく、インデックスが途中で停止しても、次回実行時に継続されます。非常にレコードの多いようなtable 型のデータ場合jq 等で検索するのは\n非常遅いです。\n\n\n以下のようなデータで\n```console\n$ cat test.json\n[\n  {\n    \"id\": 130988433,\n    \"name\": \"2011_04_24-1.m4v\"\n  },\n  {\n    \"id\": 130988434,\n    \"name\": \"2011_04_24-2.mp4\"\n  },\n  {\n    \"id\": 130988435,\n    \"name\": \"2011_04_24-3.mp4\"\n  },\n\n```\n\nindex サブコマンドでindex 登録します。登録せずに search 時にも可能です。\n\n```console \n$ go get github.com/kazu/vfs-index/cmd/vfs-index\n$ vfs-index index  --index=../idx --column=name --table=test --data=./\n100% |██████████████████████████████████████████████████| (2316/2316, 2072 it/s) [1s:0s]\n$\n```\n\nsearch command で検索\n\n```\n$ vfs-index search  -q=\"2011_04\" --index=../idx --column=name --table=test --data=./\n{\"id\":130988433,\"name\":\"2011_04_24-1.m4v\"}\n{\"id\":130988434,\"name\":\"2011_04_24-2.mp4\"}\n{\"id\":130988435,\"name\":\"2011_04_24-3.mp4\"}\nindex merging done [==============================================================] 67507 / 67507\n```\n\n\n## goで使う\n\nimport package \n\n```go\nimport vfs \"github.com/kazu/vfs-index\"\n```\n\n\nindexing \n\n```go\nfunc DefaultOption() vfs.Option {\n\treturn vfs.RootDir(\"/Users/xtakei/vfs-idx\")\n}\n\n\nidx, e := vfs.Open(\"/Users/xtakei/example/data\", DefaultOption())\ne = idx.Regist(\"test\", \"id\")\n```\n\nsearching \n```go\nfunc DefaultOption() vfs.Option {\n\treturn vfs.RootDir(\"/Users/xtakei/vfs-idx\")\n}\n\n// search number index\nidx, e := vfs.Open(\"/Users/xtakei/example/data\", DefaultOption())\nsCond := idx.On(\"test\", vfs.ReaderColumn(\"id\"), vfs.Output(vfs.MapInfOutput))\n\nrecord := sCond.Select(func(m vfs.SearchCondElem2) bool {\n    return m.Op(\"id\", \"\u003c\",  122878513)\n}).First()\n\n// search by matching substring\nsCondName := idx.On(\"test\", vfs.ReaderColumn(\"name\"), vfs.Output(vfs.MapInfOutput))\nmatches2 := sCondName.Match(\"ロシア人\").All()\n\n```\n\n\nindex merging\nstop after 1 minutes.\n\n```go\n\nidx, e := vfs.Open(\"/Users/xtakei/example/data\", DefaultOption())\nsCond := idx.On(\"test\", vfs.ReaderColumn(\"id\"), vfs.Output(vfs.MapInfOutput))\nsCond.StartMerging()\ntime.Sleep(1 * time.Minute)\nsCond.CancelAndWait()\n```\n\n## TODO\n\n- [x] file list indexの書き出し\n- [x] file list indexの読み込み\n- [x] num column index の書き出し\n- [x] num column index の読み込み\n- [x] tri-gram clumn index の書き出し\n- [x] 文字列検索の検索の実装\n- [x] index merging のサポート\n- [x] コメント、ライセンス追加\n- [x] csv のsupport\n- [ ] msgpack のsupport\n- [x] support lz4","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkazu%2Fvfs-index","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkazu%2Fvfs-index","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkazu%2Fvfs-index/lists"}