{"id":25853689,"url":"https://github.com/7sedam7/krafna","last_synced_at":"2025-03-01T15:19:08.470Z","repository":{"id":275304208,"uuid":"907744521","full_name":"7sedam7/krafna","owner":"7sedam7","description":"Obsidion dataview plugin-like tool for command line.","archived":false,"fork":false,"pushed_at":"2025-02-28T16:36:54.000Z","size":1169,"stargazers_count":23,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-28T17:13:58.580Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/7sedam7.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-24T09:32:32.000Z","updated_at":"2025-02-28T16:36:57.000Z","dependencies_parsed_at":"2025-02-01T15:29:57.698Z","dependency_job_id":"d7fd7517-e767-490f-a6e5-e1de216e47aa","html_url":"https://github.com/7sedam7/krafna","commit_stats":null,"previous_names":["7sedam7/krafna"],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/7sedam7%2Fkrafna","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/7sedam7%2Fkrafna/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/7sedam7%2Fkrafna/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/7sedam7%2Fkrafna/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/7sedam7","download_url":"https://codeload.github.com/7sedam7/krafna/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241381717,"owners_count":19953767,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-01T15:19:07.962Z","updated_at":"2025-03-01T15:19:08.449Z","avatar_url":"https://github.com/7sedam7.png","language":"Rust","funding_links":[],"categories":["Rust","\u003ca name=\"viewers\"\u003e\u003c/a\u003eViewers"],"sub_categories":[],"readme":"# Krafna\n\n[![codecov](https://codecov.io/gh/7sedam7/krafna/branch/main/graph/badge.svg)](https://codecov.io/gh/7sedam7/krafna)\n![CodeRabbit Pull Request Reviews](https://img.shields.io/coderabbit/prs/github/7sedam7/krafna)\n![Crates.io Version](https://img.shields.io/crates/v/krafna)\n![Crates.io Total Downloads](https://img.shields.io/crates/d/krafna)\n\n![Krafna is a CLI tool for SQL querying frontmatter data. Similar to Obsidian's Dataview plugin](demo.gif)\n\n## Features\n\n- Query Markdown files in a directory using SQL-like syntax\n- Support for frontmatter data extraction\n- Flexible output formats (TSV and JSON)\n- Compatible with Neovim plugin [Perec](https://github.com/7sedam7/perec.nvim)\n\n## Performance\n\nBenchmarking on a base Mac mini M4 shows that Krafna can query:\n\n- ~5000 files (without cache)\n- ~100 000 files (with cache)\n\n within ~100ms.\n\nCaching is don with [bincode](https://github.com/bincode-org/bincode?tab=readme-ov-file).\nCache files are not crazy small, but not too big. I might consider compression in the future. (Currently ~250KB for ~100 files)\nONLY files that were modified since the last cache are parsed and re-cached.\nCache files are stored at:\n\n- LINUX: $XDG_CACHE_HOME or $HOME/.cache/\n- WINDOWS: {FOLDERID_LocalAppData}\n- MAC: $HOME/Library/Caches\n\nat `com/7sedam7/krafna`\n\nFlamegraph is currently pointing to Pod (internal enum struct) deserialization as the biggest bottleneck.\n\n`cargo bench` has been giving me weird results recently, I'm not expert at using it and did not want to spend too much time on it.\n`gtime -v` might be less precise when it comes to miliseconds, but it seems to give more realistic results.\n\nRun benchmarks: (you can change the number of files that will be generated in bench/query_benchmark.rs)\n\n``` bash\ncargo bench\n```\n\nRun flamegraph: (For a cleaner flamegraph, consider temporarily disabling rayon’s parallelism by replacing `par_iter()` with `iter()`.)\n\n``` bash\ncargo install flamegraph\ncargo flamegraph --root --bin krafna -- 'select file.name, tags from frontmatter_data(\"../krafna-bench/bench/\") where \"exampl\" in tags'\n```\n\n## Installation\n\nThere are binaries available for Linux, macOS, and Windows under Releases.\n\n### Cargo\n\n```bash\ncargo install krafna\n```\n\n### Homebrew\n\n```bash\nbrew tap 7sedam7/krafna\nbrew install krafna\n```\n\n## Usage\n\n```\nUsage: krafna [OPTIONS] [QUERY]\n\nArguments:\n  [QUERY]  The query to execute\n\nOptions:\n      --select \u003cSELECT\u003e\n          OVERRIDES SELECT fields from the query with \"field1,field2\"\n      --from \u003cFROM\u003e\n          From option in case you are implementing querying for specific FROM that you don't want to specify every time. This OVERRIDES the FROM part of the query!\n      --include-fields \u003cINCLUDE_FIELDS\u003e\n          include SELECT fields with \"field1,field2\" (prepends them to th front of the SELECT fields in the query)\n      --find \u003cFIND\u003e\n          Find option to find all krafna snippets within a dir\n      --json\n          Output results in JSON format\n  -h, --help\n          Print help\n\n```\n\n### SELECT\n\n- Currently, you can only specify field names.\n- There are extra added fields for the file data itself, acessible with file.\u003coption\u003e (options: name, path, created, accessed, modified).\n- No support for *, functions, nor expressions yet.\n- No support for AS yet.\n\n### FROM\n\n#### FRONTMATTER_DATA\n\n- `FROM FRONTMATTER_DATA(\"\u003cpath\u003e\")`\n- This will find all markdown files in the specified `\u003cpath\u003e` and use their frontmatter data as rows.\n- FIELDS:\n  - `file.name` - name of the file\n  - `file.path` - path to the file\n  - `file.created` - date when the file was created\n  - `file.accessed` - date when the file was last accessed\n  - `file.modified` - date when the file was last modified\n  - All other fields are from frontmatter data\n\n#### MD_LINKS\n\n- `FROM MD_LINKS(\"\u003cpath\u003e\")`\n- This will find all the links in markdown files in the specified `\u003cpath\u003e`. Each link is a separate row.\n- FIELDS:\n  - `file.*` - file data same as above\n  - `type` - type of the link (inline, wiki)\n  - `external` - true if the link is external (not a local file)\n  - `url` - original url text from markdown file\n  - `path` - interpreted path to the local file in case link is not external. (relies on that path being within argument specified `\u003cpath\u003e`, otherwise it will be empty)\n  - `text` - text of the link\n  - `ord` - order of the link in the file\n\n#### MD_TASKS\n\n- `FROM MD_TASKS(\"\u003cpath\u003e\")`\n- This will find all the tasks in markdown files in the specified `\u003cpath\u003e`. Each link is a separate row.\n- Tasks in markdown are defined as lines starting with `- [ ]` or `- [x]`\n- FIELDS:\n  - `file.*` - file data same as above\n  - `checked` - true if the task is checked (`- [x]`)\n  - `text` - text of the task\n  - `ord` - order of the task in the file. If the task is a subtask, there is a '.' and then a number for ordering within a parent task. Nesting is supported.\n  - `parent` - parent `ord` of the task in the file. If the task is not a subtask, this will be empty\n\n\n- More functions will come.\n- No support for AS yet.\n\n### WHERE\n\n- Brackets are supported\n- Operatortors AND, OR, IN, \u003c, \u003c=, \u003e, \u003e=, ==, !=, LIKE, NOT LIKE, +, -, *, /, **, // are supported\n- Functions DATE(\u003csome-date\u003e, \u003coptional-format\u003e), DATEADD(\u003cinterval\u003e, \u003cnumber\u003e, \u003cdate\u003e, \u003coptional-format\u003e) are supported\n- Arguments to functions can be hardcoded values or field names\n- Nested functions, or expressions as arguments are NOT supported yet\n- file. fields can be used in WHERE clause as well\n\n### ORDER BY\n\n- You can only specify field names followed by ASC or DESC\n- Functions and expressions are NOT supported yet\n- file. fields can be used in ORDER BY clause as well\n\n### Other\n\n- LIMIT, OFFSET, JOIN, HAVING, GROUP BY, DISTINCT, etc. are not supported yet.\n- UPDATE and DELETE are not supported yet.\n\n\n### Examples\n\n#### Basic Query\n\n```bash\nkrafna \"SELECT title, tags FROM FRONTMATTER_DATA('~/.notes')\"\n```\n\n#### Find Files\n\n```bash\nkrafna --find ~/.notes\n```\n\n#### Output as JSON\n\n```bash\nkrafna \"SELECT * FROM FRONTMATTER_DATA('~/.notes')\" --json\n```\n\n#### Include Specific Fields\n\n```bash\nkrafna \"SELECT * FROM FRONTMATTER_DATA('~/.notes')\" --include-fields title,tags\n```\n\n## Neovim Integration\n\nUse with the [Perec](https://github.com/7sedam7/perec) Neovim plugin for seamless integration.\n\n## Roadmap\n\n(not in priority order)\n- [x] add . support for accesing sub-fields (file.name)\n- [x]  * migrate file_name, etc under file (name, path, created, accessed, modified)\n- [x] add default variables (today)\n- [ ]  * change it so that it does not need to be on every row (can have a general_values hash that can be passed around, and value getters would first check there and then from the source)\n- [ ] Implement pruning of AND and OR operators (mostly for better error messages, performance there is more than good enough)\n- [ ] TODOs\n- [x] Add tests for execution\n- [ ] add suport for functions in SELECT\n- [ ] add functions\n- [ ]  * think about which functions to add\n- [x]  * DATE(\"some-date\", \u003cformat\u003e) -\u003e new type date\n- [x]  * [DATEADD()](https://www.w3schools.com/sql/func_sqlserver_dateadd.asp)\n- [x] implement val -\u003e val operators\n- [ ] UPDATE\n- [ ] DELETE\n- [ ] add AS to SELECT\n- [ ] add querying of TODOs (think of a format similar to [todoist](https://www.todoist.com/help/articles/use-task-quick-add-in-todoist-va4Lhpzz))\n- [ ]  * maybe abstract to query by regex\n- [ ] add querying of links between notes\n- [ ] think about which other sources would be cool to add\n- [ ] add group by\n\n## Acknowledgements\n\n- [grey-matter-rs](https://github.com/the-alchemists-of-arland/gray-matter-rs) for parsing frontmatter data\n- [rayon](https://github.com/rayon-rs/rayon) for parallelizing execution\n- [bincode](https://github.com/bincode-org/bincode?tab=readme-ov-file) for binary serialization\n- [CodeRabbit](https://coderabbit.io) for code reviews\n- Various AI tools for help with answering questions faster then me searching on Google/StackOverflow\n\n\n## Author\n\n[7sedam7](https://github.com/7sedam7)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F7sedam7%2Fkrafna","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F7sedam7%2Fkrafna","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F7sedam7%2Fkrafna/lists"}