Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/russmatney/org-crud

A tool for reading and writing org content via clojure
https://github.com/russmatney/org-crud

clojure markdown org-mode

Last synced: 11 months ago
JSON representation

A tool for reading and writing org content via clojure

Awesome Lists containing this project

README 

          
#+TITLE: Readme

#+STARTUP: content



* Org CRUD



A tool for reading and writing org content via clojure, as well as converting

org to markdown.



> Ninety percent of everything is crud.

> – [[https://www.quotes.net/quote/53367][Theodore Sturgeon]]



** Status



Alpha. I've recently namespaced the keys and flattened the props a bit.

Still waiting for things to settle.



I've used it internally for months, but the public api still needs to be

proven. In particular, I'd like to iron out some things I'm doing with dynamic

vars that no user should need to figure out/configure.



It works for my current use-cases, and there are unit tests! Feel free to take

it for a spin.



** Motivation



This library was pulled out of another tool, a productivity app built on top of

org-mode. That tool needed to be able to treat org files like a database of

items.



It was pulled out to allow a few other libraries to use it independently

([[https://github.com/russmatney/ralphie][russmatney/ralphie]]), and also to add

support for converting org files to markdown. See [[#markdown][Markdown]].



Org-crud aims to provide simple interactions with org files to clojure

environments.



** Background



There is not much to the parser besides a thin layer on top of

[[https://github.com/gmorpheme/organum][organum]]. Organum does not nest the org

items - it returns a flattened list, regardless of the items' hierarchical

relationship. Org-crud provides both a flattened and nested option for parsing

org items.



This library is also [[https://github.com/borkdude/babashka][babashka]]

compatible, so you can drop it into a bb script without issue. This was

necessary for tools like [[https://github.com/russmatney/ralphie][ralphie]] to run

the exporter. You can see how ralphie consumes it [[https://github.com/russmatney/ralphie/blob/f6432e433e7e447aa1c0784e62ade2935c557cfc/src/ralphie/notes.clj*L21][in the ralphie.notes namespace]]



** Features



- [[https://github.com/borkdude/babashka][Babashka]] compatible

- List nested or flattened org items

- Update existing org items

  - Updates by :ID:

  - Add/remove tags, properties

  - Change an item's name

- Delete org items

- Convert org files to markdown files



** Org Item Model



This library parses org "items" from a given `.org` file.



An example item looks something like:



#+begin_src clojure

{:org/name "My org headline" ;; the name without the bullets/todo block

 :org/headline "** [ ] My org headline" ;; a raw headline

 :org/source-file "" ;; the file this item was parsed from

 :org/id *uuid "" ;; a unique id for the headline (parsed from the item's property bucket)

 :org/tags *{"some" "tags"}

 :org/level 2 ;; 1-6 or :level/root

 :org/body-string "raw body string\nwith lines\nof content"

 :org/body '() ;; a list of parsed lines, straight from organum TODO document this structure

 :org/status :status/not-started ;; parsed based on the

 ;; also supports :status/in-progress, :status/done, :status/cancelled



 ;; these dates are pulled through as strings

 :org/closed "2022-04-30 Sat 17:42"

 :org/deadline "2022-04-30 Sat"

 :org/scheduled "2022-04-30 Sat"



 ;; supports [#A], [#B], [#C] in headlines

 :org/priority "B"



 :org.prop/some-prop "some prop value" ;; props are lower-and-kebab-cased

 :org.prop/some-other-prop "some other prop value"

 :org.prop/created-at "2020-07-21T09:25:50-04:00[America/New_York]" ;; to be parsed by consumer



 :org/items '() ;; nested org-items (if parsed with the 'nested' helpers)



 ;; misc helper attrs

 :org/word-count 3 ;; a basic count of words in the name and body

 :org/urls '() ;; parsed urls from the body - helpful for some use-cases

}

#+end_src



Items were originally implemented to support individual org headlines, but have

been adapted to work with single org files as well (to fit org-roam tooling

use-cases).



** Install



Currently, install requires referencing the git repo.



#+begin_src clojure

;; deps.edn

{:deps

 {russmatney/org-crud {:git/url "https://github.com/russmatney/org-crud.git"

                       :sha     "a4b44022c690e1c8fb34512f1aad85bc49569d19"}}}

#+end_src



TODO add to clojars



** Usage



TODO do some work on this section!



You can see the test files for example usage.



I'm attempting to hold a public api at `org-crud.api`, but that is a WIP.



*** Parsing



#+begin_src clojure

(ns your.ns

 (:require [org-crud.api :as org-crud]))



;; a nested item represents an entire file, with items as children

(let [item (org-crud/path->nested-item "/path/to/file.org")]

  (println item))



;; parses every '.org' file in a directory into a list of nested items

(let [items (org-crud/dir->nested-items "/path/to/org/dir")]

  (println (first items)



;; 'flattened' items have no children - just a list of every headline

;; (starting with the root itself)

(let [items (org-crud/path->flattened-items "/path/to/file.org")]

  (println (first items)))

#+end_src



*** Updating



Updates are performed with a passed item and an update map that resembles the

org-item itself. It will use the passed item's id and source-file to find the

item to be updated, merge the updates in memory, then rewrite it.



#+BEGIN_SRC clojure

(ns your.ns

 (:require [org-crud.api :as org-crud]))



(-> (org-crud/path->flattened-items "/path/to/file.org")

    second ;; grabbing some item

    (org-crud/update!

      {:org/name "new item name" ;; changing the item name

       :org/tags "newtag" ;; adding a new tag

       :org.prop/some-prop "some-prop-value"

      }))

#+END_SRC



TODO document props-as-lists features

TODO document refile!, add-item!, delete-item!



*** Markdown



Org-crud provides a namespace for converting org files to markdown, and a

babashka-based cli tool for running this conversion on the command line.



In order for this to work, you'll need to have

[[https://github.com/borkdude/babashka#quickstart][Babashka]] (and [[https://clojure.org/guides/getting_started][clojure]] installed and

available on the command line as `bb` and `clojure`.



#+begin_src sh

bb org-crud.jar org-to-markdown ~/Dropbox/notes tmp-out

#+end_src



Note that this support targets a use-case for publishing an

[[https://github.com/org-roam/org-roam/][org-roam]] directory as markdown, but

otherwise is probably not a complete org->markdown conversion solution. If you

have more use-cases that you'd like to see supported, please open an issue

describing the use-case, and I'd be happy to take a shot at it.



Note that Emacs/Org supports export that is fairly similar as well - I enjoyed

putting this together and not needing to leave the joy of clojure-land.



An org file like `20200618104339-dated-example.org`:



#+begin_src org

*+TITLE: Dated Example

*+ROAM_TAGS: dated



Another org file, now with a link!



- [[file:example.org][example link]]



Dated to match the org-roam default style.

#+end_src



Will be converted to:



#+begin_src markdown

---

title: "Dated Example"

date: 2020-06-18

tags:

  - dated

  - note

---



Another org file, now with a link!



- [example link](/notes/example)



Dated to match the org-roam default style.

#+end_src



- The frontmatter pulls tags from `*+ROAM_TAGS`.

  - TODO prevent `note` from being added every time.

- The date is parsed from the filename.

  - TODO support alternate sources for the date, if users don't have timestamps

    in filenames.

- The links to other notes are prepended with `/notes/`

  - TODO support custom link handling options, not just this hardcoded /notes/ prefix.



**** Appended `Backlinks` section



When run over a directory, a `Backlinks` section is built up as a basic markdown

list.



#+begin_src org

<... rest of file>

\* Backlinks



- [Index](/notes/20200704184516-index)

#+end_src



** Notes



*** Item IDs (UUIDs)



Item IDs are more or less required for updating. Things will fallback to

matching on name if there are no ids, but this approach has a few issues,

because names are not necessarily unique throughout files.



I've updated my personal org templates/snippets in places to include IDs when

creating new items, and org-mode provides helpers that can be used to add them

without too much trouble. (Ex: `org-id-get-create`).



TODO share links to templates/snippets that create uuids



If this is a problem, let me know, there are other workarounds. Using IDs allows

for cases with repeated headlines in the same file - otherwise you might get

into tracking line numbers or parents, which did not seem worth it, especially

as my usage benefitted from the IDs elsewhere.



** Relevant/Related tools



- [[https://github.com/kaushalmodi/ox-hugo][ox-hugo]]

- [[https://github.com/gmorpheme/organum][organum]]

- [[https://github.com/org-roam/org-roam][org-roam]]



** Development

*** Running the cli using the source



Rather than the built uberjar:



#+BEGIN_SRC sh

# from this repo's root

bb -cp $(clojure -Spath) -m org-crud.cli org-to-markdown ~/Dropbox/notes tmp-out

#+END_SRC



*** Rebuild the uberjar



To rebuild the cli-based uberjar via babashka:



#+begin_src sh

bb -cp $(clojure -Spath) -m org-crud.cli --uberjar org-crud.jar

#+end_src



*** Running tests



#+begin_src sh

./bin/kaocha

#+end_src