Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/tgbugs/orgstrap
Executable Org files
https://github.com/tgbugs/orgstrap
emacs emacs-lisp org-mode
Last synced: 3 months ago
JSON representation
Executable Org files
- Host: GitHub
- URL: https://github.com/tgbugs/orgstrap
- Owner: tgbugs
- License: gpl-3.0
- Created: 2018-06-22T19:25:36.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2023-07-22T07:02:29.000Z (12 months ago)
- Last Synced: 2024-01-24T18:24:01.327Z (6 months ago)
- Topics: emacs, emacs-lisp, org-mode
- Language: Emacs Lisp
- Homepage:
- Size: 1.17 MB
- Stars: 43
- Watchers: 6
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.org
- License: LICENSE
Lists
- awesome - orgstrap - Executable Org files (Emacs Lisp)
README
# -*- org-adapt-indentation: nil; org-edit-src-content-indentation: 0; orgstrap-cypher: sha256; orgstrap-norm-func-name: orgstrap-norm-func--dprp-1-0; orgstrap-block-checksum: 24dbc0d54579db9f0972c6218bdfc1b924e54d8b230d82c2e9c55c833af19cdd; -*-
# [[orgstrap][jump to orgstrap block for this file]]
#+title: Orgstrap: Executable Org files
#+startup: showall
#+options: num:nil \n:nil
#+todo: TODO STARTED | DONE EXPIRED
#+property: header-args :eval no-export
#+property: header-args:elisp :lexical yes
#+latex_header: \usepackage[margin=0.8in]{geometry}
#+latex_header: \setlength\parindent{0pt}
#+link: yt https://youtu.be/
#+link: gh https://github.com/
# [[file:./README.pdf]]
# [[file:./README.html]]
#+HTML: 
#+HTML: 
# orgstrap
# Not quite a unicorn.
# If you want to grow up to be a unicorn you're going to have to
# pull yourself up by your own bootstraps!
#+name: orgstrap-shebang
#+begin_src bash :eval never :results none :exports none
set -e "-C" "-e" "-e"
{ null=/dev/null;} > "${null:=/dev/null}"
{ args=;file=;MyInvocation=;__p=$(mktemp -d);touch ${__p}/=;chmod +x ${__p}/=;__op=$PATH;PATH=${__p}:$PATH;} > "${null}"
$file = $MyInvocation.MyCommand.Source
{ file=$0;PATH=$__op;rm ${__p}/=;rmdir ${__p};} > "${null}"
emacs -batch -no-site-file -eval "(let (vc-follow-symlinks) (defun orgstrap--confirm-eval (l _) (not (memq (intern l) '(elisp emacs-lisp)))) (let ((file (pop argv)) enable-local-variables) (find-file-literally file) (end-of-line) (when (eq (char-before) ?\^m) (let ((coding-system-for-read 'utf-8)) (revert-buffer nil t t)))) (let ((enable-local-eval t) (enable-local-variables :all) (major-mode 'org-mode) find-file-literally) (require 'org) (org-set-regexps-and-options) (hack-local-variables)))" "${file}" -- ${args} "${@}"
exit
<# powershell open
#+end_src
=orgstrap= allows Org files to describe their own requirements and
define their own functionality, making them self-contained standalone
computational artifacts dependent only on Emacs or other
implementations of the Org Babel protocol in the future.
This file bootstraps itself to provide the tools to use =orgstrap= in
any Org file.
=orgstrap= has a formal [[#specification][specification]] and this
file contains 3 implementations that support 3 slightly different use
cases along with tooling for common =orgstrap= workflows.
=orgstrap= works with all versions of Emacs since =24.4= and Org since =8.2.10=.
Please see the [[#changelog][changelog]] for the latest updates.
* Contents :noexport:
:PROPERTIES:
:TOC: :include all :depth 1
:visibility: folded
:END:
:CONTENTS:
- [[#getting-started][Getting started]]
- [[#hello-orgstrap][Hello orgstrap]]
- [[#inspiration][Inspiration]]
- [[#use-cases][Use cases]]
- [[#details][Details]]
- [[#specification][Specification]]
- [[#local-variables][Local Variables]]
- [[#code][Code]]
- [[#changelog][Changelog]]
- [[#contributing][Contributing]]
- [[#guides][Guides]]
- [[#best-practices][Best practices]]
- [[#bootstrapping-to-emacs-bootstrapping-to-org][Bootstrapping to Emacs, bootstrapping to Org]]
- [[#examples][Examples]]
- [[#background-file-local-variables-and-checksums][Background, file local variables, and checksums]]
- [[#experience-reports][Experience reports]]
- [[#future-work][Future work]]
:END:
* Getting started
:PROPERTIES:
:CUSTOM_ID: getting-started
:END:
Using =orgstrap= is easy.
If you already have =orgstrap= installed you can enable it for any
Org file by running =M-x= =orgstrap-init= which will add the basic
=orgstrap= machinery to the current buffer.
=orgstrap= is available on [[https://melpa.org/#/orgstrap][melpa]]. You can install it via
=M-x= =package-install= =orgstrap= or ~(use-package orgstrap)~.
You can also try out =orgstrap= without installing just by opening
this file in Emacs!
1. Obtain the org mode source for this file. (e.g.
[[https://raw.githubusercontent.com/tgbugs/orgstrap/master/orgstrap.org][from GitHub]]).
2. Open the source in Emacs[[#bootstrapping-to-emacs-bootstrapping-to-org][*]].
(e.g. =M-x= =url-handler-mode= then =C-x C-f=
# @@latex: \\@@
=https://raw.githubusercontent.com/tgbugs/orgstrap/master/README.org=).
3. Decline the file local variables.
4. Inspect the [[#details][orgstrap block]] and file local variables.
5. Reload the file and accept the file local variables.
6. Congratulations you can now use =orgstrap= with your own files!
If you install =orgstrap= in this way you have to open the file again
every time you open a new Emacs, so installing [[file:./orgstrap.el][orgstrap.el]]
via ~package.el~ or by some other means is recommended.
A minor mode for editing orgstrapped files is included as =orgstrap-edit-mode=.
It is activated by =orgstrap-init=. When enabled it automatically updates
the =orgstrap-block-checksum= prop line local variable whenever the
=orgstrap= block changes.
If you do not use =orgstrap-edit-mode= then the easiest way to add the
orgstrap checksum to a file is to invoke =M-x= =orgstrap-add-block-checksum=.
=orgstrap= also includes =orgstrap-mode=, which is a regional minor mode
for =org-mode=. When enabled, =orgstrap-mode= detects, checks, and runs
orgstrap blocks when visiting Org files, superseding any embedded =eval:=
local variables.
The rest of this file is an overview of the use cases for =orgstrap= and
the implementation of =orgstrap= along with discussion and commentary.
If you are looking for examples of how to use =orgstrap= this files is a good place to start.
* Hello =orgstrap=
:PROPERTIES:
:CUSTOM_ID: hello-orgstrap
:END:
The bare minimum needed to make an =org-mode= file executable (with a bit of safety).
#+caption: [[file:./orgstrap-minimal.org]]
#+begin_src org :tangle ./orgstrap-minimal.org
# -*- orgstrap-cypher: sha256; orgstrap-block-checksum: 66ba9b040e22cc1d30b6f1d428b2641758ce1e5f6ff9ac8afd32ce7d2f4a1bae; orgstrap-norm-func-name: orgstrap-norm-func--dprp-1-0; -*-
# [[orgstrap][jump to orgstrap block for this file]]
,#+name: orgstrap
,#+begin_src elisp :results none
(message "orgstrap successful!") ; (ref:im-a-coderef-and-thats-ok)
,#+end_src
=orgstrap= a plain-text executable format. Powered by Org mode and Emacs.
# Local Variables:
# eval: (progn (setq-local orgstrap-min-org-version "8.2.10") (let ((a (org-version)) (n orgstrap-min-org-version)) (or (fboundp #'orgstrap--confirm-eval) (not n) (string< n a) (string= n a) (error "Your Org is too old! %s < %s" a n))) (defun orgstrap-norm-func--dprp-1-0 (body) (let ((p (read (concat "(progn\n" body "\n)"))) (m '(defun defun-local defmacro defvar defvar-local defconst defcustom)) print-quoted print-length print-level) (cl-labels ((f (b) (cl-loop for e in b when (listp e) do (or (and (memq (car e) m) (let ((n (nthcdr 4 e))) (and (stringp (nth 3 e)) (or (cl-subseq m 3) n) (f n) (or (setcdr (cddr e) n) t)))) (f e))) p)) (prin1-to-string (f p))))) (unless (boundp 'orgstrap-norm-func) (defvar-local orgstrap-norm-func orgstrap-norm-func-name)) (defun orgstrap-norm-embd (body) (funcall orgstrap-norm-func body)) (unless (fboundp #'orgstrap-norm) (defalias 'orgstrap-norm #'orgstrap-norm-embd)) (defun orgstrap--confirm-eval-minimal (lang body) (not (and (member lang '("elisp" "emacs-lisp")) (eq orgstrap-block-checksum (intern (secure-hash orgstrap-cypher (orgstrap-norm body))))))) (unless (fboundp #'orgstrap--confirm-eval) (defalias 'orgstrap--confirm-eval #'orgstrap--confirm-eval-minimal)) (let (enable-local-eval) (vc-find-file-hook)) (let ((ocbe org-confirm-babel-evaluate) (obs (org-babel-find-named-block "orgstrap"))) (if obs (unwind-protect (save-excursion (setq-local orgstrap-norm-func orgstrap-norm-func-name) (setq-local org-confirm-babel-evaluate #'orgstrap--confirm-eval) (goto-char obs) (org-babel-execute-src-block)) (when (eq org-confirm-babel-evaluate #'orgstrap--confirm-eval) (setq-local org-confirm-babel-evaluate ocbe)) (ignore-errors (org-set-visibility-according-to-property))) (warn "No orgstrap block."))))
# End:
#+end_src
* Inspiration
:PROPERTIES:
:CUSTOM_ID: inspiration
:END:
By default =org-mode= source block headers only take existing elisp functions as arguments.
This means that header arguments can become extremely verbose.
Wouldn't it be great if you could use the magical mystical power of =defun=
inside an org file itself to provide simple, reusable functionality rather
than +copying and pasting+ +yanking and putting+ killing and yanking raw
elisp around the buffer?
With =orgstrap= you can.
=orgstrap= makes sure that the functionality that you need is available when you need it.
Whether it is =(defun dir-tramp-sudo (host) (format "/ssh:%s|sudo:%s:" host host))= to
simplify a pattern for remote execution when using the =:dir= header argument, or a
function to detect and set the right environment variables, =orgstrap= is there for you.
* Use cases
:PROPERTIES:
:CUSTOM_ID: use-cases
:END:
=orgstrap= specifies what is essentially a plain-text executable file format.
Thus, it can be used for nearly everything[fn::Now, whether it *should* be....].
While many (including the author) might find this to be totally radically awesome,
there are much better, saner, and safer ways to execute arbitrary code than to hash
some elisp blocks and use Emacs file local variables to automatically eval a specially
named source block only when it matches the hash.
#+caption: Things you can do with arbitrary code execution and checksums.
#+name: table-use-cases
|----------------------------------------+------------------+------------------------------------|
| Use case | Good idea | Alternative |
|----------------------------------------+------------------+------------------------------------|
| Always run defuns used in file | ✅ Yes | init.el, =C-c C-c= |
| Install elisp code directly | ❌ No | Use =package.el=, =straight=, etc. |
| Self tangling files | ✅ I do it | =C-c C-v C-t= |
| Install packages required by file | Probably | System package manager |
| Create an Emacs based botnet | ✅ ✅ Definitely | ??? |
| Create Orgware for non-technical users | ✅ Yes | Web server and the unholy trinity. |
| Replace hard to follow instructions | ✅ Yes | Hard to follow instructions |
| Tangle git hook files for publishing | ✅ Yes | Manually tangle |
| System specific behavior without edits | ✅ Yes | #+name: literal blocks via =:= |
| Version control for source blocks | ❌ ❌ Please no | git, hg, svn, anything please |
| Detect and set environment variables | ✅ Yes | |
|----------------------------------------+------------------+------------------------------------|
# Actually I'm kind of hyped for though of describing the system used to version
# control the code in the file itself. Not so simple to pull off though.
# It only sort of works in this case because we have the rest of the file under
# version control in another system. Without git, developing this would have been
# a complete nightmare.
* Details
:PROPERTIES:
:CUSTOM_ID: details
:END:
The first elisp source block named =orgstrap= in an org file is
automatically run using an =eval:= file local variable. Users can
review and add the file local variables to their known safe list
so that the code can be run in the future without the need to bother
them again.
When opening a file for the first time, users should decline the local
variables, review the =eval:= local variable and the =orgstrap= block
directly, and then reload, revisit, or =M-x= =org-mode= and only then
accept the local variables. This only needs to be done once for the
=eval:= local variables (unless they are updated).
** The orgstrap block
:PROPERTIES:
:CUSTOM_ID: the-orgstrap-block
:END:
This is the =orgstrap= block that is used for this file.
# FIXME the -r -l is kind of needed here deal with ref blocks
# FIXME this is an internal inconsistency in babel
#+caption: The =orgstrap= block that is used for this file.
#+name: orgstrap
#+begin_src elisp :results none :noweb no-export :lexical yes
;; This is an example that also nowebs in the source for
;; `orgstrap-init' and `orgstrap-add-block-checksum' along
;; with the rest of the orgstrap machinery so it is easy to
;; use orgstrap to create and update orgstrap blocks
<>
<>
<>
<>
<>
;; tangle helpers
(defun ow---strip-empty-lines-and-refs ()
(save-excursion
(goto-char (point-min))
(while (re-search-forward "^ +$\\|[ ;]*(ref:.+)$" nil t)
;; FIXME stripping the refs here can cause a divergence for the checksums
;; FIXME this incorrectly strips refs from orgstrap-minimal.org due to wrong mode
(replace-match ""))))
;; XXX reminder that this cannot be a buffer local hook because it
;; doesn't run in this buffer this is likely a bug
(add-hook 'org-babel-tangle-body-hook #'ow---strip-empty-lines-and-refs)
;; helper functions to update examples
(defun orgstrap--update-examples ()
"Use with `orgstrap-on-change-hook' to automatically keep the contents
of the example blocks in sync."
;; XXX WARNING if you update the orgstrap-file-local-variables-common block
;; you MUST re`eval-defun' for `orgstrap--local-variables--eval-common' and
;; `orgstrap--lv-common-with-block-name' otherwise the changes will not take
(let ((pairs `(("local-variables-prop-line-example" ,(orgstrap--local-variables-prop-line-string))
("local-variables-portable-example" ,(orgstrap--file-local-variables-string))
("local-variables-minimal-example" ,(let ((orgstrap-use-minimal-local-variables t))
(orgstrap--file-local-variables-string))))))
(mapcar (lambda (name-content) (apply #'orgstrap-update-src-block name-content)) pairs)))
(defun orgstrap--local-variables-prop-line-string ()
"Copy the first logical line of the file since it is easier and faster
than trying to sort out which variables were or were not in the prop line."
;; XXX NOTE There are some cases involving bootstrapping to emacs where the first line of
;;an org-mode file is a shebang, but we will deal with those if and when they arrise
(buffer-substring-no-properties 1 (save-excursion (goto-char 0) (next-logical-line) (point))))
(defun orgstrap--file-local-variables-string ()
(let (print-length)
(with-temp-buffer
(org-mode)
(goto-char 0)
(insert "#+name: orgstrap\n#+begin_src elisp :lexical yes\n#+end_src\n")
(orgstrap--add-file-local-variables orgstrap-use-minimal-local-variables)
(goto-char 0)
(kill-whole-line 4)
(buffer-string))))
;; tangle blocks and update examples on change
(add-hook 'orgstrap-on-change-hook #'org-babel-tangle nil t) ;; FIXME should fire on non-semantic changes
(add-hook 'orgstrap-on-change-hook #'orgstrap--update-examples nil t)
;; enable orgstrap mode locally for this file when this block runs
(orgstrap-edit-mode 1)
(message "orgstrap complete!")
#+end_src
The headers for the block above look like this.
#+name: orgstrap-example
#+begin_example org :eval never :noweb no
,#+name: orgstrap
,#+begin_src elisp :results none :noweb no-export :lexical yes
<>
,#+end_src
#+end_example
Additional machinery is provided as part of this file to update the local
variable value of =orgstrap-block-checksum= so that only known blocks can
be run. Note that this DOES NOT PROTECT against someone changing the block
and the checksum at the same time and sending you a malicious file! You need
an alternate and trusted source against which to verify the checksum of the
=orgstrap= block.
** Portability
A couple of notes on portability and backward compatibility with older
versions of Emacs. I have tried to get =orgstrap= running on emacs-23,
however the differences between org =6.33x= and org =8.2.10= are too
large to be overcome without significant additional code. First, all
uses of =(setq-local var "value")= have to be changed to
=(set (make-local-variable 'var) "value")= so that the local variable
eval code can run. However once that is done, you discover that all of
the org-babel functions are missing, and then you will discover that
emacs-23 doesn't support lexical binding. Therefore, we don't support
emacs-23 and older versions.
** Version specific behavior
There is a major usability issue for =orgstrap= when running Emacs
< 27. Specifically, prior to Emacs 27 it is not possible to view the
file whose local variables are about to be set because it is
impossible to switch out of the file local variables confirmation
buffer. Starting in Emacs 27 it is possible to change buffer to view
the file that is about to have its file local variables set.
* Specification
:PROPERTIES:
:CUSTOM_ID: specification
:END:
# Except for this comment, comments in the spec are not official parts of the spec.
** Terminology
The specification for orgstrap makes extensive use of terminology
derived from the Emacs manual section on
[[info:emacs#Specifying File Variables][Specifying File Variables]]
and the Org manual section on the
[[info:org#Structure of Code Blocks][Structure of Code Blocks]].
What the Emacs manual calls the first line or prop-line is referred
to in this document as the =prop line= and the variables specified in
it are referred to as =prop line local variables=. What the Emacs
manual explicitly calls the =local variables list= we refer to in the
same way[fn::In other sections of the readme that contains this
specification the nomenclature is inconsistent, and refers to these
variously as end local variables or simply as local variables or file
local variables.].
What the Org manual refers to as a =source code block= we refer to in the
same way.
** File contents
In order for an Org mode file to support the use of =orgstrap= it must
contain the following.
The =prop line= of the Org file must include three local variables:
=orgstrap-cypher=, =orgstrap-norm-func-name=, and =orgstrap-block-checksum=.
Anywhere in the rest of the file there must be an Org =source code block=
that has the == =orgstrap= with whitespace preceding the =o= and only
whitespace following the =p= until a newline. Newline and whitespace are as
defined by [[https://orgmode.org/worg/dev/org-syntax.html][Org mode syntax]].
This =source code block= is henceforth referred to as the =orgstrap block=.
If there is more than one =source code block= with the == =orgstrap=
then the =source code block= that starts closest to the beginning of the file
is the =orgstrap block=.
The == for the =orgstrap block= must be =elisp= or =emacs-lisp=. [fn::
It is possible that other languages might be supported in the future. However,
that is somewhat challenging given that Org and Orb-babel only implicitly
specify that a conforming implementation that can execute =source code blocks=
must support Emacs lisp =source code blocks= and the use of Emacs lisp in
header arguments. There is an infinitesimal possibility that Org-babel will
support the use of other languages for inline header arguments since it
already supports them via blocks and it is not trivial to allow additional
languages to be used inline without some additional way to indicate the language
in use for a particular block. On the other hand, there is a small possibility
that other languages could be supported in the =orgstrap block= by specifying
them as part of the =local variables list=. However it is not clear that this
is needed, because it is possible to specify a small orgstrap block that can
ensure that the required Org-babel language implementations are installed and
then securely run those blocks. This block can probably be stripped down
sufficiently to make it possible to implement only the subset of elisp
required to run that block.]
Everything else about the =orgstrap block= is delegated to Org mode, including
header arguments, and noweb expansion.
# TODO With the possible exception being that a header of the form
# :var orgstrap-enable-optional=(identity nil) might be added to
# make it possible for the user to toggle optional dependencies
# obviously authors can do whatever they want with the block and
# set as many :vars to t or nil as they want to give users as much
# or as little control over what is run as they desire, this should
# probably just go in as an example, with note that this is one of the
# reasons why we don't hash :vars but also why users need to check those
# I'm 99% certain that embedding orgstrap-norm-func in the local variables list
# should NOT be required as part of the specification. I do that in the current
# implementation, but the 3000 char limit for the local variables list is going
# to pose quite the challenge for the portable implementation, and thus I think
# all the spec needs to say is that an implementation must be able to reproduce
# the orgstrap block hash when the whole file hash is the same.
** Implementation behavior
When provided with the same file whose =orgstrap block= was originally hashed
(where "the same file" means a file with the same checksum when hashed using
the algorithm specified by the =orgstrap-cypher= variable), a conforming
implementation must be able to do the following.
A conforming implementation must be able to reproduce the =orgstrap-block-checksum=
using only the information contained in the =orgstrap-cypher= and
=orgstrap-norm-func-name= =prop line local varaibles=, and information
contained in the rest of the file explicitly excluding the contents of
the =orgstrap-block-checksum= =prop line local varaible=. The most obvious
additional information required being the contents of the =orgstrap block= [fn::
The reference implementation provided in the readme containing this specification
uses an Emacs =eval:= local variable (elv) in the =local variables list=. Embedding
an elv is not required by this specification. However, such an implementation allows
files to depend only on the core Emacs implementation.
In the future an optional extension may be added to this document that specifies the
behavior for files using an elv in the =local variables list=.
A minimal implementation that works without elvs is also provided.
Files that contain only the prop line local variables are dependent on an implementation
of orgstrap already being present on the system running the file.
There is a fine balance between portability and compactness since a minimal implementation
has to make more assumptions about the systems it will run on.
Multi-stage orgstrap or other means of bootstrapping a working runtime for an Org file such
as the process implemented in the
[[#bootstrapping-to-emacs-bootstrapping-to-org][Bootstrapping to Emacs, bootstrapping to Org]]
section of this readme are ongoing areas of exploration.].
#+begin_quote
One implementation detail is that conforming implementations
must implement noweb expansion and coderef removal prior to
passing the contents of the =orgstrap block= to a normalization
function.
#+end_quote
Normalization functions that produce different output given the same
input for at least one input must have different names. One way this
can be achieved is by suffixing a name with a version number.
In order for an orgstrap normalization function name to be considered
official it must have an implementation bearing that name in the
[[#normalization-functions][Normalization functions]] section of the
readme that contains this specification. Once a function has been
named, no other function shall ever bear the same name unless for
all inputs it produces output that is byte-identical to the output
of all other previous implementations of the function bearing that
name.
#+begin_quote
A key point about =orgstrap-norm-func-name= is that the implementation
of these functions must be agreed upon by various implementations, if a user
inserts a fake hash, implementations should deal with it by running the
normalization and hashing process again using a known-conforming implementation
on a system that they control.
#+end_quote
* Local Variables
:PROPERTIES:
:CUSTOM_ID: local-variables
:END:
** Contents :noexport:
:PROPERTIES:
:TOC: :include siblings :exclude this :depth 1
:visibility: folded
:END:
:CONTENTS:
- [[#overview][Overview]]
- [[#org-version-support][Org version support]]
- [[#normalization][Normalization]]
- [[#definitions][Definitions]]
- [[#note-on-noweb-support][Note on noweb support]]
- [[#note-on-coderefs][Note on coderefs]]
- [[#how-local-variables-appear-in-the-file][How local variables appear in the file]]
:END:
** Overview
:PROPERTIES:
:CUSTOM_ID: overview
:END:
This section contains two implementations of =orgstrap= (minimal and
portable) that are small enough to fit in the local variables list at
the end of a file. *The local variables list must start less than 3000*
*chars from the end of the file*.
We use =setq-local= in =eval:= to set =org-confirm-babel-evaluate=
because it is a =safe-local-variable= only when the value is =t= and
cannot be set directly as a file local variable. In this context this
workaround seems reasonable and not malicious because the use of
=eval:= should alert users that some arbitrary stuff is going on and
that they should be on high alert to check it.
Below in [[#definitions][Definitions]] there is a more readable
version of what the compacted local variables code at the end of the
file is doing. *Always check that the =eval:= local variables in*
*unknown orgstrapped files match a known set when reviewing and*
*accepting local variables*.
=orgstrap= eval local variables, or *elvs* for short, are little
helpers at the end of the file that make everything work in a portable
manner when =orgstrap.el= is not present on a system.
While elvs are not required by the specification, they greatly reduce
the complexity of implementation. They also simplify the instructions
to two steps: 1. install Emacs, 2. open the file.
# the orgstrap block can the install orgstrap.el if needed
# TODO it is entirely possible to automate that check
# but not without already having orgstrap available.
# TODO publish the hashes of the eval sexps.
** Org version support
:PROPERTIES:
:CUSTOM_ID: org-version-support
:END:
Different versions of the =orgstrap= local variables work with
different versions of =org-mode=. We include an explicit version
check and fail so that strange partial successes can be avoided
and so that newer versions of the local variables can be simplified
when backward compatibility is not needed. For example one might
imagine a future where no local variables are needed in the file
at all, only the cypher and the checksum because we managed to
get support for the convention built into =org-mode= directly.
This will also allow us to streamline which block to use based
on whether noweb is being used. If it is not then we can decide
automatically.
If orgstrap is installed, we use the installed version of orgstrap
anyway so don't bother.
#+name: orgstrap-check-org-version
#+begin_src elisp
(let ((a (org-version)) ; actual
(n orgstrap-min-org-version)) ; need
(or (fboundp #'orgstrap--confirm-eval) ; orgstrap with portable is already present on the system
(not n)
(string< n a)
(string= n a)
(error "Your Org is too old! %s < %s" a n)))
#+end_src
#+caption: Portability note.
#+begin_quote
=string<= must be used in order to support emacs-24
#+end_quote
** Normalization
:PROPERTIES:
:CUSTOM_ID: normalization
:END:
*** Shared normalization machinery
Shared normalization code embedded as elvs.
# FIXME should orgstrap-norm-func be a local variable ?!?
#+caption: Shared normalization code embedded as elvs.
#+name: orgstrap-normalization-common-embed
#+begin_src elisp
(unless (boundp 'orgstrap-norm-func)
(defvar-local orgstrap-norm-func orgstrap-norm-func-name))
(defun orgstrap-norm-embd (body)
"Normalize BODY."
(funcall orgstrap-norm-func body))
(unless (fboundp #'orgstrap-norm)
(defalias 'orgstrap-norm #'orgstrap-norm-embd))
#+end_src
Normalization functions for orgstrap.el.
#+caption: Normalization functions for orgstrap.el.
#+name: orgstrap-code-normalization-functions
#+begin_src elisp :eval never :noweb yes
(defun orgstrap-norm (body)
"Normalize BODY."
(if orgstrap--debug
(orgstrap-norm-debug body)
(funcall orgstrap-norm-func body)))
(defun orgstrap-norm-debug (body)
"Insert BODY normalized with NORM-FUNC into a buffer for easier debug."
(let* ((print-quoted nil)
(bname (format "body-norm-%s" emacs-major-version))
(buffer (let ((existing (get-buffer bname)))
(if existing existing
(create-file-buffer bname))))
(body-normalized (funcall orgstrap-norm-func body)))
(with-current-buffer buffer
(erase-buffer)
(insert body-normalized))
body-normalized))
;; orgstrap normalization functions
<>
<>
<>
#+end_src
#+caption: XXX portability note
#+begin_quote
For emacs < 26 (org < 9) either lowercase =#+caption:= must be placed
_BEFORE_ =#+name:=, OR =#+CAPTION:= must be uppercase and can come
after =#+name:=, otherwise =#+name:= will not be associated with the
block. What a fun bug.
Addendum. Apparently in the older version of Org =:noweb= is always
yes. As a result, testing against Emacs 24 or 25 will alert you if
you forget to set =:noweb= on a block.
#+end_quote
*** Normalization functions
:PROPERTIES:
:CUSTOM_ID: normalization-functions
:END:
**** prp-1.0 :obsolete:
*This normalization function is obsolete*
#+name: orgstrap-code-normalization--prin1-read-progn-1-0
#+begin_src elisp :eval never
(let ((print-quoted nil))
(prin1-to-string (read (concat "(progn\n" body "\n)"))))
#+end_src
#+name: block-orgstrap-norm-func--prp-1-0
#+begin_src elisp :noweb yes :eval never
(defun orgstrap-norm-func--prp-1-0 (body)
"Normalize BODY using prp-1-0."
<>)
(make-obsolete #'orgstrap-norm-func--prp-1-0 #'orgstrap-norm-func--prp-1-1 "1.2")
#+end_src
Normalize BODY by wrapping in =progn=, calling =read=, and then =prin1-to-string=.
There are still unresolved issues if tabs are present in the orgstrap block which
is why 1.0 is included. =print-quoted= is critical for consistent hashing.
=prin1-to-string= is used to normalize the code in the orgstrap block,
removing any comments and formatting irregularities. This is important
for two reasons.
First it helps prevent denial of service attacks against human auditors
who have low bandwidth for detecting fiddly changes.
Second, normalization that ignores comments makes it possible to improve
the documentation of code without changing the checksum. Hopefully this
will reduce one of the obstacles to enhancing the documentation of orgstrap
code and blocks over time since rehashing will not be required when the
meaningful code itself has not changed.
=(print-quoted nil)= is needed for backward compatibility due to a change
to the default from =nil= to =t= in emacs-27 (sigh). See
[[orgit-rev:~/git/NOFORK/emacs::72ee93d68daea00e2ee69417afd4e31b3145a9fa][emacs commit 72ee93d68daea00e2ee69417afd4e31b3145a9fa]].
**** prp-1.1
#+name: orgstrap-code-normalization--prin1-read-progn-1-1
#+begin_src elisp :eval never
(let (print-quoted print-length print-level)
(prin1-to-string (read (concat "(progn\n" body "\n)"))))
#+end_src
#+name: block-orgstrap-norm-func--prp-1-1
#+begin_src elisp :noweb yes :eval never
(defun orgstrap-norm-func--prp-1-1 (body)
"Normalize BODY using prp-1-1."
<>)
#+end_src
I learned that =print-length= and =print-level= exist in the usual
way, which is that somehow they got set to something other than =nil=
and as a result checksums started failing left and right because the
number of expressions in the body of the progn eval was greater than
the value of =print-length=, resulting in truncation and replacement
with =...=. This can also happens inside =add-file-local-variable= and
possibly even inside =format=!? Therefore I'm updating to version to
1.1 of the normalization procedure so that I can defensively bind
those variables to =nil=.
**** dprp-1.0
A normalization function that is invariant to changes in docstrings.
Walk the tree and ~setcdr~ out docstrings.
This normalization function must be portable between versions, which
means that the forms that get spliced must be from a static set that
does not change between versions.
Since this normalization is mostly a quality of life improvement to
allow docstrings to be changed without rehashing, limiting to a
specific set of forms is ok. If you use something like ~cl-defun~ and
change the docstring, then you will have to rehash. The 90% use case
is covered here in a compact manner.
#+name: orgstrap-code-normalization--dedoc-prin1-read-progn-1-0
#+begin_src elisp :eval never
(let ((p (read (concat "(progn\n" body "\n)")))
(m '(defun defun-local defmacro defvar defvar-local defconst defcustom))
print-quoted print-length print-level)
(cl-labels
((f
(b)
(cl-loop
for e in b when (listp e) do ; for expression in body when the expression is a list
(or
(and
(memq (car e) m) ; is a form with docstrings
(let ((n (nthcdr 4 e))) ; body after docstring
(and
(stringp (nth 3 e)) ; has a docstring
(or (cl-subseq m 3) n) ; var or doc not last
(f n) ; recurse for nested
;; splice out the docstring and return t to avoid the other branch
(or (setcdr (cddr e) n) t))))
;; recurse e.g. for (when x (defvar y t))
(f e)))
p))
(prin1-to-string (f p))))
#+end_src
#+name: block-orgstrap-norm-func--dprp-1-0
#+begin_src elisp :noweb yes :results none :eval never :eval yes
(defun orgstrap-norm-func--dprp-1-0 (body)
"Normalize BODY using dprp-1-0."
<>)
#+end_src
#+begin_src elisp :results code
(orgstrap--with-block "orgstrap"
(prog1 (read (orgstrap-norm-func--dprp-1-0 body)) nil))
#+end_src
** Definitions
:PROPERTIES:
:CUSTOM_ID: definitions
:END:
These blocks are nowebbed into ref:orgstrap-init-helper-defuns and are
used directly by =orgstrap-init= to populate file local variables.
The portable confirm eval is extracted to its own block so that we can
include it as a backstop for users who have orgstrap installed but are
running an older version of =org-mode= than is supported by the file
that they are trying to load.
#+caption: Portable confirm eval.
#+name: orgstrap-portable-confirm-eval
#+begin_src elisp :eval never :noweb yes
;;;###autoload
(defun orgstrap--confirm-eval-portable (lang _body)
"A backwards compatible, portable implementation for confirm-eval.
This should be called by `org-confirm-babel-evaluate'. As implemented
the only LANG that is supported is emacs-lisp or elisp. The argument
_BODY is rederived for portability and thus not used."
;; `org-confirm-babel-evaluate' will prompt the user when the value
;; that is returned is non-nil, therefore we negate positive matchs
(not (and (member lang '("elisp" "emacs-lisp"))
(let* ((body (orgstrap--expand-body (org-babel-get-src-block-info)))
(body-normalized (orgstrap-norm body))
(content-checksum
(intern
(secure-hash
orgstrap-cypher
body-normalized))))
;;(message "%s %s" orgstrap-block-checksum content-checksum)
;;(message "%s" body-normalized)
(eq orgstrap-block-checksum content-checksum)))))
;; portable eval is used as the default implementation in orgstrap.el
;;;###autoload
(unless (fboundp #'orgstrap--confirm-eval)
(defalias 'orgstrap--confirm-eval #'orgstrap--confirm-eval-portable))
#+end_src
#+caption: Minimal confirm eval.
#+name: orgstrap-minimal-confirm-eval
#+begin_src elisp
(defun orgstrap--confirm-eval-minimal (lang body)
(not (and (member lang '("elisp" "emacs-lisp"))
(eq orgstrap-block-checksum
(intern
(secure-hash
orgstrap-cypher
(orgstrap-norm body)))))))
(unless (fboundp #'orgstrap--confirm-eval)
;; if `orgstrap--confirm-eval' is bound use it since it is
;; is the portable version XXX NOTE the minimal version will
;; not be installed as local variables if it detects that there
;; are unescaped coderefs since those will cause portable and minimal
;; to produce different hashes
(defalias 'orgstrap--confirm-eval #'orgstrap--confirm-eval-minimal))
#+end_src
Once =orgstrap--confirm-eval= is defined the rest of the =eval:= local variables are the same.
# FIXME vc-find-file-hook aka vc-refresh-state uses find-file NOT
# find-file-literally the issue is in vc-link-follow which calls
# find-file-noselect blindly we would need to overwrite it or advise
# it, I have a workaround which is to set vc-follow-symlinks to nil
# for the shebang block, but that causes variant behavior, vc-mode
# probably needs a variable to control whether it opens in the mode
# that the symlinked file was opened in or the mode of the target to
# avoid issues like this
# XXX If you modify this block call =M-x orgstrap-run-block=
# so that functions that use it internally will be updated.
#+caption: common local variables
#+name: orgstrap-file-local-variables-common
#+begin_src elisp :eval never
(let (enable-local-eval) (vc-find-file-hook)) ; use the obsolete alias since it works in 24
(let ((ocbe org-confirm-babel-evaluate)
(obs (org-babel-find-named-block ,orgstrap-orgstrap-block-name))) ; quasiquoted when nowebbed
(if obs
(unwind-protect
(save-excursion
(setq-local orgstrap-norm-func orgstrap-norm-func-name)
(setq-local org-confirm-babel-evaluate #'orgstrap--confirm-eval)
(goto-char obs) ; FIXME `org-save-outline-visibility' but that is not portable
(org-babel-execute-src-block))
(when (eq org-confirm-babel-evaluate #'orgstrap--confirm-eval)
;; XXX allow orgstrap blocks to set ocbe so audit for that
(setq-local org-confirm-babel-evaluate ocbe))
(ignore-errors
(org-set-visibility-according-to-property)))
;; FIXME warn or error here?
(warn "No orgstrap block.")))
#+end_src
Since =orgstrap-norm-func= is a dynamic variable it simplifies the
potential future case where we don't embed the normalization function,
still not sure if we really want to do that though
#+caption: common local variables once lexical is enabled by default
#+name: orgstrap-file-local-variables-common-lexical
#+begin_src elisp :eval never
(let ((orgstrap-norm-func orgstrap-norm-func-name)
(org-confirm-babel-evaluate #'orgstrap--confirm-eval)
(obs (org-babel-find-named-block ,orgstrap-orgstrap-block-name))) ; quasiquoted when nowebbed
(if obs
(unwind-protect
(save-excursion
(goto-char obs)
(org-babel-execute-src-block))
(org-set-startup-visibility))
;; FIXME warn or error here?
(warn "No orgstrap block.")))
#+end_src
** Note on noweb support
:PROPERTIES:
:CUSTOM_ID: note-on-noweb-support
:END:
The minimal set of local variables only works if you don't use noweb
or if you are using Org =>== =9.3.8=.
The portable set of local variables described below works with versions of
Org as far back as =8.2.10= (the version bundled with =emacs-24.5=).
** Note on coderefs
:PROPERTIES:
:CUSTOM_ID: note-on-coderefs
:END:
Older versions of =org-mode= do not know what to do with coderefs.
The simplest solution is to hide them in comments as =;(ref:coderef)=
if you need them. See [[(clrin)]] and [[(oab)]] for examples in this file.
** How local variables appear in the file
:PROPERTIES:
:CUSTOM_ID: how-local-variables-appear-in-the-file
:END:
# DO NOT EDIT THESE BLOCKS THEY ARE UPDATED AUTOMATICALLY
Here is the prop line from the first line of this file that
includes the cypher and checksum of the =orgstrap= block.
#+name: local-variables-prop-line-example
#+begin_src org :eval never
# -*- org-adapt-indentation: nil; org-edit-src-content-indentation: 0; orgstrap-cypher: sha256; orgstrap-norm-func-name: orgstrap-norm-func--dprp-1-0; orgstrap-block-checksum: d33bdc8924478fedbd92ff73836c43e136d90e4c18393ff7c5e0aeda37f892d2; -*-
#+end_src
# BE VERY CAREFUL WITH MANUAL EDITS
# If this block is being edited manually the automatic update will not work.
Here are the portable local variables from the end of the file.
#+name: local-variables-portable-example
#+begin_src org :eval never
# Local Variables:
# eval: (progn (setq-local orgstrap-min-org-version "8.2.10") (let ((a (org-version)) (n orgstrap-min-org-version)) (or (fboundp #'orgstrap--confirm-eval) (not n) (string< n a) (string= n a) (error "Your Org is too old! %s < %s" a n))) (defun orgstrap-norm-func--dprp-1-0 (body) (let ((p (read (concat "(progn\n" body "\n)"))) (m '(defun defun-local defmacro defvar defvar-local defconst defcustom)) print-quoted print-length print-level) (cl-labels ((f (b) (cl-loop for e in b when (listp e) do (or (and (memq (car e) m) (let ((n (nthcdr 4 e))) (and (stringp (nth 3 e)) (or (cl-subseq m 3) n) (f n) (or (setcdr (cddr e) n) t)))) (f e))) p)) (prin1-to-string (f p))))) (unless (boundp 'orgstrap-norm-func) (defvar-local orgstrap-norm-func orgstrap-norm-func-name)) (defun orgstrap-norm-embd (body) (funcall orgstrap-norm-func body)) (unless (fboundp #'orgstrap-norm) (defalias 'orgstrap-norm #'orgstrap-norm-embd)) (defun orgstrap-org-src-coderef-regexp (_fmt &optional label) (let ((fmt org-coderef-label-format)) (format "\\([:blank:]*\\(%s\\)[:blank:]*\\)$" (replace-regexp-in-string "%s" (if label (regexp-quote label) "\\([-a-zA-Z0-9_][-a-zA-Z0-9_ ]*\\)") (regexp-quote fmt) nil t)))) (unless (fboundp #'org-src-coderef-regexp) (defalias 'org-src-coderef-regexp #'orgstrap-org-src-coderef-regexp)) (defun orgstrap--expand-body (info) (let ((coderef (nth 6 info)) (expand (if (org-babel-noweb-p (nth 2 info) :eval) (org-babel-expand-noweb-references info) (nth 1 info)))) (if (not coderef) expand (replace-regexp-in-string (org-src-coderef-regexp coderef) "" expand nil nil 1)))) (defun orgstrap--confirm-eval-portable (lang _body) (not (and (member lang '("elisp" "emacs-lisp")) (let* ((body (orgstrap--expand-body (org-babel-get-src-block-info))) (body-normalized (orgstrap-norm body)) (content-checksum (intern (secure-hash orgstrap-cypher body-normalized)))) (eq orgstrap-block-checksum content-checksum))))) (unless (fboundp #'orgstrap--confirm-eval) (defalias 'orgstrap--confirm-eval #'orgstrap--confirm-eval-portable)) (let (enable-local-eval) (vc-find-file-hook)) (let ((ocbe org-confirm-babel-evaluate) (obs (org-babel-find-named-block "orgstrap"))) (if obs (unwind-protect (save-excursion (setq-local orgstrap-norm-func orgstrap-norm-func-name) (setq-local org-confirm-babel-evaluate #'orgstrap--confirm-eval) (goto-char obs) (org-babel-execute-src-block)) (when (eq org-confirm-babel-evaluate #'orgstrap--confirm-eval) (setq-local org-confirm-babel-evaluate ocbe)) (ignore-errors (org-set-visibility-according-to-property))) (warn "No orgstrap block."))))
# End:
#+end_src
Here are the minimal local variables from the end of the file.
#+name: local-variables-minimal-example
#+begin_src org :eval never
# Local Variables:
# eval: (progn (setq-local orgstrap-min-org-version "8.2.10") (let ((a (org-version)) (n orgstrap-min-org-version)) (or (fboundp #'orgstrap--confirm-eval) (not n) (string< n a) (string= n a) (error "Your Org is too old! %s < %s" a n))) (defun orgstrap-norm-func--dprp-1-0 (body) (let ((p (read (concat "(progn\n" body "\n)"))) (m '(defun defun-local defmacro defvar defvar-local defconst defcustom)) print-quoted print-length print-level) (cl-labels ((f (b) (cl-loop for e in b when (listp e) do (or (and (memq (car e) m) (let ((n (nthcdr 4 e))) (and (stringp (nth 3 e)) (or (cl-subseq m 3) n) (f n) (or (setcdr (cddr e) n) t)))) (f e))) p)) (prin1-to-string (f p))))) (unless (boundp 'orgstrap-norm-func) (defvar-local orgstrap-norm-func orgstrap-norm-func-name)) (defun orgstrap-norm-embd (body) (funcall orgstrap-norm-func body)) (unless (fboundp #'orgstrap-norm) (defalias 'orgstrap-norm #'orgstrap-norm-embd)) (defun orgstrap--confirm-eval-minimal (lang body) (not (and (member lang '("elisp" "emacs-lisp")) (eq orgstrap-block-checksum (intern (secure-hash orgstrap-cypher (orgstrap-norm body))))))) (unless (fboundp #'orgstrap--confirm-eval) (defalias 'orgstrap--confirm-eval #'orgstrap--confirm-eval-minimal)) (let (enable-local-eval) (vc-find-file-hook)) (let ((ocbe org-confirm-babel-evaluate) (obs (org-babel-find-named-block "orgstrap"))) (if obs (unwind-protect (save-excursion (setq-local orgstrap-norm-func orgstrap-norm-func-name) (setq-local org-confirm-babel-evaluate #'orgstrap--confirm-eval) (goto-char obs) (org-babel-execute-src-block)) (when (eq org-confirm-babel-evaluate #'orgstrap--confirm-eval) (setq-local org-confirm-babel-evaluate ocbe)) (ignore-errors (org-set-visibility-according-to-property))) (warn "No orgstrap block."))))
# End:
#+end_src
* Code
:PROPERTIES:
:CUSTOM_ID: code
:END:
** =orgstrap= implementation
This section contains the implementation of functions to calculate
=orgstrap-block-checksum= and set it as a prop line local variable.
It also contains functions to embed the bootstrapping code as an
=eval:= local variable in the local variables list, along with other
quality of life functionality for the user such as =orgstrap-mode=,
=orgstrap-edit-mode=, and =orgstrap-init=.
# [[info:elisp#File Local Variables][info:elisp#File Local Variables]] is a useful reference
*** Expand
Testing =org-src-coderef-regexp= with =fboundp= in ref:orgstrap-expand-body
is needed due to changes in the behavior of =org-babel-get-src-block-info=
roughly around the =9.0= release.
The changes in behavior for =org-babel-get-src-block-info= are commits
orgit-rev:~/git/NOFORK/org-mode::88659208793dca18b7672428175e9a712af7b5ad and
orgit-rev:~/git/NOFORK/org-mode::9738da473277712804e0d004899388ad71c6b791. They
both occur before the introduction of =org-src-coderef-regexp= in
orgit-rev:~/git/NOFORK/org-mode::9f47b37231b3c45afcd604a191e346200bd76e98.
All of this happend before orgit-rev:~/git/NOFORK/org-mode::release_9.0. By
testing =org-src-coderef-regexp= with =fboundp= there are only a tiny number
of versions where there might be some inconsistent behavior, e.g.
orgit-rev:~/git/NOFORK/org-mode::release_8.3.6, but I suspect that the probability
that anyone anywhere is running one of those versions is approximately zero.
#+name: orgstrap-expand-body
#+begin_src elisp :eval never
(defun orgstrap-org-src-coderef-regexp (_fmt &optional label)
"Backport `org-src-coderef-regexp' for 24 and 25.
See the upstream docstring for info on LABEL.
_FMT has the wrong meaning in 24 and 25."
(let ((fmt org-coderef-label-format))
(format "\\([:blank:]*\\(%s\\)[:blank:]*\\)$"
(replace-regexp-in-string
"%s"
(if label
(regexp-quote label)
"\\([-a-zA-Z0-9_][-a-zA-Z0-9_ ]*\\)")
(regexp-quote fmt)
nil t))))
(unless (fboundp #'org-src-coderef-regexp)
(defalias 'org-src-coderef-regexp #'orgstrap-org-src-coderef-regexp))
(defun orgstrap--expand-body (info)
"Expand noweb references in INFO body and remove any coderefs."
;; this is a backport of `org-babel--expand-body'
(let ((coderef (nth 6 info))
(expand
(if (org-babel-noweb-p (nth 2 info) :eval)
(org-babel-expand-noweb-references info)
(nth 1 info))))
(if (not coderef)
expand
(replace-regexp-in-string
(org-src-coderef-regexp coderef) "" expand nil nil 1))))
#+end_src
*** Run
In order for orgstrap to be maximally portable and not depend on
already being installed, the implementation needs to work with the
local variables list eval variable without complicating the situation
when orgstrap is installed as a package.
While ideally this would be done using only the standard hooks around
=hack-local-variables= such an approach does not work because the
variables are filtered before those hooks can run. Therefore, we have
to advise =hack-local-variables-confirm= in order to capture and
remove any orgstrap elvs that we find. For maximum safety this
minimally requires mutation of the =all-vars= list passed to
=hack-local-variables-confirm=.
This is a fairly deep tampering with the way that hack-local-variables works,
so special attention should be given when reviewing the security implications
of any changes.
#+caption: run helpers
#+name: orgstrap-run-helper-defuns
#+begin_src elisp :noweb yes
(require 'cl-lib)
;;;###autoload
(defvar orgstrap-mode nil
"Variable to track whether `orgstrap-mode' is enabled.")
(cl-eval-when (eval compile load)
;; prevent warnings since this is used as a variable in a macro
(defvar orgstrap-orgstrap-block-name "orgstrap"
"Set the default blockname to orgstrap by convention.
This makes it easier to search for orgstrap if someone encounters
an orgstrapped file and wants to know what is going on."))
(defvar orgstrap-default-cypher 'sha256
"The default cypher passed to `secure-hash' when hashing blocks.")
(defvar-local orgstrap-cypher orgstrap-default-cypher
"Local variable for the cypher for the current buffer.
If you change `orgstrap-default-cypher' you should update this as well
using `setq-default' since it will not change automatically.")
(put 'orgstrap-cypher 'safe-local-variable (lambda (v) (ignore v) t))
(defvar-local orgstrap-block-checksum nil
"Local variable for the expected checksum for the current orgstrap block.")
;; `orgstrap-block-checksum' is not a safe local variable, if it is set
;; as safe then there will be no check and code will execute without a check
;; it is also not risky, so we leave it unmarked
(defconst orgstrap--internal-norm-funcs
'(orgstrap-norm-func--prp-1-0
orgstrap-norm-func--prp-1-1
orgstrap-norm-func--dprp-1-0)
"List internally implemented normalization functions.
Used to determine which norm func names are safe local variables.")
(defvar-local orgstrap-norm-func-name nil
"Local variable for the name of the current orgstrap-norm-func.")
(put 'orgstrap-norm-func-name 'safe-local-variable
(lambda (value) (and orgstrap-mode (memq value orgstrap--internal-norm-funcs))))
;; Unless `orgstrap-mode' is enabled and the name is in the list of
;; functions that are implemented internally this is not safe
(defvar-local orgstrap-norm-func #'orgstrap-norm-func--dprp-1-0
"Dynamic variable to simplify calling normalizaiton functions.
Defaults to `orgstrap-norm-func--dprp-1-0'.")
(defvar orgstrap--debug nil
"If non-nil run `orgstrap-norm' in debug mode.")
(defgroup orgstrap nil
"Tools for bootstraping Org mode files using Org Babel."
:tag "orgstrap"
:group 'org
:link '(url-link :tag "README on GitHub"
"https://github.com/tgbugs/orgstrap/blob/master/README.org"))
(defcustom orgstrap-always-edit nil
"If non-nil command `orgstrap-mode' activates command `orgstrap-edit-mode'."
:type 'boolean
:group 'orgstrap)
(defcustom orgstrap-always-eval nil
"Always try to run orgstrap blocks even when populating `org-agenda'."
:type 'boolean
:group 'orgstrap)
(defcustom orgstrap-always-eval-whitelist nil
"List of files that should always try to run orgstrap blocks."
:type 'list
:group 'orgstrap)
(defcustom orgstrap-file-blacklist nil
"List of files that should never run orgstrap blocks.
For files on the blacklist `orgstrap-block-checksum' is removed from
the local variables list so that the checksum will not be added to
the `safe-local-variable-values' list. If it were added it would then
be impossible to prevent execution of the source block when `orgstrap-mode'
is disabled.
This is useful when developing a block that modifies Emacs' configuration.
NOTE this variable only works if `orgstrap-mode' is enabled."
:type 'list
:group 'orgstrap)
;; orgstrap blacklist
(defun orgstrap-blacklist-current-file (&optional universal-argument)
"Add the current file to `orgstrap-file-blacklist'.
If UNIVERSAL-ARGUMENT is provided do not run `orgstrap-revoke-current-buffer'."
;; It is usually better to revoke a checksum when its file is blacklisted since
;; it is easier for the user to add the checksum again when needed than it is
;; for them to revoke manually. The prefix argument allows users who know that
;; they only want to blacklist the file and not revoke to do so though such
;; cases are expected to be fairly rare.
;; FIXME blacklisting a bad file that has already been approved is painful
;; right now, you have to manually set `enable-local-eval' to nil, load the
;; file, run this function, and then reset `enable-local-eval'.
(interactive "P")
(unless universal-argument
(orgstrap-revoke-current-buffer))
(add-to-list 'orgstrap-file-blacklist (buffer-file-name))
(customize-save-variable 'orgstrap-file-blacklist orgstrap-file-blacklist))
(defun orgstrap-unblacklist-current-file ()
"Remove the current file from `orgstrap-file-blacklist'."
(interactive)
(setq orgstrap-file-blacklist (delete (buffer-file-name) orgstrap-file-blacklist))
(customize-save-variable 'orgstrap-file-blacklist orgstrap-file-blacklist))
;; orgstrap revoke
(defun orgstrap-revoke-checksums (&rest checksums)
"Delete CHECKSUMS or all checksums if nil from `safe-local-variables-values'."
(interactive)
(cl-delete-if (lambda (pair)
(cl-destructuring-bind (key . value)
pair
(and
(eq key 'orgstrap-block-checksum)
(or (null checksums) (memq value checksums)))))
safe-local-variable-values)
(customize-save-variable 'safe-local-variable-values safe-local-variable-values))
(defun orgstrap-revoke-current-buffer ()
"Delete checksum(s) for current buffer from `safe-local-variable-values'.
Deletes embedded and current values of `orgstrap-block-checksum'."
(interactive)
(let* ((elv (orgstrap--read-current-local-variables))
(cpair (assoc 'orgstrap-block-checksum elv))
(checksum-existing (and cpair (cdr cpair))))
(orgstrap-revoke-checksums orgstrap-block-checksum checksum-existing)))
(defun orgstrap-revoke-elvs ()
"Delete all approved orgstrap elvs from `safe-local-variable-values'."
(interactive)
(cl-delete-if #'orgstrap--match-elvs safe-local-variable-values)
(customize-save-variable 'safe-local-variable-values safe-local-variable-values))
(define-obsolete-function-alias
'orgstrap-revoke-eval-local-variables
#'orgstrap-revoke-elvs
"1.2.4"
"Replaced by the more compact `orgstrap-revoke-elvs'.")
;; orgstrap run helpers
<>
;; orgstrap-mode implementation
(defun orgstrap--org-buffer ()
"Only run when in `org-mode' and command `orgstrap-mode' is enabled.
Sets further hooks."
(when enable-local-eval
;; if `enable-local-eval' is nil we honor it and will not run
;; orgstrap blocks natively, this matches the behavior of the
;; embedded elvs and simplifies logic for cases
;; where orgstrap should not run (e.g. when populating `org-agenda')
(advice-add #'hack-local-variables-confirm :around #'orgstrap--hack-lv-confirm)
(unless (member (buffer-file-name) orgstrap-file-blacklist)
(add-hook 'before-hack-local-variables-hook #'orgstrap--before-hack-lv nil t))))
(defun orgstrap--hack-lv-confirm (command &rest args)
"Advise `hack-local-variables-confirm' to remove orgstrap eval variables.
COMMAND should be `hack-local-variables-confirm' with ARGS (all-vars
unsafe-vars risky-vars dir-name)."
(advice-remove #'hack-local-variables-confirm #'orgstrap--hack-lv-confirm)
(cl-destructuring-bind (all-vars unsafe-vars risky-vars dir-name)
(cl-loop
for arg in
(if (member (buffer-file-name) orgstrap-file-blacklist)
(cl-loop ; zap checksums for blacklisted
for arg in args collect
(if (listp arg)
(cl-delete-if
(lambda (pair) (eq (car pair) 'orgstrap-block-checksum))
arg)
arg))
args)
collect ; use `cl-delete-if' to mutate the lists in calling scope
(if (listp arg) (cl-delete-if #'orgstrap--match-elvs arg) arg))
;; After removal we have to recheck to see if unsafe-vars and
;; risky-vars are empty so we can skip the confirm dialogue. If we
;; do not, then the dialogue breaks the flow.
(or (and (null unsafe-vars)
(null risky-vars))
(funcall command all-vars unsafe-vars risky-vars dir-name))))
(defun orgstrap--before-hack-lv ()
"If `orgstrap' is in the current buffer, add hook to run the orgstrap block."
;; This approach is safer than trying to introspect some of the implementation
;; internals. This hook will only run if there are actually local variables to
;; hack, so there is little to no chance of lingering hooks if an error occures
(remove-hook 'before-hack-local-variables-hook #'orgstrap--before-hack-lv t)
;; XXX we have to remove elvs here since `hack-local-variables-confirm' is not called
;; if all variables are marked as safe, e.g. via `orgstrap-whitelist-file'
;; FIXME other interactions between blacklist and whitelist may need to be handled here
(setq file-local-variables-alist (cl-delete-if #'orgstrap--match-elvs file-local-variables-alist))
(add-hook 'hack-local-variables-hook #'orgstrap--hack-lv nil t))
(defun orgstrap--used-in-current-buffer-p ()
"Return t if all the required orgstrap prop line local variables are present."
(and (boundp 'orgstrap-cypher) orgstrap-cypher
(boundp 'orgstrap-block-checksum) orgstrap-block-checksum
(boundp 'orgstrap-norm-func-name) orgstrap-norm-func-name))
(defmacro orgstrap--lv-common-with-block-name ()
"Helper macro to allow use of same code between core and lv impls."
`(progn
<>))
(defun orgstrap--hack-lv ()
"If orgstrap is present, run the orgstrap block for the current buffer."
;; we remove this hook here and we do not have to worry because
;; it is always added by `orgstrap--before-hack-lv'
(remove-hook 'hack-local-variables-hook #'orgstrap--hack-lv t)
(when (orgstrap--used-in-current-buffer-p)
(orgstrap--lv-common-with-block-name)
(when orgstrap-always-edit
(orgstrap-edit-mode 1))))
(defun orgstrap--match-elvs (pair)
"Return nil if PAIR matchs any elv used by orgstrap.
Avoid false positives if possible if at all possible."
(and (eq (car pair) 'eval)
;;(message "%s" (cdr pair))
;; keep the detection simple for now, any eval lv that
;; so much as mentions orgstrap is nuked, and in the future
;; if orgstrap-nb is used we may need to nuke that too
(string-match "orgstrap" (prin1-to-string (cdr pair)))))
;;;###autoload
(defun orgstrap-mode (&optional arg)
"A regional minor mode for `org-mode' that automatically runs orgstrap blocks.
When visiting an Org file or activating `org-mode', if orgstrap prop line local
variables are detect then use the installed orgstrap implementation to run the
orgstrap block. If orgstrap embedded local variables are present, they will not
be executed. `orgstrap-mode' is not a normal minor mode since it does not run
any hooks and when enabled only adds a function to `org-mode-hook'. ARG is the
universal prefix argument."
(interactive "P")
(ignore arg)
(let ((turn-on (not orgstrap-mode)))
(cond (turn-on
(add-hook 'org-mode-hook #'orgstrap--org-buffer)
(setq orgstrap-mode t)
(message "orgstrap-mode enabled"))
(arg) ; orgstrap-mode already enabled so don't disable it
(t
(remove-hook 'org-mode-hook #'orgstrap--org-buffer)
(setq orgstrap-mode nil)
(message "orgstrap-mode disabled")))))
;; orgstrap do not run aka `org-agenda' eval protection
(defun orgstrap--advise-no-eval-lv (command &rest args)
"Advise COMMAND to disable elvs for files loaded inside it.
ARGS vary by COMMAND.
If the elvs are disabled then `orgstrap-block-checksum' is added
to the `ignored-local-variables' list for files loaded inside
COMMAND. This makes it possible to open orgstrapped files where
the elvs will not run without having to accept the irrelevant
variable for `orgstrap-block-checksum'."
;; continually prompting users to accept a local variable when they
;; cannot inspect the file and when accidentally accepting could
;; allow unchecked execution at some point in the future is bad
;; better to simply pretend that the elvs and the block checksum
;; do not even exist unless the file is explicitly on a whitelist
;; orgstrapped files are just plain old org files in this context
;; since agenda doesn't use any babel functionality ... of course
;; I can totally imagine using orgstrap to automatically populate
;; an org file or update an org file using orgstrap to keep the
;; agenda in sync with some external source ... so need a variable
;; to control this
(if orgstrap-always-eval
(apply command args)
(let* ((enable-local-eval
(and args
orgstrap-always-eval-whitelist
(member (car args)
orgstrap-always-eval-whitelist)
enable-local-eval))
(ignored-local-variables
(if enable-local-eval ignored-local-variables
(cons 'orgstrap-block-checksum ignored-local-variables))))
(apply command args))))
(advice-add #'org-get-agenda-file-buffer :around #'orgstrap--advise-no-eval-lv)
#+end_src
*** Edit
#+caption: edit helpers
#+name: orgstrap-edit-helper-defuns
#+begin_src emacs-lisp :results none :lexical yes :noweb yes
;;; edit helpers
(defvar orgstrap--clone-stamp-source-buffer-block nil
"Source code buffer and block for `orgstrap-stamp'.")
(defcustom orgstrap-on-change-hook nil
"Hook run via `before-save-hook' when command `orgstrap-edit-mode' is enabled.
Only runs when the contents of the orgstrap block have changed."
:type 'hook
:group 'orgstrap)
(defcustom orgstrap-use-minimal-local-variables nil
"Set whether minimal, smaller but less portable variables are used.
If nil then backward compatible local variables are used instead.
If the value is customized to be non-nil then compact local variables
are used and `orgstrap-min-org-version' is set accordingly. If the
current version of org mode does not support the features required to
use the minimal variables then the portable variables are used instead."
:type 'boolean
:group 'orgstrap)
;; edit utility functions
(defun orgstrap--current-buffer-cypher ()
"Return the cypher used for the current buffer.
The value is `orgstrap-cypher' if it is bound otherwise
`orgstrap-default-cypher' is returned."
(if (boundp 'orgstrap-cypher) orgstrap-cypher orgstrap-default-cypher))
<>
<>
(defun orgstrap--goto-named-src-block (blockname)
"Goto org block named BLOCKNAME.
Like `org-babel-goto-named-src-block' but non-interactive, does
not use the mark ring, and errors if the block is not found."
(let ((obs (org-babel-find-named-block blockname)))
(if obs (goto-char obs)
(error "No block named %s" blockname))))
(defmacro orgstrap--with-block (blockname &rest macro-body)
"Go to the source block named BLOCKNAME and execute MACRO-BODY.
The macro provides local bindings for four names:
`info', `params', `body-unexpanded', and `body'."
(declare (indent defun))
`(save-excursion
(let* ((info
(org-save-outline-visibility 'use-markers
(orgstrap--goto-named-src-block ,blockname)
(org-babel-get-src-block-info)))
(params (nth 2 info))
(body-unexpanded (nth 1 info))
(body (orgstrap--expand-body info)))
,@macro-body)))
(defun orgstrap--update-on-change ()
"Run via the `before-save-hook' local variable.
Test if the checksum of the orgstrap block has changed,
if so update the `orgstrap-block-checksum' local variable
and then run `orgstrap-on-change-hook'."
(let* ((elv (orgstrap--read-current-local-variables))
(cpair (assoc 'orgstrap-block-checksum elv))
(checksum-existing (and cpair (cdr cpair)))
(checksum (orgstrap-get-block-checksum)))
(unless (eq checksum-existing (intern checksum))
(remove-hook 'before-save-hook #'orgstrap--update-on-change t)
;; for some reason tangling from a buffer counts as saving from that buffer
;; so have to remove the hook to avoid infinite loop
(unwind-protect
(save-excursion
(undo)
(undo-boundary) ; insert an undo boundary so that the
;; changes to the checksum are transparent to the user
(undo) ; undo the undo above
(orgstrap-add-block-checksum nil checksum)
(run-hooks 'orgstrap-on-change-hook))
(add-hook 'before-save-hook #'orgstrap--update-on-change nil t)))))
(defun orgstrap--get-actual-params (params)
"Filter defaults, nulls, and junk from src block PARAMS."
(let ((defaults (append org-babel-default-header-args
org-babel-default-header-args:emacs-lisp)))
(cl-remove-if (lambda (pair)
(or (member pair defaults)
(memq (car pair) '(:result-params :result-type))
(null (cdr pair))))
params)))
(defun orgstrap-header-source-element (header-name &optional block-name &rest more-names)
"Given HEADER-NAME find the element that provides its value.
If BLOCK-NAME is non-nil then search for headers for that block,
otherwise search for headers associated with the current block.
If MORE-NAMES are provided return the value for each (or nil)."
;; get the current headers, see if the value is set anywhere
;; or if it is default, search for default anyway just to be sure
;; return nil if not found
;; when searching for any header go to the end of the src line
;; `re-search-backward' from that point for :header-arg but not
;; going beyond the affiliated keywords for the current element
;; (if you can get affiliated keywords for the current element
;; that might simplify the search as well? check the impl for how
;; the actual values are obtained during execution etc)
;; when found use `org-element-at-point' to obtain the element
;; in another function the operates on the element
;; the element will give start, end, value, etc.
;; find bounds of value from element or sub element
;; delete the value, replace with new value
(ignore header-name block-name more-names)
(error "Not implemented TODO"))
(defun orgstrap-update-src-block-header (name new-params &optional update)
"Add header arguments to block NAME from NEW-PARAMS from some other block.
Existing header arguments will NOT be removed if they are not included in
NEW-PARAMS. If UPDATE is non-nil existing header arguments are updated."
(let ((new-act-params (orgstrap--get-actual-params new-params)))
(orgstrap--with-block name
(ignore body body-unexpanded)
(let ((existing-act-params (orgstrap--get-actual-params params)))
(dolist (pair new-act-params)
(cl-destructuring-bind (key . value)
pair
(let ((header-arg (substring (symbol-name key) 1)))
(if (assq key existing-act-params)
(if update
(unless (member pair existing-act-params)
;; TODO remove existing
;; `org-babel-insert-header-arg' does not remove
;; and it is not trivial to find the actual location
;; of an existing header argument there are 4 places
;; that we will have to look and then in some cases
;; we will have to append even if we do find them
(org-babel-insert-header-arg header-arg value)
;; This message works around the fact that we don't
;; have replace here, only append TODO consider
;; changing the way update works to be nil, replace,
;; or append once an in-place replace is implemented
(message "%s superseded for block %s." key name))
(warn "%s already defined for block %s!" key name))
(org-babel-insert-header-arg header-arg value)))))))))
(unless (fboundp #'flatten-tree)
;; backwards compatibility for Emacs < 27
(defun flatten-tree (tree)
(let (elems)
(while (consp tree)
(let ((elem (pop tree)))
(while (consp elem)
(push (cdr elem) tree)
(setq elem (car elem)))
(if elem (push elem elems))))
(if tree (push tree elems))
(nreverse elems))))
(defun orgstrap--check-portable-subset (body)
"Ensure that BODY uses only symbols that are portable for `prin1-to-string'."
;; XXX Note that [.] may diverge again because `.asdf' can be read without
;; escaping the leading . whereas `\?asdf' cannot. This is an artifact of
;; the c implementation of `prin1' being reused to handle both chars.
(let* ((l (flatten-tree (read (concat "(progn\n" body "\n)"))))
(symbols (cl-remove-duplicates (cl-remove-if-not #'symbolp l)))
(bads (cl-remove-if-not
(lambda (s) (string-match "[.?]" (cl-subseq (symbol-name s) 1)))
symbols)))
(when bads
(error "checksum failed: non-portable symbols detected: %S" bads))))
;; edit user facing functions
(defun orgstrap-get-block-checksum (&optional cypher)
"Calculate the `orgstrap-block-checksum' for the current buffer using CYPHER."
(interactive)
(orgstrap--with-block orgstrap-orgstrap-block-name
(ignore params body-unexpanded)
(orgstrap--check-portable-subset body)
(let ((cypher (or cypher (orgstrap--current-buffer-cypher)))
(body-normalized (orgstrap-norm body)))
(secure-hash cypher body-normalized))))
(defun orgstrap-add-block-checksum (&optional cypher checksum)
"Add `orgstrap-block-checksum' to file local variables of `current-buffer'.
The optional CYPHER argument should almost never be used,
instead change the value of `orgstrap-default-cypher' or manually
change the file property line variable. CHECKSUM can be passed
directly if it has been calculated before and only needs to be set.
If `orgstrap-save-developer-checksums' is non-nil then add the checksum to
`orsgrap-developer-checksums'."
(interactive)
(let* ((cypher (or cypher (orgstrap--current-buffer-cypher)))
(orgstrap-block-checksum (or checksum (orgstrap-get-block-checksum cypher))))
(when orgstrap-block-checksum
(save-excursion
(add-file-local-variable-prop-line 'orgstrap-cypher cypher)
(add-file-local-variable-prop-line 'orgstrap-norm-func-name orgstrap-norm-func)
(add-file-local-variable-prop-line 'orgstrap-block-checksum (intern orgstrap-block-checksum)))
(when orgstrap-save-developer-checksums
(add-to-list 'orgstrap-developer-checksums (intern orgstrap-block-checksum))))
orgstrap-block-checksum))
(defun orgstrap-run-block ()
"Evaluate the orgstrap block for the current buffer."
;; bind to :orb or something like that
(interactive)
(save-excursion
(orgstrap--goto-named-src-block orgstrap-orgstrap-block-name)
(org-babel-execute-src-block)))
(defun orgstrap-clone (&optional universal-argument)
"Set current block or orgstrap block as the source for `orgstrap-stamp'.
If a UNIVERSAL-ARGUMENT is supplied then the orgstrap block is always used."
;; TODO consider whether to avoid the inversion of behavior around C-u
;; namely that nil -> always from orgstrap block, C-u -> current block
;; this would avoid confusion where unprefixed could produce both
;; behaviors and only switch when already on a src block
(interactive "P")
(let ((current-element (org-element-at-point))
(current-buffer (current-buffer)))
(if (and (eq (org-element-type current-element) 'src-block)
(not universal-argument))
(let ((block-name (org-element-property :name current-element)))
(if block-name
(setq orgstrap--clone-stamp-source-buffer-block
(cons current-buffer block-name))
(warn "The current block has no name, it cannot be a clone source!")))
(if (orgstrap--used-in-current-buffer-p)
(setq orgstrap--clone-stamp-source-buffer-block
(cons current-buffer orgstrap-orgstrap-block-name))
(warn "orgstrap is not used in the current buffer!")))))
(defun orgstrap-stamp (&optional universal-argument overwrite)
"Stamp orgstrap block via `orgstrap-clone' to current buffer.
If UNIVERSAL-ARGUMENT is \\='(16) aka (C-u C-u) this will OVERWRITE any existing
block. If you are not calling this interactively all as (orgstrap-stamp nil t)
for calirty. You cannot stamp an orgstrap block into its own buffer."
(interactive "P")
(unless (eq major-mode 'org-mode)
(user-error "`orgstrap-stamp' only works in org-mode buffers"))
(unless orgstrap--clone-stamp-source-buffer-block
(user-error "No value to clone! Use `orgstrap-clone' first"))
(let ((overwrite (or overwrite (equal universal-argument '(16))))
(source-buffer (car orgstrap--clone-stamp-source-buffer-block))
(source-block-name (cdr orgstrap--clone-stamp-source-buffer-block))
(target-buffer (current-buffer)))
(when (eq source-buffer target-buffer)
(error "Source and target are the same buffer. Not stamping!"))
(cl-destructuring-bind (source-body
source-params
org-adapt-indentation
org-edit-src-content-indentation)
(save-window-excursion
(with-current-buffer source-buffer
(orgstrap--with-block source-block-name
(ignore body-unexpanded)
(list body
params
org-adapt-indentation
org-edit-src-content-indentation))))
(if (and (not overwrite)
(member orgstrap-orgstrap-block-name
(org-babel-src-block-names)))
(warn "orgstrap block already exists not stamping!")
(orgstrap--add-orgstrap-block source-body) ; FIXME somehow the hash is different !?!??!
(orgstrap-update-src-block-header orgstrap-orgstrap-block-name source-params t)
(orgstrap-add-block-checksum) ; I think it is correct to add the checksum here
(message "Stamped orgsrap block from %s" (buffer-file-name source-buffer))))))
;;;###autoload
(define-minor-mode orgstrap-edit-mode
"Minor mode for editing with orgstrapped files."
:init-value nil :lighter "" :keymap nil
(unless (eq major-mode 'org-mode)
(setq orgstrap-edit-mode 0)
(user-error "`orgstrap-edit-mode' only works with org-mode buffers"))
(cond (orgstrap-edit-mode
(add-hook 'before-save-hook #'orgstrap--update-on-change nil t))
(t
(remove-hook 'before-save-hook #'orgstrap--update-on-change t))))
#+end_src
# orgstrap-embed-normalization-code
# is a potential future variable but for sanity
# I am leaving it out for now because it is easier
# to have a rule that says "always use orgstrap-embedded-norm-func"
# and then we don't have to wonder about it, the size tradeoff can
# be made by the user based on their use case
*** Dev
#+caption: dev helpers
#+name: orgstrap-dev-helper-defuns
#+begin_src elisp
;;; dev helpers
(defcustom orgstrap-developer-checksums-file (concat user-emacs-directory "orgstrap-developer-checksums.el")
"Path to developer checksums file."
:type 'path
:group 'orgstrap)
(defcustom orgstrap-save-developer-checksums nil ; FIXME naming
"Whether or not to save checksums of orgstrap blocks under development."
:type 'boolean
:group 'orgstrap
:set (lambda (variable value)
(set-default variable value)
(if value
(add-hook 'orgstrap-on-change-hook #'orgstrap-save-developer-checksums)
(remove-hook 'orgstrap-on-change-hook #'orgstrap-save-developer-checksums))))
(defvar orgstrap-developer-checksums nil ; not custom because it is saved elsewhere
"List of checksums for orgstrap blocks created or modified by the user.")
(defun orgstrap--pp-to-string (value)
"Ensure that we actually print the whole VALUE not just the summarized subset."
(let (print-level print-length)
(pp-to-string value)))
(defun orgstrap-revoke-developer-checksums (&optional universal-argument)
"Remove all saved developer checksums. UNIVERSAL-ARGUMENT is a placeholder."
(interactive "P") (ignore universal-argument)
(setq orgstrap-developer-checksums nil)
(orgstrap-save-developer-checksums t))
(defun orgstrap-save-developer-checksums (&optional overwrite)
"Function to update `orgstrap-developer-checksums-file'.
If OVERWRITE is non-nil then overwrite the existing checksums."
(interactive "P")
(if orgstrap-save-developer-checksums
(let* ((checksums orgstrap-developer-checksums)
(buffer (find-file-noselect orgstrap-developer-checksums-file)))
(with-current-buffer buffer
(unwind-protect
(progn
(lock-buffer)
(let* ((saved (and (not (= (buffer-size) 0)) (cadr (nth 2 (read (buffer-string))))))
;; XXX NOTE saved is not used to updated `orgstrap-developer-checksums' here
;; FIXME massively inefficient
(combined (or (and (not overwrite)
(cl-remove-duplicates (append checksums saved)))
checksums)))
;; TODO do we need to check whether combined and saved are different?
;; (message "checksums: %s\nsaved: %s\ncombined: %s" checksums saved combined)
(erase-buffer)
(insert ";;; -*- mode: emacs-lisp; lexical-binding: t -*-\n")
(insert ";;; DO NOT EDIT THIS FILE IT IS AUTOGENERATED AND WILL BE OVERWRITTEN!\n\n")
(insert (string-replace
" " "\n"
(orgstrap--pp-to-string `(setq orgstrap-developer-checksums ',combined))))
(insert "\n;;; set developer checksums as safe local variables\n\n")
(insert
(orgstrap--pp-to-string
'(mapcar (lambda (checksum-value)
(add-to-list 'safe-local-variable-values
(cons 'orgstrap-block-checksum checksum-value)))
orgstrap-developer-checksums)))
(pp-buffer)
(indent-region (point-min) (point-max))
(save-buffer)))
(unlock-buffer)
(kill-buffer))))
(warn "No checksums were saved because `orgstrap-save-developer-checksums' is not set.")))
#+end_src
*** Init
A note on filter aka =cl-remove-if-not= in =orgstrap--add-file-local-variables= at [[(clrin)]].
| emacs version | require |
|---------------+---------|
| < 24 | 'cl |
| < 25 | 'cl-lib |
| < 27 | 'seq |
The most portable thing to do for now is =(require 'cl-lib)= since we
don't currently support anything below 23. Then use =cl-remove-if-not=.
There is a similar issue with =pcase=, which is that in =emacs-24= the
syntax was closer to =cl-case= when dealing with symbols. Since =cl-lib=
is already in use, =cl-case= is the logical solution for portability.
Not all functionality works in older versions of Org. For example see
[[(obubb-issue)][update block issue]] which is caused by the fact that
~org-babel-update-block-body~ is broken prior to revision
orgit-rev:~/git/NOFORK/org-mode::7d6b8f51ec1993a66a385b98b2df42d0853fe289
which is not present in the versions of Org released with Emacs < 26.
# We have to cache this result to avoid ocbe issues when tangling
# XXX this also has to be manually converted to the : style because
# old versions of org don't support :cache yes and we wind up with
# circular dependencies
#+name: orgstrap-shebang-body-command
#+begin_src elisp :results code :exports none :cache yes
(orgstrap--with-block "orgstrap-shebang" body)
#+end_src
# XXX must keep this in sync manually to support old versions of org
#+name: orgstrap-shebang-body
: "set -e \"-C\" \"-e\" \"-e\"\n{ null=/dev/null;} > \"${null:=/dev/null}\"\n{ args=;file=;MyInvocation=;__p=$(mktemp -d);touch ${__p}/=;chmod +x ${__p}/=;__op=$PATH;PATH=${__p}:$PATH;} > \"${null}\"\n$file = $MyInvocation.MyCommand.Source\n{ file=$0;PATH=$__op;rm ${__p}/=;rmdir ${__p};} > \"${null}\"\nemacs -batch -no-site-file -eval \"(let (vc-follow-symlinks) (defun orgstrap--confirm-eval (l _) (not (memq (intern l) '(elisp emacs-lisp)))) (let ((file (pop argv)) enable-local-variables) (find-file-literally file) (end-of-line) (when (eq (char-before) ?\\^m) (let ((coding-system-for-read 'utf-8)) (revert-buffer nil t t)))) (let ((enable-local-eval t) (enable-local-variables :all) (major-mode 'org-mode) find-file-literally) (require 'org) (org-set-regexps-and-options) (hack-local-variables)))\" \"${file}\" -- ${args} \"${@}\"\nexit\n<# powershell open"
#+caption: init helpers
#+name: orgstrap-init-helper-defuns
#+begin_src emacs-lisp :results none :lexical yes :noweb yes
;;; init helpers
(defvar orgstrap-link-message "jump to the orgstrap block for this file"
"Default message for file internal links.")
(defvar-local orgstrap--local-variables nil
"Variable to capture local variables from `hack-local-variables'.")
;; local variable generation functions
(defun orgstrap--get-min-org-version (info minimal)
"Get minimum org mode version needed by the orgstrap block for this file.
INFO is the source block info. MINIMAL sets whether to use minimal local vars."
(if minimal
(let ((coderef (or (nth 6 info) org-coderef-label-format))
(noweb (org-babel-noweb-p (nth 2 info) :eval)))
(if noweb
"9.3.8"
(let* ((body (or (nth 1 info) ""))
(crrx (org-src-coderef-regexp coderef))
(pos (string-match crrx body))
(commented
(and pos (string-match
(concat (rx ";" (zero-or-more whitespace)) crrx) body))))
;; FIXME the right way to do this is similar to what is done in
;; `org-export-resolve-coderef' but for now we know we are in elisp
(if (or (not pos) commented)
"8.2.10"
"9.3.8"))))
"8.2.10"))
(defun orgstrap--have-min-org-version (info minimal)
"See if current version of org meets minimum requirements for orgstrap block.
INFO is the source block info.
MINIMAL is passed to `orgstrap--get-min-org-version'."
(let ((actual (org-version))
(need (orgstrap--get-min-org-version info minimal)))
(or (not need)
(string< need actual)
(string= need actual))))
(defun orgstrap--dedoc (sexp)
"Remove docstrings from SEXP. WARNING mutates sexp!"
(let ((m '(defun defun-local defmacro defvar defvar-local defconst defcustom)))
(cl-loop
for e in sexp when (listp e) do ; for expression in sexp when the expression is a list
(or
(and
(memq (car e) m) ; is a form with docstrings
(let ((n (nthcdr 4 e))) ; body after docstring
(and
(stringp (nth 3 e)) ; has a docstring
(or (cl-subseq m 3) n) ; var or doc not last
(orgstrap--dedoc n) ; recurse for nested
;; splice out the docstring and return t to avoid the other branch
(or (setcdr (cddr e) n) t))))
;; recurse e.g. for (when x (defvar y t))
(orgstrap--dedoc e))))
sexp)
(defun orgstrap--local-variables--check-version (info &optional minimal)
"Return the version check local variables given INFO and MINIMAL."
`(
(setq-local orgstrap-min-org-version ,(orgstrap--get-min-org-version info minimal))
<>))
(defun orgstrap--local-variables--norm (&optional norm-func-name)
"Return the normalization function for local variables given NORM-FUNC-NAME."
(let ((norm-func-name (or norm-func-name (default-value 'orgstrap-norm-func))))
(cl-case norm-func-name
(orgstrap-norm-func--dprp-1-0
'(
<>))
(orgstrap-norm-func--prp-1-1
'(
<>))
(orgstrap-norm-func--prp-1-0
(error "`orgstrap-norm-func--prp-1-0' is deprecated.
Please update `orgstrap-norm-func-name' to `orgstrap-norm-func--prp-1-1'"))
(otherwise (error "Don't know that normalization function %s" norm-func-name)))))
(defun orgstrap--local-variables--norm-common ()
"Return the common normalization functions for local variables."
'(
<>))
(defun orgstrap--local-variables--eval (info &optional minimal)
"Return the portable or MINIMAL elvs given INFO."
(let* ((minimal (or minimal orgstrap-use-minimal-local-variables))
(minimal (and minimal (orgstrap--have-min-org-version info minimal))))
(if minimal
'(
<>)
'( ;(ref:elv-noweb-issue)
;; if you automatically reindent it will break these two
<>
<>))))
(defun orgstrap--local-variables--eval-common ()
"Return the common eval check functions for local variables."
`( ; quasiquote to fill in `orgstrap-orgstrap-block-name'
<>))
;; init utility functions
(defun orgstrap--new-heading-elisp-block (heading block-name &optional header-args noexport)
"Create a new elisp block named BLOCK-NAME in a new heading titled HEADING.
The heading is inserted at the top of the current file.
HEADER-ARGS is an alist of symbols that are converted to strings.
If NOEXPORT is non-nil then the :noexport: tag is added to the heading."
(declare (indent 1))
(save-excursion
(goto-char (point-min))
(outline-next-heading) ;; alternately outline-next-heading
(org-meta-return)
(insert (format "%s%s\n" heading (if noexport " :noexport:" "")))
;;(org-edit-headline heading)
;;(when noexport (org-set-tags "noexport"))
(move-end-of-line 1)
(insert "\n#+name: " block-name "\n")
(insert "#+begin_src elisp")
(mapc (lambda (header-arg-value)
(insert " :" (symbol-name (car header-arg-value))
" " (symbol-name (cdr header-arg-value))))
header-args)
(insert "\n#+end_src\n")))
(defun orgstrap--trap-hack-locals (command &rest args)
"Advice for `hack-local-variables-filter' to do nothing except the following.
Set `orgstrap--local-variables' to the reversed list of read variables which
are the first argument in the lambda list ARGS.
COMMAND is unused since we don't actually want to hack the local variables,
just get their current values."
(ignore command)
(setq-local orgstrap--local-variables (reverse (car args)))
nil)
(defun orgstrap--read-current-local-variables ()
"Return the local variables for the current file without applying them."
(interactive)
;; orgstrap--local-variables is a temporary local variable that is used to
;; capture the input to `hack-local-variables-filter' it is unset at the end
;; of this function so that it cannot accidentally be used when it might be stale
(setq-local orgstrap--local-variables nil)
(let ((enable-local-variables t))
(advice-add #'hack-local-variables-filter :around #'orgstrap--trap-hack-locals)
(unwind-protect
(hack-local-variables nil)
(advice-remove #'hack-local-variables-filter #'orgstrap--trap-hack-locals))
(let ((local-variables orgstrap--local-variables))
(makunbound 'orgstrap--local-variables)
local-variables)))
(defun orgstrap--add-link-to-orgstrap-block (&optional link-message)
"Add an `org-mode' link pointing to the orgstrap block for the current file.
The link is placed in comment on the second line of the file. LINK-MESSAGE
can be used to override the default value set via `orgstrap-link-message'"
(interactive) ; TODO prompt for message with C-u ?
(goto-char (point-min))
(next-logical-line) ; required to get correct behavior?
(let ((link-message (or link-message orgstrap-link-message)))
(unless (save-excursion
(re-search-forward
(format "^# \\[\\[%s\\]\\[.+\\]\\]$"
orgstrap-orgstrap-block-name)
nil t)) ; XXX for some reason save-excursion fails so we have to reset
(goto-char (point-min))
(next-logical-line) ; use logical-line to avoid issues with visual line mode
(insert (format "# [[%s][%s]]\n"
orgstrap-orgstrap-block-name
(or link-message orgstrap-link-message))))))
(defun orgstrap--add-orgstrap-block (&optional block-contents)
"Add a new elisp source block with #+name: orgstrap to the current buffer.
If a block with that name already exists raise an error.
Insert BLOCK-CONTENTS if they are supplied."
(interactive)
(let ((all-block-names (org-babel-src-block-names)))
(if (member orgstrap-orgstrap-block-name all-block-names)
(warn "orgstrap block already exists not adding!")
(goto-char (point-max))
(insert "\n")
(orgstrap--new-heading-elisp-block "Bootstrap"
orgstrap-orgstrap-block-name
'((results . none)
(exports . none)
(lexical . yes))
'noexport)
(goto-char (point-max))
(insert "\n** Local Variables :ARCHIVE:\n")
(orgstrap--with-block orgstrap-orgstrap-block-name
(ignore params body-unexpanded body)
(when block-contents
;; FIXME `org-babel-update-block-body' is broken in < 26 (ref:obubb-issue)
;; for now warn and fail if the version is known bad NOTE trying to backport
;; is not simple because there are changes to the function signatures
(if (string< org-version "8.3.4")
(warn "Your version of Org is too old to use this feature! %s < 8.3.4"
org-version)
(org-babel-update-block-body block-contents)))
nil))))
(defun orgstrap--lv-command (info &optional minimal norm-func-name)
"Create the elvs for an orgstrapped file.
INFO is the output of `org-babel-get-src-block-info' for the orgstrap block.
MINIMAL determines whether a non-portable block has been requested.
NORM-FUNC-NAME names the function used to normalize orgstrap blocks."
(let ((lv-cver (orgstrap--local-variables--check-version
info
minimal))
(lv-norm (orgstrap--local-variables--norm
norm-func-name))
(lv-ncom (orgstrap--local-variables--norm-common))
(lv-eval (orgstrap--local-variables--eval
info
minimal))
(lv-ecom (orgstrap--local-variables--eval-common)))
(cons 'progn (orgstrap--dedoc (append lv-cver lv-norm lv-ncom lv-eval lv-ecom)))))
(defun orgstrap--add-file-local-variables (&optional minimal norm-func-name)
"Add the file local variables needed to make orgstrap work.
MINIMAL is used to control whether the portable or minimal block is used.
If MINIMAL is set but the orgstrap block uses features like noweb and
uncommented coderefs and function `org-version' is too old, then the portable
block will be used. NORM-FUNC-NAME is an optional argument that can be provided
to determine which normalization function is used independent of the current
buffer or global setting for `orgstrap-norm-func'.
When run, this function replaces any existing orgstrap elv with the latest
implementation available according to the preferences for the current buffer
and configuration. Other elvs are retained if they are present, and the
orgstrap elv is always added first."
;; switching comments probably wont work ? we can try
;; Use a prefix argument (i.e. C-u) to add file local variables comments instead of in a :noexport:
(interactive)
(let ((info (save-excursion
(orgstrap--goto-named-src-block orgstrap-orgstrap-block-name)
(org-babel-get-src-block-info)))
(elv (orgstrap--read-current-local-variables)))
(let ((lv-command (orgstrap--lv-command info minimal norm-func-name))
(commands-existing (mapcar #'cdr (cl-remove-if-not (lambda (l) (eq (car l) 'eval)) elv)))) ;(ref:clrin)
(let* ((stripped
(cl-remove-if
(lambda (cmd) (orgstrap--match-elvs (cons 'eval cmd)))
commands-existing))
(eval-commands (cons lv-command stripped)))
(when commands-existing
(delete-file-local-variable 'eval))
(let ((print-escape-newlines t) ; needed to preserve the escaped newlines
;; if `print-length' or `print-level' is accidentally set
;; `add-file-local-variable' will truncate the sexp with and elispsis
;; this is clearly a bug in `add-file-local-variable' and possibly in
;; something deeper, `print-length' is the only one that has actually
;; caused issues, but better safe than sorry
print-length print-level)
(mapcar (lambda (sexp) (add-file-local-variable 'eval sexp)) eval-commands))))))
(defun orgstrap--before-first-dull ()
"Goto the first non-empty line not starting with a sharp sign."
(goto-char (point-min))
(re-search-forward "\n[^#\n \t]")
(beginning-of-line))
(defun orgstrap--goto-elvs ()
"Goto the start of the elvs for the current buffer.
If no elvs are found goto `point-max' instead."
(widen)
(goto-char (point-max))
(search-backward "\n\^L" (max (- (point-max) 3000) (point-min)) 'move)
(when (let ((case-fold-search t))
(search-forward "Local Variables:" nil t))
(beginning-of-line)))
(defconst orgstrap--shebang-body
<>
"Shebang block body content.")
(defun orgstrap--add-shebang-block (&optional update)
"Add a shebang block to the current buffer."
;; goto correct location
;; create empty bash block
;; fill block
;; go to start of elvs
;; add powershell closing line
(let ((block-name "orgstrap-shebang")
(header-args '((eval . never) (results . none) (exports . none))))
(if (org-babel-find-named-block block-name)
(if update
(orgstrap-update-src-block "orgstrap-shebang" orgstrap--shebang-body)
(warn "A shebang block already exists. Not adding."))
(if update
(warn "A shebang block does not exist. Not updating.")
(save-excursion
(orgstrap--before-first-dull)
(insert "\n#+name: " block-name "\n")
(insert "#+begin_src bash")
(mapc (lambda (header-arg-value)
(insert " :" (symbol-name (car header-arg-value))
" " (symbol-name (cdr header-arg-value))))
header-args)
(insert "\n#+end_src\n")
(orgstrap-update-src-block "orgstrap-shebang" orgstrap--shebang-body)
(orgstrap--goto-elvs)
(insert (format "# close powershell comment %s>\n" "#")))))))
(defun orgstrap-update-shebang-block (&optional universal-argument)
"Update an existing shebang block. UNIVERSAL-ARGUMENT is ignored."
(interactive "P")
(ignore universal-argument)
(orgstrap--add-shebang-block 'update))
;; init user facing functions
;;;###autoload
(defun orgstrap-init (&optional prefix-argument shebang)
"Initialize orgstrap in a buffer and enable command `orgstrap-edit-mode'.
If PREFIX-ARGUMENT is non-nil and has a value of 4 or 64 init will attempt
to use the minimal local variables if possible.
If SHEBANG is non-nil or PREFIX-ARGUMENT is greater than or equal to 16
then a shebang block will also be added to the file.
Example usage.
M-x orgstrap-init -> portable elvs
C-u M-x orgstrap-init -> minimal elvs
C-u C-u M-x orgstrap-init -> portable elvs + shebang
C-u C-u C-u M-x orgstrap-init -> minimal elvs + shebang"
(interactive "P")
(unless (eq major-mode 'org-mode)
(error "Cannot orgstrap, buffer not in `org-mode' it is in %s!" major-mode))
;; TODO option for no link?
;; TODO option for local variables in comments vs noexport
(let (onf)
(let ((shebang (or shebang (and prefix-argument (>= (car prefix-argument) 16))))
(orgstrap-norm-func
(or (cdr (assoc 'orgstrap-norm-func-name (orgstrap--read-current-local-variables)))
(default-value 'orgstrap-norm-func))))
(save-excursion
(orgstrap--add-orgstrap-block)
(orgstrap-add-block-checksum)
(orgstrap--add-link-to-orgstrap-block)
;; FIXME sometimes local variables don't populate due to an out of range error
(orgstrap--add-file-local-variables
(or (and prefix-argument (memq (car prefix-argument) '(4 64))) orgstrap-use-minimal-local-variables))
(when shebang (orgstrap--add-shebang-block))
(orgstrap-edit-mode 1)
(setq onf orgstrap-norm-func)))
;; reset to ensure that a stale value is not inserted on next save
(setq-local orgstrap-norm-func onf)))
#+end_src
# Note that multi-line strings cause issues with indentation if they are
# nowebbed with leading whitespace. We avoid this by left aligning the
# [[(elv-noweb-issue)][noweb references]] so that no leading whitespace
# is inserted. This is something to watch out for in general when trying
# to ensure consistent hashing.
# I suspect that the underlying issue may be an org-mode bug related
# to incorrect handling of leading whitespace when inserting contents
# into a new block
# dedoc testing
# (orgstrap--dedoc '(defvar lol 'hello))
# (orgstrap--dedoc '(defvar lol 'hello "there"))
# (orgstrap--dedoc '(defun lol () "there" 1))
# (orgstrap--dedoc '(defun lol (arg) "there" arg))
# (orgstrap--dedoc '(defun lol (arg) "there"))
# (orgstrap--dedoc '(defmacro lol (arg) "there"))
# (orgstrap--dedoc '(defmacro lol (arg) "there" arg))
# (orgstrap--dedoc '((defmacro lol (arg) "there" arg) (defvar lol 'hello "there")))
*** Extras
#+caption: extra helpers
#+name: orgstrap-extra-helper-defuns
#+begin_src elisp :noweb yes
;;; extra helpers
(defun orgstrap-update-src-block (name content)
"Set the content of source block named NAME to string CONTENT.
XXX NOTE THAT THIS CANNOT BE USED WITH #+BEGIN_EXAMPLE BLOCKS."
;; FIXME this seems to fail if the existing block is empty?
;; or at least adding file local variables fails?
(let ((block (org-babel-find-named-block name)))
(if block
(save-excursion
(orgstrap--goto-named-src-block name)
(org-babel-update-block-body content))
(error "No block with name %s" name))))
(defun orgstrap-get-src-block-checksum (&optional cypher)
"Calculate of the checksum of the current source block using CYPHER."
(interactive)
(let* ((info (org-babel-get-src-block-info))
(params (nth 2 info))
(body-unexpanded (nth 1 info))
(body (orgstrap--expand-body info))
(body-normalized
(orgstrap-norm body))
(cypher (or cypher (orgstrap--current-buffer-cypher))))
(ignore params body-unexpanded)
(secure-hash cypher body-normalized)))
(defun orgstrap-get-named-src-block-checksum (name &optional cypher)
"Calculate the checksum of the first sourc block named NAME using CYPHER."
(interactive)
(orgstrap--with-block name
(ignore params body-unexpanded)
(let ((cypher (or cypher (orgstrap--current-buffer-cypher)))
(body-normalized
(orgstrap-norm body)))
(secure-hash cypher body-normalized))))
(defun orgstrap-run-additional-blocks (&rest name-checksum) ;(ref:oab)
"Securely run additional blocks in languages other than elisp.
Do this by providing the name of the block and the checksum to be embedded
in the orgstrap block as NAME-CHECKSUM pairs."
(ignore name-checksum)
(error "TODO"))
(defun orgstrap--get-elvs (&optional from-flv-alist)
"Return the elvs as they are written in the current buffer.
If FROM-FLV-ALIST is not null display the elvs that are in
`file-local-variables-alist'."
(cl-loop
for var in
(if from-flv-alist
file-local-variables-alist
(orgstrap--read-current-local-variables))
when (orgstrap--match-elvs var)
return (cdr var)))
(defun orgstrap-inspect-elvs (&optional from-flv-alist)
"Display the elvs for the current buffer.
If FROM-FLV-ALIST is not null display the elvs that are in
`file-local-variables-alist'."
(interactive "P")
(let ((buffer (get-buffer-create (format "%s orgstrap elvs" (buffer-file-name))))
(elvs (orgstrap--get-elvs from-flv-alist))
(cypher orgstrap-cypher)
print-length print-level)
(with-current-buffer buffer
(emacs-lisp-mode)
(read-only-mode)
(let ((inhibit-read-only t))
(erase-buffer)
;; TODO insert checksum in comment
(insert ";; -*- elvs-checksum: "
(secure-hash cypher (orgstrap-norm (pp-to-string elvs)))
"; -*-\n")
(insert (pp-to-string elvs))
(goto-char (point-min))
(while (re-search-forward "(\\(let\\|defun\\|when\\|unless\\|if\\|read\\)" nil t)
(join-line 1))
(indent-region (point-min) (point-max)))
(local-set-key (kbd "q") #'quit-window)
(goto-char (point-min)))
(display-buffer buffer)))
(defun orgstrap--whitelist-current-buffer ()
"Mark local variable values in the current buffer as safe."
(let ((lvs (orgstrap--read-current-local-variables)))
(customize-push-and-save 'safe-local-variable-values lvs)))
;; extra user facing functions
;;;###autoload
(defun orgstrap-whitelist-file (path)
"Add local variables in PATH as safe custom variable values.
This is useful when distributing orgstrapped files.
Use with a command similar to the following.
Since -batch implies -q, `user-init-file' must be passed explicitly.
emacs -batch -eval \\
\"(let ((user-init-file (pop argv)) (file (pop argv))) (package-initialize) (orgstrap-whitelist-file file))\" \\
~/.emacs.d/init.el /path/to/whitelist.org"
(let (enable-local-variables) ; < 28 don't run orgstrap block
;; `find-file-literally' is broken on 27 so regularize behavior
(with-current-buffer (find-file-literally path)
(orgstrap--whitelist-current-buffer)
(kill-buffer))))
#+end_src
Ideally we want to call [[(oab)][orgstrap-run-additional-blocks]] as
=(orgstrap-run-additional-blocks "additional-block-name" "checksum-value-hash-thing" "ab2" "cs2")=
It probably makes sense to house this in its own orgstrap-aux block or something.
I want to keep the file local variables as minimal as possible, so having another
aux block that could be automatically updated with the names and hashes of additional
blocks would be nice ... probably via something like =orgstrap-add-additional-block=
but it will not go in the local variables because we want there to be some hope of
orgstrap being portable to other platforms outside of Emacs at some point in the
very distant future, so keeping the machinery outside of the org file itself as
minimal as possible is critical.
** orgstrap.el :noexport:
# XXX TODO it would be a super cool feature if xref could resolve to elisp source
# blocks in org-mode files, because then half the need for the .el file would go away
#+caption: Retangle this if something changes.
#+name: orgstrap.el
#+header: :exports none
#+begin_src elisp -r -l "\([[:space:]]\|;\)*(ref:%s)$" :noweb yes :eval never :tangle ./orgstrap.el
;;; orgstrap.el --- Bootstrap an Org file using file local variables -*- lexical-binding: t -*-
;; Author: Tom Gillespie
;; URL: https://github.com/tgbugs/orgstrap
;; Keywords: lisp org org-mode bootstrap
;; Version: 1.5.5 (ref:orgstrap.el-version)
;; Package-Requires: ((emacs "24.4"))
;;;; License and Commentary
;; License:
;; SPDX-License-Identifier: GPL-3.0-or-later
;;; Commentary:
;; orgstrap is a specification and tooling for bootstrapping Org files.
;; It allows Org files to describe their own requirements, and
;; define their own functionality, making them self-contained,
;; standalone computational artifacts, dependent only on Emacs,
;; or other implementations of the Org-babel protocol in the future.
;; orgstrap.el is an elisp implementation of the orgstrap conventions.
;; It defines a regional minor mode for `org-mode' that runs orgstrap
;; blocks. It also provides `orgstrap-init' and `orgstrap-edit-mode'
;; to simplify authoring of orgstrapped files. For more details see
;; README.org which is also the literate source for this orgstrap.el
;; file in the git repo at
;; https://github.com/tgbugs/orgstrap/blob/master/README.org
;; or wherever you can find git:c1b28526ef9931654b72dff559da2205feb87f75
;; Code in an orgstrap block is usually meant to be executed directly by its
;; containing Org file. However, if the code is something that will be reused
;; over time outside the defining Org file, then it may be better to tangle and
;; load the file so that it is easier to debug/xref functions. The code in
;; this orgstrap.el file in particular is tangled for inclusion in one of the
;; *elpas so as to protect the orgstrap namespace and to make it eaiser to
;; use orgstrap in Emacs.
;; The license for the orgstrap.el code reflects the fact that the
;; code for expanding and hashing blocks reuses code from ob-core.el,
;; which at the time of writing is licensed as part of Emacs.
;;; Code:
(require 'org)
(require 'org-element)
<>
<>
<>
<>
<>
(provide 'orgstrap)
;;; orgstrap.el ends here
#+end_src
# have to have an empty line at the end so that a newline shows up
# when tangled ... surely this is a bug?
** Testing :noexport:
*** Simple
#+name: test-portable
#+begin_src bash :var THIS_FILE=(buffer-file-name) :results none
emacs-24 -Q $THIS_FILE
emacs-25 -Q $THIS_FILE
emacs-26 -Q $THIS_FILE
emacs-27 -Q $THIS_FILE
emacs-28 -Q $THIS_FILE
emacs-29-vcs -Q $THIS_FILE
#+end_src
#+name: test-minimal
#+begin_src bash :var THIS_FILE=(buffer-file-name) :results none
emacs-24 -Q orgstrap-minimal.org
emacs-25 -Q orgstrap-minimal.org
emacs-26 -Q orgstrap-minimal.org
emacs-27 -Q orgstrap-minimal.org
emacs-28 -Q orgstrap-minimal.org
emacs-29-vcs -Q orgstrap-minimal.org
#+end_src
*** Matrix
Before running the tests below you need to generate [[file:./orgstrap-autoloads.el]].
Newer version of =autoload-generate-file-autoloads= add functions that may not be
supported by older versions of Emacs. Thus you should run this on the oldest version
of Emacs you will be testing against.
# XXX This must be run with emacs-26 to avoid bytecode compatibility issues
#+name: generate-autoloads-for-test
#+begin_src elisp :results none
(require 'autoload)
(with-current-buffer (find-file-noselect "orgstrap-autoloads.el")
(erase-buffer)
(let* ((cb (current-buffer))
(fn (buffer-file-name cb))
(generated-autoload-file fn))
(autoload-generate-file-autoloads "orgstrap.el" cb fn))
(save-buffer)
(kill-buffer))
#+end_src
# XXX note that you cannot use append t with add-to-list for local variables
# it puts any additional values in the wrong place, need to check other cases
# reminder that you can't use bare bangs ! anywhere in bash scripts :/
# XXX also reminder that unbound variables will break at export time
# XXX remove elc files before running this due to bytecode mismatches
# XXX regenerate autoloads, it seems that the version in 26 works for everyone
#+name: test-matrix-run
#+begin_src bash :results drawer output
versions=( 24 25 26 27 28 29-vcs )
test_files=( test-no-lv-list.org test-lv-list-portable test-lv-list-minimal )
for v in ${versions[@]}; do
[ -d test/emacs-$v ] || mkdir -p test/emacs-$v
for f in ${test_files[@]};do
# uncomment and reorder to debug tests
#f=test-lv-list-minimal
#emacs -Q \
emacs-$v -Q -batch \
-eval "(setq user-init-file (concat default-directory \"test/emacs-${v}/init.el\"))" \
-l orgstrap-autoloads.el \
-eval "(message \"\n%s\"(emacs-version))" \
-f toggle-debug-on-error \
-eval "(add-to-list 'load-path \"$(pwd)/\")" \
-eval "(orgstrap-mode)" \
-eval "(defun orgstrap-test () (error \"test failed.\"))" \
-eval "(orgstrap-whitelist-file \"${f}\")" \
-visit $f \
-eval "(orgstrap-test)" 2>&1
done
done
#+end_src
**** TODO Full matrix
#+begin_src elisp
;; create buffer
;; fill buffer
;; set code
;; hash
;; save buffer
;; open in all the other impls having set the checksum as accepted
;; with orgstrap-mode enabled
;; without orgstrap-mode enabled
;; with minimal
;; with portable
;; with noweb
;; without noweb
;; iterate over norm funcs
;; orgstrap-norm-func-name mismatch
;; orgstrap-norm-func-name not in internal list
;; .org extension and mode: org local variable
(defconst orgstrap--test-matrix
`(((orgstrap-mode (nil t))
(lv-type (nil minimal portable))
(noweb (nil t))
;; (comments (nil link noweb)) ; comments aren't actually relevant here I think?
(norm-func ,orgstrap--internal-norm-funcs)
(path-suffix-lv ((".org" . (mode . org))
(".org" nil)
("" . (mode . org))
("" . (mode . nil))))))
"The dimensions of the test files that need to be generated."
)
#+end_src
**** TODO One time tests
1. Blacklist and open, then unblacklist and open.
This requires an actual file since we need buffer file name.
2. Need a way to test revoke as well.
**** Test files
# TODO automatically generate all of these test files
# maybe even generate them as buffers that can be run
# by loading this files with orgstrap--test t?
# #+header: :comments link # broken atm
#+name: test-no-lv-list
#+begin_src org :tangle ./test-no-lv-list.org
# -*- orgstrap-cypher: sha256; orgstrap-norm-func-name: orgstrap-norm-func--dprp-1-0; orgstrap-block-checksum: 8d941e14e89664b834f5b28c070f9f7b0ec55b092b55cc23dd903c010fdaeda5; -*-
# [[orgstrap][jump to the orgstrap block for this file]]
,* Bootstrap :noexport:
,#+name: orgstrap
,#+begin_src elisp :results none :lexical yes
(defun orgstrap-test ()
(if (cl-remove-if-not #'orgstrap--match-elvs
file-local-variables-alist)
(error "elv is still present!")
(message "No local variables here!")))
(message "orgstrap successful")
,#+end_src
#+end_src
#+name: test-lv-list-portable
#+begin_src org :tangle ./test-lv-list-portable :noweb yes
# -*- mode: org; orgstrap-cypher: sha256; orgstrap-norm-func-name: orgstrap-norm-func--dprp-1-0; orgstrap-block-checksum: 14e85d1213ef7a6739ca6ca7361a227b0a55346d4c7c6457bdd5f7ba91ff5dff; -*-
# [[orgstrap][jump to the orgstrap block for this file]]
,* Bootstrap :noexport:
,#+name: orgstrap
,#+begin_src elisp :results none :lexical yes
(defun orgstrap-test ()
(if (cl-remove-if-not #'orgstrap--match-elvs
file-local-variables-alist)
(error "elv is still present!")
(message "Portable local variables here!")))
(message "orgstrap successful")
,#+end_src
,** Local Variables :ARCHIVE:
<>
#+end_src
#+name: test-lv-list-minimal
#+begin_src org :tangle ./test-lv-list-minimal :noweb yes
# -*- mode: org; orgstrap-cypher: sha256; orgstrap-norm-func-name: orgstrap-norm-func--dprp-1-0; orgstrap-block-checksum: 3008580fd616cdfca904c7508ae023f782585229a87adab33fb8c2d391f89561; -*-
# [[orgstrap][jump to the orgstrap block for this file]]
,* Bootstrap :noexport:
,#+name: orgstrap
,#+begin_src elisp :results none :lexical yes
(defun orgstrap-test ()
(if (cl-remove-if-not #'orgstrap--match-elvs
file-local-variables-alist)
(error "elv is still present!")
(message "Minimal local variables here!")))
(message "orgstrap successful")
,#+end_src
,** Local Variables :ARCHIVE:
<>
#+end_src
** Release :noexport:
*** Flycheck
Use ~flycheck-mode~ on [[file:./orgstrap.el]] to checkdoc for melpa.
Don't forget to run ~flycheck-package-setup~ to get better reports.
*** Byte compile
Before a release run the following block and fix any byte compile
errors and warnings. Using Emacs 26 ensures that bytecode is forward
and backward compatible.
#+begin_src elisp :noweb yes :results drawer
(ow-run-command "emacs" "-batch" "-f" "batch-byte-compile" "orgstrap.el")
(ow-run-command "emacs-26" "-batch" "-f" "batch-byte-compile" "orgstrap.el")
(delete-file "orgstrap.elc")
#+end_src
*** Run test matrix
Run ref:test-matrix-run.
#+call: test-matrix-run()
*** Run init tests
These are not automated at the moment. Run ref:test-portable and
do the following for each version of Emacs.
1. Accept lvs.
2. For Emacs >= 26 ~orgstrap-clone~.
3. switch to ~*scratch*~
4. enable ~org-mode~
5. ~orgstrap-init~
6. optional edit block
7. optional save to file and test reload the saved file
8. For Emacs >= 26 in ~*scratch*~ buffer undo
9. For Emacs >= 26 ~orgstrap-stamp~
10. For Emacs >= 26 check that the checksum matches the checksum for this file
11. quit
#+call: test-portable()
*** Final steps
Things that need to be done for a release.
- Bump the version number in the [[(orgstrap.el-version)][orgstrap.el header comment]].
You will need to manually retangle after this step.
- Update the [[#changelog][changelog]].
- Convert the changelog entry to markdown for the GitHub release.
=C-c C-e C-s m M=.
* Changelog
:PROPERTIES:
:CUSTOM_ID: changelog
:END:
** 1.5.5
- Fix missing paren.
Missed due to a duplicate orgstrap block for debug in the same file.
** 1.5.4
- Update ~orgstrap--shebang-body~ to set ~find-file-literally~ to nil.
If you use shebang blocks you should update them so that a call to
~find-file~ on ~buffer-file-name~ of the orgstrapped file will
continue without prompting the user.
** 1.5.3
- Ignore errors from ~org-set-visibility-according-to-property~.
Recent changes to ~org-set-visibility-according-to-property~ result
in errors if it is called when a file is not in ~org-mode~ and/or
when a buffer is in ~org-mode~ but Emacs is ~noninteractive~.
This requires and update to the elvs which wraps the call in
~ignore-errors~ which is more space efficient than e.g. testing
whether we are non-interactive and also mostly +fool+ future proof.
If you use orgstrap shebang blocks you should updated your elvs.
** 1.5.2
- Update ~orgstrap--shebang-body~ to simplify and improve safety.
If you use shebang blocks you should update them so that shebang
blocks are not affected by leaking environment variables. See
[[./shebang.org]] for a full explication of the operation of each
line of shell/powershell code.
- Add ability to update shebang blocks to latest version.
New interactive function ~orgstrap-update-shebang-block~ updates an
existing shebang block to the latest ~orgstrap--shebang-body~.
Internally ~orgstrap--add-shebang-block~ was updated to accept an
optional ~update~ argument which is used to control whether to add a
new or update an existing block.
** 1.5.1
- Update ~orgstrap--shebang-body~ so that shebang blocks pass args correctly.
If you use shebang blocks you should update them so that they pass
the correct args to emacs. See [[./shebang.org]] for more details on
the changes.
** 1.5
- Regularize behavior of ~orgstrap-whitelist-file~.
An internal call to ~find-file-literally~ has different behavior in
27 vs 28 (see notes about that in [[./shebang.org]]).
- Rename all normalization functions to remove use of =.=.
See https://debbugs.gnu.org/cgi/bugreport.cgi?bug=55645 for details.
*You will need to update any orgstrapped files to use the new names.*
The simplest way to do this is to update to to 1.5 and then delete
the prop line local variables and then run ~orgstrap-init~.
- Add a check to ensure that only portable symbol names are used.
Symbols with =[.?]= appearing anywhere other than at the start of
the symbol are banned since their prin1 representation diverges
across the 28/29 boundary.
If you have orgstrap blocks that contain such symbols you will need
to change the symbol names. An error will be raised when calling
~orgstrap-add-block-checksum~ with a list of symbols that need to be
changed.
** 1.4
- Update ~orgstrap-init~ so that it can insert a shebang block.
Use 2 or more prefix arguments to add a shebang block when using
~orgsgrap-init~. For example =C-u C-u C-u M-x orgstrap-init=.
** 1.3
- Remove all ~orgstrap-do~ variables.
The core of orgstrap is not the right place to maintain these. They
will reappear under ~ow-do~ in the future.
- Add ~orgstrap-norm-func--dprp-1.0~ and make it the default norm func.
The default normalization function for orgstrap is now invariant to
changes in the docstring for ~defun~, ~defun-local~, ~defmacro~,
~defvar~, ~defvar-local~, ~defconst~, and ~defcustom~. This allows
improved documentation without requiring the user to re-audit.
Note that ~orgstrap-norm-func--prp-1.1~ has NOT been deprecated, but
is no longer the default. It is still useful if for whatever reason
you want to minimize the elvs.
- Add support for batch execution.
The preferred method is to use an org shebang block (see
[[./shebang.org]]) It is also possible to maintain an automatically
updating list of developer checksums. This approach was deemed to be
silly given shebang blocks, however the functionality is retained.
- Add ~orgstrap-whitelist-file~ to make it easier to mark known safe
files in batch.
See the docstring for example usage.
- Add ~orgstrap-inspect-elvs~ to inspect the elvs for the current buffer.
The command also calculates the elvs checksum for comparison.
Known elv checksums for this release are below. The order is
minimal, minimal-noweb, and portable (aka minimal-noweb-eval).
For prp-1.1 \\
=446d0c80d72bb89dd149181e6a24eafa011d12d6dc99fad958a03ddebd9a95ad= \\
=b294539a74f2a1932d39790d6377a4229bd3d5e84df64d968baa8ff3f85349cc= \\
=543e3400c80e2cc7b9bf94b1799d1460b240776c2c7415a2f6c0c7a9507978ef= \\
For dprp-1.0 \\
=aa080a6469c22dfe960c43fa3bff3b92a6bc3da9383ec8fcd7d0a019192e7aa0= \\
=72c52a3483905aff6b83c6cd2c36899a2a8d1cbc603b6e5ce8c9e98f0dd7b099= \\
=9e33bc67b8850147962edcece4ea1193bb6cf3711264a26bde91b1f838912ffc= \\
- Fix ~org-edit-mode~ so that it now activates correctly.
- Fix ~orgstrap-init~ so that it no longer misplaces the link to the
orgstrap block if there is already content in the buffer.
- Fix ~orgstrap-init~ so that invocation in files with existing elvs
updates only the existing orgstrap elv and preserves other elvs.
- Fix ~orgstrap-init~ to read and pass the current value of
~orgstrap-norm-func-name~ when creating local variables.
If ~orgstrap-norm-func-name~ is missing, the default value of
~orgstrap-norm-func~ is used.
This prevents klobbering while also providing an easy way to update
the normalization function --- just update the variable value and
run ~orgstrap-init~.
- Fix ~orgstrap-norm-func~ by always declaring it with ~defvar-local~.
- Update the elvs to handle issues with symlinks and vc mode. The
core functionality remains compatible.
- Update the elvs so that ~org-confirm-babel-evaluate~ can be set by
an orgstrap block without having to modify the elvs.
- Update the elvs so that they only restore visibility set via
property drawers.
This makes it possible to use the orgstrap block to control initial
visibility and narrowing to simplify the presentation of orgstrapped
files (and avoid distracting users with the orgstrap machinery).
- Update ~orgstrap-init~ to put the Bootstrap section at the end of
the file and to put the elvs in an archived heading inside that.
This pattern has been found to be quite effective for a number of
different use cases.
** 1.2.7
- Fix bad defaults on ~orgstrap-do-*~ custom variables.
If these are not set to t by default then it is impossible
individual blocks to know whether a nil value was intentional on the
part of the user or not. Users _must_ set values to nil in their
config if they do not want certain sections to run.
** 1.2.6
- Ignore ~orgstrap-block-checksum~ when loading ~org-agenda~.
If ~enable-local-eval~ is t (following the behavior described in the
[[#122][1.2.2]] changelog), then ~orgstrap-block-checksum~ is not
ignored and if the local variable value has not been added to the
safe list then the user will be prompted.
- Add ~orgstrap-do-*~ variables.
Boolean control variables that can be used enable/disable standard
functionality/steps needed by org files. See [[file:./do.org]] for
more details.
** 1.2.5
- Improve behavior of ~orgstrap-blacklist-current-file~.
Now revokes the current buffer checksum by default, this can be
overridden by providing a universal argument. The blacklist is
immediately saved via ~customize-save-variable~.
- Fix ~orgstrap-mode~ to use the universal argument.
Behavior is now correct when ~(orgstrap-mode t)~ is called.
** 1.2.4
- Add ~orgstrap-clone~ and ~orgstrap-stamp~ commands.
~orgstrap-clone~ stores the current buffer and current or orgstrap
block ~orgstrap-stamp~ copies the expanded contents and headers of
that block to a new orgstrap block in a new file. Useful in cases
where users want to duplicate functionality in a new file.
NOTE ~orgstrap-stamp~ only works for org version >= 8.3.4 which
means that it does not work for versions of Emacs < 26.
- ~orgstrap--add-orgstrap-block~ add ~block-contents~ argument.
This simplifies the implementation of stamp, and makes it possible
to set the initial contents of the orgstrap block programmatically.
There are some lingering issues with indentation that may need to
be resolved for this to work seamlessly.
- Fix byte compile bug from destructuring-bind not being aliased.
- Fix for issues with noweb blocks containing multi-line docstrings.
We need this to test ~orgstrap-clone~ with this readme file. Without it
the checksum will not match in the stamped file due to differences in
the leading whitespace in docstrings.
- Make ~orgstrap-revoke-eval-local-variables~ obsolete.
Replaced by the more compact ~orgstrap-revoke-elvs~.
** 1.2.3
- Add ~orgstrap-file-blacklist~ to block eval of specific files.
The functionality only works if ~orgstrap-mode~ is enabled.
** 1.2.2
:PROPERTIES:
:CUSTOM_ID: 122
:END:
- Do not run eval local variables when loading org-agenda.
~orgstrap-always-eval~ can be set to ~t~ by users who want to
evaluate orgstrap blocks in all situations. More granular control is
provided by adding the full path of files that should always try to
run their orgstrap block to ~orgstrap-always-eval-whitelist~.
In all cases, if the global setting for ~enable-local-eval~ is more
restrictive then it is honored (i.e., nil will block any execution
and the default ~'maybe~ will continue to prompt).
- Add ability to revoke previously approved orgstrap-block-checksums.
Rapid revocation of permissions is an important part of any security
system. Therefore we now provide a way to revoke all previously
approved values for ~orgstrap-block-checksum~ in a single command
~orgstrap-revoke-checksums~. This command can also be provided
with a specific list of checksums to revoke. Another convenience function
~orgstrap-revoke-current-buffer~ is provided that revokes the checksum
of the orgstrap block for the current buffer.
- Add ability to revoke previously approved eval local variables.
As with revocations for ~orgstrap-block-checksum~ values, we also
need a way to revoke eval local variables. There is no granular
control. Use ~orgstrap-revoke-eval-local-variables~ to nuke all
orgstrap eval local variables from orbit.
** 1.2.1
- Fix bad startup visibility when using orgstrap.
*You should run =M-:= =(orgstrap--add-file-local-variables)= to*
*update embedded eval local variables.*
Running =org-babel-execute-src-block= changes the visibility of the
tree holding the orgstrap block. As a result the startup visibility
of any org file using orgstrap was incorrect. Adding a call to
=org-set-startup-visibility= in the unwind forms ensures that
startup visibility is correct.
** 1.2
- Add =orgstrap-norm-func--prp-1.1= and make it the default norm func.
This change does not effect the default behavior of =orgstrap=. The
reason for the change is to defensively shadow =print-length= and
=print-level= to =nil= so that if they are somehow non-nil, Emacs
will not truncate the contents of the src block prior to hashing.
- Mark =orgstrap-norm-func--prp-1.0= as obsolete.
*You should update any files using prp-1.0 to use prp-1.1.*
Whenever there is a case where a change in the environment can cause
a change in the output of a normalization function there is a risk
that it could be exploited.
** 1.1.1
- Fix =orgstrap--hack-lv= to remove itself from the local
=hack-local-variables-hook=.
** 1.1
:PROPERTIES:
:CUSTOM_ID: 11
:END:
- Renamed existing =orgstrap-mode= to =orgstrap-edit-mode=.
This is a *BREAKING CHANGE*. Please update your workflows.
- Added the new =orgstrap-mode= implementation.
This is a regional minor mode for =org-mode= which makes it possible
to use orgstrap without the embedded local variables. This allows
for greater security at the expense of portability, depending on the
exact use case. By a stroke of good fortune it is possible to use
the =hack-local-variables= hooks to trap and remove the embedded
local variables if they are present so that the orgstrap block is
not evaluated twice.
- Added =orgstrap-always-edit= as a custom variable.
If non-nil then =orgstrap-edit-mode= will be automatically activated
by =orgstrap-mode=.
- =orgstrap--add-file-local-variables= update existing =eval:= vars.
This change makes it vastly easier to switch between portable and
minimal implementations, and should make it easier to switch the
normalization function once we get that implemented.
If an existing orgstrap eval file local variable is detected it is
removed and the latest version is added. Other =eval:= variables are
not modified. Note however that the orgstrap eval variable will
always be placed first.
* Contributing
:PROPERTIES:
:CUSTOM_ID: contributing
:END:
The primary =orgstrap= repository lives at https://github.com/tgbugs/orgstrap.
There are a number of ways to contribute to =orgstrap=.
1. Have an Org file that use =orgstrap=? Create an issue or a pull
request to add it to the list of [[#examples-from-around-the-web][examples from around the web]].
2. Encounter a bug? Please [[https://github.com/tgbugs/orgstrap/issues/new/choose][submit an issue]]!
3. Feel like writing some elisp? Check out [[#future-work][future work]]
for a list of potential projects (TODO status not visible on GitHub).
* Guides
** User guide
This guide is for users of Org files that have =orgstrap= blocks.
This includes Emacs users who have their own configs as well as users
who might be encountering Emacs for the first time.
** Developer guide
This guide is for developers who want to use =orgstrap= in their own
org files.
It covers workflows for development, distribution, and maintenance of
orgstrap files.
It also covers best practices and effective strategies for making Org
files accessible to users.
*** There can be only one. Dealing with elisp's absent namespaces.
A key issue that orgstrap must contend with is the fact that there is
only one global namespace for all elisp functions. Variables are not
an issue for orgstrap because buffer local variables provide
sufficient separation.
To this end there are two conventions that orgstrap and orgware follow.
The ersatz namespaces ~orgstrap---~ and ~ow---~ may be used in any
orgstrap block. Developers may assume that any such definitions will
remain unchanged at least until another orgstrap block runs. Clearly it
is impossible to know for sure that no one else will use a function
with the same name ~ow---you-re-standing-on-my-toe-!~ that has different
behavior. We can make an attempt to keep a record of all known ~ow---~
function names, but ultimately it is up to the user to run a check.
# TODO need to implement a local ow--- collision checker that searchers
# all orgs files for ow--- usage and reports any collisions
# TODO implement a way to report a name collision to upstream
** Contributor guide
This guide is for developers who want to contribute to the core
=orgstrap= implementation or documentation.
*** Setup
*** Testing
*** Making changes
*** Submitting a pull request
* Best practices
:PROPERTIES:
:CUSTOM_ID: best-practices
:END:
** Accepting local variables
In short. If you use orgstrap.el don't accept the elvs.
Suggestions for users. If you run an orgstrapped file via -Q or
similar, then you have to accept the file. The elvs probably aren't
going to include package initialize and require orgstrap, but maybe
they could. Essentially, in -Q mode the user should know that they
have to initialize it all themselves (orgware could do it for them).
When you aren't running in -Q mode and you DO have orgstrap.el
installed on your system then I strongly suggest that you should NOT
accept, or even actively remove, any and all approved elvs and use
orgstrap-mode since it has a number of features that can enhance the
security of execution such as blacklists etc.
** Use the system package manager.
There is a big difference between using a script to install a program directly
from the internet and using a script to ask the host system to install a program.
Even if you audit a random script from the internet it is unlikely that you will
be able to do due diligence. On the other hand, if you ask your system package
manager to install something for you, there is a much better chance that it has
at least been somewhat audited, and there is usually an existing process for
getting a package into the system which helps to mitigate certain types of attacks.
To give a military example it is the difference between inspecting and accepting a
package from a random person because they say you asked for it yesterday (maybe you
did!) versus only every allowing packages to come through procurement. You are much
less likely to get a bomb or a packaged rigged to exfil data if you go through
procurement because there is an established process for how to do things and that
process enshrines generations experience about how to not get blown up by the pizza guy.
So, if you are writing instructions that require a certain tool, it is better to tell
whoever is following them to ask procurement to get the tool for them than to tell them
to going out to the hardware store and get it themselves, or worse, give them the address
of a random tool delivery man who happens to be a good buddy of yours. Even if everyone
involved is trustworthy those kinds of relationships are much easier for some third party
to compromise and use for their own purposes.
The obvious corollary when you are the user rather than the author, is that if you
encounter instructions that ask you to directly install software from a random place
you should be suspicious, even, perhaps especially, if that random place is housed
within a larger reputable site. If you're not in a hurry, ask for the software to be
packaged, or package it yourself so that it can go through the process.
** Opening orgstrapped files as an Emacs user
One problem that orgstrap has is that an orgstrap block can modify the
Emacs configuration. Modifying the config is often critical to get the
desired behavior for the particular target user population. This is
not an issue if the users do not use Emacs for anything else. However,
if the user opening the file is an Emacs user then blocks that modify
the configuration are a serious problem.
There are three ways around this issue: one a standard convention for
naming a variable to control evaluation of config related code or two
always use ~emacs -q~ and of course three a combination of both.
In the first case a set of standard conventions for variable names can
be used to control whether some, or all configuration variables are
set. However, this can only attain the status of a best practice
because orgstrap blocks run arbitrary code and there is no way to
enforce the convention (thus why it is in this section).
One convention that I have been testing is to ...
# TODO enable-*
# orgstrap-enable-config ? not a good name ...
In the second case we suggest that users always open orgstrapped files
with ~emacs -q~. This is as close as we can get to having a sandboxed
execution environment. If the user also wants to load their config
then they would have to use ~-l~ as well. This is a pain, but without
converting all of the variables into buffer local variables this is
unlikely to work.
Another major drawback of using ~emacs -q~ is that there are many
more advanced features enabled by orgstrap.el that cannot be used
without running ~emacs -q -l orgstrap.el~ or some equivalent. This
adds significant complexity to the command line invocation. There
is no easy way to work around this since the features of interest
are ones that must be available before the orgstrap block is run.
Another approach is to use ~emacs -q -f package-initialize~.
There is also the issue of how to handle potential name collisions.
# TODO orgstrap--- prefix is too long, but is one solution
In summary, Emacs and orgstrap are _very_ sharp tools. I have tried to
provide a bit of protection via the checksum mechanism, however if you
are an Emacs user, you should probably always check the orgstrap block
to make sure that it won't completely klobber your config.
If you are authoring a file that uses orgstrap, it is friendly to put
any config related code in its own block so that it can be controlled
globally via ~the-orgstrap-variable-name-to-be-determined~.
* Bootstrapping to Emacs, bootstrapping to Org
:PROPERTIES:
:CUSTOM_ID: bootstrapping-to-emacs-bootstrapping-to-org
:END:
See [[file:./get-emacs.org]].
* Examples
:PROPERTIES:
:CUSTOM_ID: examples
:END:
** Examples from around the web
:PROPERTIES:
:CUSTOM_ID: examples-from-around-the-web
:END:
- [[https://raw.githubusercontent.com/SciCrunch/sparc-curation/master/docs/apinatomy.org][apinatomy.org]] \\
An executable Org file for running pipelines that build computational models of anatomy.
- [[https://raw.githubusercontent.com/SciCrunch/sparc-curation/master/docs/queries.org][queries.org]] \\
An org file set up as an interface to query knowledge bases that is
configured to provide a familiar (cua+) interface for users who may
be unfamiliar with Emacs. Has examples for loading packages, and of
one way to hide configuration to avoid information overload. A
simpler version of the file (which leverages the same orgstrap block
because the files are distributed together) is at [[https://raw.githubusercontent.com/SciCrunch/sparc-curation/master/docs/sckan/scratch.org][scratch.org]].
- [[https://raw.githubusercontent.com/SciCrunch/sparc-curation/master/docs/sckan/welcome.org][welcome.org]] \\
An org file that acts as a static welcome page for a docker image,
with links to other interactive org files. Has an example of how to
use narrowing to hide the orgstrap machinery from the user.
# TODO release.org
# TODO git-share/README.org
** Tooling for orgstrap
- [[./reval.org][reval]]
- [[./orgware.org][orgware]]
- [[./shebang.org][shebang]]
*** Services
Configurations for services that can be reused across orgstrap files.
- [[./services/blazegraph.org][Blazegraph]]
** Useful orgstrap blocks
Include these in part or whole to simplify common orgstrap workflows.
- [[file:reval.org::#minimal][reval-minimal]]
# - [[file:orgware.org::#run-command][run-process-as-command]]
# - [[file:orgware.org::#securl][securl]]
* Background, file local variables, and checksums
:PROPERTIES:
:CUSTOM_ID: background-file-local-variables-and-checksums
:END:
As mentioned above, the primary use case for =orgstrap= was that I was sick of having
to work around the limitation that I had to do one of four things. I either one, had
to remember to eval the source block containing defuns used later before I could
eval other source blocks that used those functions in headers, or two, had to put those
functions in =init.el=, destroying the ability to use org files as standalone self describing
portable and reusable computational artifacts, three, had to copy and paste verbose
elisp bits around to achieve what I wanted, or four, had to double tangle a file so that
the results of the first tangle could be loaded before calling the second tangle so that
the functionality would be available (this also produces the situation described in three).
Furthermore, it is hard for humans to follow all the steps needed to get everything
working -- even when 'everything' is just invoking =C-c C-c= on a single source block
I still forget. This can lead to _bad things_ if some of those source blocks were
interdependent, or proceeded with a nil, etc.
File local variables to the rescue!
I'm slightly embarrassed to say how long it took me to arrive at the current solution.
I had known for quite a while that file local variables are a pathway to +abilities that+
the evils of arbitrary code execution, but it didn't click that all I was looking for was
the ability to just run some arbitrary elisp code every time a particular file was loaded,
which of course is exactly what file local variables are for.
The only question then was how to avoid the very real dangers of enabling arbitrary code
execution of plain text. Actually it was more along the lines of "How can I keep org-babel
happy without also pwning myself?" Fortunately =org-confirm-babel-evaluate= can be customized
to be a function that accepts the body of the code to be evaluated. Therefore we can do the
following.
When creating a file.
1. *Hash the block to be run before distributing the file.*
Make sure to test if there are any changes to the header.
For example I have a bad habit of accidentally setting
=:noweb no-export= incorrectly without the dash and that will
prevent the checksum from updating if a nowebbed block changes.
2. *Embed the checksum in the file local variable property line.*
The property line is highly visible as the first line of the
file. This makes it easy for users to verify that the embedded
checksum matches a known independent checksum (running step 2).
Thus if the embedded checksum does not match a known checksum
the user will notice, and if the code to be executed does not
match the embedded checksum then the user will at least be
prompted by org-mode to run the block even in the case where
they accepted the file local variables. Emacs also prompts for
verification of the property line value which is another
opportunity for the user to check.
3. *Publish the checksum independent of the file itself.*
It is trivial for someone to change the contents of the orgstrap block
and rerun =M-x= =orgstrap-add-block-checksum=. Therefore known checksums
need to be published independent of the files themselves.
When running a file.
1. *Audit, accept, and store permanently the eval file local variables.*
Storing audited variables permanently is critical for improving signal to noise
so that unexpected mismatches retain their salience and can elicit the correct
response (i.e., suspicion).
# XXX there may be an issue here if the property line tags along with the rest
# because we want to be able to mark the exact variables used in this file
# as safe and if they are couple to a random hash that is bad
2. *Audit the orgstrap block*
I assume most people are not going to do this. However, one of the advantages
of the current approach is that the same orgstrap blocks can be reused across
multiple files which reduces the audit load such that one only needs to review
unique orgstrap blocks, not all files. [fn::NOTE there are certain patterns inside
blocks that are NOT safe to accept because they introduce a level of indirection
that orgstrap cannot verify. Examples of these kinds of dangerous blocks are ones
that make any reference to other blocks in the file via some means other than noweb.
This isn't really surprising, and for use cases where =org-babel-execute-src-block=
is called multiple times on different blocks, the default execution protection will
work. In addition, any blocks which want to run automatically without prompting should
use the =orgstrap--confirm-eval= function (see [[file:::#future-work][Future work]]).]
3. *Verify that the embedded checksum matches the independent checksum.*
A known embedded checksum matching the content checksum only means that the content
matches the content observed by the provider of the independent checksum
(assuming no hash collisions).
4. *Observe whether org-mode complains that the orgstrap block has changed.*
* Experience reports
:PROPERTIES:
:CUSTOM_ID: experience-reports
:END:
** First trial
The first use case for ~orgstrap~ beyond personal use was to create a
stand alone application that would allow a user to run, modify, and
create SPARQL queries running from a local server[fn::The file itself
is in a private git repo at the moment, but the functionality will be
extracted and made public, and the repo itself will be as well.]
The primary users had no prior Emacs experience. Under supervision via
a video call they were able to follow the instructions to download
Emacs, download a zip of the file plus data and additional software,
unzip, and click on the file (which opened in Emacs by default), and
accept the local variables.
There were a couple of hiccups. In one case Java for macos was
missing. In the other case the built-in version of Org mode was not
correctly replaced so Emacs had to be restarted. In the second case
there was also an issue with multiple windows being opened and the
confirmation window for the local variables thus being hard to find.
Even with the slowdown they were able to get up and running within
about 20 minutes. This is an enormous improvement over previous
attempts which involved many hard to follow instructions that could
take nearly 3 hours to complete and debug.
* Future work
:PROPERTIES:
:CUSTOM_ID: future-work
:visibility: children
:END:
** TODO separate user-emacs-directory
:PROPERTIES:
:CREATED: [2023-02-01 Wed 21:22]
:END:
Emacs 29 introduces the long desired command line option
~--init-directory=DIR~. This solves a number of issues related to
sandboxing the impact on existing Emacs configurations of using
orgstrap for various things. That is somewhat secondary however.
More importantly, after a significant amount of experience working
with this setup, it seems fairly clear to me that the default behavior
when using orgstrap shebang blocks is that orgstrap configuration
files and more importantly packages should be kept separate from the
default ~user-emacs-directory~ to avoid a wide variety of issues.
Probably add this to ~ow-enable-use-packages~ with options to use
something like ~~/.config/orgstrap/~ or ~~/.orgstrap.d/~ and
additionally to sandbox packages for the whole file. I'm thinking that
probably won't be necessary for most use cases.
** TODO detect whether dprp-1.0 is needed, otherwise use prp-1.1
:PROPERTIES:
:CREATED: [2021-09-26 Sun 14:26]
:END:
The hashes that these generate are the same so long as there aren't
any docstrings. For orgstrap blocks that don't use comments, we can
save quite a bit of space in this way.
** DONE bug secondary runs of the orgstrap block klobber modified obce
:PROPERTIES:
:CREATED: [2021-08-19 Thu 21:47]
:END:
[[file:~/git/sparc-curation/docs/queries.org::orgstrap][orgstrap]]
the quick fix is just to manually remove the step where we restore ocbe
** DONE somehow a stale orgstrap norm func managed to sneak in I have no idea how
:PROPERTIES:
:CREATED: [2021-08-11 Wed 12:32]
:END:
I'm 99% sure that this was coming from release.org and
orgstrap-norm-func was not being reset and sticking around and messing stuff up.
This was part of the issue, though it wasn't that it was not being
reset, it was that ~orgstrap-init~ did not source the default value
because ~orgstrap-norm-func~ was incorrectly marked as a global
dynamic variable instead of as ~defvar-local~.
The other part of the issue was the we were not using the current
~orgstrap-norm-func-name~ for the buffer during ~orgstrap-init~.
Even all that wasn't quite right. ~orgstrap-norm-func~ has to be
overwritten before the checksum is added for existing files, otherwise
the stale value will persist for files where local variables were
actually accepted instead of ignored.
** TODO data section
base64 (or whatever) encode a compressed data blob, stick it in an
archived heading and then add =--pack= and =--unpack= and =--repack=
or something equivalent. This is probably the most reasonable way
to manage distributing org files that have dynamic data associated
with them, such as an sqlite database or something.
# TODO consider adding --check sha256 or one of the others
# TODO -7 -8 and -9 when the input file is > 8 16 and 32 mb
#+begin_src bash :eval never
xz -zk file
base64 -w 127 file.xz > file.xz.base64 # 127 is a nicely sized prime
xz -d file.xz
xz -dc file.lz > /dev/null
#+end_src
#+begin_src elisp
(math-prime-test 109 9999)
(math-prime-test 113 9999)
(math-prime-test 127 9999) ; this one
#+end_src
These are likely to be of interest since they can also be used to tar a whole
directory, at which point it is possible to deal with the dissociation issue.
=dired-compress-files-alist=
=dired-do-compress=
=dired-compress-file=
=base64-encode-region=
=base64-decode-region=
Well, that was productive for figuring out what was going wrong.
Turns out that =:ARCHIVE:= sections are still seen by flyspell and by
auto-complete. =narrow-to-region= seems to have the behavior we want
but it doesn't do multiple. For some reason archived text is still
being searched by =auto-complete-mode= which causes massive slowdowns.
I have looked into modifying =org-hide-archived-subtrees= so that
isearch does not open and search inside, however it does not seem to
make any difference.
apparently flyspell doesn't honor read-only so we have to use something else
and it also seems to ignore the first regexp I list here, so no good solutions
thus far, I still think that narrowing to before and after are the best solution ...
#+begin_src elisp
(add-to-list 'ispell-skip-region-alist '("^\\* data :ARCHIVE:$" . "=$"))
(add-to-list 'ispell-skip-region-alist '("^#+begin_data$" . "^#+end_data$"))
(auto-complete-mode 0)
(rainbow-delimiters-org-mode 0)
(flyspell-mode 0)
#+end_src
#+begin_src elisp
(use-package zones) ; not part of the core so hard to make use of
#+end_src
#+begin_src org
,#+startup: showall
,* data :ARCHIVE:
:PROPERTIES:
:visibility: folded
:END:
put the big stuff here
,* Bootstrap :noexport:
,#+begin_src elisp
(let ((inhibit-read-only t))
(add-text-properties 21 44543976
'(read-only t)))
,#+end_src
#+end_src
** DONE orgstrap-inspect-elvs
** DONE symlinks and vc
:PROPERTIES:
:CREATED: [2021-08-02 Mon 01:23]
:END:
vc is called via find-file-hook which explicitly runs after
hack-local-variables which explains how we are getting in so early
with the orgstrap blocks, a solution has been found
** Safe local variable values need backlinks.
~orgstrap-block-checksum-sources~ alist as a custom variable so that it is
easier for people to know what came from where in a summary even though
we have the revoke functionality.
** DONE orgstrap-edit-mode fails to active when orgstrap-mode is enabled
:PROPERTIES:
:CREATED: [2020-11-29 Sun 00:10]
:END:
annoying
this is because orgstrap-mode is idiotically broken and can't be activated
globally despite being a global minor mode >_<
** Tutorial videos for various workflows
Record a series of short screen casts to illustrate common orgstrap
authoring and consumption workflows.
** TODO Display contents of orgstrap block in other window during confirm
:PROPERTIES:
:CREATED: [2020-10-08 Thu 23:27]
:END:
One major usability feature would be to figure out how to display the
full body of the orgstrap block in the other window when the confirm
local variables dialogue was presented. It seems easy enough when
orgstrap.el is installed, however it seems like it might be hard to
implement a minimal version, but maybe not, it is basically just save
excursion and rearrange so that the local variables confirm buffer and
the orgstrap block are the only two windows visible.
Another issue is whether it is possible to do this in Emacs < 27,
since it is not possible to switch out of the confirm dialogue.
Probably use =org-babel-expand-src-block= via =call-interactively=.
This will eat into the elvs budget. This also addresses some of the
security concerns.
** EXPIRED Async blocks
:PROPERTIES:
:CREATED: [2020-10-02 Fri 17:03]
:END:
One issue that needs to be resolved is how to ensure that slow running blocks
don't freeze Emacs. Ideally this could be done via ob-async, however using
ob-async essentially means that we have to either tangle the block or we have
to sneakily noweb it in if it is an elisp block, or something, to ensure that
the inferior Emacs process has the definitions. Tangle and inject a call to
load the file in the prologue or something?
The answer is that orgstrap isn't the place to handle these issues beyond
providing documentation on best practices.
The best practice for this is to use the orgstrap block in a sane manner.
For interactive use orgstrap blocks should include variable settings and
defuns at most. Little to no actual computation should be done at that stage.
Longer running processes, such as tangling or building etc, should be masked
in =(when noninteractive body ...)=. That allows orgstrap blocks to make
the functionality defined in the file available via command line arguments
(including editing via ./orgstrapped.org --edit).
In this context the evolution of orgstrap do will be to provide a library to
make command line interaction with orgstrapped files discoverable. We are most
of the way there because there is already an implementation of docopt for elisp.
** Spec extension to support arbitrary orgstrap block names.
Since all the conventions for how this is done are defined locally by each file, you could
in principle rename the special block as you see fit, perhaps from =orgstrap= to =main= if
you need to pretend that the file is actually c source code with some special syntax.
However, this is not advisable if you care about portability since it depends on an
implementation detail of orgstrap.el which is not required by the specification. Namely
that =orgstrap-orgstrap-block-name= is not required as one of the prop line local
variables. Given the desire for the orgstrap machinery to be as unobtrusive as possible,
it is unlikely that support for an arbitrary name for the block will be added to the spec.
That said, it is worth considering how and whether to update the spec so that a conforming
implementation could do this if it wanted to. All the change does is move a convention to
an optional variable. Maybe a compact variable such as orgstrap-bn could be specified as
an optional prop line variable, and if absent the orgstrap block name defaults to
orgstrap, otherwise the block name searched is the value of the variable.
Still not 100% sure about this. It would increase the complexity of the implementation for
sure. It will require updating how and when we populate the link to orgstrap block, and
makes auditing more difficult. It also opens up a way to trick the user, namely by having
a link to an innocent looking orgstrap block, and no convention set in the prop line, or
maybe even having it in the prop line (people are habitual and assume things are not
present when they are), and then using setq-local, or some other means to change the block
name to a malicious block elsewhere in the file. Implementations would have to know to
check for this and fail if it was detected. Basically we would have to specify that if a
block named orgstrap is present in a file when orgstrap-bn is present and not set to
orgstrap, then it is a fatal error and orgstrap will not continue. Sticking this in the
900 or so chars we have left for further features seems like it would be a stretch, but
might be possible.
** Security considerations
=orgstrap= currently does not check all the headers or vars properties that materialized
onto a source block we probably need to do this. For the time being users need to check
for any hidden header properties that might be attached if the source block is buried
within a tree somewhere. See ~org-babel-one-header-arg-safe-p~ for one way that this
might be implemented in the elvs.
** DONE Batch mode
This is more effectively implemented in [[./shebang.org]] because it
bypasses the orgstrap checksum entirely. It is still secure because
the user has to intentionally run the org file as a script. The
overall complexity for the user is lower as well since they do not
have to maintain or worry about the batch helper file, and the churn
in the batch helper file is also eliminated. This section is retained
for the record.
There are a number of use cases for being able to process orgstrapped files in batch mode.
For example being able to load a file and have it automatically tangle itself vastly simplifies
a number of different workflows. =emacs -q --batch -l orgstrap-known-safe.el my-file.org= seems
like a reasonable approach. Essentially =orgstrap-known-safe.el= needs to contain the safe eval
blocks and the audited hashes so that local variable prompts will not be triggered since they
always return no when in batch mode. One additional feature is be to able to pass the checksum
on the command line. The eval variables would still have to be loaded in some way, but avoiding
the need to open and edit =orgstrap-known-safe.el= for each new file, and possibly edit it again
to remove the approval in the future.
#+begin_src elisp :results none
(let ((buffer (find-file-noselect "orgstrap-batch-helper.el.example")))
(with-current-buffer buffer
(erase-buffer)
(let (print-length print-level (print-escape-newlines t))
(insert ";;; -*- mode: emacs-lisp; lexical-binding: t -*-\n\n")
(insert ";;; add audited checksums here\n\n")
(insert "(setq-local\n orgstrap-audited-checksums\n '(\n\n ))\n\n")
;; TODO insert test file block checksums
(insert ";;; set audiated checksums as safe local variables\n\n")
(insert
(pp-to-string
'(mapcar (lambda (checksum-value)
(add-to-list 'safe-local-variable-values (cons 'orgstrap-block-checksum checksum-value)))
orgstrap-audited-checksums))))
(insert "\n;;; helper local variables\n\n")
(cl-loop
for local-variable in '((orgstrap-cypher . sha256)
(orgstrap-norm-func-name . orgstrap-norm-func--prp-1-1)
(orgstrap-norm-func-name . orgstrap-norm-func--dprp-1-0))
do
(let (print-length print-level (print-escape-newlines t))
(insert (prin1-to-string
`(add-to-list 'safe-local-variable-values ',local-variable)))
(insert "\n")))
(insert "\n;;; known eval local variables\n\n")
)
(cl-loop
for (eval-local-variable elv-checksum) in
(cl-remove-duplicates
(cl-loop
for block-name in '("example-noweb-no" "example-noweb-yes" "example-noweb-eval")
append
(cl-loop
for minimal in '(nil t)
append
(cl-loop
for norm-func-name in
'(;; orgstrap-norm-func--prp-1-0 ; deprecated do not include
orgstrap-norm-func--prp-1-1
orgstrap-norm-func--dprp-1-0)
collect
(let ((info (save-excursion
(orgstrap--goto-named-src-block block-name)
(org-babel-get-src-block-info))))
(setf (nth 6 info) "hrm")
(list (orgstrap--lv-command info minimal norm-func-name)
(secure-hash
orgstrap-cypher
(orgstrap-norm
(let (print-quoted print-length print-level)
(prin1-to-string (orgstrap--lv-command info minimal norm-func-name)))))
)))))
:test #'equal)
do
(with-current-buffer buffer
(let (print-length print-level (print-escape-newlines t))
(insert (prin1-to-string `(add-to-list 'orgstrap-known-elvs ,elv-checksum)))
(insert "\n")
(insert (prin1-to-string
`(add-to-list 'safe-local-eval-forms ',eval-local-variable)))
(insert "\n"))))
(with-current-buffer buffer
(save-buffer)))
#+end_src
#+name: example-noweb-no
#+begin_src elisp :noweb no
#+end_src
#+name: example-noweb-yes
#+begin_src elisp :noweb yes
#+end_src
#+name: example-noweb-eval
#+begin_src elisp :noweb eval
#+end_src
#+name: test-batch-helper
#+begin_src bash
emacs -q --batch \
-l orgstrap-batch-helper.el \
--eval "(message \"\n%s\"(emacs-version))" \
--eval "(defun orgstrap-test () (error \"Test failed.\"))" \
-f toggle-debug-on-error \
--visit test-lv-list-minimal \
--eval "(message \"done\")" 2>&1
emacs -q --batch \
-l orgstrap-batch-helper.el \
--eval "(message \"\n%s\"(emacs-version))" \
--eval "(defun orgstrap-test () (error \"Test failed.\"))" \
-f toggle-debug-on-error \
--visit test-lv-list-portable \
--eval "(message \"done\")" 2>&1
#+end_src
** Run once
In principle the simplest way to do this is to use the =:cache yes= header on a block.
However, unless the state is persisted into a users =init.el= file or equivalent, then
the file would need a way to know that it had not been run when opened again in a new
Emacs session. Similar issue with opening the same file in multiple Emacs sessions at
the same time. The block simply will not run again if the cached result is present.
Therefore, since =:cache yes= by itself is a dead end for ensuring that functionality
is always available any time a file is loaded there are a couple of options.
1. Persist to =init.el=. This is evil.
2. Request to tangle and install as package.
A variant of this is simply to use package.el to install
the desired functionality in a persistent way in combination
with accept klobbering.
3. Figure out how to transparently wrap an elisp block in =unless=.
4. Advise =defun= (say what!?)? @@comment: TERROR@@
5. Figure out how to un-cache a block when Emacs exits.
This will fail in nasty, unpredictable, and hard to debug ways.
6. Set =:cache (if (boundp 'orgstrap-already-run) "yes" "no")=.
This ALMOST works. If =:cache no= embedded the sha1 sum then
we would be golden. *This seems like the best bet.*
7. Accept klobbering.
8. Advise org-babel-eval to run with org-babel-sha1-sum even when cache is not set to yes
Another possibility would be
1. put checksums of orgstrap blocks that have been run in a list
2. use a special header arg ~:run-once yes~ to mark blocks that should not run if
their checksum is already on the list.
If used with multiple blocks/multiple revals this would make it possible to
run only a subset
** Tangle once
When bootstrapping a new system there are many times when want to create a
file only if it does not already exist. The =:tangle= header does not support
this use case, but we can implement it anyway using the example below.
#+name: tangle-once-example
#+begin_src org
,#+name: orgstrap
,#+begin_src elisp
(defun tangle-once (path) (if (file-exists-p path) "no" path))
,#+end_src
,#+begin_src bash :tangle (tangle-once "./path-to-tangle")
echo lol
,#+end_src
# I think I've seen this before but you apparently can't have ,#+end_src on the line before #+end_src ... fun bug
#+end_src
** Multiple blocks
There must be only a single one of those blocks so that the rest of
the blocks can safely use the functions defined in the orgstrap block.
A single elisp block is sufficient to enable nearly all use cases involving
tangling source blocks to file without having to fight the prompts. However,
it is very much not sufficient for any use cases that involve other languages.
This is particularly an issue for org files that want to bootstrap whole systems.
The simplest solution to me seems to be to add a second prompt variable which is
an alist of source block checksums and names[fn::the names are not technically required
but are for human readability]. As soon as the =orgstrap= block is run
=orgstrap--confirm-eval= is no longer needed and can be replace with a function
that validates the other blocks from the prompt variable.
This seems like a tractable approach, but also over complicated because it is surely
easier in a case like this where blocks are very unlikely to be reused across org files
to simply =(setq-local org-confirm-babel-evaluate nil)= and tell people to audit the
whole file. The alternative in that case might be to hash all the source blocks and
validate all of them at once at the start of the orgstrap block. This might need some
additional machinery, not entirely sure, maybe just have =orgstrap-all-blocks-checksum=
that can be used in cases like that. The advantage here is that the core of the process
can be verified once and then the documentation around it can change and grow as needed.
** STARTED Support for orgstrap blocks beyond elisp
See https://github.com/tgbugs/laundry for the start of work
implementing org mode in Racket that would make it possible to have a
practical discussion about how to approach Org babel beyond Emacs.
The spec may need to be extended to allow multiple orgstrap blocks
with the same name. We might need to make a provision for the
implementation specific language blocks to allow multiple names
with the block for the main implementation language getting priority,
however that may add significantly more complexity than e.g. just
adding =orgstrap-elisp=, =orgstrap-racket=, =orgstrap-{lang}= as
block names that are also searched.
This is unlikely to be useful any time soon given that there aren't
full implementations of org mode outside of Emacs. If some setup is
written in another language the best approach is to call the other
block using the orgstrap block.
If at some point it becomes possible to use another language in the
top level of org, then I imagine that such functionality will go in as
a property or a file local variable or something like that. Early
candidates for other languages that might be supportable are other
lisp dialects. With such mechanisms in place, it would be relatively
straight forward to lift the restriction on the language type, or
rather, to include the normalized language name as part of the hash,
or possibly to include it as a prop line local variable. More
exploration would be required, but until there is some other
implementation that has an extension language that is not elisp, this
is a moot point.
** DONE Remove defun docstrings from hashing
One additional source of noise in addition to comments are defun and
defmacro docstrings. These should be dropped from the tree if they are
present. This is now partially implemented via =orgstrap--dedoc=.
The issues in [[*1.2.4][1.2.4]] with inconsistent noweb block
indentation. Is yet another reason for this. There is some lingering
inconsistency in exactly how many leading spaces are included when
nowebbing in elisp forms that include a multi-line string. This is one
of those extremely annoying but hard to fix issues related to noweb
and leading whitespace.
** Deterministic semantics preserving reordering
Reorder the expressions used in the orgstrap block alphabetically (or something like that)
according to a deterministic rule, but not in a way that changes program semantics.
For example a function definition cannot be moved after a top level invocation of that
function.
1. defuns with different names can be reordered
2. defuns with the same name can be reordered as a block but cannot
internally be reordered because the order of shadowing matters
3. While it might be nice to completely erase the names of functions as well
as internal variable names, this would make it trivial to shadow existing
function names in ways that are malicious. The exact names matter, so we
have to preserve them. Also the cost of not being able to tell that
=(lambda (a) (+ a a))= and =(lambda (b) (+ b b))= are the same seems fairly
small.
4. One potential approach is to lift all defuns to the top, and then function calls
or whatever the more generic procedure invocation means. The simple local rule
is that all definitions must occur before usage except in the case where there is
a shadowing event that happens after a first invocation. This is annoying, but
if a call to a function happens before that function is defined we have to assume
that the call is calling some other function and those statements cannot be reordered.
So the ordering is calls to functions with names matching any later defuns or
any later assignment. Then defuns and assignments, finally procedure invocations
which might also include assignments. I get the sense that this is covered under
some part of compiler theory but can't quite put my finger on it.
** Figure out how to demo loading the packages used in this file
#+begin_src elisp
(use-package org-ref) ; for ref:
(use-package ox-gfm) ; simplify export of changelog for release
(use-package flycheck-package) ; linting for melpa
#+end_src
** Orgware run
One potential way to simplify command line execution of orgstrapped
files is to write a wrapper that is aware of the internal variables to
control various aspects of orgstrap behavior, such as config, testing,
tangling, setup, build/make, dependencies, exporting, publishing, etc.
Essentially we wind up with a set of functionalities that are commonly
needed and set conventions for how to run only the desired subset.
Dealing with dependencies between functionalities is probably out of
scope. The assumption is that Emacs and system packages are required
for all further functionality, but beyond that it is not clear.
# This is also related to the self describing nature of the orgstrap
# approach which means that the diversity explosion steps, such as
# dealing with the huge variety of package managers (discussed above),
# will have to accumulate over time in the files themselves, or in a
# portability layer that outside the files (e.g. via reval)
** Testing and continuous integration
Ideally would like to move toward using ert for testing in this file.
Generally it would be good to have an established workflow for testing
orgstrap files beyond just nowebbing into ~when orgstrap-do-test~.
As those workflows become established it would be nice to have a set
of minimal wrappers for the various CI systems that will be sufficient
to kick off the testing. See https://github.com/purcell/nix-emacs-ci
for one way we might approach this.
** STARTED Lexical variant
This saves about 300 chars over the current minimal version.
#+begin_src elisp :results none
(cl-defun orgstrap--length-of-sexp-at-point (&aux print-level print-length (print-escape-newlines t))
(interactive)
(message
"%s"
(length
(prin1-to-string
(read
;; (org-babel-get-src-block-info)
(thing-at-point 'sexp))))))
(defalias 'length-of-sexp-at-point #'orgstrap--length-of-sexp-at-point)
#+end_src
#+begin_src elisp
(let ((n "8.2.10") ; need
(a (org-version)) ; actual
(org-confirm-babel-evaluate #'orgstrap--confirm-eval)
(obs (org-babel-find-named-block "orgstrap")))
(or
(fboundp #'orgstrap--confirm-eval)
(not n)
(string< n a)
(string= n a)
(error "Org too old! %s < %s" a n))
(defun orgstrap-norm-func--prp-1-1 (body)
(let (print-quoted print-length print-level)
(prin1-to-string (read (concat "(progn\n" body "\n)")))))
(unless (fboundp #'orgstrap-norm)
(defun orgstrap-norm (body) (funcall orgstrap-norm-func-name body)))
(unless (fboundp 'orgstrap--confirm-eval)
(defun orgstrap--confirm-eval (lang body)
(not (and (member lang '("elisp" "emacs-lisp"))
(eq orgstrap-block-checksum
(intern (secure-hash orgstrap-cypher (orgstrap-norm body))))))))
(unwind-protect
(save-excursion
(goto-char obs)
(org-babel-execute-src-block))
(org-set-startup-visibility)))
#+end_src
** DONE Buffer local functions
See [[file:./defl.org]] for an awful hack.
** DONE fix orgstrap--update-on-change hook so that it does not modify buffer fold state
CLOSED: [2021-08-29 Sun 16:36]
:PROPERTIES:
:CREATED: [2020-10-21 Wed 01:28]
:END:
Not sure exactly which function is causing this but getting the
checksum of the block seems like it might be the one, unfolding and
not restoring the state.
The issue was in =orgstrap--with-block= because =goto-char= will cause
outlines to open and we have to use =org-save-outline-visibility= with
=use-markers= set to ensure that folding is restored.
=save-restriction= was not what we needed.
https://stackoverflow.com/a/44158824 put me on the right track.
** DONE fix orgstrap-init formatting issue where comment goes in headline
:PROPERTIES:
:CREATED: [2020-10-19 Mon 18:01]
:END:
** DONE blacklist and whitelist
:PROPERTIES:
:CREATED: [2020-10-15 Thu 22:32]
:END:
#+begin_example elisp
(defcustom orgstrap-blacklist-files nil
"List of files that should not be orgstrapped."
;; TODO but that should still have orgstrap-edit-mode enabled?
:type 'list
:group 'orgstrap)
#+end_example
** TODO BUG :comments link breaks prop line :noexport:
or rather not breaks, but tries to co-exist with it which duplicates the line
which seems to break the ability to detangle among other things
** TODO behavior for activating orgstrap-mode in a buffer
What to do if we are in a buffer that has an orgstrap block?
I think the answer is to check for the local variables, and if they
are present, do nothing, if they are absent run the block?
** TODO resolve the issue with tabs in < 26 :noexport:
~prin1-to-string~ has inconsistent behavior when the string in question contains
an escaped tab character ~\t~. This can make checksums inconsistent.
** DONE command to checksum the file local variables :noexport:
This is now done as part of ~orgstrap-inspect-elvs~.
** TODO ruby org so that github can render footnotes correctly :noexport:
[[file:~/git/NOFORK/org-ruby]]
** DONE orgstrap-mode should not skip filter
:PROPERTIES:
:CREATED: [2020-10-01 Thu 21:04]
:END:
A potential issue in when running in =orgstrap-mode=. When the block
checksum has not been confirmed =orgstrap--hack-lv-confirm= might
filter/skip the checksum. I don't think this is actually what I
observed (see below).
*** I think this was never an issue
:PROPERTIES:
:CREATED: [2020-10-08 Thu 22:59]
:END:
I can't seem to reproduce this issue using the following.
#+begin_src bash
emacs -q -l orgstrap-autoloads.el \
-f toggle-debug-on-error \
--eval "(add-to-list 'load-path \"$(pwd)/\")" \
--eval "(orgstrap-mode)" orgstrap-minimal.org
#+end_src
I'm fairly certain that the issue that I was actually observing was
the fact that when there is a checksum _mismatch_, then nothing is
displayed and the block asks you to eval it -- you should decline and
inspect the block.
The issue it seems is actually that the block that we are being asked
to evaluate is not visible in the other window. So I have added a todo
item for that.
** DONE Smart update/upgrade using the version specifier
This is much easier now that I have put all the normalization and hashing
machinery in a single elv, and it no longer needs to rely
on the version of the block, just the version of the normalization function,
which is currently embedded along with everything else.
** DONE orgstrap-mode
If orgstrap is installed, then having a orgstrap-mode which could add itself to
org-mode-hook when enabled would supersede the local variables list implementation, or
possibly just do it when the local variables version was absent. Disabling the local
variables when orgstrap-mode is detect to be on might be possible, but without
it the user would have to explicitly decline the variables or risk having the orgstrap
block run twice. Going to have to get this resolved before doing any announcements since
it could leave users with quite a bit of pain if we don't get that check in before people
start using it.
** DONE Auto update block checksum on save
Before save hook and/or before commit hook to automatically update the block checksum.
** DONE use orgstrap to automatically keep example blocks in sync :noexport:
** DONE melpa :noexport:
Woo! https://melpa.org/#/orgstrap
gh:melpa/melpa/pull/7111
** DONE Determine whether to use minimal or portable based on the :noweb header
This is essentially what is implemented via =orgstrap--have-min-org-version=.
The choice to prefer the minimal local variables is left up to the user and
is controlled via the =orgstrap-use-minimal-local-variables= custom variable.
The only time when minimal is not used is if the version of org that is drafting
the file is too old (i.e. < =9.3.8=).
* Local Variables Footer :noexport:
:PROPERTIES:
:visibility: folded
:END:
# close powershell comment #>
# Local Variables:
# eval: (progn (setq-local orgstrap-min-org-version "8.2.10") (let ((a (org-version)) (n orgstrap-min-org-version)) (or (fboundp #'orgstrap--confirm-eval) (not n) (string< n a) (string= n a) (error "Your Org is too old! %s < %s" a n))) (defun orgstrap-norm-func--dprp-1-0 (body) (let ((p (read (concat "(progn\n" body "\n)"))) (m '(defun defun-local defmacro defvar defvar-local defconst defcustom)) print-quoted print-length print-level) (cl-labels ((f (b) (cl-loop for e in b when (listp e) do (or (and (memq (car e) m) (let ((n (nthcdr 4 e))) (and (stringp (nth 3 e)) (or (cl-subseq m 3) n) (f n) (or (setcdr (cddr e) n) t)))) (f e))) p)) (prin1-to-string (f p))))) (unless (boundp 'orgstrap-norm-func) (defvar-local orgstrap-norm-func orgstrap-norm-func-name)) (defun orgstrap-norm-embd (body) (funcall orgstrap-norm-func body)) (unless (fboundp #'orgstrap-norm) (defalias 'orgstrap-norm #'orgstrap-norm-embd)) (defun orgstrap-org-src-coderef-regexp (_fmt &optional label) (let ((fmt org-coderef-label-format)) (format "\\([:blank:]*\\(%s\\)[:blank:]*\\)$" (replace-regexp-in-string "%s" (if label (regexp-quote label) "\\([-a-zA-Z0-9_][-a-zA-Z0-9_ ]*\\)") (regexp-quote fmt) nil t)))) (unless (fboundp #'org-src-coderef-regexp) (defalias 'org-src-coderef-regexp #'orgstrap-org-src-coderef-regexp)) (defun orgstrap--expand-body (info) (let ((coderef (nth 6 info)) (expand (if (org-babel-noweb-p (nth 2 info) :eval) (org-babel-expand-noweb-references info) (nth 1 info)))) (if (not coderef) expand (replace-regexp-in-string (org-src-coderef-regexp coderef) "" expand nil nil 1)))) (defun orgstrap--confirm-eval-portable (lang _body) (not (and (member lang '("elisp" "emacs-lisp")) (let* ((body (orgstrap--expand-body (org-babel-get-src-block-info))) (body-normalized (orgstrap-norm body)) (content-checksum (intern (secure-hash orgstrap-cypher body-normalized)))) (eq orgstrap-block-checksum content-checksum))))) (unless (fboundp #'orgstrap--confirm-eval) (defalias 'orgstrap--confirm-eval #'orgstrap--confirm-eval-portable)) (let (enable-local-eval) (vc-find-file-hook)) (let ((ocbe org-confirm-babel-evaluate) (obs (org-babel-find-named-block "orgstrap"))) (if obs (unwind-protect (save-excursion (setq-local orgstrap-norm-func orgstrap-norm-func-name) (setq-local org-confirm-babel-evaluate #'orgstrap--confirm-eval) (goto-char obs) (org-babel-execute-src-block)) (when (eq org-confirm-babel-evaluate #'orgstrap--confirm-eval) (setq-local org-confirm-babel-evaluate ocbe)) (ignore-errors (org-set-visibility-according-to-property))) (warn "No orgstrap block."))))
# End: