Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/atlas-engineer/nfiles

User configuration and data file management
https://github.com/atlas-engineer/nfiles

Last synced: about 1 month ago
JSON representation

User configuration and data file management

Awesome Lists containing this project

README 

          
#+TITLE: NFiles



NFiles is a Common Lisp library that deals with customizable path resolution,

file persistence and loading.



Its main use case is help manage user-centric files like configuration files.



In some aspects, it can be seen as "Common Lisp 'logical pathnames' over CLOS".



** Goals



- Performance ::

  - Data is not persisted to disk if it does not need to.

  - Files are read only once (unless modified externally).

- Extensibility :: Persist any data structure the way you want it.

- Reliability :: no corruption and no data loss should occur.



** Features



- Dynamic and customizable path expansion.

- Extensible serialization and deserialization.

- Cached reads and writes ::

  When a =file= object expands to the same path as another one, a =read= or

  =write= on it won't do anything in case there was no change since last write.

- (*Experimental!*) On-the-fly PGP encryption.

- Profile support.

- On read errors, existing files are backed up.

- On write errors, no file is written to disk, the existing file is preserved.

- A =remote-file= can point to a URL, which is automatically downloaded if the

  local file is not found.



** Motivation



This package was developed after dealing with a problem when delivering Common

Lisp images: when an image is generated, path expansion may already be resolved

and thus hard-coded within the image, which makes it unfit for delivery.

Consider this:



#+begin_src lisp

> (defvar *foo-path* (uiop:xdg-config-home))

*FOO-PATH*

> *foo-path*

#P"/home/johndoe/.config/"

#+end_src



Now if I ship this image to my friend Kaboom, =*foo-path*= will expand to



#+begin_src lisp

#P"/home/johndoe/.config/"

#+end_src



on their machine instead of the expected



#+begin_src lisp

#P"/home/kaboom/.config/"

#+end_src



** Examples



A basic session:



#+begin_src lisp

(defvar *config-file* (make-instance 'nfiles:config-file :base-path #p"my-app/init.lisp"))



(nfiles:expand *config-file*)

; => #P"/home/johndoe/.config/my-app/init.lisp"



(setf (nfiles:content *config-file*) "Hello file!") ; The file is written to disk.



(nfiles:content *config-file*)

; => "Hello file!"

#+end_src



The following convenience macro ensures the file is updated when done with the

body:



#+begin_src lisp

  (nfiles:with-file-content (content *config-file*)

    (format t "Length: ~a~%" (length content))

    (setf content (serapeum:string-replace "file" content "config")))

#+end_src



The =with-paths= helper allows for let-style bindings of the expanded paths:



#+begin_src lisp

  (let ((file1 (make-instance 'nfiles:file))

        (file2 (make-instance 'nfiles:file :base-path #p"alt")))

    (nfiles:with-paths ((path1 file1)

                        (path2 file2))

      (list path1 path2)))

#+end_src



A =remote-file= works the same but needs some specialization:



#+begin_src lisp

  (defmethod nfiles:fetch ((profile nfiles:profile) (file remote-counter-file) &key)

    (dex:get (nfiles:url file)))



  ;; Optional:

  (defmethod nfiles:check ((profile nfiles:profile) (file remote-counter-file) content &key)

    (let ((path (nfiles:expand file)))

      (ironclad:byte-array-to-hex-string

       (ironclad:digest-file :sha3 path))))



  (let ((file (make-instance 'nfiles:remote-file

                             ;; The URL to download from if the file is not found on disk.

                             :url (quri:uri "https://example.org")

                             ;; Without base-path, the file won't be saved to disk.

                             :base-path #p"/tmp/index.html"

                             :checksum "794df316afac91572b899b52b54f53f04ef71f275a01c44b776013573445868c95317fc4a173a973e90addec7792ff8b637bdd80b1a6c60b03814a6544652a90")))

    ;; On access, file is automatically downloaded if needed and the checksum is verified:

    (nfiles:content file)

    ;; ...

    )

#+end_src



See the [[file:package.lisp][package]] documentation for a usage guide and more examples.



** Configuration



NFiles was designed with configurability in mind.  All configuration happens through

subclassing of the =file= and =profile= classes together with method

specialization.



All configuration methods are specialized against =profile= and =file= so that

the user can easily *compose* the behaviour:

- Profile-customization impacts all files using that profile;

- File-customization impacts the files of that specific type (or subtype)

  regardless of their profile.



Of course you can specialize against both!



The specialization methods are divided into the following:



- =resolve= :: This is where path resolution is done.  On call site, prefer the

  =expand= convenience wrapper.



- =deserialize= and =serialize= :: This is how the content is transformed

  to the file on disk.  These functions are meant to be called by the

  =read-file= and =write-file= methods.



- =read-file= and =write-file= :: This is how the file is read and written to

  disk.  These functions are responsible for calling the =deserialize= and

  =serialize= methods.



- =fetch= :: This generic function is only called for =remote-file= objects.  You

  _must_ define its methods.  It does not have any method by default so as to

  not burden NFiles with undesirable dependencies.



- =check :=: Like =fetch=, this generic function is only called for =remote-file=

  objects to test the integrity of the downloaded file.  You _must_ define its

  methods.  It does not have any method by default so as to not burden NFiles

  with undesirable dependencies.



** Conditions and restarts



Some NFiles-specific conditions are raised in case of exceptional situations to

provide for interactive and customizable behaviour:



- =external-modification= :: The file was modified externally.  See the

  =on-external-modification= slot to automate what to do in this case.



- Read error restarts can also customized, see the =on-read-error= slot to

  automate what to do in this case.



- =process-error= :: This may be raised for instance when =gpg= fails to encrypt.

  The =use-recipient= restart is provided to retry with the given recipient.



** Shadowing



NFiles 1 shadows =cl:delete=, thus you should not =:use= the package (as with

any other library anyways).



** Platform support



It's pure Common Lisp and all compilers plus all operating systems should be

supported.



Some notes:



- All compilers but SBCL depend on [[https://github.com/sionescu/iolib][IOlib]] to preserve file attributes.

- Android devices also depend on [[https://github.com/sionescu/iolib][IOlib]] to preserve file attributes,

  regardless of the compiler.

- File attributes might not be preserved on Windows.



** Roadmap



- Improve PGP support.

- Support OS-level locks (à-la Emacs / LibreOffice).

- Improve portability.

  - In particular, preservation of file attributes may not work on

    Windows.

- Compressing =write-file= and =read-file= (for instance with zstd / lz).

  - But is it such a good idea?  Users should prefer compression at

    the level of the file system.



** Change log



*** 1.1.4



- Remove =NASDF= as a dependency.



*** 1.1.3



- Ensure a =cl:pathname= is returned from =resolve=.

- Complete some missing documentation.



*** 1.1.2



- Add restarter functions for =ask=, =reload=, etc.

  Mind that =cl:delete= is now shadowed (this was necessary to preserve backward

  compatibility).

  Do not =:use= the package!



- Switch from =hu.dwim.defclass-star= to [[https://github.com/atlas-engineer/nclasses/][nclasses]].



*** 1.1.1



- Allow path expansion for =virtual-file= (as in 1.0.0).



  This restores the usefulness of virtual-files, namely to handle the

  path-expansion business while deferring the read/write business to a

  third-party.



  =virtual-profile= still nullifies path expansions (as in 1.1.0).



*** 1.1.0



- Add support for Android.

- Nullify path expansion for =virtual-file= and =virtual-profile=.

- Ensure that the =deserialize= method of =virtual-file= and =virtual-profile=

  return nil.

- Fix =basename= corner case.

- Add report messages to all restarts.



** History



NFiles was originally developed for user file management in [[https://nyxt.atlas.engineer][Nyxt]], so the "N"

may stand for it, or "New", or whatever poetic meaning you may find behind it!