Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/scymtym/configuration.options
Extensible, schema-based configuration system
https://github.com/scymtym/configuration.options
commandline configuration environment-variables schema
Last synced: 6 days ago
JSON representation
Extensible, schema-based configuration system
- Host: GitHub
- URL: https://github.com/scymtym/configuration.options
- Owner: scymtym
- License: other
- Created: 2013-03-19T12:40:25.000Z (almost 12 years ago)
- Default Branch: master
- Last Pushed: 2021-04-12T09:42:48.000Z (almost 4 years ago)
- Last Synced: 2024-12-03T23:42:09.181Z (2 months ago)
- Topics: commandline, configuration, environment-variables, schema
- Language: Common Lisp
- Homepage:
- Size: 416 KB
- Stars: 5
- Watchers: 4
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.org
- License: COPYING
Awesome Lists containing this project
README
#+TITLE: configuration.options README
#+AUTHOR: Jan Moringen
#+EMAIL: [email protected]
#+DESCRIPTION: Description, tutorial and reference for the configuration.options system
#+KEYWORDS: common lisp, options, configuration
#+LANGUAGE: en
* Introduction
The =configuration.options= system provides
+ data structures and functions for hierarchical configuration
schemata and options
+ sources of option values (builtin sources are configuration files,
environment variables and commandline options)
+ and handling of changes of option values
All of these aspects are extensible via protocols.
#+ATTR_HTML: :alt "build status image" :title Build Status :align right
[[https://travis-ci.org/scymtym/configuration.options][https://travis-ci.org/scymtym/configuration.options.svg]]
* STARTED Tutorial
#+BEGIN_SRC lisp :exports results :results silent
(defun first-line-or-less (string)
(let ((end (min 70 (or (position #\Newline string) (length string)))))
(subseq string 0 end)))
(defun provider-table (service)
(loop :for (name . provider) :in (service-provider:service-providers/alist service)
:collect `(,name ,(first-line-or-less (documentation provider t)))))
(defun provider-table/sorted (service)
(sort (provider-table service) #'string< :key #'first))
#+END_SRC
Implementing configuration processing using the
=configuration.options= system involves at least three steps:
1. [[*Specifying a Schema]]
2. [[*Constructing and Populating a Configuration]] based on the schema
3. [[*Querying a Configuration]]
** Names
Since options (and the corresponding schema items) are organized
into a hierarchy, option names are a sequence of multiple
components. The notation =COMPONENT₁.COMPONENT₂.…= is used when
representing names as strings.
"Wildcard names" are names in which one or more components is
~:wild~ or ~:wild-inferiors~.
The following functions deal with names:
#+BEGIN_SRC lisp :results values :exports results :colnames '("Form" "Result")
(mapcar (lambda (example)
(destructuring-bind (function arguments) example
(let ((*package* (find-package :configuration.options)))
(list (format nil "(~(~A~)~{ ~S~})" (symbol-name function) arguments)
(prin1-to-string (apply function arguments))))))
`((configuration.options:parse-name ("a.b.\"c.d\""))
(configuration.options:make-name ("a.b.c"))
(configuration.options:make-name ("a.*.c"))
(configuration.options:make-name ("a.**.c"))
(configuration.options:make-name (("a" "b" "c")))
(configuration.options:name-components (,(configuration.options:make-name "a.**.c")))
(configuration.options:name-equal (,(configuration.options:make-name "a.b.c")
,(configuration.options:make-name "d.e.f")))
(configuration.options:name-matches (,(configuration.options:make-name "d.**.g")
,(configuration.options:make-name "d.e.f.g")))
(configuration.options:name-equal (,(configuration.options:make-name "a.b.c")
,(configuration.options:make-name "d.e.f")))
(configuration.options:merge-names (,(configuration.options:make-name "a.b.c")
,(configuration.options:make-name "d.e.f")))))
#+END_SRC
#+RESULTS:
| Form | Result |
|-----------------------------------------------------------------------+--------------------------------------|
| (parse-name "a.b.\"c.d\"") | ("a" "b" "c.d") |
| (make-name "a.b.c") | ("a" "b" "c") |
| (make-name "a.*.c") | # |
| (make-name "a.**.c") | # |
| (make-name ("a" "b" "c")) | ("a" "b" "c") |
| (name-components #) | ("a" :WILD-INFERIORS "c") |
| (name-equal ("a" "b" "c") ("d" "e" "f")) | NIL |
| (name-matches # ("d" "e" "f" "g")) | T |
| (name-equal ("a" "b" "c") ("d" "e" "f")) | NIL |
| (merge-names ("a" "b" "c") ("d" "e" "f")) | ("a" "b" "c" "d" "e" "f") |
** Specifying a Schema
A schema can be defined in multiple ways:
+ "Manually" via multiple function and method calls
+ Declaratively using ~configuration.options:eval-schema-spec~
+ Declaratively using ~configuration.options:define-schema~
Since the third method is likely the most commonly used (and uses
the same syntax as the second method), it is probably sufficient to
only discuss ~configuration.options:define-schema~. Here is an
example:
#+BEGIN_SRC lisp :results silent :exports both
(configuration.options:define-schema *my-schema*
"Configuration schema for my program."
("logging"
("appender" :type '(member :file :standard-output)
:default :standard-output
:documentation
"Appender to use.")
((:wild-inferiors "level") :type '(member :info :warning :error)
:documentation
"Package/module/component log level.")))
#+END_SRC
The above code creates a schema object and stores it in the
parameter ~*my-schema*~. The schema consists of two items:
1. ~logging.appender~ with allowed values ~:file~ and
~:standard-output~ and default value ~:standard-output~
2. a "template" option named =logging.**.level= with allowed values
~:info~, ~:warning~ and ~:error~ and without a default value
#+BEGIN_SRC lisp :results output :exports both
(describe *my-schema*)
#+END_SRC
#+RESULTS:
#+begin_example
#
Tree:
│ Configuration schema for my program.
└─logging
├─appender
│ Type (MEMBER FILE STANDARD-OUTPUT)
│ Default :STANDARD-OUTPUT
│ Appender to use.
└─**
└─level
Type (MEMBER INFO WARNING ERROR)
Default
Package/module/component log level.
#+end_example
*** STARTED Types
As demonstrated [[*Specifying a Schema][above]], each schema item has an associated type
that describes the allowed values of associated options, as types
tend to do. In addition to that, types are used to control the
parsing and unparsing of option values. For better or worse,
schema item types are specified using Common Lisp type specifiers
such as ~(member :info :warning :error)~ in the above example. The
validation, parsing and unparsing behavior for types is
implemented using an extensible protocol. This protocol is used
by, for example, the =configuration.options-and-puri= system to
add support for additional types.
The builtin types are:
#+BEGIN_SRC lisp :results value table :exports results
(sort
(remove-duplicates
(alexandria:mappend
(lambda (method)
(let ((specializer (third (c2mop:method-specializers method))))
(when (typep specializer 'c2mop:eql-specializer)
(let* ((type-specifier (c2mop:eql-specializer-object specializer))
(documentation (documentation type-specifier 'type))
(description (cond
((eq (symbol-package type-specifier)
(find-package '#:common-lisp))
"«standard»")
(documentation
(first-line-or-less documentation))
(t
"«not documented»"))))
(list (list type-specifier description))))))
(c2mop:generic-function-methods #'configuration.options:value->string-using-type))
:test #'eq :key #'first)
#'string-lessp :key #'first)
#+END_SRC
#+RESULTS:
| AND | «standard» |
| BOOLEAN | «standard» |
| CONFIGURATION.OPTIONS:DIRECTORY-PATHNAME | A pathname syntactically suitable for designating a directory. |
| CONFIGURATION.OPTIONS:FILE-PATHNAME | A pathname syntactically suitable for designating a file. |
| INTEGER | «standard» |
| LIST | «standard» |
| MEMBER | «standard» |
| NULL | «standard» |
| OR | «standard» |
| PATHNAME | «standard» |
| STRING | «standard» |
*** TODO Sub-schemata
** Constructing and Populating a Configuration
Configurations are created from schemata by first creating an empty
configuration object and then populating it with option objects
corresponding to schema item objects in the schema:
#+BEGIN_SRC lisp :results silent :exports both
(defparameter *my-configuration* (configuration.options:make-configuration *my-schema*))
#+END_SRC
The created configuration is empty:
#+BEGIN_SRC lisp :results output :exports both
(describe *my-configuration*)
#+END_SRC
#+RESULTS:
#+begin_example
#
Tree:
#+end_example
There are several ways to create option objects from schema item
objects:
1. "Manually", options can be created using the ~make-option~
generic function (this also works if the corresponding to schema
items have wild names):
#+BEGIN_SRC lisp :exports both
(let* ((name "logging.mypackage.myparser.level")
(schema-item (configuration.options:find-option
name *my-schema*
:interpret-wildcards? :container)))
(setf (configuration.options:find-option name *my-configuration*)
(configuration.options:make-option schema-item name)))
#+END_SRC
#+RESULTS:
: # {100B8A4FE3}>
Note that the schema item named =logging.**.level= matches the
requested name because of its ~:wild-inferiors~ name
component. Also note that creating an option object does not
automatically assign a value to it (even if the schema item
specifies a default value).
The schema item lookup and ~make-option~ call in the above code
can be done automatically, shortening the example to:
#+BEGIN_SRC lisp :exports both
(configuration.options:find-option
"logging.mypackage.mylexer.level" *my-configuration*
:if-does-not-exist :create)
#+END_SRC
#+RESULTS:
: # {100B8DD5C3}>
2. Using a "synchronizer" which integrates data from sources such
as configuration files into configuration objects:
#+BEGIN_SRC lisp :results silent :exports both
(defun populate-configuration (schema configuration)
(let ((synchronizer (make-instance 'configuration.options:standard-synchronizer
:target configuration))
(source (configuration.options.sources:make-source :defaults)))
(configuration.options.sources:initialize source schema)
(configuration.options.sources:process source synchronizer)))
(populate-configuration *my-schema* *my-configuration*)
#+END_SRC
The above example uses the simple "default values" source which
instantiates option objects for all schema items with non-wild
names and sets their values to the respective default values (if
any) stored in corresponding schema items.
After creating these option objects, the configuration looks like
this:
#+BEGIN_SRC lisp :results output :exports both
(describe *my-configuration*)
#+END_SRC
#+RESULTS:
#+begin_example
#
Tree:
└─logging
├─appender
│ Type (MEMBER FILE STANDARD-OUTPUT)
│ Default :STANDARD-OUTPUT
│ Value :STANDARD-OUTPUT
│ Sources DEFAULT:
│ :STANDARD-OUTPUT
│ Appender to use.
└─mypackage
├─mylexer
│ └─level
│ Type (MEMBER INFO WARNING ERROR)
│ Default
│ Value
│ Package/module/component log level.
└─myparser
└─level
Type (MEMBER INFO WARNING ERROR)
Default
Value
Package/module/component log level.
#+end_example
In a more realistic setting, populating the configuration would be
done exclusively using a synchronizer but with a "cascade" of
sources [fn:1] instead of just the "default values" source.
** TODO Querying a Configuration
** TODO Tracking Changes of Option Values
** More on Sources
[[*Constructing and Populating a Configuration]] introduced the
"source" and "synchronizer" concepts by demonstrating the default
values source.
In more realistic settings, a combination of multiple sources like
(from highest to lowest priority)
1. Commandline options
2. Environment variables
3. Configuration file(s) and directories
4. Default values
will be used. Cascades of this kind can be constructed by
instantiating the ~:cascade~ source with appropriate subordinate
sources:
#+BEGIN_SRC lisp :exports both :resuts value
(configuration.options.sources:make-source
:cascade
:sources '((:commandline)
(:environment-variables)
(:config-file-cascade :config-file "my-program.conf"
:syntax :ini)
(:defaults)))
#+END_SRC
#+RESULTS:
: #
A similar cascade of sources is constructed by the
~:common-cascade~ source without the need for manually specifying
the involved sources.
#+BEGIN_SRC lisp :exports both :results value
(configuration.options.sources:make-source
:common-cascade :basename "my-program" :syntax :ini)
#+END_SRC
#+RESULTS:
: #
Currently available sources are:
#+BEGIN_SRC lisp :exports results :results value table :colnames '("Name" "Documentation")
(provider-table/sorted 'configuration.options.sources::source)
#+END_SRC
#+RESULTS:
| Name | Documentation |
|------------------------+----------------------------------------------------------------------|
| :CASCADE | This source organizes a set of sources into a prioritized cascade. |
| :COMMANDLINE | This source obtains option values from commandline arguments. |
| :COMMON-CASCADE | This source implements a typical cascade for commandline programs. |
| :CONFIG-FILE-CASCADE | This source implements a cascade of file-based sources. |
| :DEFAULTS | This source assigns default values to options. |
| :DIRECTORY | Collects config files and creates corresponding subordinate sources. |
| :ENVIRONMENT-VARIABLES | This source reads values of environment variables. |
| :FILE | This source reads configuration data from files. |
| :STREAM | This source reads and configuration data from streams. |
The ~:stream~ (and therefore ~:file~, ~:config-file-cascade~ and
~:common-cascade~) source supports the following syntaxes:
#+BEGIN_SRC lisp :exports results :results value table :colnames '("Name" "Documentation")
(provider-table/sorted 'configuration.options.sources::syntax)
#+END_SRC
#+RESULTS:
| Name | Documentation |
|------+----------------------------------------------------------|
| :INI | Parse textual configuration information in "ini" syntax. |
| :XML | This syntax allows using some kinds of XML documents as |
** STARTED Configuration Debugging
With multiple configuration sources such as environment variables
and various configuration files, it can sometimes be hard to
understand how a particular option got its value (or did not get an
expected value). This is true in particular for users who cannot
poke around inside the program.
To alleviate this problem, the =configuration.options= system
provides a simple configuration debugging facility aimed at
users. This facility can be enabled by calling
+ ~(configuration.options.debug:enable-debugging STREAM)~ ::
To enable debug output to ~STREAM~ unconditionally
+ ~(configuration.options.debug:maybe-enable-debugging PREFIX :stream STREAM)~ ::
To enable debug output to ~STREAM~ if the environment variable
=PREFIXCONFIG_DEBUG= is set
The intention is that a program using this system calls one of
these functions before configuration processing starts.
For example, using the schema defined [[*Specifying a Schema][above]]:
#+BEGIN_SRC lisp :exports both :results output
(setf (uiop:getenv "MY_PROGRAM_LOGGING_APPENDER") "file")
(configuration.options.debug:enable-debugging *standard-output*)
(let* ((schema *my-schema*)
(configuration (configuration.options:make-configuration schema))
(synchronizer (make-instance 'configuration.options:standard-synchronizer
:target configuration))
(source (configuration.options.sources:make-source
:common-cascade :basename "my-program" :syntax :ini)))
(configuration.options.sources:initialize source schema)
(configuration.options.sources:process source synchronizer))
#+END_SRC
#+RESULTS:
#+begin_example
Configuring COMMON-CASCADE-SOURCE with child sources (highest priority first)
1. Environment variables with prefix mapping
MY_PROGRAM_LOGGING_APPENDER=file (mapped to logging.appender) -> "file"
2. Configuring CONFIG-FILE-CASCADE-SOURCE with child sources (highest priority first)
1. Current directory file "my-program.conf" does not exist
2. User config file "/home/jmoringe/.config/my-program.conf" does not exist
3. System-wide config file "/etc/my-program.conf" does not exist
#+end_example
* STARTED Integration with the =architecture.service-provider= System
The [[https://github.com/scymtym/architecture.service-provider][architecture.service-provider system]] allows defining services
and providers of these services. The integration described here adds
the ability to automatically define a configuration schema for a
given service and use a configuration object to choose, instantiate
and configure a provider:
#+BEGIN_SRC dot :file "service-provider-integration.png" :exports results
digraph {
rankdir="LR"
node [shape="box"]
service
provider1 [shape=record,label="provider 1|slot a: boolean"]
service -> provider1 [arrowhead="none",weight=1000]
service -> provider2 [arrowhead="none"]
derive_schema [label="derive schema",shape="ellipse"]
schema [shape="record",label="schema|item provider: (member 1 2)|item 1.a: boolean"]
service -> derive_schema -> schema
make_configuration [label="make configuration",shape="ellipse"]
configuration [shape="record",label="configuration|option provider = 1|option 1.a = t"]
schema -> make_configuration
make_configuration -> configuration
provider1 -> configuration [style="dashed",label="selects",arrowtail="normal",dir="back",weight=0]
instantiate_provider [label="instantiate provider", shape="ellipse"]
instance [shape="record",label="instance|slot a = t"]
configuration -> instantiate_provider
instantiate_provider -> instance
provider1 -> instance [arrowtail="emptytriangle",dir="back",weight=0]
}
#+END_SRC
#+RESULTS:
[[file:service-provider-integration.png]]
This functionionality is provided in the separate
~configuration.options-and-service-provider~ system:
#+BEGIN_SRC lisp :results silent :exports both
(asdf:load-system :configuration.options-and-service-provider)
#+END_SRC
** STARTED Deriving a Schema for A Schema
#+BEGIN_SRC lisp :results output :exports both
(service-provider:define-service my-service)
(defclass my-provider () ((a :initarg :a :type string)))
(service-provider:register-provider/class
'my-service :my-provider :class 'my-provider)
(describe
(configuration.options.service-provider:service-schema
(service-provider:find-service 'my-service)))
#+END_SRC
#+RESULTS:
#+begin_example
#
Tree:
│ Configuration options of the MY-SERVICE service.
├─provider
│ Type (PROVIDER-DESIGNATOR-MEMBER MY-PROVIDER)
│ Default
│ Selects one of the providers of the MY-SERVICE service for
│ instantiation.
└─my-provider
│ Configuration of the MY-PROVIDER provider.
└─a
Type STRING
Default
#+end_example
** STARTED Creating a Configured Provider
#+BEGIN_SRC lisp :results output :exports both
(let* ((schema (configuration.options.service-provider:service-schema
'my-service))
(configuration (configuration.options:make-configuration schema)))
(populate-configuration schema configuration)
(setf (configuration.options:option-value
(configuration.options:find-option "provider" configuration))
:my-provider
(configuration.options:option-value
(configuration.options:find-option "my-provider.a" configuration))
"foo")
(describe (service-provider:make-provider 'my-service configuration)))
#+END_SRC
#+RESULTS:
#+begin_example
#
[standard-object]
Slots with :INSTANCE allocation:
A = "foo"
#+end_example
** TODO Tracking Service Changes
* TODO Reference
* TODO Related Work
+ https://github.com/Shinmera/universal-config/
+ https://github.com/Shinmera/ubiquitous
+ https://docs.python.org/3/library/configparser.html
+ cl-config
* Settings :noexport:
#+OPTIONS: H:2 num:nil toc:t \n:nil @:t ::t |:t ^:t -:t f:t *:t <:t
#+OPTIONS: TeX:t LaTeX:t skip:nil d:nil todo:t pri:nil tags:not-in-toc
#+SEQ_TODO: TODO STARTED | DONE
* Footnotes
[fn:1] See [[*More on Sources]]