Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/avisonovate/config

Configure a system using EDN files and clojure.spec
https://github.com/avisonovate/config

clojure component

Last synced: 2 months ago
JSON representation

Configure a system using EDN files and clojure.spec

Awesome Lists containing this project

README 

        
= Config - Smart and flexible system configuration



image:http://clojars.org/io.aviso/config/latest-version.svg[Clojars Project, link="http://clojars.org/io.aviso/config"]



Config is a very small library used to handle configuration of a server; it works

quite well with a system defined in terms of

link:https://github.com/stuartsierra/component[Stuart Sierra's component library].



link:https://medium.com/@hlship/microservices-configuration-and-clojure-4f6807ef9bea[This posting] provides

a lot of detail on the requirements and capabilities of config.



link:http://avisonovate.github.io/docs/config/[API Documentation]



== Overview



Config reads a series of files, primarily from the classpath.

The files contain contain configuration data in

link:https://github.com/edn-format/edn[EDN] format.



The files are read in a specific order, based on a set of _profiles_.

The name of the file to read is based on the profile and the variant (described shortly).



The contents of all the configuration files are converted to Clojure maps and are

deep-merged together.



The intent of profiles is that there is an approximate mapping between components and profiles:

generally, each component will have exactly one profile.



Each component may, optionally, define a link:http://clojure.org/guides/spec[spec] for its configuration

data.



But what about the

link:http://12factor.net/config[12 Factor App]'s guideline to store configuration only as environment

variables?

This is embraced by config, because the files may contain environment variable references that are expanded

at runtime.



At link:http://www.aviso.io/[Aviso], we use these features in a number of ways.

For example, for quick testing we combine a number of microservices (each of which

has its own configuration profile and schema) together into a single system.

Meanwhile, in production (on AWS) we can build a smaller system with a single microservice.

We can also provide an additional configuration file that enables configuration overrides based on environment variables

set by CloudFormation.



== Implementing Components



You might define a web service as:



[source,clojure]

----

(require '[clojure.spec :as s]

         '[io.aviso.config :as config]

         '[com.example.jetty :as jetty]

         '[com.stuartsierra.component :as component])



(defrecord WebService [port request-handler jetty-instance]



  config/Configurable



  (configure [this configuration]

    (merge this configuration))



  component/Lifecycle



  (start [this]

    (assoc :jetty-instance (jetty/run-jetty

                              request-handler

                              {:port port

                               :join? false})))



  (stop [this]

    (.stop jetty-instance)

    (assoc this :jetty-instance nil))



(s/def ::port (s/and int? pos?)

(s/def ::config (s/keys :req-un [::port])



(defn web-service

  []

  (-> (map->WebService {})

      (component/using [:request-handler])

      (config/with-config-spec :web-service ::config)))

----



This is a standard component, with the `start` and `stop` lifecycle methods,

a dependency on another component (:request-handler), and a local field

for the instance of Jetty managed by the component.



In addition, the component is configurable: it implements the `Configurable`

protocol, and receives its specific configuration as a map.

The configuration passed to the component conforms to the ::config spec;

it will have a :port key.



The `configure` method is invoked before the `start` method.



== Providing Configuration Files



Without configuration files, your application will not start up; you will see

errors about invalid specs, because there is (in this example)

no :web-service top-level key, and no :port key below that.



Configuration files are located on the class path, within a `conf` package; this means inside

the `resources/conf` folder in a typical project.



For the :web-service component, you would

provide a default configuration file, `web-service.edn`:



[source,clojure]

----

{:web-service {:port 8080}}

----



== Starting the System



And finally, build and start a system from all this:



[source,clojure]

----

(let [system (component/system-map

               :web-service (web-service)

               :request-handler (request-handler))]

    (-> system

        (config/configure-using nil)

        component/start-system))

----



The `configure-using` function reads the configuration files and assembles the configuration map,

then applies the the configuration to each component.



The second parameter to `configure-using` is map of options.



`configure-using` generates default profiles from components in the system.

Any component that declared a configuration key using `with-config-spec`

will be included.



Here, the default list of profiles is just :web-service.



The :web-service keyword is being used in three ways here:



* As the component key in the system map

* As the name of the profile for the component, identifying the configuration file(s) for the component

* As the key within the system configuration containing the component's specific configuration



Unless you have a compelling reason otherwise, you should always follow this pattern; the profile name

should match the configuration key, which should match the component key in the system map.



The default profiles are in dependency order.

If :request-handler has a configuration key, then it will be ordered ahead of :web-service, because

the :web-service component depends on the :request-handler component.



For each component that defines a configuration spec, `configure-using` will:



* Extract the component's configuration

* Conform the configuration

* Throw an exception if the configuration contains invalid data

* Either invoke the `configure` method, or associate a :configuration key, providing the conformed configuration



== Configuration Overrides



But what if you want to override part of the :web-service configuration ...

for example, to specify a different port?

This is very common ... your local development configuration is going to vary considerably from

your deployed production configuration.



This can be accomplished in a number of ways.



=== Explicit Overrides



First off all, it is possible to provide an explicit map of overrides

when constructing the configuration map:



[source,clojure]

----

   (config/configure-using {:overrides {:web-service {:port 9999}}})

----



However, that option is generally intended for special cases, such as overrides

during testing.



Most other approaches involve controlling which files are loaded to form the system configuration.



=== Explicit Profiles



So if you wish to have some overrides, you could provide a configuration file named `overrides.edn`

and ensure that is loaded after the :web-service profile:



[source,clojure]

----

   (config/configure-using {:profiles [:overrides]})

----



Implicit profiles, via `with-config-spec` are loaded first, then explicit profiles in the options.

Order can be important here, and later-loaded profiles will override earlier profiles

if there are conflicts.



=== Variants



Another option is to support an additional _variant_ to customize the configuration.



For each profile, config searches for any variant.



In this case, the file name would be `web-service-production.edn`.

`web-service` comes from the profile and `production` from the variant.



[source,clojure]

----

   (config/configure-using {:variants [:production]})

----



The nil variant (`web-service.edn`) is always loaded first to provide the defaults,

the provided variants (when they exist) overlay the nil variant.



In this example, the normal configuration is safe; it's for local testing.

Only when deploying to production does the :production variant get added in.



=== Additional Files



You could also explicitly load one or more configuration files stored on the file system

(rather than as classpath resources):



[source,clojure]

----

   (config/configure-using {:additional-files ["overrides/production.edn"]})

----



This is another possible way to provide overrides that only apply in production;

the difference being that this file is on the file system, not packaged inside the

application JARs.



== Runtime Properties



Often, especially in production, you don't know all of the configuration until

your application is actually started. For example, in a cloud provider,

important IP addresses and port numbers are often assigned dynamically.

This information is provided to the processes via environment variables.



Although this information _could_ be extracted by startup code, and provided

to the `configure-using` function using the :overrides configuration, that

is both rigid and clumsy.



Instead, it is possible to reference these dynamic properties inside the configuration

files using the special reader macros supplied by config.



Properties are:



  * Shell environment variables.



  * JVM System properties.



  * The :properties option, passed to `configure-using`.



The following reader macros are available:



#config/prop::

    Accesses dynamic properties.

    The value is either a single string key, or a vector

    of string key followed by a default value.



#config/join::

    Joins a number of values together to form a single string; this is used when

    an building a single string from a mix of properties and static text.



#config/long::

    Converts a string to a long value.  Typically used with #config/prop.



#config/keyword::

    Converts a string to a keyword value. Typically used with #config/prop.



Here's an example showing all the variants:



[source,clojure]

----

{:connection-pool

  {:user-name #config/prop ["DB_USER" "accountsuser"]

   :user-pw #config/prop "DB_PW"

   :url  #config/join ["jdbc:postgresql://"

                       #config/prop "DB_HOST"

                       ":"

                       #config/prop "DB_PORT"

                       "/accounts"]}

 :web-server

 {:port #config/long #config/prop "WEB_PORT"}}

----



In this example, the `DB_USER`, `DB_PW`, `DB_HOST`, and `DB_PORT`, and WEB_PORT environment variables

all play a role (though `DB_USER` is optional, since it has a default value).



In the final configuration, the key [:connection-pool :url] is a single string, and the key

[:web-server :port] is a long (not a string).



== License



Config is available under the terms of the Apache Software License 2.0.