Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/etingof/apacheconfig
Apache / Config::General configuration file parser
https://github.com/etingof/apacheconfig
apache httpd-conf perl-config-general
Last synced: 30 days ago
JSON representation
Apache / Config::General configuration file parser
- Host: GitHub
- URL: https://github.com/etingof/apacheconfig
- Owner: etingof
- License: bsd-2-clause
- Created: 2018-03-10T12:00:23.000Z (almost 7 years ago)
- Default Branch: master
- Last Pushed: 2020-08-12T18:25:18.000Z (over 4 years ago)
- Last Synced: 2024-10-06T13:17:48.841Z (2 months ago)
- Topics: apache, httpd-conf, perl-config-general
- Language: Python
- Homepage:
- Size: 270 KB
- Stars: 48
- Watchers: 6
- Forks: 14
- Open Issues: 24
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGES.rst
- License: LICENSE.rst
Awesome Lists containing this project
README
# Apache-style config parser
[](https://pypi.org/project/apacheconfig)
[](https://pypi.org/project/apacheconfig/)
[](https://secure.travis-ci.org/etingof/apacheconfig)
[](https://codecov.io/github/etingof/apacheconfig)
[](https://raw.githubusercontent.com/etingof/apacheconfig/master/LICENSE.rst)
This is a pure-Python implementation of Apache-like configuration file parser into Python built-in
types. Similar file format is utilized by Perl's [Config::General](http://search.cpan.org/dist/Config-General/General.pm)
module.
## How to use apacheconfig
With apacheconfig you can build a tree of Python objects from Apache configuration
file.
For example, the following Apache configuration:
```bash
name stein
age 25
color \#000000
```
Would be transformed by `apacheconfig` into:
```python
{
'cops': {
'name': 'stein',
'age': '25',
'colors': {
'color': '#000000'
}
}
```
By running the following code:
```python
from apacheconfig import *
with make_loader() as loader:
config = loader.load('httpd.conf')
print(config)
```
You can also dump the config `dict` back into Apache configuration file form
by calling `dump` or `dumps` method:
```python
from apacheconfig import *
with make_loader() as loader:
config = loader.load('httpd.conf')
# ...potentially modify the `config` dict anyhow...
print(loader.dumps(config))
```
Please, note that dumped config file might differ in its representation from the original
because `dict` form does not contain all pieces of the original file (like comments),
some items may get modified on load (like variable expansion) and included files
rendered into the single `dict`.
## Parsing options
Parser behavior can be modified by passing it one or more options. The options are passed as a dictionary:
```python
from apacheconfig import *
options = {
'lowercasenames': True
}
with make_loader(**options) as loader:
config = loader.load('httpd.conf')
print(config)
```
The options are largely patterned after Perl's Config::General module, documentation for the options is also
borrowed there.
The following options are currently supported:
### allowmultioptions
If the value is `False`, then multiple identical options are disallowed. The default is `True` in which
case values of the identical options are collected into a list.
### forcearray
You may force a single config line to get parsed into a list by turning on the option *forcearray* and by
surrounding the value of the config entry by *[]*.
Example:
```python
hostlist = [foo.bar]
```
Will result in a single value array entry if the *forcearray* option is turned on.
### lowercasenames
If set to `True`, then all options found in the config will be converted to lowercase. This allows you to
provide case-in-sensitive configs. The values of the options will not lowercased.
### nostripvalues
If set to `False`, then each value found in the config will not be right-stripped. This allows you to gather the options as long as their trailing whitespaces.
### useapacheinclude
If set to `True`, the parser will consider "include ..." as valid include statement (just like the well known
Apache include statement).
It also supports apache's "IncludeOptional" statement with the same behavior, that is, if the include file
doesn't exist no error will be thrown.
### includeagain
If set to `True`, you will be able to include a sub-configfile multiple times. With the default, `False`, duplicate
includes are silently ignored and only the first include will succeed.
Reincluding a configfile can be useful if it contains data that you want to be present in multiple places in the
data tree.
Note, however, that there is currently no check for include recursion.
### includerelative
If set to `True`, included files with a relative path (i.e. "cfg/blah.conf") will be opened from within the location
of the configfile instead from within the location of the script ($0). This works only if the configfile has a
absolute pathname (i.e. "/etc/main.conf").
If the *configpath* option has been set and if the file to be included could not be found in the location relative
to the current config file, the module will search within *configpath* for the file.
### includedirectories
If set to `True`, you may specify include a directory, in which case all files inside the directory will be loaded
in ASCII order. Directory includes will not recurse into subdirectories. This is comparable to including a directory
in Apache-style config files.
### includeglob
If set to `True`, you may specify a glob pattern for an include to include all matching files
(e.g. <>).
An include option will not cause a parser error if the glob didn't return anything.
### configpath
You can use this variable to specify a search path for relative config files which have to be included.
The apacheconfig tool will search within this path for the file if it cannot find the file at the location
relative to the current config file.
To provide multiple search paths you can specify an array reference for the path. For example:
```python
options = {
'configpath': ['dira', 'dirb']
}
```
### mergeduplicateblocks
If set to `True`, then duplicate blocks, that means blocks and named blocks, will be merged into a single one
The default behavior is to create an array if some junk in a config appears more than once.
### mergeduplicateoptions
If set to `True`, then duplicate options will be merged. That means, if the same option occurs more than once, the
last one will be used in the resulting config dictionary.
### autotrue
If set to `True`, then options in your config file, whose values are set to *true* or *false* values, will be
normalised to *1* or *0* respectively.
The following values will be considered as *true*:
- yes
- on
- 1
- true
The following values will be considered as *false*:
- no
- off
- 0
- false
This effect is case-insensitive, i.e. both *Yes* or *No* will result in *1*.
### namedblocks
This option enables tag splitting by the first whitespace and turning the trailing
piece into a nested block. `True` by default.
### flagbits
This option takes one required parameter, which must be a dictionary defining variables for which you want to preset
values. Each variable you have defined in this dictionary and which occurs in your config file, will cause this
variable being set to the preset values to which the value in the config file refers to.
Multiple flags can be used, separated by the pipe character |.
For example, this option:
```python
options = {
'flagbits': {
'mode': {
'CLEAR': '1',
'STRONG': '1',
'UNSECURE': '32bit'
}
}
}
```
In this example we are defining a variable named *mode* which may contain one or more of *CLEAR*, *STRONG* and
*UNSECURE* as value.
The appropriate config entry may look like this:
```
mode = CLEAR | UNSECURE
```
The parser will create a dictionary which will be the value of the key *mode*. This dictionary will contain all
flags which you have pre-defined, but only those which were set in the config will contain the pre-defined value,
the other ones will be undefined.
The resulting config structure would look like this after parsing:
```python
config = {
'mode': {
'CLEAR': '1',
'UNSECURE': '32bit',
'STRONG': None
}
}
```
This method allows the user to set multiple pre-defined values for one option.
Please beware, that all occurrences of those variables will be handled this way, there is no way to distinguish between
variables in different scopes. That means, if *mode* would also occur inside a named block, it would also parsed this
way.
Values which are not defined in the dictionary supplied to the parameter *flagbits* and used in the corresponding
variable in the config will be ignored.
### defaultconfig
The value of this option should be a dictionary holding default options-values. This causes the module to populate
the resulting config dictionary with the given values, which allows you to set default values for particular config
options directly.
Note that you probably want to use this with *mergeduplicateoptions*, otherwise a default value already in the
configuration file will produce an array of two values.
### interpolatevars
If set to `True`, variable interpolation will be done on your config input.
Variables can be defined everywhere in the config and can be used afterwards as the value of an option. Variables
cannot be used as keys or as part of keys.
If you define a variable inside a block or a named block then it is only visible within this block or within blocks
which are defined inside this block.
For example:
```python
# sample config which uses variables
basedir = /opt/ora
user = t_space
sys = unix
instance = INTERN
owner = $user # "t_space"
logdir = $basedir/log # "/opt/ora/log"
sys = macos
misc1 = ${sys}_${instance} # macos_INTERN
misc2 = $user # "t_space"
```
This will result in the following structure:
```python
{
'basedir': '/opt/ora',
'user': 't_space'
'sys': 'unix',
'table': {
'intern': {
'sys': 'macos',
'logdir': '/opt/ora/log',
'instance': 'INTERN',
'owner': 't_space',
'procs': {
'misc1': 'macos_INTERN',
'misc2': 't_space'
}
}
}
}
```
As you can see, the variable *sys* has been defined twice. Inside the ** block a variable *${sys}* has been
used, which then were interpolated into the value of *sys* defined inside the ** block, not the *sys* variable
one level above. If *sys* were not defined inside the ** block then the "global" variable *sys* would have
been used instead with the value of *unix*.
Variables inside double quotes will be interpolated, but variables inside single quotes will not interpolated unless
*allowsinglequoteinterpolation* option is set.
In addition you can surround variable names with curly braces to avoid misinterpretation by the parser.
### interpolateenv
If set to `True`, environment variables can be referenced in configs, their values will be substituted in place of
their reference in the value.
This option enables *interpolatevars*.
### allowsinglequoteinterpolation
By default variables inside single quotes will not be interpolated. If you turn on this option, they will be
interpolated as well.
### strictvars
By default this is set to `True`, which causes parser to fail if an undefined variable with *interpolatevars* turned
on occurs in a config. Set to `False` to avoid such error messages.
### ccomments
The parser will process C-style comments as well as hash-style comments. By default C-style comments are processed,
you can disable that by setting *ccomments* option to `False`.
### noescape
If you want to preserve escaping special characters ($\"#) in the configuration data, turn this parameter on. It is
set to `False` by default.
### preservewhitespace
When this option is enabled, the parser would collect insignificant whitespaces
into AST. This information could then be used for code generation. The default
is `False`.
### multilinehashcomments
Escaped newlines at the end of hash comments add the following line to the comment. Apache's native config parser processes hash comments this way. Set to `False` by default.
### disableemptyelementtags
When this option is enabled, the parser does not recognize empty element tags
such as ``. The former, for instance, would be re-interpreted
as opening tag with tag name `block`, and argument `name/`.
The default is `False`.
## Parser plugins
You can alter the behavior of the parser by supplying callables which will be invoked on certain hooks during
config file processing and parsing.
The general approach works like this:
```python
def pre_open_hook(file, base):
print('trying to open %s... ' % file)
if 'blah' in file:
print('ignored')
return False, file, base
else:
print('allowed')
return True, file, base
options = {
'plug': {
'pre_open': pre_open_hook
}
}
```
Output:
```bash
trying to open cfg ... allowed
trying to open x/*.conf ... allowed
trying to open x/1.conf ... allowed
trying to open x/2.conf ... allowed
trying to open x/blah.conf ... ignored
```
As you can see, we wrote a little function which takes a filename and a base directory as parameters. We tell
the parser via the *plug* option to call this sub every time before it attempts to open a file.
General processing continues as usual if the first value of the returned array is `True`. The second value of that
tuple depends on the kind of hook being called.
The following hooks are available so far:
### pre_open
Takes two parameters: *filename* and *basedirectory*.
Has to return a tuple consisting of 3 values:
- `True` or `False` (continue processing or not)
- filename
- base directory
The returned *basedirectory* and *filename* will be used for opening the file.
### pre_read
Takes two parameters: the source of the contents read from and a string containing the raw contents.
This hook gets the unaltered, original contents as it's read from a file (or some other source).
Has to return an array of 3 values:
- `True` or `False` (continue processing or not)
- the source of the contents or `None` if `loads()` is invoked rather than `load()`
- a string that replaces the read contents
You can use this hook to apply your own normalizations or whatever.
## Command-line tool
The library comes with a simple command-line tool `apacheconfigtool` which can convert Apache-style
config files into JSON. The tool is also useful for playing with config file formats and parser
options.
```bash
$ apacheconfigtool --help
usage: apacheconfigtool [-h] [-v] [--json-input] [--allowmultioptions]
[--forcearray] [--lowercasenames] [--nostripvalues]
[--useapacheinclude] [--includeagain]
[--includerelative] [--includedirectories]
[--includeglob] [--mergeduplicateblocks]
[--mergeduplicateoptions] [--namedblocks] [--autotrue]
[--interpolatevars] [--interpolateenv]
[--allowsinglequoteinterpolation] [--strictvars]
[--noescape] [--ccomments]
[--configpath CONFIGPATH] [--flagbits ]
[--defaultconfig ]
file [file ...]
Dump Apache config files into JSON
positional arguments:
file Path to the configuration file to dump
optional arguments:
-h, --help show this help message and exit
-v, --version show program's version number and exit
--json-input Expect JSON file(s) on input, produce Apache
configuration
parsing options:
--allowmultioptions Collect multiple identical options into a list
--forcearray Force a single config line to get parsed into a list
by turning on this option and by surrounding the value
of the config entry by []
--lowercasenames All options found in the config will be converted to
lowercase
--nostripvalues All values found in the config will not be right-
stripped
--useapacheinclude Consider "include ..." as valid include statement
--includeagain Allow including sub-configfiles multiple times
--includerelative Open included config files from within the location of
the configfile instead from within the location of the
script
--includedirectories Include statement may point to a directory, in which
case all files inside the directory will be loaded in
ASCII order
--includeglob Include statement may point to a glob pattern, in
which case all files matching the pattern will be
loaded in ASCII order
--mergeduplicateblocks
Duplicate blocks (blocks and named blocks), will be
merged into a single one
--mergeduplicateoptions
If the same option occurs more than once, the last one
will be used in the resulting config dictionary
--namedblocks Do not split tags by the first whitespace and turn the
trailing part into a nested block
--autotrue Turn various forms of binary values in config into "1"
and "0"
--interpolatevars Enable variable interpolation
--interpolateenv Enable process environment variable interpolation
--allowsinglequoteinterpolation
Perform variable interpolation even when being in
single quotes
--strictvars Do not fail on an undefined variable when performing
interpolation
--noescape Preserve special escape characters left outs in the
configuration values
--ccomments Do not parse C-style comments
--configpath CONFIGPATH
Search path for the configuration files being
included. Can repeat.
--flagbits Named bits for an option in form of a JSON object of
the following structure {"OPTION": {"NAME": "VALUE"}}
--defaultconfig
Default values for parsed configuration in form of a
JSON object
$ apacheconfigtool --includedirectories include-directory-test.conf
{
"final_include": "true",
"seen_first_config": "true",
"seen_second_config": "true",
"inner": {
"final_include": "true",
"seen_third_config": "true"
},
"seen_third_config": "true"
}
```
## How to get apacheconfig
The apacheconfig package is distributed under terms and conditions of 2-clause
BSD [license](https://github.com/etingof/apacheconfig/LICENSE.rst). Source code is freely
available as a GitHub [repo](https://github.com/etingof/apacheconfig).
You could `pip install apacheconfig` or download it from [PyPI](https://pypi.org/project/apacheconfig).
If something does not work as expected,
[open an issue](https://github.com/etingof/apacheconfig/issues) at GitHub.
Copyright (c) 2018-2020, [Ilya Etingof](mailto:[email protected]). All rights reserved.