etingof/apacheconfig

Apache / Config::General configuration file parser

apacheconfig's Issues

Build docs

Let's build & upload Sphinx docs to PyPi!

NoEscape option of Config::General

Are there plans to add some option for "NoEscape" equivalent in this python version?

From perldoc:
-NoEscape
If you want to use the data ( scalar or final leaf ) without escaping special character, turn this parameter on. It is not enabled by default.

Add linter to CI

Common linters include pep8 (which is now called pycodestyle), flake8, or pylint. From talking to folks, it seems flake8 might be a good choice.

Handle backslash at end so that it doesn't consider line after blank line

For config file:

<main>
   key1 abc \
        pqr\

   key2 value2
</main>

This is parsing it into:

{"main": {"key1": "abc pqr key2 value2"}}

while expected parsing is
{"main": {"key1": "abc pqr ", "key2": "value2"}}

Also for config:

<main>
   key1 abc \
        pqr\

   <tag>
        key2 value2
    </tag>
</main>

It is raising below error:

raise ApacheConfigError("Parser error at '%s'" % p.value if p else 'Unexpected EOF')

apacheconfig.error.ApacheConfigError: Parser error at 'main'

Can you please check?

Option for closing brackets on the same line.

Apache 2.2, which is the version of Apache on CentOS 6, sometimes allows syntax like:

<Block1>
<Block2>
</Block2> </Block1>

Where the final closing brackets are on the same line. @etingof, what do you think about enabling an option to support this? It seems to actually be a bug in Apache 2.2's parser, but we may have to support it (to prevent regressions) until CentOS 6 is EOL'ed.

Unable to parse heredoc inside a block properly

Config:

<main>
    PYTHON <<MYPYTHON
        def a():
            x = y
            return
    MYPYTHON
</main>

With python version of config parser I am getting parse error

But with perl version:

$VAR1 = {
          'main' => {
                      'PYTHON' => 'def a():
        x = y
        return'
                    }
        };

Can you please check?

When starting config with backslash parsing fails

For below config:

# comment
\
<main>
    key value
</main>

This is raising an error:

raise ApacheConfigError("Illegal character '%s'" % t.value[0])

apacheconfig.error.ApacheConfigError: Illegal character ''

while perl parser parse it properly.

I am not sure whether perl's behavior is correct but just wanted to bring it to your notice.

Also it might be related to #34.

Can you please check?

Unicode support

Currently, the loader crashes on Unicode input.

What should happen when a block and option collide?

Should blocks and options have separate namespaces during parsing, or should this behavior be disallowed?
For instance:

a = 1
<a/>

The parsing results seem to vary quite a bit using Config::General as well, depending on the order of the lines.

Reading from a remote host.

Hello,
I was thinking about using your parser to create a new module for testinfra. But for this to work I would have to read from the target host via a different command then open. If you could give me some pointers on how you would implement this, I would like to create a PR with the changes.
Thanks in advance
Barnabas

Some troubles with `ApacheIncludesLexer`

Hey Ilya, one more issue from me !

It looks like there is a problem with the ApacheIncludesLexer 'cause Include operations do not work when they looks like :

Include conf.d/*.conf

... due to the 'I', and because the token is 'include[\t ]+[^\n\r]+' (and case-sensitive).

It has been a headache to debug, and I don't know how to solve this problem as you instanciate only one lex element for the whole parsing, and because of course the case-sensitivity could be important in some cases.

It works well if we change the line to :

include conf.d/*.conf

Thanks again, have a good WE 👌

EDIT : Well... We could add a new token doing the same job but with a 'I'... Maybe there is a cleaner solution ?

with make_loader() as loader: (SyntaxError: invalid syntax)

Hi,
I'm trying to parse a config file with python2.4...
when trying to load the conf file as a dict i get the following error:

CODE:
with make_loader() as loader:
httpd_dict = loader.load(httpd_conf_file)

ERROR:
with make_loader() as loader:
^
SyntaxError: invalid syntax

Can someone give me a clue?

ConfigFileReadError: Config file "ports.conf" not found in search path

Hi Ilya,

First, thank you very much for your helpful project.

if i run :

from apacheconfig import *

options = {
    'lowercasenames': True,
    'useapacheinclude': True,
    'includerelative': True,
    'includedirectories': True,
    'includeglob': True
}

with make_loader() as loader:
    config = loader.load('/etc/apache2/apache2.conf')

print(config)

and i get

$ python apache_config.py
Traceback (most recent call last):
  File "apache_config.py", line 12, in <module>
    config = loader.load('/etc/apache2/apache2.conf')
  File "/usr/local/lib/python2.7/dist-packages/apacheconfig/loader.py", line 385, in load
    return self.loads(f.read(), source=filepath)
  File "/usr/local/lib/python2.7/dist-packages/apacheconfig/loader.py", line 349, in loads
    self._ast_cache[source] = self._walkast(ast)
  File "/usr/local/lib/python2.7/dist-packages/apacheconfig/loader.py", line 325, in _walkast
    return handler(ast[1:])
  File "/usr/local/lib/python2.7/dist-packages/apacheconfig/loader.py", line 39, in g_config
    items = self._walkast(subtree)
  File "/usr/local/lib/python2.7/dist-packages/apacheconfig/loader.py", line 325, in _walkast
    return handler(ast[1:])
  File "/usr/local/lib/python2.7/dist-packages/apacheconfig/loader.py", line 92, in g_contents
    items = self._walkast(subtree)
  File "/usr/local/lib/python2.7/dist-packages/apacheconfig/loader.py", line 325, in _walkast
    return handler(ast[1:])
  File "/usr/local/lib/python2.7/dist-packages/apacheconfig/loader.py", line 261, in g_include
    raise error.ConfigFileReadError('Config file "%s" not found in search path %s' % (filename, ':'.join(configpath)))
apacheconfig.error.ConfigFileReadError: Config file "ports.conf" not found in search path .

i have an Ubuntu 16.04.1 LTS with Apache/2.4.18 (Ubuntu)

kind regards!
Oliver

Parsing incorrectly to nested array when there is comment between two lines with same key

I am using following configuration to parse

<main>
    <body>
        key value

        # comment
        key value
        key value
    </body>
</main>

and expecting result to be

{  
   "main":{  
      "body":{  
         "key":[  
            "value",
            "value",
            "value"
         ]
      }
   }
}

but parsing result of the library is

{  
   "main":{  
      "body":{  
         "key":[  
            "value",
            [  
               "value",
               "value"
            ]
         ]
      }
   }
}

Here is code used for parsing:

from apacheconfig import *
options = {
    'lowercasenames': True
}
with make_loader(**options) as loader:
    config = loader.load('test.conf')

Comment at end of value are not considered as comments

Config

<main>
    key value # comment1
    # comment2
</main>

With python parser

{
    "main": {
        "key": "value # comment1"
    }
}

with perl parser

$VAR1 = {
          'main' => {
                      'key' => 'value'
                    }
        };

Can you please check?

P.S.: You are awesome.

white space not ignored when key value pair splits in multiple lines

for the file

<mytag>
		key1 		this is a \
				very long line
</mytag>

the perl parser produces

{
  "mytag": {
    "key1": "this is a very long line"
  }
}

where as the python parser produces

{
  "mytag": {
    "key1": "this is a \t\t\t\tvery long line"
  }
}

The python parser doesn't ignores spaces if key value pair is splitting in multiple lines

Comment and escaping # in same line

Config

<main>
    key value req\#123 # comment1
    # comment2
</main>

With python parser

{
    "main": {
        "key": "value req#123 # comment1"
    }
}

with perl parser

$VAR1 = {
          'main' => {
                      'key' => 'value req#123'
                    }
        };

Can you please check?

Writable loader.py interface

Thought I'd create an issue for more thorough discussion about this!

loader.py transforms the apache configuration into a Python dict. Although we can write this configuration to a new file, a lot of information isn't saved by a Python dict-- for instance, whitespace and field ordering.

We're looking to be able to alter fields, then write them back to a file without changing the parts of the file that we did not change.

Option for disabling `OPEN_CLOSE_TAG`

Hiya! So the Apache parser doesn't support the "open/close tag" (AKA empty-element tag) -- that is, using <block/> as a shortcut for <block> </block>. This enables people to have a section like:

<Directory />
</Directory>

(a pretty common apache block!) without breaking the parser. Perl's Config::General throws an error when parsing the above since the first line is interpreted as an open/close tag, but Apache's parser is fine with it.

To resolve this conflict, I might propose adding an option to disable OPEN_CLOSE_TAG parsing.

Provide ability to save config similar to Perl's Config::General module.

Can we please add ability like save config (https://metacpan.org/pod/Config::General#SaveConfig()) which are available in Perl's Config::General module.

New line and tab characters get removed when '\' is used to split a line in heredoc

If we use '\' to split a line inside a heredoc then all the new line and tab characters are getting removed.
For example:

PYTHON <<END
def fn():
	print "hi"
	return 1 + fn2()

def fn2():
	return 3
END

produces:

{'PYTHON': 'def fn():\n\tprint "hi"\n\treturn 1 + fn2()\n\ndef fn2():\n\treturn 3'}

where as

PYTHON2 <<END2
def fn():
        print "hi"
        return 1 + \
	fn2()

def fn2():
        return 3

END2

produces:

{'PYTHON2': 'def fn(): print "hi" return 1 + fn2() def fn2(): return 3'}

Option for multi-line hash comments

Native apache has support for multi-line hash comments, such as:

# here is a \
   multi-line hash comment

where the entire body is interpreted to be a part of the hash comment.

Perl's General::Config doesn't seem to support this, but maybe we can provide an option for it.

dumps does not handle certain options correctly

The issue

I'm currently running on the latest master branch of apacheconfig for testing purposes. I am unable to correctly grab options such as LocationMatch, Directory, and VirtualHost from an Apache config file and dump them to string. The option gets correctly picked up into the config dict, but when I run dumps, it converts the directive's option into it's own directive.

Minimal Test Case

import apacheconfig as ac

configFile = """<VirtualHost *:80>
  ServerName orange.com
</VirtualHost>
"""

_options = {
    'configpath': '/etc/apache2',
    'includeglob': True,
    'multilinehashcomments': True
}
with ac.make_loader(**_options) as loader:
    configStruct = loader.loads(configFile)
    print(loader.dumps(configStruct))

I would expect it to output essentially the same data as what was contained in the string, however, it returns the following:

<VirtualHost>
  <*:80>
    ServerName "orange.com"
  </*:80>
</VirtualHost>

Use of same `ApacheConfigError` makes debugging harder

Hey,

Let's take a look at a likely case :

• A first config file uses IncludeOptional instruction
• This makes the parser trying to load some extra files
• If a parsing error is thrown during the loading of those files, it will be catched by the https://github.com/etingof/apacheconfig/blob/master/apacheconfig/loader.py#L202 statement and thus totally silenced for the developer 😕

The point is : The module should use different Error classes to allow distinction between them.

Tell me what you think about that !

Have a good day 🌞

EDIT : FYI, if there is a problem with interpolated variables in one file for instance, the whole iteration (for includeglob and surely includedirectories) is stopped.
Each g_include call should watch for throwed exceptions coming from load or _merge_contents instead of letting them being catched by g_includeoptional parent call. WDYT ?

Simplify parser.py whitespace BNF

Discussion from this comment.

Initial newline and whitespaces ignored by perl parser but not by python parser

Perl Config::General module ignores the initial newline characters and white-spaces in the beginning of heredoc but this python parser doesn't. eg:

Config :

PERL <<END_OF_CODE


    line1;
line2;
END_OF_CODE

With Perl parser:
{"PERL":"line1;\nline2;"}

With Python parser:
{'PERL': '\n\n line1;\nline2;'}

Do you think if this behavior should be improved ?

Dropping Python 2 support

Python 2 is going to be EOL'ed in 2020, so here's an issue to track dropping python 2 support

Drop support for deprecated Pythons

<2.7 and <3.4 are deprecated, I think we can drop support by changing the python_requires in setup.py.

Once we do this, we can re-enable Sphinx builds on Travis for all builds: see discussion in #77.

dumps result is different than expected

Hello,

I would have a question.
I have a config like the following:

<thing-pool 123>
  thing thing1
  thing thing2
</thing-pool 123>

After loading it the given dict looks ok.

"thing-pool": [
          {
            "123": {
                "thing": "thing1"
            }
        }
]

But after dumping this dict I get the following:

<thing-pool>
  <123>
    thing "thing1"
    thing "thing2"
  </123>
</thing-pool>

My question is if I can somehow force the dump to use the structure like on the top?

Missing PyPi dependency

Hi Ilya,

'starting working with your module around here, and it looks like the dependency to ply is not explicitly set for PyPi.
Installing apacheconfig with pip does not automatically install ply.

Are you aware of this ?

Thanks, bye 👋

`ApacheConfigError` with "<Directory />"

Hey, it's me again (already !),

Looks like the lexer parser can't handle config files with something like :

<Directory />
# ...
</Directory>

We got this error thrown :

apacheconfig.error.ApacheConfigError: Parser error at 'Directory'

This seems to come from the > not separated from the /.

Nevertheless, this configuration is likely possible with Apache default files 😨

Thanks for your time,
Bye

Support for hash escape.

Can we please have support for hash escaping.

<main>
    key value\#123 
</main>

Perl parser:

{
    "main": {
        "key": "'value\\#123"
    }
}

Python parser:

{'main': {'key': 'value#123'}}

Extra spaces at end are not trimmed

Config:

<main>
    key value 
</main>

(There is one space extra after value. i.e, key value)

With perl parser, extra space at end is trimmed off.

$VAR1 = {
          'main' => {
                      'key' => 'value'
                    }
        };

with python parser, space at end is not trimmed

{
    "main": {
        "key": "value "
    }
}

Incorrect parsing of line with only key

For config file:

<main>
   key1
   key2 value2
</main>

python parser is parsing into

{  
   "main":{  
      "key1":"key2 value2"
   }
}

while perl parse will parse into

{  
   "main" => {  
      "key1"=> undef,
      "key2"=> "value2"
   }
}

Can this be handled?

Constant-ize tag names

Turn literals in the code into a "constant" variables to make the code easier to manage and allow code analyzers to catch possible issues.

Suggested here!

Can this be extended to support expression tags in apache config?

I am not sure whether this parser can be extended to support expression tags, please comment. Something like jsp expression tags or something in these lines http://jinja.pocoo.org/

Support for multi-line block tags

Perl's General::Config has support for multi-line block tags. Something like:

<block \
  with a \
  multiline name\
>
</block>

gets turned into:
'block' => { 'with a multiline name' => {} }

As it is now, multi-line block tags are parsed as option-value statements, which leads to parsing errors later down the line.

Blank space after heredoc terminator is causing incorrect parsing

Config:

# comment
  
TAG << MYTAG

some line

MYTAG # comment2

<tag>
    key value
</tag>

Please note extra space after second MYTAG keyword.

with python parser:

{}

Also for config:

TAG << MYTAG
some line
MYTAG # comment

I am getting

raise ApacheConfigError("Parser error at '%s'" % p.value if p else 'Unexpected EOF')

apacheconfig.error.ApacheConfigError: Unexpected EOF

Can you please check?

Consolidate default options in one place

From #72 (comment) -- we could make a distinct "flavor" that lists all the default options in one place.

Problem with mergeduplicateblocks

Maybe I'm missunderstanding something but right know I have multiple Header options set in my httpd.conf.

Header ...
FileETag ...
Header ...
HttpProtocolOptions ...
Header ...

Arent't these options and should not be affected by mergeduplicateblocs but by mergeduplicateoptions? Because right know I get following Error:
ApacheConfigError: Cannot merge duplicate items "Header"

The only options I'm using are configpath and mergeduplicateblocks=True

Parsing then writing a config file adds syntax errors

Hi,
Thanks for apacheconfig. It has issues however, and some Apache drective are not handled as they should.

For instance, the following code turns the example config file into a defective version
code

import apacheconfig
with apacheconfig.make_loader(useapacheinclude=False) as loader:
        config = loader.load("/path/to/config.conf")
print(loader.dumps(config))

original config file

<IfModule mod_ssl.c>
    Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains
    <Directory />
    Require all denied
    Options None
    AllowOverride None
    </Directory>
</IfModule>

modified config file

<IfModule>
  <mod_ssl.c>
        Header "always set Strict-Transport-Security "max-age=31536000; includeSubDomains""
        <Directory>
          </>
            Require "all denied"
            Options None
            AllowOverride None
          <//>
        </Directory>
  </mod_ssl.c>
</IfModule>

(as stated in the Readme), this module changes the way the config file looks, but it also changes what they mean, which is not expected.
In this example, it has broken directives that have arguments (and Apache complains that, e.g. <IfModule> directive requires additional arguments), and has added quotes where it shouldn't.

Also, it would be great that comments were kept, even though I understand that it is hard to keep them due to the dictionary-like object the file is converted into.

I am using Python 3.7.3 and apacheconfig version 0.3.2 (installed from pip).

Complete docs migration to Sphinx

Sphinx docs infrastructure has been introduced since 0.3.0. We already have some docs (mostly, CLI usage) in README. This issue is a reminder to:

• Move CLI docs from README to Sphinx
• Make Sphinx gathering docstrings and publishing them as an API doc

Getting line number of config where error has occurred ?

It will be more intuitive if the Exceptions thrown also includes the line number of the config where error has occurred. for eg:
<open>
a 1
</close>

Then we may get something like "In line no:3 closing tag without any opening tag" or something similar.

Provide option to avoid quoting values while dumping data

Is there a way we can provide option to avoid this quoting string values while dumping strings when they have special characters?

def _dump_dict():
..........
if val.isalnum():
text += '%s%s %s\n' % (spacing, key, val)
else:
text += '%s%s "%s"\n' % (spacing, key, val)

Getting syntax error on trying to use the Parser.

Hi,

On trying to parse a config file using the parser, I get the following syntax error:

ERROR:/my_project/apacheconfig/lexer.py:358: Invalid regular expression for rule 't_OPTION_AND_VALUE'. unbalanced parenthesis
ERROR:/my_project/apacheconfig/lexer.py:358. Make sure '#' in rule 't_OPTION_AND_VALUE' is escaped with '#'
Traceback (most recent call last):
File "/my_project/test.py", line 114, in
create_object_from_config()
File "/my_project/test.py", line 75, in create_object_from_config
dict = load_config(config_path)
File "/my_project/test.py", line 39, in load_config
with make_loader(**options) as config_loader:
File "/opt/python/python-2.7/lib64/python2.7/contextlib.py", line 17, in enter
return self.gen.next()
File "/my_project/apacheconfig/init.py", line 24, in make_loader
yield ApacheConfigLoader(ApacheConfigParser(ApacheConfigLexer()),
File "/my_project/apacheconfig/lexer.py", line 144, in init
self.reset()
File "/my_project/apacheconfig/lexer.py", line 152, in reset
errorlog=log if self._debug else None
File "/usr/local/python/python-2.7/std/lib/python2.7/site-packages/ply/lex.py", line 910, in lex
raise SyntaxError("Can't build lexer")
SyntaxError: Can't build lexer

I am using the following options:

options = {
'noescape': True,
'autotrue': True,
'ccomments': False
}

Disabling multiple options doesn't work when statements are split

With the allowmultioptions set to False, the following config should throw an error but doesn't:

a = 1
<b/>
a = 2

Investigate DeprecationWarnings from ply/lex.py

On some Python 3 tests, we're getting warnings that look like the following:

ply/lex.py:760: DeprecationWarning: Flags not at the start of the expression '(?P(<t_TAG>'
ply/lex.py:498: DeprecationWarning: Flags not at the start of the expression '(?P(<t_TAG>(?'

Convert inmemory loaded file into dict

Currently to use the library we need to provide the file path as below

with make_loader() as loader:
    config = loader.load('httpd.conf')

However if the file is already loaded in the memory (say as a string), is there any way to convert directly from that in-memory loaded file into dict.

Sort imports alphabetically throughout the codebase.

With a plugin like this: https://github.com/PyCQA/flake8-import-order

Incorrect parsing of line with only key

For config file:

<main>
   key1
   key2 value2
</main>

This is raising following error

raise ApacheConfigError("Illegal character '%s'" % t.value[0])
apacheconfig.error.ApacheConfigError: Illegal character 'k'

while perl parse will parse into

{  
   "main" => {  
      "key1"=> undef,
      "key2"=> "value2"
   }
}

Can you please check?