bw2/configargparse

A drop-in replacement for argparse that allows options to also be set via config files and/or environment variables.

configargparse's Issues

Without argument groups, all options are in the 'optional arguments:" group

Seems like if the required=True argument is set for any argument, by default 2 argument groups should be created... required and optional. Even if required=True is set, in the help message, these show up under the text of: optional arguments:.

singleton -h and --help only show the first group of options to be parsed

Using the singleton option from the readme results in the following

$ python main.py --help
usage: main.py [-h] [--utils-setting UTILS_SETTING]

optional arguments:
  -h, --help            show this help message and exit
  --utils-setting UTILS_SETTING
                        Config-file-settable option for utils (default: None)

As you can see the main options aren't added because the -h is being processed by the first p.parse_known_args. Basically this make this great feature useless.

I've tried using a named parser too and got the same result.

I think a decent fix would be to not process -h until you hit a parse_args and completely ignore it in parse_known_args, but I haven't looked at the code at all to see how feasible that is.

Regex tests fail on 0.10.0

While testing 0.10.0, a couple of regex tests fail. This happens under both python 2.7 and 3.4.

Here is the failing portion of the log when building with python 2.7.10:

======================================================================
FAIL: testBasicCase2 (tests.test_configargparse.TestBasicUseCases)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/leo/tmp/nix-build-python2-configargparse-0.10.0.drv-0/ConfigArgParse-0.10.0/tests/test_configargparse.py", line 230, in testBasicCase2
    'positional arguments:\n'
AssertionError: Regexp didn't match: 'usage: .* \\[-h\\] --genome GENOME \\[-v\\] -g MY_CFG_FILE \\[-d DBSNP\\]\\s+\\[-f FRMT\\]\\s+vcf \\[vcf ...\\]\n\n(.+\\s+)(.+\\s+)(.+\\s+)(.+\\s+)(.+\\s+)(.+\\s+)(.+\\s+)(.+\\s+)positional arguments:\n  vcf \\s+ Variant file\\(s\\)\n\noptional arguments:\n  -h, --help \\s+ show this help message and exit\n  --genome GENOME \\s+ Path to genome file\n  -v\n  -g MY_CFG_FILE, --my-cfg-file MY_CFG_FILE\n  -d DBSNP, --dbsnp DBSNP\\s+\\[env var: DBSNP_PATH\\]\n  -f FRMT, --format FRMT\\s+\\[env var: OUTPUT_FORMAT\\]\n' not found in "usage: setup.py [-h] --genome GENOME [-v] -g MY_CFG_FILE [-d DBSNP] [-f FRMT]\n                vcf [vcf ...]\n\nArgs that start with '--' (eg. --genome) can also be set in a config file\n(/etc/settings.ini or /home/jeff/.user_settings or /home/leo/tmp/nix-build-\npython2-configargparse-0.10.0.drv-0/tmpVTIlXW or specified via -g). The\nrecognized syntax for setting (key, value) pairs is based on the INI and YAML\nformats (e.g. key=value or foo=TRUE). For full documentation of the\ndifferences from the standards please refer to the ConfigArgParse\ndocumentation. If an arg is specified in more than one place, then commandline\nvalues override environment variables which override config file values which\noverride defaults.\n\npositional arguments:\n  vcf                   Variant file(s)\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --genome GENOME       Path to genome file\n  -v\n  -g MY_CFG_FILE, --my-cfg-file MY_CFG_FILE\n  -d DBSNP, --dbsnp DBSNP\n                        [env var: DBSNP_PATH]\n  -f FRMT, --format FRMT\n                        [env var: OUTPUT_FORMAT]\n"

======================================================================
FAIL: testBasicCase2_WithGroups (tests.test_configargparse.TestBasicUseCases)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/leo/tmp/nix-build-python2-configargparse-0.10.0.drv-0/ConfigArgParse-0.10.0/tests/test_configargparse.py", line 266, in testBasicCase2_WithGroups
    self.testBasicCase2(use_groups=True)
  File "/home/leo/tmp/nix-build-python2-configargparse-0.10.0.drv-0/ConfigArgParse-0.10.0/tests/test_configargparse.py", line 244, in testBasicCase2
    'positional arguments:\n'
AssertionError: Regexp didn't match: 'usage: .* \\[-h\\] --genome GENOME \\[-v\\] -g MY_CFG_FILE \\[-d DBSNP\\]\\s+\\[-f FRMT\\]\\s+vcf \\[vcf ...\\]\n\n.+\\s+.+\\s+.+\\s+.+\\s+.+\\s+.+\\s+.+\\s+.+\\s+positional arguments:\n  vcf \\s+ Variant file\\(s\\)\n\noptional arguments:\n  -h, --help \\s+ show this help message and exit\n\ng1:\n  --genome GENOME \\s+ Path to genome file\n  -v\n  -g MY_CFG_FILE, --my-cfg-file MY_CFG_FILE\n\ng2:\n  -d DBSNP, --dbsnp DBSNP\\s+\\[env var: DBSNP_PATH\\]\n  -f FRMT, --format FRMT\\s+\\[env var: OUTPUT_FORMAT\\]\n' not found in "usage: setup.py [-h] --genome GENOME [-v] -g MY_CFG_FILE [-d DBSNP] [-f FRMT]\n                vcf [vcf ...]\n\nArgs that start with '--' (eg. --genome) can also be set in a config file\n(/etc/settings.ini or /home/jeff/.user_settings or /home/leo/tmp/nix-build-\npython2-configargparse-0.10.0.drv-0/tmptLIJ5R or specified via -g). The\nrecognized syntax for setting (key, value) pairs is based on the INI and YAML\nformats (e.g. key=value or foo=TRUE). For full documentation of the\ndifferences from the standards please refer to the ConfigArgParse\ndocumentation. If an arg is specified in more than one place, then commandline\nvalues override environment variables which override config file values which\noverride defaults.\n\npositional arguments:\n  vcf                   Variant file(s)\n\noptional arguments:\n  -h, --help            show this help message and exit\n\ng1:\n  --genome GENOME       Path to genome file\n  -v\n  -g MY_CFG_FILE, --my-cfg-file MY_CFG_FILE\n\ng2:\n  -d DBSNP, --dbsnp DBSNP\n                        [env var: DBSNP_PATH]\n  -f FRMT, --format FRMT\n                        [env var: OUTPUT_FORMAT]\n"

----------------------------------------------------------------------
Ran 1646 tests in 2.850s

FAILED (failures=2)

And with python 3.4.3:

======================================================================
FAIL: testBasicCase2 (tests.test_configargparse.TestBasicUseCases)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/leo/tmp/nix-build-python-configargparse-0.10.0.drv-0/ConfigArgParse-0.10.0/tests/test_configargparse.py", line 230, in testBasicCase2
    'positional arguments:\n'
AssertionError: Regex didn't match: 'usage: .* \\[-h\\] --genome GENOME \\[-v\\] -g MY_CFG_FILE \\[-d DBSNP\\]\\s+\\[-f FRMT\\]\\s+vcf \\[vcf ...\\]\n\n(.+\\s+)(.+\\s+)(.+\\s+)(.+\\s+)(.+\\s+)(.+\\s+)(.+\\s+)(.+\\s+)positional arguments:\n  vcf \\s+ Variant file\\(s\\)\n\noptional arguments:\n  -h, --help \\s+ show this help message and exit\n  --genome GENOME \\s+ Path to genome file\n  -v\n  -g MY_CFG_FILE, --my-cfg-file MY_CFG_FILE\n  -d DBSNP, --dbsnp DBSNP\\s+\\[env var: DBSNP_PATH\\]\n  -f FRMT, --format FRMT\\s+\\[env var: OUTPUT_FORMAT\\]\n' not found in "usage: setup.py [-h] --genome GENOME [-v] -g MY_CFG_FILE [-d DBSNP] [-f FRMT]\n                vcf [vcf ...]\n\nArgs that start with '--' (eg. --genome) can also be set in a config file\n(/etc/settings.ini or /home/jeff/.user_settings or /home/leo/tmp/nix-build-\npython-configargparse-0.10.0.drv-0/tmpa49kdjqu or specified via -g). The\nrecognized syntax for setting (key, value) pairs is based on the INI and YAML\nformats (e.g. key=value or foo=TRUE). For full documentation of the\ndifferences from the standards please refer to the ConfigArgParse\ndocumentation. If an arg is specified in more than one place, then commandline\nvalues override environment variables which override config file values which\noverride defaults.\n\npositional arguments:\n  vcf                   Variant file(s)\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --genome GENOME       Path to genome file\n  -v\n  -g MY_CFG_FILE, --my-cfg-file MY_CFG_FILE\n  -d DBSNP, --dbsnp DBSNP\n                        [env var: DBSNP_PATH]\n  -f FRMT, --format FRMT\n                        [env var: OUTPUT_FORMAT]\n"

======================================================================
FAIL: testBasicCase2_WithGroups (tests.test_configargparse.TestBasicUseCases)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/leo/tmp/nix-build-python-configargparse-0.10.0.drv-0/ConfigArgParse-0.10.0/tests/test_configargparse.py", line 266, in testBasicCase2_WithGroups
    self.testBasicCase2(use_groups=True)
  File "/home/leo/tmp/nix-build-python-configargparse-0.10.0.drv-0/ConfigArgParse-0.10.0/tests/test_configargparse.py", line 244, in testBasicCase2
    'positional arguments:\n'
AssertionError: Regex didn't match: 'usage: .* \\[-h\\] --genome GENOME \\[-v\\] -g MY_CFG_FILE \\[-d DBSNP\\]\\s+\\[-f FRMT\\]\\s+vcf \\[vcf ...\\]\n\n.+\\s+.+\\s+.+\\s+.+\\s+.+\\s+.+\\s+.+\\s+.+\\s+positional arguments:\n  vcf \\s+ Variant file\\(s\\)\n\noptional arguments:\n  -h, --help \\s+ show this help message and exit\n\ng1:\n  --genome GENOME \\s+ Path to genome file\n  -v\n  -g MY_CFG_FILE, --my-cfg-file MY_CFG_FILE\n\ng2:\n  -d DBSNP, --dbsnp DBSNP\\s+\\[env var: DBSNP_PATH\\]\n  -f FRMT, --format FRMT\\s+\\[env var: OUTPUT_FORMAT\\]\n' not found in "usage: setup.py [-h] --genome GENOME [-v] -g MY_CFG_FILE [-d DBSNP] [-f FRMT]\n                vcf [vcf ...]\n\nArgs that start with '--' (eg. --genome) can also be set in a config file\n(/etc/settings.ini or /home/jeff/.user_settings or /home/leo/tmp/nix-build-\npython-configargparse-0.10.0.drv-0/tmpy1sza7e8 or specified via -g). The\nrecognized syntax for setting (key, value) pairs is based on the INI and YAML\nformats (e.g. key=value or foo=TRUE). For full documentation of the\ndifferences from the standards please refer to the ConfigArgParse\ndocumentation. If an arg is specified in more than one place, then commandline\nvalues override environment variables which override config file values which\noverride defaults.\n\npositional arguments:\n  vcf                   Variant file(s)\n\noptional arguments:\n  -h, --help            show this help message and exit\n\ng1:\n  --genome GENOME       Path to genome file\n  -v\n  -g MY_CFG_FILE, --my-cfg-file MY_CFG_FILE\n\ng2:\n  -d DBSNP, --dbsnp DBSNP\n                        [env var: DBSNP_PATH]\n  -f FRMT, --format FRMT\n                        [env var: OUTPUT_FORMAT]\n"

----------------------------------------------------------------------
Ran 1554 tests in 3.005s

FAILED (failures=2)

Add all argparse classes to the configargparse module namespace so that the 2 modules are truly interchangeable

Argument parsing fails with --option=equals_delimiter and a config file

Test Files

test.py

#!/usr/bin/env python

import argparse
from pprint import pprint

import configargparse

a = argparse.ArgumentParser()
a.add_argument("--config")
a.add_argument("--foo")
args = a.parse_args()
pprint(args)

p = configargparse.getArgumentParser()
p.add_argument("--config", is_config_file=True)
p.add_argument("--foo")
args = p.parse_args()
pprint(args)

test.ini

foo=VALUE_IN_CONFIG

Example of Issue

Using --foo=bar (with equals) works without config parsing:

./test.py --foo=bar
Namespace(config=None, foo='bar')
Namespace(config=None, foo='bar')

However it breaks with config parsing (but not for --config):

./test.py --foo=bar --config=test.ini
Namespace(config='test.ini', foo='bar')
Namespace(config='test.ini', foo='VALUE_IN_CONFIG')

Using --foo bar (with space instead of equals) shows us the expected behavior:

./test.py --foo bar --config test.ini
Namespace(config='test.ini', foo='bar')
Namespace(config='test.ini', foo='bar')

Feature: Parse additional config file information to be merged into existing config object

Maybe this is possible already, but it isn't obvious. I'd like to parse arguments and config files at startup, but then add an additional config (either file or string contents of file) at a later time. The existing config would be updated with the settings from the new information.

arguments with config file string True get automatically converted to bool

Supposed we have the following in the INI files:

value = True

and the add command

p.add( "-v","--value", type=str, default="False" ) 

This results in an error value set to 'True' rather than a value (see below line 322)
But the user said it wants to have a type str. So it should be parsed as a string in any case and not transformed into bool automatically, this should only happen if the user did not specify a type , or really requested a type bool. (suggestion)
(I need this to happens since I reuse the parsed stuff to interpolate with other parsed ini files somewhere else in the code, and to make that happen everything needs to be strings )

I would use the following safe bool conversion:

def toBool(self,s, optionHint=None):
        """Safe converting to a boolean value, raises exception if s has wrong type or does not have the appropriate value"""
        if type(s) == str:
            try:
                if ['true','yes','on','false','no','off'].index(s.lower()) <= 2:
                    return True
                else:
                    return False
            except:
                raise ValueError("Value: %s %s !" % (s, "at option: %s" % optionsHint if optionHint else ""))
        else:
            raise ValueError("Converting type: %s with value: %s to bool is deliberately not supported!" % (type(s),s))

It would be good to be able to not automatically convert to bool here (line 322):
in convert_setting_to_command_line_arg

if value.lower() == "true":
            if type(action) not in ACTION_TYPES_THAT_DONT_NEED_A_VALUE:
                self.error("%s set to 'True' rather than a value" % key)
            args.append( action.option_strings[-1] )
``

Change API To conform to PEP 8 names

There doesn't seem to be a good reason for violating PEP 8 for some of the API.

For example, configargparse.getArgumentParser should better be named configargparse.get_argument_parser or similar.

Setting is_config_file to true causes help to cut off

When I add the config argument with is_config_file=True and I run my program with the -h option, the help only shows the -h and the -c flags. Here is the example output using my_script.py from the documentation:

~$ python ./my_script.py -h
usage: my_script.py [-h] -c MY_CONFIG

optional arguments:
  -h, --help            show this help message and exit
  -c MY_CONFIG, --my-config MY_CONFIG
                    config file path

I am running python 2.7.5 on osx 10.9.4 with configargparse 0.9.2 installed via pip.

Refactor config file parsing, add support for more robust config file format(s)

The current config file format is overly simplistic. Support for lists and booleans was added as an after-thought and works only for very simple use-cases, but doesn't generalize.

This issue coalesces Issue #10 and other format-related Issues (#16, #19).

I think the right way to move forward on this will be:

• step 1 - refactor / extract out config file-format-aware code into a separate class that can be overriden more easily.
• step 2 - implement the class to provide a more robust alternative to the current format.

Any suggestions and/or pull requests will be appreciated.

Test failures under Python 3.5

I am getting the following test failures with python 3.5.0. Maybe allow_abbrev needs to be supported?

======================================================================
ERROR: test_failures_many_groups_listargs (tests.test_configargparse.TestOptionalsDisallowLongAbbreviation)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<string>", line 208, in wrapper
  File "<string>", line 223, in test_failures
  File "<string>", line 218, in _get_parser
TypeError: __init__() got an unexpected keyword argument 'allow_abbrev'

======================================================================
ERROR: test_failures_many_groups_sysargs (tests.test_configargparse.TestOptionalsDisallowLongAbbreviation)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<string>", line 208, in wrapper
  File "<string>", line 223, in test_failures
  File "<string>", line 218, in _get_parser
TypeError: __init__() got an unexpected keyword argument 'allow_abbrev'

======================================================================
ERROR: test_failures_no_groups_listargs (tests.test_configargparse.TestOptionalsDisallowLongAbbreviation)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<string>", line 208, in wrapper
  File "<string>", line 223, in test_failures
  File "<string>", line 218, in _get_parser
TypeError: __init__() got an unexpected keyword argument 'allow_abbrev'

======================================================================
ERROR: test_failures_no_groups_sysargs (tests.test_configargparse.TestOptionalsDisallowLongAbbreviation)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<string>", line 208, in wrapper
  File "<string>", line 223, in test_failures
  File "<string>", line 218, in _get_parser
TypeError: __init__() got an unexpected keyword argument 'allow_abbrev'

======================================================================
ERROR: test_failures_one_group_listargs (tests.test_configargparse.TestOptionalsDisallowLongAbbreviation)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<string>", line 208, in wrapper
  File "<string>", line 223, in test_failures
  File "<string>", line 218, in _get_parser
TypeError: __init__() got an unexpected keyword argument 'allow_abbrev'

======================================================================
ERROR: test_failures_one_group_sysargs (tests.test_configargparse.TestOptionalsDisallowLongAbbreviation)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<string>", line 208, in wrapper
  File "<string>", line 223, in test_failures
  File "<string>", line 218, in _get_parser
TypeError: __init__() got an unexpected keyword argument 'allow_abbrev'

======================================================================
ERROR: test_successes_many_groups_listargs (tests.test_configargparse.TestOptionalsDisallowLongAbbreviation)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<string>", line 208, in wrapper
  File "<string>", line 230, in test_successes
  File "<string>", line 218, in _get_parser
TypeError: __init__() got an unexpected keyword argument 'allow_abbrev'

======================================================================
ERROR: test_successes_many_groups_sysargs (tests.test_configargparse.TestOptionalsDisallowLongAbbreviation)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<string>", line 208, in wrapper
  File "<string>", line 230, in test_successes
  File "<string>", line 218, in _get_parser
TypeError: __init__() got an unexpected keyword argument 'allow_abbrev'

======================================================================
ERROR: test_successes_no_groups_listargs (tests.test_configargparse.TestOptionalsDisallowLongAbbreviation)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<string>", line 208, in wrapper
  File "<string>", line 230, in test_successes
  File "<string>", line 218, in _get_parser
TypeError: __init__() got an unexpected keyword argument 'allow_abbrev'

======================================================================
ERROR: test_successes_no_groups_sysargs (tests.test_configargparse.TestOptionalsDisallowLongAbbreviation)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<string>", line 208, in wrapper
  File "<string>", line 230, in test_successes
  File "<string>", line 218, in _get_parser
TypeError: __init__() got an unexpected keyword argument 'allow_abbrev'

======================================================================
ERROR: test_successes_one_group_listargs (tests.test_configargparse.TestOptionalsDisallowLongAbbreviation)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<string>", line 208, in wrapper
  File "<string>", line 230, in test_successes
  File "<string>", line 218, in _get_parser
TypeError: __init__() got an unexpected keyword argument 'allow_abbrev'

======================================================================
ERROR: test_successes_one_group_sysargs (tests.test_configargparse.TestOptionalsDisallowLongAbbreviation)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<string>", line 208, in wrapper
  File "<string>", line 230, in test_successes
  File "<string>", line 218, in _get_parser
TypeError: __init__() got an unexpected keyword argument 'allow_abbrev'

----------------------------------------------------------------------
Ran 1578 tests in 1.713s

FAILED (errors=12)

feature request: Better formatting of help text

Current help text is output very strangely:

  --user USER           user to login to foobar with [env var: FOOBAR_USER]
  --password PASSWORD   user password to login to foobar with [env var:
                        FOOBAR_PASSWORD]

Please consider providing a formater that formats the help text as such:

  --user USER           user to login to foobar with
                        [env var: FOOBAR_USER]
  --password PASSWORD   user password to login to foobar with
                        [env var: FOOBAR_PASSWORD]

Help page adds "CONFIGFILE or " for each configfile, including the final one (so always have trailing "or" + " ")

Repro:

$ cat >junktest <<EOF
import configargparse
p = configargparse.Parser(default_config_files=['woot1', 'woot2'])
p.add('--foo', help="Store some foo")
print(p.parse())
EOF

$ python junktest -h
usage: junktest [-h] [--foo FOO]

Args that start with '--' (eg. --foo) can also be set in a config file (woot1
or woot2 or ) by using .ini or .yaml-style syntax (eg. foo=value). If an arg
is specified in more than one place, then command-line values override config
file values which override defaults.

optional arguments:
  -h, --help  show this help message and exit
  --foo FOO   Store some foo

Notice the relevant part:

... set in a config file (woot1 or woot2 or ) by using .ini ...

Not the end of the world, but you know.

bug when setting a float value to -10. and not -10.0

When fixing a value defined with:
parser.add('--'+par_name,type=types.FloatType,required=False,help=par_help, default=par_default)

I get an error if in the config file i set
par_name = -10.
or
par_name = -9.
The error doesn't occur in the following:
`
par_name = -10.0

par_name = 10.

par_name = 10.0
`

Command line arguments do not override config file values

If the following scheme is used and arguments are specified both in a config-file and from command line
the correct priority is not always followed.
E.g. if we specify --testarg in the config-file and --no-testarg in the command line the result will be testarg=True. This is due to the fact that --testarg and --no-testarg are treated as distinct arguments even if they refer to the same destination.

parser.add(
        '--testarg', dest='testarg', default=None, required=False,
        action='store_true'
    )
parser.add(
        '--no-testarg', dest='testarg', default=None, required=False,
        action='store_false'
    )

Allow some command line options to be merged with those in the config

Taking from the first point of Features section of the readme:

If a value is specified in more than one way then: command line > environment variables > config file values > defaults.

ConfigArgParse (CAP) overrides the values specified in the config file with those specified in the command line arguments. For most of the cases this behaviour is nice, except when it isn't.

Consider the case of a mitmproxy user who has specified some scripts in the config file, and when they specify another script in the command line, the config ones get replaced.

The expected behaviour here is the merging of the list of scripts.

I propose that CAP should add a new parameter (say multiple) to parser.add_argument that determines this override behaviour. So, for eg, the following snippet would result in an option that merges the values:

parser.add_argument(
    "-s", "--script", 
    action="append", type=str, dest="scripts" default=[],
    multiple="merge"
)

The multiple parameter could have values:

• merge - Merge the values specified in the command line, env variables & config files & defaults.
• override - Follow the normal override process we follow now.

Unable to set boolean value to False

If I have a boolean with a default of True, it appears impossible to set it to False from the command line since you can only set it to true with a --foobar singleton and you can't set it to false with an environment variable either.

(cap) /tmp $ cat ./cap.py
#!/usr/bin/env python

import configargparse

parser = configargparse.ArgParser()
parser.add('--foobar', action='store_true', default=True)
args = parser.parse_args()
parser.print_values()
if args.foobar:
    print('foobar is True')
else:
    print('foobar is False or undefined')
(cap) /tmp $ ./cap.py
foobar is True
(cap) /tmp $ export foobar=False
(cap) /tmp $ ./cap.py
foobar is True
(cap) /tmp $ export FOOBAR=False
(cap) /tmp $ ./cap.py
foobar is True
(cap) /tmp $ echo $FOOBAR $foobar
False False

new release soon?

Some important bugfixes have landed in develop but not made it into a release yet ... wondering if you can make one so that a "good" version can be resolved from PyPI without having to put GitHub URIs into my setup.py .... thanks!

help option should not be configurable through config files or environment variables

It's a bit pointless to set -h in a config file or in an environment variable. This parameter should explicitly be banned (e.g. prevented from being read) from being set in either a config file or environment variable. Also, the help text for it should not list the environment variable name.

Value from env-var doesn't work for options defined on main parser when used with subparsers

Here's a minimal example:

parser = configargparse.getArgumentParser()
parser.add('--foo', env_var='MY_FOO')
subparsers = parser.add_subparsers(help='Choose command')
sub_parser = subparsers.add_parser('sub')
args = parser.parse()

when running MY_FOO=bar ./my_script.py this works fine, but MY_FOO=bar ./my_script.py sub throws this errors:

my_script.py: error: unrecognized arguments: --foo

I assume this is because --foo is not an argument of the subparser, and this works by appending --foo $MY_FOO at the end of sys.argv...

Add multiline YAML (and/or INI).

YAML supports multiline input (https://gist.github.com/rjattrill/7523554) and lists (http://docs.ansible.com/YAMLSyntax.html#yaml-basics) but neither of those can be used in the ini file:

listoffruits:
    - Apple
    - Orange
theelements: >
    [my_extraorbitantly_long_element_address1,
    my_extraorbitantly_long_element_address2,
    my_extraorbitantly_long_element_address3]

these produce errors. I mainly wanted to use your library because I expect user to provide tens of long strings as configuration. Putting them in one-line would clutter the INI file.

Do you consider such feature useful and would like to implement it?

I also didn't find a way to provide an INI style multiline content but this is not officially supported (INI is not even a well defined standard and the most formal definitions I saw didn't mention multiline though some libs allow that with indentation).

Allow config file paths to use ~/ for specifying user's home dir

Cannot escape comment characters within value in configuration file

For a configuration file ConfigfileParser.parse(), using a semicolon embedded in the value string (my case being SQL connection strings) cause the parser to fail. There should be a way to 1.) turn off comment parsing (the whole line becomes the value) or 2.) escape individual semicolon and hash characters or 3.) quote the value to separate it from comments. (Using 0.10.0 from PyPI)

regression: unrecognized arguments error when using subparsers and config file

Test file:

#!/usr/bin/env python
import configargparse
parser = configargparse.ArgParser(args_for_setting_config_path=["-c"])
parser.add_subparsers().add_parser('foo')
parser.add_argument("-b", "--bar")
print parser.parse_args()

Test config:

bar = baz

Problem

(venv)jakub@iks:~/dev/letsencrypt/lets-encrypt-preview$ pip install git+https://github.com/bw2/[email protected]
Downloading/unpacking git+https://github.com/bw2/[email protected]
  Cloning https://github.com/bw2/ConfigArgParse.git (to 0.9.3) to /tmp/pip-80_Rgc-build
  Running setup.py (path:/tmp/pip-80_Rgc-build/setup.py) egg_info for package from git+https://github.com/bw2/[email protected]

Installing collected packages: ConfigArgParse
  Running setup.py install for ConfigArgParse

Successfully installed ConfigArgParse
Cleaning up...
(venv)jakub@iks:~/dev/letsencrypt/lets-encrypt-preview$ ./t.py -c t.ini foo                                                                                                                         
Namespace(bar='baz', config_file='t.ini')
(venv)jakub@iks:~/dev/letsencrypt/lets-encrypt-preview$ pip uninstall ConfigArgParse                                                                                                                
Uninstalling ConfigArgParse:
  /home/jakub/dev/letsencrypt/lets-encrypt-preview/venv/lib/python2.7/site-packages/ConfigArgParse-0.9.3-py2.7.egg-info
  /home/jakub/dev/letsencrypt/lets-encrypt-preview/venv/lib/python2.7/site-packages/configargparse.py
  /home/jakub/dev/letsencrypt/lets-encrypt-preview/venv/lib/python2.7/site-packages/configargparse.pyc
Proceed (y/n)? y
  Successfully uninstalled ConfigArgParse
(venv)jakub@iks:~/dev/letsencrypt/lets-encrypt-preview$ pip install git+https://github.com/bw2/ConfigArgParse.git@e0463e0ec5fbd1f453ed833abebc633c77d4e0c6                                        
Downloading/unpacking git+https://github.com/bw2/ConfigArgParse.git@e0463e0ec5fbd1f453ed833abebc633c77d4e0c6
  Cloning https://github.com/bw2/ConfigArgParse.git (to e0463e0ec5fbd1f453ed833abebc633c77d4e0c6) to /tmp/pip-6fNLp7-build
  Could not find a tag or branch 'e0463e0ec5fbd1f453ed833abebc633c77d4e0c6', assuming commit.
  Running setup.py (path:/tmp/pip-6fNLp7-build/setup.py) egg_info for package from git+https://github.com/bw2/ConfigArgParse.git@e0463e0ec5fbd1f453ed833abebc633c77d4e0c6

Installing collected packages: ConfigArgParse
  Running setup.py install for ConfigArgParse

Successfully installed ConfigArgParse
Cleaning up...
(venv)jakub@iks:~/dev/letsencrypt/lets-encrypt-preview$ ./t.py -c t.ini foo
usage: t.py [-h] [-c CONFIG_FILE] [-b BAR] {foo} ...
t.py: error: unrecognized arguments: --bar baz

Subcommands params in config?

Is there way to have global and subparsers params in one config file?

For example:

#!/usr/bin/env python3

import configargparse
parser = configargparse.ArgParser(
    default_config_files=["config.ini"],
    args_for_setting_config_path=["-c"])

sub = parser.add_subparsers().add_parser('foo')
parser.add_argument("-b", "--bar")
sub.add_argument("-f", "--foo")
print(parser.parse_args())

And config.ini:

bar = baz
foo = fool

Now I get: contains unknown config key(s): foo

PS: I know about #26 , but this is not exactly same.

Error with # in setting file values.

For example i have
SECRET_KEY=s#
Will produce error
configargparse.ConfigFileParserException: Unexpected line 0 in 123.env:

Please include the tests directory in your sdist tarball

I am packaging the module for Kali Linux (needed for a new version of mitmproxy) and I noticed that trying to run the test suite fails because the "tests" directory is missing from the tarball sent to PyPi:
https://pypi.python.org/packages/source/C/ConfigArgParse/ConfigArgParse-0.9.3.tar.gz

You might also want to create proper tags in your Git repository so that one can find out the precise commit used for each release sent to PyPi (and so that we can download tarballs from GitHub as a work-around when the PyPi tarball doesn't suit our needs, like is the case right now).

Thank you!

Positional argument seperator '--' strips config/env values

example.py

import configargparse

p = configargparse.ArgParser()
p.add('--required', required=True, env_var='TEST_REQUIRED')

print(p.parse_args())

Results

export TEST_REQUIRED=value;

python3 example.py
# Namespace(required='value')

python3 example.py --
# usage: example.py [-h] --required REQUIRED
# example.py: error: the following arguments are required: --required

Python 2.6 support

Would it be feasible to support Python 2.6 or is it a no go? Failing build log: https://travis-ci.org/letsencrypt/lets-encrypt-preview/jobs/60949344#L1366

Backref: certbot/certbot#376

Comparison with other packages is out of date

https://pypi.python.org/pypi/ConfArgParse is now 1.1.20 vs 1.0.15

https://pypi.python.org/pypi/appsettings is now 0.7 vs 0.5

https://pypi.python.org/pypi/yconf is now 0.3.3 vs 0.3.2

adding certain types of config directives to config files breaks argcomplete tab-completion

I would love to get some help with this from someone awesome.

kislyuk/argcomplete#123

missing SUPPRESS

Changed my program as follows:
import configargparse as argparse

And now:

$ tag -h
Traceback (most recent call last):
File "/home/user/make-deployment-group.py", line 31, in
parser.add_argument("--user-name", help=argparse.SUPPRESS)
AttributeError: 'module' object has no attribute 'SUPPRESS'

Pypi is not up to date 0.11.0

Hi,

Could you update Pypi, because it would be convinient as I'm using a lot Yaml configuration files?

Kind regards,

AttributeError: 'module' object has no attribute 'ArgParser'

If I try to use the package (in either Python 2.7.8 or 3.4.1), it's as if there is nothing in the module.

For instance, from within iPython I do import configargparse, then if I type configargparse. and hit the tab key there is no auto-complete.

Trying to use the package in an app yields the following error:

Traceback (most recent call last):
  File "test/server.py", line 62, in <module>
    main(args=sys.argv)
  File "test/server.py", line 47, in main
    parser = configargparse.ArgParser(default_configfiles=['/etc/settings.ini'])
AttributeError: 'module' object has no attribute 'ArgParser'

Config file output can generate illigitimate values for keys when using custom Actions

Cause: ArgumentParser.get_items_for_config_file_output makes the incorrect assumption that an argument's config file value can always be found in the parsed namespace's attribute named after action.dest. With custom Actions this value can be arbitrary.

Effect: Using the action's computed value rather than the command-line's string value results in a config key-value pair that does not reproduce the original specified options.

class CustomAction(configargparse.Action):
    def __call__(self, parser, namespace, values, option_string=None):
        setattr(namespace, self.dest, 'arbitrary')

p = configargparse.ArgumentParser(args_for_setting_config_path=['-c'],
                                  args_for_writing_out_config_file=['-w'])
p.add_argument('--number', type=int, dest='result', action=CustomAction)
p.parse_args('--number 2' -w bad.conf)

The resulting namespace:

Namespace(result='arbitrary')  # Great, CustomAction did its job

But the resulting bad.conf

number = arbitrary  # This is not going to reproduce

Now consider when bad.conf is specified to -c

error: argument --number: invalid int value: 'arbitrary'

We only recognize the issue due to the type constraint on --number. If there were none, it might be more confusing as to why our option's arg is ignored or worse depending on our action's __call__ behavior.

The main problem here is the generated config does not reproduce the behavior of the original passed in arguments when passed back in as a config file.

Solution?: Basically we need to store the value passed on the command line to obtain reproducible behavior in situations like this, but it's not clear to me how best to do so. This information is available to the action's __call__ as arguments, and so could be stored on the parser for later reference in ArgumentParser.get_items_for_config_file_output.

Do you think this is a bug?

Thank you,
Brad

ConfigFileParse does not handle action='append'

Only the last key/value pair in the file gets stored, eg:

--foo a --foo b --foo c

Will produce options.foo ['a', 'b', 'c']

But a file with:

foo = a
foo = b
foo = c

Will only produce ['c']

Allow getting values from a specific section of config file

I have a use case where my config file has a DEFAULT section, then a section for each code environment (test, dev, qa, etc.). I see in the documentation that you recommend not forcing users to put settings in specific sections, but in this case am I defining the same variables in each section. Is there a way to have my values merge only with one section? For example, I might have:

[DEFAULT]
color = blue

[dev]
name = Simon

[qa]
color = red
name = Bill

When my environment is dev, I'm going to want color: blue, name: Simon, with the possibility of overriding either from the CLI. Am I missing this functionality, or is it something you might accept a pull request for?

Add a "dump config" option

Any "--" argument can be used in a config file, and a user can see what those options are by parsing the help text, but there are other tools (like "clang-format") which will dump the config file equivalent of your command line if you ask them to.

You can then edit and modify this file as required, but you don't start from scratch, as you currently must do for ConfigArgParse.

If there's a straightforward way to do so outside of the module, that's cool, too, but I dug in a little and didn't see one.

My naive solution was to format the output of "vars(args)", but that fails to cover the cases where a flag is used (but not actually used, so its value is still "False"), or where a value is being stored to a destination different than its CLI name (via the "dest" kwarg), or where the CLI is not a "--" option.

To do this correctly, I really need to get access to the parser internals, so I'm requesting it as a new feature of the module itself.

allow unknown options in config file

Currently, only defined args are parsed from the config file, and unknown keys are ignored or cause an error (depending on the allow_unknown_config_file_keys constructor arg).

I want to change this so that unknown keys are always appended to the command line and whether they cause an error depends on whether parsing is done with parse_args() (which results in an error) or using parse_known_args() (in which case the unknown args are returned the same way as unknown command line args).

I'm still debating whether to also deprecate allow_unknown_config_file_keys constructor arg.

ConfigArgParse depends on pypandoc when installing via pip

The package is not installable via pip:

$ pip install ConfigArgParse
Downloading/unpacking ConfigArgParse
  Downloading ConfigArgParse-0.9.1.tar.gz
  Running setup.py (path:/.../build/ConfigArgParse/setup.py) egg_info for package ConfigArgParse
    ERROR: pypandoc module not found, could not convert Markdown to RST for PyPI
    Complete output from command python setup.py egg_info:
    ERROR: pypandoc module not found, could not convert Markdown to RST for PyPI

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /.../build/ConfigArgParse
Storing debug log for failure in /.../.pip/pip.log

$ python --version
Python 2.7.8

Installing pypandoc solves the problem, but the README states the library has no 3rd-party requirements. The README.md file is not converted anyway, as it is missing in the PyPI package:

$ pip install pypandoc
[...]
$ pip install ConfigArgParse
Downloading/unpacking ConfigArgParse
  Downloading ConfigArgParse-0.9.1.tar.gz
  Running setup.py (path:/.../build/ConfigArgParse/setup.py) egg_info for package ConfigArgParse
    WARNING: couldn't find README.md

Installing collected packages: ConfigArgParse
  Running setup.py install for ConfigArgParse
    WARNING: couldn't find README.md

Successfully installed ConfigArgParse
Cleaning up...

Create a more robust config file format implemented as a subclass of ConfigFileParser

Write a new implementation of ConfigFileParser to provide a more robust alternative to the current config file format which:

• ideally doesn't introduce dependencies on any 3rd party libraries
• allows multiline values (#10, #23 )
• has better support for lists ( #16 )

This issue combines and closes #10, #16, #23

ArgParser() "Description' parameter when outputted via -h option prints no newline characters

Effectively, a help text ("Args that start with ...") starts at the same line where description ends, for example:

A special tool Args that start with

Description is set to 'A special tool'

default for default_config_files=[...]

I look at the README and see this:

p = configargparse.ArgParser(default_config_files=['/etc/settings.ini', '~/.my_settings'])

I think: everybody is going to have a different list here.

But I guess there could be a default which satisfies a lot of people.

My prefered API: just provide a basename.

Then the library checks below os.environ['VIRTUAL_ENV'], os.environ['HOME'], '/etc'.

Of course the new feature should not make old code break.

LICENSE file is missing year and copyright holder's name

It just says, "Copyright (c) <year> <copyright holders>"

Combination of nargs='+' and type=float fails when reading from config file

I love the work you've done with this package, it has spared me from so many parsing-induced headaches.

I did, however, run into a slight problem with the nargs-option (using ConfigArgParse 0.9.3). If I have a file foo.py and I add an argument with parser.add('--bar', type=float, nargs='+'), then the following works as expected:

./foo.py --bar 1.0 2.0 3.0

However, having bar = 1.0 2.0 3.0 in the config file conf.ini does not work:

./foo.py --conf conf.ini
foo.py: error: argument --bar: invalid float value: '1.0 2.0 3.0'

As far as I understand it from reading the source code, the problem arises because config file values are passed to argparse as a single string, so the config file setting above would actually be equivalent to

./foo.py --bar "1.0 2.0 3.0"
foo.py: error: argument --bar: invalid float value: '1.0 2.0 3.0'

This of course fails since that string is not a valid float.

convert_setting_to_command_line_arg not support nargs in ('+', '*')

replace:

        elif value.startswith("[") and value.endswith("]"):
            if action is not None:
                if type(action) != argparse._AppendAction:
                    self.error(("%s can't be set to a list '%s' unless its "
                        "action type is changed to 'append'") % (key, value))
            for list_elem in value[1:-1].split(","):
                args.append( command_line_key )
                args.append( list_elem.strip() )

with:

        elif value.startswith("[") and value.endswith("]"):
            if type(action) == argparse._AppendAction:
                for list_elem in value[1:-1].split(","):
                    args.append( list_elem.strip() )
            elif type(action) == argparse._StoreAction and action.nargs in ('+', '*'):
                args.append( command_line_key )
                for list_elem in value[1:-1].split(","):
                    args.append( list_elem.strip() )
            else:
                self.error(("%s can't be set to a list '%s', unsupported action type: %s") % (key, value, type(action)))

will work around this BUG. but, I have not fully tested, so just submit as an issue not pull a request.

support for config directories

Hello,

Thanks for writing configargparse - it is great.

It would be nice to extend the software to include arbitrary config snippets in a directory without having to specify each snippet file. Perhaps adding a named parameter to the constructor:

default_config_directories = [ '/etc/software/conf-enabled' ]
default_config_directories_globs = [ '.conf', '.yaml' ]

or

default config_directories = [
{ 'directory' = '/etc/software/conf-enabled', glob = '*.conf' },
]

Pros and cons to each.

Also, a way to specify the config-directory on the command line:

config_arg_parser.add(
'--config-directory',
is_config_directory = True,
glob = '*.conf',
)

For larger software projects, it is nice to break the configurations into snippets. This would be a very useful feature. It would also allow mimicking a conf-available, conf-enabled paradigm.

Thanks for considering it!

-m

Optionally allow to address positional arguments

This function should be disabled per default since it might produce problems for some software. I would suggest that the metavar is used as the config parameter.

Probably it is hard to figure out if a config value is a positional argument (and in principle there could be some ambiguity). Maybe 'uppercase' is a good convention to label positional arguments?

Feature request: Python string values

I'm just about to release the first version of mitmproxy with config parsing based on ConfigArgParse. Thanks a bunch for your solid work.

Something we'd find immensely useful is the ability to specify values in key/value pairs as Python strings and triple-quoted multi-line strings, with Python-style escape sequences. This would make the config files for our tools hugely more powerful.

This would require converting the current parser to something more sophisticated. PyParsing has built-in definitions for quoted strings, and implementing the parser in it would not be very much work.

I don't have the bandwidth for this myself, but please let me know if a patch like this would be pulled, and I may get to it some time down the track.

feature - automatic env variables

I'd like to have an app where command line arguments, config file keys, and env vars are all "the same"

Given an app named "foo_bar", and command line options --single --multiple-word, without passing env_var, these config file keys would work (current behavior):

single = lonely
multiple-word = logorrhea

and these env vars would be automatic:

FOO_BAR__SINGLE=finally
FOO_BAR__MULTIPLE_WORD=unending

I used two underscores to separate the uniqueness prefix (FOO_BAR (defaulting to the uppercase app name)) but it could be one.

p = configargparse.ArgParser(automatic_env_vars=true)

p = configargparse.ArgParser(automatic_env_vars=true, env_var_prefix='FOO_BAR')