continuumio/ciocheck

Continuum Analytics linter, formatter and test suite helper.

ciocheck's Issues

Create a standard output for the different tools

Enhance current print and display to capture all info from all tolls and unify the display.

Maybe add some colors to display?

Dont apply formatters to files if check only

Checks should not change files when using formatters

Dicts exceeding char limits are badly modified

Here's the diff:

-            return { "errors": [
-                {
-                    "detail": "HTTP 400: Bad path provided: Path %s does not exist!" % nbs_root,
-                    "title": "HTTP 400",
-                    "status": 400
-                }
-            ]}
+            return {
+                "errors": [
+                    {
+                        "detail":
+                        "HTTP 400: Bad path provided: Path %s does not exist!"
+                        % nbs_root,
+                        "title":
+                        "HTTP 400",
+                        "status":
+                        400
+                    }
+                ]
+            }

Is there a way to fix it or at least skip those specific checks? I'd rather violate line lengths in this case...

Add error loging

Enable enforce only option if check and enforce are the same

flake8 link broken

http://flake8.readthedocs.io/en/latest/config.html in the readme is broken. Consider https://flake8.readthedocs.io/en/latest/user/configuration.html

pytest-cov --cov does not work on submodule

https for readthedocs.io

readthedocs.io supports https, but the readme sometimes uses http links.

Add a debug option for printing errors etc..

create a [common] settings section to group redundant values

This can initially focus on max-line-length and exclude

Add exclude for main config

Two "t"s in formatter

There are two "t"s in formatter. Using only one t is a misspelling.

More ascii decoding errors in formatters

/home/circleci/miniconda/lib/python2.7/site-packages/ciocheck/formaters.py:43: UnicodeWarning: Unicode unequal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  changed = new_contents != old_contents
/home/circleci/miniconda/lib/python2.7/difflib.py:437: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  a[besti+bestsize] == b[bestj+bestsize]:
/home/circleci/miniconda/lib/python2.7/difflib.py:433: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  a[besti-1] == b[bestj-1]:
Traceback (most recent call last):
  File "/home/circleci/miniconda/lib/python2.7/site-packages/ciocheck/format_task.py", line 53, in <module>
    main()
  File "/home/circleci/miniconda/lib/python2.7/site-packages/ciocheck/format_task.py", line 45, in main
    task_result = format_file(filename)
  File "/home/circleci/miniconda/lib/python2.7/site-packages/ciocheck/format_task.py", line 35, in format_file
    result = formater.format_task(path)
  File "/home/circleci/miniconda/lib/python2.7/site-packages/ciocheck/formaters.py", line 54, in format_task
    'diff': diff(old_contents, new_contents),
  File "/home/circleci/miniconda/lib/python2.7/site-packages/ciocheck/utils.py", line 209, in diff
    return ''.join(result)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 36: ordinal not in range(128)

ascii decoding errors

Running "isort" ...
Traceback (most recent call last):
  File "/home/circleci/miniconda/bin/ciocheck", line 11, in <module>
    sys.exit(main())
  File "/home/circleci/miniconda/lib/python2.7/site-packages/ciocheck/main.py", line 389, in main
    test.run()
  File "/home/circleci/miniconda/lib/python2.7/site-packages/ciocheck/main.py", line 76, in run
    extensions=tool.extensions)
  File "/home/circleci/miniconda/lib/python2.7/site-packages/ciocheck/files.py", line 50, in get_files
    branch=branch, diff_mode=diff_mode, extensions=extensions)
  File "/home/circleci/miniconda/lib/python2.7/site-packages/ciocheck/files.py", line 65, in get_modified_file_lines
    results = self.diff_tool.commited_file_lines(branch=branch)
  File "/home/circleci/miniconda/lib/python2.7/site-packages/ciocheck/vcs.py", line 542, in commited_file_lines
    results.update(diff_tool.commited_file_lines(branch=branch))
  File "/home/circleci/miniconda/lib/python2.7/site-packages/ciocheck/vcs.py", line 423, in commited_file_lines
    result = self._parse_diff_str(self._diff_committed(branch=branch))
  File "/home/circleci/miniconda/lib/python2.7/site-packages/ciocheck/vcs.py", line 364, in _diff_committed
    branch=branch, files_only=False, mode=COMMITED_MODE)
  File "/home/circleci/miniconda/lib/python2.7/site-packages/ciocheck/vcs.py", line 151, in _git_run_helper
    result, error = run_command(command, cwd=self.path)
  File "/home/circleci/miniconda/lib/python2.7/site-packages/ciocheck/utils.py", line 100, in run_command
    output = output.decode()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 9777: ordinal not in range(128)
Exited with code 1

Ciocheck config

unix: cd ${PROJECT_DIR}/deploy && python -m pytest --cov=anaconda_platform.deploy --cov-report=term-missing:skip-covered --cov-report=html --cov-fail-under=100 --no-cov-on-fail --cov-config=.coveragerc anaconda_platform/deploy/tests

@goanpeca so this is essentially what I want to do with ciocheck (see also the .coveragerc in this PR). The reason ciocheck doesn't work (I think?) is that if I cd into deploy subdir as I do here, ciocheck can't find the .ciocheck in the root dir one level up. (I have .ciocheck-test which does an inherit_from .ciocheck, and I think the inherit_from looks relative to cwd not relative to ciocheck-test?). If I don't cd into the deploy subdir then the module paths are messed up which causes its own trouble.

if you want you can check this branch out and try ./apb test deploy to see what my command here does and play with using ciocheck in kapsel.yml instead..

@havocp could please add any additional context here. The initial version of ciocheck was very limited in terms of where to look for config and stuff. But if you define some "rules" that make sense to you I can make a new release in a week or so.

Avoid endless loop on config referring to itself when using inherit

Limit folders that contain .py file to be the ones that get __init__ included

Would it be useful to only do this if a .py[o,c,w,d] file exists in that directory?

setup.py broken

$ pip install -e .
Obtaining file:///home/antoine/ciocheck
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/antoine/ciocheck/setup.py", line 44
        description='A stand alone PyQt/PySide GUI application for managing conda '
           ^
    SyntaxError: keyword argument repeated

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /home/antoine/ciocheck/

Prepend options with the program that runs them

[flake8:section] instead of [section]

This will clarify what each section refers to.

Add import sorter step

https://github.com/timothycrosley/isort

CioCheck should exit with nonzero exit code if any linter fails

Look for full command path based on sys.prefix

Option for exclude some folders from generation of __init__.py files

This will be useful for excluding pytest folders, adding __init__files will convert them to modules and will break the tests (in some cases).

I'm aware that there is an option add_init = false, but maybe it could be more configurable

Add RELEASE.md

Release to continuumcrew

Waiting for conda-forge/staged-recipes#1770 to have all dependencies ready.

pyformat does not work correctly on windows!

the encoding file wants to replace line endings...

Add logic to capture dates inside top comments

In order to have a smarter copyright section replace

Add diff flake8

Add support for no module/file given and run on the root folder

MNT: Stop using ci-helpers in appveyor.yml

To whom it may concern,

If you are using https://github.com/astropy/ci-helpers in your appveyor.yml , please know that the Astropy project has dropped active development/support for Appveyor CI. If it still works, good for you, because we did not remove the relevant files (yet). But if it ever stops working, we have no plans to fix anything for Appveyor CI. Please consider using native Windows support other CI, e.g., Travis CI (see https://docs.travis-ci.com/user/reference/windows/). We apologize for any inconvenience caused.

If this issue is opened in error or irrelevant to you, feel free to close. Thank you.

xref astropy/ci-helpers#464

pytest check fails

$ ciocheck .

================
Running ciocheck
================

Running "pyformat" ...
Running "isort" ...
Running "yapf" ...
Running "autopep8" ...
Running "Multi formatter"
Running "pydocstyle" ...
Running "flake8" ...
Running "pylint" ...
Running "coverage" ...
Running "pytest" ...
['/path/to/main.py', '/path/to/utils/concurrency.py', '/path/to/utils/messaging.py', '--json=.pytestreport.json', '-n', '4', '--cov-config', '/path/to/.coveragerc', '--cov=/path/to/.']
usage: ciocheck [options] [file_or_dir] [file_or_dir] [...]
ciocheck: error: unrecognized arguments: --json=.pytestreport.json -n 4
  inifile: /path/to/pytest.ini
  rootdir: /path/to/.

Config:

[ciocheck]
inherit_config = .ciocheck
branch = origin/master
diff_mode = commited
file_mode = lines
check = pydocstyle,flake8,pylint,pyformat,isort,autopep8,yapf,coverage,pytest
enforce = pydocstyle,flake8,pylint,pyformat,isort,autopep8,yapf,coverage,pytest

header = # -*- coding: utf-8 -*-
copyright_file = LICENSE
add_copyright = true
add_header = true
add_init = false

add hooks folder and auto command

https://github.com/sjungwirth/githooks

http://stackoverflow.com/questions/3462955/putting-git-hooks-into-repository/3464399#3464399

http://stackoverflow.com/questions/3462955/putting-git-hooks-into-repository/3464399#3464399

pyformat is not including a \n when adding encoding header

Exclude files that do not belong to the module passed in the CLI

This affects pep257 and flake8

Make default behavior less agressive

Add CI testing

Copyright is pasted into each file every time ciocheck runs

This may however be an issue with ciocheck crashing due to #44

committed mode with lines not detecting new files

Pylint has no JSON output when no errors are found, which messes up json.loads

https://github.com/ContinuumIO/ciocheck/blob/master/ciocheck/linters.py#L46

An easy fix would be to do this:

    def _parse_json(self, string):
        """Parse output with json keys."""
        if not string:
            return []  # No response, successful
        data = json.loads(string)
        ...

Ciocheck fails with unhandled JSONDecodeError

Run fails with weird decoder errors:

Traceback (most recent call last):
  File "/Users/fpliger/miniconda3/envs/fusion/bin/ciocheck", line 9, in <module>
    load_entry_point('ciocheck', 'gui_scripts', 'ciocheck')()
  File "/Users/fpliger/dev/repos/ciocheck/ciocheck/main.py", line 389, in main
    test.run()
  File "/Users/fpliger/dev/repos/ciocheck/ciocheck/main.py", line 97, in run
    multi_results = tool.run(files)
  File "/Users/fpliger/dev/repos/ciocheck/ciocheck/formatters.py", line 273, in run
    results = await_all_processes()
  File "/Users/fpliger/dev/repos/ciocheck/ciocheck/formatters.py", line 245, in await_all_processes
    output, error = await_one_process()
  File "/Users/fpliger/dev/repos/ciocheck/ciocheck/formatters.py", line 235, in await_one_process
    output = json.loads(output)
  File "/Users/fpliger/miniconda3/envs/fusion/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/Users/fpliger/miniconda3/envs/fusion/lib/python3.5/json/decoder.py", line 342, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 3)

The text triggering the errors is like:

[{"yapf": {"error": null, "diff": "--- \\n+++ \\n@@ -22,6 +22,7 @@\\n from anacondafusion.core.server import FusionApp\\n \\n pemfile = os.path.join(settings.HERE, \\"assets\\", \\"fusion.pem\\")\\n+\\n \\n def get_nbs_root():\\n \'\'\'\\n", "created": false, "path": "/Users/fpliger/dev/repos/anaconda-fusion/anacondafusion/__main__.py"}}]\n\x1b[?1034h

or

[]\n\x1b[?1034h

Remove dependency on xdist

xdist seems to create random segfaults on CI systems, we should no longer use it

pycodestyle

pep8 is no more. pycodestyle is its replacement.