The medikit's discuss from python-medikit

"Nice fatals"

Some fatals are really normal, and the stack trace dump misleads the user into thinking there is something wrong with the software.

For example:

 FileExistsError  Already started, use `medikit pipeline release start --force` to force a restart, or use `medikit pipeline release continue`.

Find a better way to output this, while not hiding anything (stack trace should be accessible, even if maybe hidden by default).

Maybe this should be a https://github.com/python-mondrian/mondrian feature ?

Refactor Commands into its own module, Refactor common code out in bureau

Remove tornado dependency

Tornado is only used to format logs nicely, and it's a bit overkill to depend on a web framework for this job. Let's code a minimal and similar log formatter and remove it.

No python Projectfiles

Let's see if we can use projectfiles for no-python projects, like okdocker or maybe a node only project.

Release pipeline: when publishing to pypi, "bad marshal data (unknown type code)" error

Probably linked to releasing with multiple version targets. Works with 3.5/3.6 but complains with 3.7. Need bytecode cache clearing between versions and/or regeneration? Maybe it's just because py3.7 is not released yet?

Traceback (most recent call last):
  File "setup.py", line 78, in <module>
    download_url='https://github.com/python-medikit/medikit/archive/{version}.tar.gz'.format(version=version),
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 209, in run
    os.path.join(archive_root, 'EGG-INFO'), self.zip_safe()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 245, in zip_safe
    return analyze_egg(self.bdist_dir, self.stubs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 355, in analyze_egg
    safe = scan_module(egg_dir, base, name, stubs) and safe
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 392, in scan_module
    code = marshal.load(f)

Make's "%" rule is a fallback. Allow to override it, and maybe have a default

Maybe delegate all unknown tasks to and entrypoint provided by the project ? At elast allow to do so ?

How would that work with "help" task, as there would then be 2 different command lists ?

If set_remote is called after use_rocker_builder, image is not properly overriden

docker.set_remote(registry=...)
docker.use_rocker_builder()

works but not the other way round, because the variables are computed at use_* call time.

Find a way to make it work both ways.

Find a way to allow prereleases, but on an optin basis.

Easy to do the following, but one should opt-in to prereleases.

--- a/medikit/feature/python.py
+++ b/medikit/feature/python.py
@@ -309,7 +309,7 @@ class PythonFeature(Feature):
                     tmpfile.name, finder=repository.finder, session=repository.session, options=pip_options
                 )
             )
-            resolver = Resolver(constraints, repository, prereleases=False, clear_caches=False, allow_unsafe=False)
+            resolver = Resolver(constraints, repository, prereleases=True, clear_caches=False, allow_unsafe=False)

             self.render_file_inline(
                 'requirements{}.txt'.format('-' + extra if extra else ''),

add "packaging" to dependencies.

Disable or Rename some files

README.rst vs README.md, travis config, etc ...

Add twine and git-semver to dependencies ?

Coul help for releases, both of this project and managed ones.

Use prettier for make format in javascript, and eventually allow it instead of yapf for python when they finish python support

https://github.com/prettier/prettier

Enhance output for humans, not for implementation details.

Event bubbling should still be visible if someone wants to see it, but this should not be the default.

Supports working out of git ?

Should be already enabled by the git feature, but we need to test that, and maybe have command line flags to avoid updating git repo if unwanted.

Add python_requires support both for project and managed projects

See django/django#9413

Make all config objects use a fluid interface

So we can do:

(python
    .setup(...)
    .add_requirements(...)
)

This is already available for the nodejs feature, generalize this.

Pipelines polishing

Pipelines are very raw, for now. Enhancements that would be a quick win:

Warnings in generated file.

This file is generated by medikit ... Recipe is Projectfile, regenerate using "make update", "make update-requirements" or medikit update (if make update broken).

Kube - Patch generation is too verbose, and output order not predictible.

$(shell which foo) will return an empty string if foo is not installed, causing the make targets to cause unreadable errors.

Add a "grep XXX|TODO" in release pipeline

Git: (optionally) manage gitignores ?

do not generate requirements if file exist (its long but useless work, as no override is done if exist).

Now that every config goes to config object, add a --dry-run mode and a config dump command

Better auto-install / upgrade medikit in make update/update-requirements

There should be a version check in there, maybe better output too.
And maybe refuse to auto-install/upgrade if systemwide python.

And use python variables instead of hardcode.

Sphinx feature: extends `clean` task to also clean the sphinx build directories

Add a "make help" task, eventually first (at least as default) so "make" by itself outputs a list of commands.

This is what sphinx does, we don't have the full list available outside of the makefile but we may do it with simple "echo" lines (and I believe until recently, sphinx was doing the same).

# Put it first so that "make" without argument is like "make help".
help:
	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

Website: entry points by role

Guide for project's operators (make help, etc.)
Guide for project's developers (make update, etc.)
Guide for project's maintainers (features, etc.)

Make sure all setup options are possible, link to documentation in generated template

https://packaging.python.org/tutorials/distributing-packages/

py2 issue : argparse aliases not implemented

pipelines continue although there is an error.

pip._vendor.pkg_resources.ContextualVersionConflict: (stevedore 1.28.0 (/Users/rd/.virtualenvs/rdnet/lib/python3.6/site-packages), Requirement.parse('stevedore<1.28,>=1.27'), {'bonobo'})
make: *** [install] Error 2
Make('clean install') is complete, moving forward.

Release pipeline: Sanity check before accepting a version number

While releasing, no check is done on version number. Add the following:

If version is lower than current, warn the user he's probably making a mistake and ask for "hard" confirmation.
If version is equal to current, warn and refuse if already pushed
If there is a version jump, warn
Add default ? Next patch ?
Add "patch"/"p", "minor"/"m", "major"/"M" string to compute the version instead of typing it.

Current version: 0.5.8 Git version: 0.5.8
Next version? 0.5.9

Better config using «mock recorders».

Mock/MagicMock calls are recorded in order, we may be able to improve the configuration process with a really nice syntax by recording all calls to mock objects then replaying them on real configurator objects once the config parsing is finished.

how to overcome distributionnotfound errors due to too strict requirements ?

pkg_resources.DistributionNotFound: The 'stevedore<1.20,>=1.19' distribution was not found and is required by edgy.project

Python 2 not supported anymore, document it.

Check system & dependency make target ?

A lightweight "make check" task could run a lightweight environment check :

Are all (system) deps satisfied ?
Are versions correct ?
etc.

Can enhance the DX / onboarding for medikit-unaware developpers on managed projects.

Problem when git repository is higher in the directory tree than the project root

edgy-project will recreate an empty git repo if, for example, git repo is in ..

Should not happen.

extras_require is buggy with old setuptools versions

Michael had an old setuptools version, and looks that setup.py was not parseable.

(bonobo)bash-3.2$ pip install --upgrade setuptools
Collecting setuptools
  Using cached setuptools-36.6.0-py2.py3-none-any.whl
Installing collected packages: setuptools
  Found existing installation: setuptools 18.2
    Uninstalling setuptools-18.2:
      Successfully uninstalled setuptools-18.2
Successfully installed setuptools-36.6.0

(Do not work with 18.2, but works with 36.6)

Error was:

(bonobo)bash-3.2$ pip install --editable bonobo
Obtaining file:///Users/misha/pyconde17/bonobo
    Complete output from command python setup.py egg_info:
    error in bonobo setup command: 'extras_require' must be a dictionary whose values are strings or lists of strings containing valid project/version requirement specifiers.

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /Users/misha/pyconde17/bonobo/

Django: use yapf to adjust settings?

Following code kept for reference even if not working, but it would be amazing to be able to tune settings (like ensure that something is in INSTALLED_APPS). Maybe not achievable easily and safely, though.

    # Use yapf to adjust settings (for example, add some apps).
    @subscribe('medikit.feature.django.on_configure')
    def on_django_configure(self, event):
        original_source, newline, encoding = yapf_api.ReadFile('config/settings.py')
        print(original_source, newline, encoding)
        reformatted_code, encoding, has_change = yapf_api.FormatFile(
            'config/settings.py',
            in_place=True,
        )
        print(reformatted_code, encoding, has_change)