zulip/zulint

A lightweight linting framework designed for complex applications using a mix of third-party linters and custom rules.

zulint

zulint is a lightweight linting framework designed for complex applications using a mix of third-party linters and custom rules.

Why zulint

Modern full-stack web applications generally involve code written in several programming languages, each of which have their own standard linter tools. For example, Zulip uses Python (mypy, Ruff), JavaScript (eslint), CSS (stylelint), puppet (puppet-lint), shell (shellcheck), and several more. For many codebases, this results in linting being an unpleasantly slow experience, resulting in even more unpleasant secondary problems like developers merging code that doesn't pass lint, not enforcing linter rules, and debates about whether a useful linter is "worth the time".

Zulint is the linter framework we built for Zulip to create a reliable, lightning-fast linter experience to solve these problems. It has the following features:

• Integrates with git to only checks files in source control (not automatically generated, untracked, or .gitignore files).
• Runs the linters in parallel, so you only have to wait for the slowest linter. For Zulip, this is a ~4x performance improvement over running our third-party linters in series.
• Produduces easy-to-read, clear terminal output, with each independent linter given its own color.
• Can check just modified files, or even as a pre-commit hook, only checking files that have changed (and only starting linters which check files that have changed).
• Handles all the annoying details of flushing stdout and managing color codes.
• Highly configurable.
  • Integrate a third-party linter with just a couple lines of code.
  • Every feature supports convenient include/exclude rules.
  • Add custom lint rules with a powerful regular expression framework. E.g. in Zulip, we want all access to Message objects in views code to be done via our access_message_by_id functions (which do security checks to ensure the user the request is being done on behalf of has access to the message), and that is enforced in part by custom regular expression lint rules. This system is optimized Python: Zulip has a few hundred custom linter rules of this type.
  • Easily add custom options to check subsets of your codebase, subsets of rules, etc.
• Has a nice automated testing framework for custom lint rules, so you can make sure your rules actually work.

This codebase has been in production use in Zulip for several years, but only in 2019 was generalized for use by other projects. Its API to be beta and may change (with notice in the release notes) if we discover a better API, and patches to further extend it for more use cases are encouraged.

Using zulint

Once a project is setup with zulint, you'll have a top-level linter script with at least the following options:

$ ./example-lint --help
usage: example-lint [-h] [--modified] [--verbose-timing] [--skip SKIP]
                    [--only ONLY] [--list] [--list-groups] [--groups GROUPS]
                    [--verbose] [--fix]
                    [targets [targets ...]]

positional arguments:
  targets               Specify directories to check

optional arguments:
  -h, --help            show this help message and exit
  --modified, -m        Only check modified files
  --verbose-timing, -vt
                        Print verbose timing output
  --skip SKIP           Specify linters to skip, eg: --skip=mypy,gitlint
  --only ONLY           Specify linters to run, eg: --only=mypy,gitlint
  --list, -l            List all the registered linters
  --list-groups, -lg    List all the registered linter groups
  --groups GROUPS, -g GROUPS
                        Only run linter for languages in the group(s), e.g.:
                        --groups=backend,frontend
  --verbose, -v         Print verbose output where available
  --fix                 Automatically fix problems where supported

pre-commit hook mode

See https://github.com/zulip/zulip/blob/master/tools/pre-commit for an example pre-commit hook (Zulip's has some extra complexity because we use Vagrant from our development environment, and want to be able to run the hook from outside Vagrant).

Adding zulint to a codebase

TODO. Will roughly include pip install zulint, copying an example lint script, and adding your rules.

Adding third-party linters

TODO: Document the linter_config API.

Writing custom rules

TODO: Document all the features of the RuleList and custom_check system.

Development Setup

Run the following commands in a terminal to install zulint.

git clone [email protected]:zulip/zulint.git
python3 -m venv zulint_env
source zulint_env/bin/activate
python3 setup.py install

zulint's Issues

Improve zulint documentation in README.md

This repository needs a guidance on code base of the Zulint. Writing proper documentation in the README.md would be good.

Something like these.

• Docs explaining zulint
• Usage of zulint in Zulip developer's workflow.
• What does zulint/printer.py do?

Parallel linters with --fix can corrupt files

With --fix, one linter may be writing a file while a second linter is reading it. This may result in the second linter getting a corrupted version. If the second linter then decides to make additional changes, this corruption leaks into the final output.

This unfortunately means we need to run --fix-aware linters in series when --fix is enabled.

Suppress color codes when stdout is not a terminal.

Discussion here:

https://chat.zulip.org/#narrow/stream/43-automated-testing/topic/lint.20color.20codes/near/953792

`exclude_files_in` is always overwritten when passed as a parameter to `RuleList`.

In zulint/custom_rules.py, theoretically exclude_files_in can be passed as a parameter when creating a RuleList, but it's always set as "\\" no matter what string is passed as a parameter. See code block below.

I'm not sure what the intention was as exclude_files_in is not listed in the documentation comment "Rule help" as a way to exclude files. There's only one current instance of it's use in zulip/zulip, here.

class RuleList:
    """Defines and runs custom linting rules for the specified language."""

    def __init__(
        self,
        langs: Sequence[str],
        rules: Sequence[Rule],
        exclude_files_in: Optional[str] = None,
    ) -> None:
        self.langs = langs
        self.rules = rules
        # Exclude the files in this folder from rules
        self.exclude_files_in = "\\"
        self.verbose = False

Issue installing from Git using pip

I keep getting this issue when installing from pip, trying multiple Ubuntu machines and python venvs. I get this error:

ERROR: Command errored out with exit status 1:
   command: /project/zulinttest/venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-k5nb2u_7/zulint/setup.py'"'"'; __file__='"'"'/tmp/pip-install-k5nb2u_7/zulint/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-1eim5wzx
       cwd: /tmp/pip-install-k5nb2u_7/zulint/
  Complete output (6 lines):
  usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: setup.py --help [cmd1 cmd2 ...]
     or: setup.py --help-commands
     or: setup.py cmd --help
  
  error: invalid command 'bdist_wheel'
  ----------------------------------------
  ERROR: Failed building wheel for zulint
  Running setup.py clean for zulint
Failed to build zulint

If I try it again after doing pip install wheel it works fine. I also tried in my fork setting wheel as required, but this still causes the error. But as wheel did get installed it succeeds the second time of running.

This is being tried in a requirements.txt file using pip3 install -r requirements.txt

Can this be released on PyPi?

It would be great if this could be released on PyPi.

custom_rules - matches based on multiline pattern cant exclude based on multiline matching

currently it's possible to match based on multiline patterns. eg 'pattern': r'(\?\>\n.)',
However it is not possible to exclude based that next line content. It would be nice to be able to exclude based on next line(s) matching

Generalize exclude rules support for long lines for URLs (etc.)

See zulip/zulip#13078 (comment) for details