Write important tests

[dense-analysis]

Write important tests

[w0rp]

Thanks to the work of @prashcr and some minor tweaking from yours truly, we now have tests running with a Docker image on Travis CI. The results are speedy, and the output is readable enough. You can see the log from one of the builds here: https://travis-ci.org/w0rp/ale/builds/167205251

If you can get Docker installed on your machine, you can run them locally by running just make. (I was able to set up so make is the same as make test.)

Let's all thank @prashcr for the work he put into this. It's good stuff. 👍

Now we can go about adding some of these tests so they'll actually run on Travis CI. I'll have a look at some of the other pull requests tomorrow, now I've gotten the test suite to run.

[w0rp]

I was finally able to get a test to run a job. I added a function which checks the job status and waits with sleep in a loop with a timeout to support blocking until linting is complete for the benefit of tests. I didn't add it to the documentation, as I don't intend for the function to be used outside of this plugin.

[KabbAmine]

And how do we install linters? There are a lot of dependencies involved (pip, npm...). Maybe we should focus first on the ones installable from repositories like gcc and shellcheck?

Take a look at the sh test implemented in validator, its a very simple one and check only if the pattern used in linter handler is valid.

[w0rp]

We can install whatever via pip, npm, etc on Travis CI. We could start with just one linter which will cover most of the cases above. Travis CI currently caches pip installed modules, so it shouldn't slow builds down that much.

[w0rp]

We could also perhaps make use of virtualenv, but Docker is also being introduced into the mix. Either way, it can be done.

[KabbAmine]

Good to know, I'm not familiar at all with Docker and Travis.

[KabbAmine]

Unit test the loclist sorting.

Actually, the loclist is sorted by line numbers, and IMO it should be sorted by severity (Errors first then warnings).

[w0rp]

The loclist has to be sorted by line and then column for the binary search used by the cursor functions to work.

[w0rp]

The current sorting is also used by the signs functions for showing errors instead of warnings when a line contains both warnings and errors.

[neersighted]

Why not just bundle the linters in our docker image?

[w0rp]

Yep, whatever programs can be put in the Docker image.

[w0rp]

I have started adding tests. This should complement the work from @prashcr on adding the Docker image for running tests on Travis CI. We can start adding tests which pass locally, and get them on Travis CI when that is ready to go.

I managed to catch a bug and fix it in the process, so Vader is looking good to me.

[w0rp]

I'll make a note, if anyone can figure out a decent way to get NeoVim included in the build too, then that would be a very good next step.

[neersighted]

There is a pending PR on the vim-testbench repo adding neovim support, so I would say wait a few days and then see what they come up with.

[KabbAmine]

Good job guys, the plugin is far better than before 👍
Sorry for not being available, but I was and still very busy with personal problems (f*****g divorce..).

[w0rp]

We'll keep working on it.

I'm sorry to hear about your personal problems! Good luck to you.

[w0rp]

Here is something we should keep in mind, which I just fixed, and I've gotten wrong myself. The order of the AssertEqual arguments is expected, actual, and not actual, expected. Only one way produces the correct output when the assertion fails.

[w0rp]

I've been trying to get a test to execute a job with buffer input, and so far I haven't succeeded. If anyone finds a way to do that, let me know.

[w0rp]

I added an ID to the Makefile so running tests locally will pull a more recent Docker image if we update the ID in the Makefile. So for updating the image, we'll have to do the following.

1. Update the Dockerfile in the repo, and push that to the master branch. Get the build to run again. (It should be automatic, but so far... it doesn't seem to be.)
2. Pull the latest Docker image on a local machine, and get the ID for that image.
3. Update the ID in the Makefile again, and push that to the repo.

If we follow those steps, then anyone who pulls the repo and runs make again afterwards will run the tests with the most up to date image automatically.

[neersighted]

Why not just make the Make file run docker pull unconditionally?

[neersighted]

I do believe the purpose of the docker images -q was just to print the image version to the test output -- there is no reason not to pull every time.

[w0rp]

I believe that will mean running make will always use your Internet connection.

[neersighted]

If you have the latest image it will just noop, and if you don't have internet it will just error and continue.

[w0rp]

Hmm, I'll have to try it with my Internet connection turned off later. As long as it runs quickly enough, I'll go with that.

[w0rp]

I tried running docker pull with my Internet connection off, and the command fails. I could just ignore the error, but then that will cause other weird errors.

[w0rp]

I tried to find a way to test the cursor moving triggering the function, but I couldn't get it to work.

[w0rp]

I tried last night, but I had trouble testing anything around functions triggered from autocmd events. I'll close this issue now, and open another asking for help.