exercism/haskell-test-runner

Exercism Haskell Test Runner

The Docker image to automatically run tests on Haskell solutions submitted to Exercism.

Run the test runner

To run the tests of an arbitrary exercise, do the following:

1. Open a terminal in the project's root
2. Run ./bin/run.sh <exercise-slug> <solution-dir> <output-dir>

Once the test runner has finished, its results will be written to <output-dir>/results.json.

Run the test runner on an exercise using Docker

This script is provided for testing purposes, as it mimics how test runners run in Exercism's production environment.

To run the tests of an arbitrary exercise using the Docker image, do the following:

1. Open a terminal in the project's root
2. Run ./bin/run-in-docker.sh <exercise-slug> <solution-dir> <output-dir>

Once the test runner has finished, its results will be written to <output-dir>/results.json.

Run the tests

To run the tests to verify the behavior of the test runner, do the following:

1. Open a terminal in the project's root
2. Run ./bin/run-tests.sh

These are golden tests that compare the results.json generated by running the current state of the code against the "known good" tests/<test-name>/results.json. All files created during the test run itself are discarded.

When you've made modifications to the code that will result in a new "golden" state, you'll need to generate and commit a new tests/<test-name>/results.json file.

Run the tests using Docker

This script is provided for testing purposes, as it mimics how test runners run in Exercism's production environment.

To run the tests to verify the behavior of the test runner using the Docker image, do the following:

1. Open a terminal in the project's root
2. Run ./bin/run-tests-in-docker.sh

These are golden tests that compare the results.json generated by running the current state of the code against the "known good" tests/<test-name>/results.json. All files created during the test run itself are discarded.

When you've made modifications to the code that will result in a new "golden" state, you'll need to generate and commit a new tests/<test-name>/results.json file.

haskell-test-runner's Issues

Upgrade to version 3 spec

If possible, this test runner should be updated to version 3 of the test runner interface specification. In version 3, one additional feature is enabled: the ability to link individual tests to tasks.
This allows the website to show which tests belong to which tasks.

The way tests are linked to tasks is via an (optional) task_id field, which is an integer that matches the number of the task as defined in the exercise's instructions.md file (note: the instructions start at index 1).

This is an example of a test in the results.json file:

{
  "name": "Expected oven time in minutes",
  "status": "pass",
  "task_id": 1,
  "test_code": "Assert.Equal(40, Lasagna.ExpectedMinutesInOven());"
}

You are completely free in how to implement this. Some options are:

1. Add metadata to a test that the test runner can then discover while running the tests (e.g. an attribute or annotation)
2. Define a test name/task id mapping (e.g. in the exercise's .meta/config.json file)
3. Any other option you can think of...

As this test runner currently implements version 1 of the test runner specification, it might be good to first upgrade to version 2 of the test runner specification before upgrading to version 3.

Let me know if there are any questions.

Upgrade to version 2 spec

If possible, this test runner should be updated to version 2 of the test runner interface specification.

The version 2 spec differs from version 1 in that the results.json file contains detailed information on individual tests.
This information includes things like:

• The test status (passing/failing)
• The error message (if failing)
• The test code that was used to run the test

You are completely free on how you want to extract the test code, either via the AST or via finding the test code in the source code text via string manipulation.

Let me know if there are any questions.

Upgrade to version 2 spec

If possible, this test runner should be updated to version 2 of the test runner interface specification.

The version 2 spec differs from version 1 in that the results.json file contains detailed information on individual tests.
This information includes things like:

• The test status (passing/failing)
• The error message (if failing)
• The test code that was used to run the test

You are completely free on how you want to extract the test code, either via the AST or via finding the test code in the source code text via string manipulation.

Let me know if there are any questions.

Matrix tests can't find vector package

The tests for the matrix exercise pass locally but fail when using exercism submit or the online editor with this error message:

Cabal file info not found for vector-0.12.1.2@sha256:9291bc581f36e51d5bda9fce57cb980fbec3dd292996896f285fef39eb80a9a0,7364, updating
Selected mirror https://hackage.haskell.org/
Downloading root
HttpExceptionRequest Request {
  host                 = "hackage.haskell.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","Haskell pantry package")]
  path                 = "/root.json"
  queryString          = ""
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 (ConnectionFailure Network.Socket.getAddrInfo (called with preferred socket type/protocol: AddrInfo {addrFlags = [AI_ADDRCONFIG], addrFamily = AF_UNSPEC, addrSocketType = Stream, addrProtocol = 0, addrAddress = <assumed to be undefined>, addrCanonName = <assumed to be undefined>}, host name: Just "hackage.haskell.org", service name: Just "443"): does not exist (Try again))

Community submissions from more than 2 months ago were successful, but recent submissions also fail.

Haskell track maintainers for PRs to pre-compiled/package.yaml

Looking through the history of PRs to pre-compiled/package.yaml in https://github.com/exercism/haskell-test-runner/commits/main/pre-compiled/package.yaml, it looks like for most of them (all but one), you've sought an opinion from a Haskell track maintainer. That makes sense if the track maintainers are making sure that the packages being added are reasonable. So if that's going to happen every time, maybe an update to https://github.com/exercism/haskell-test-runner/blob/main/.github/CODEOWNERS would be warranted, so that the track maintainers will automatically get added to PRs for that file?

It's just a suggestion; if you think there are cases where a track maintainer's input isn't necessary and therefore it should be left on a case-by-case basis instead of automatically, that's fine too.

Upgrade to version 3 spec

If possible, this test runner should be updated to version 3 of the test runner interface specification. In version 3, one additional feature is enabled: the ability to link individual tests to tasks.
This allows the website to show which tests belong to which tasks.

The way tests are linked to tasks is via an (optional) task_id field, which is an integer that matches the number of the task as defined in the exercise's instructions.md file (note: the instructions start at index 1).

This is an example of a test in the results.json file:

{
  "name": "Expected oven time in minutes",
  "status": "pass",
  "task_id": 1,
  "test_code": "Assert.Equal(40, Lasagna.ExpectedMinutesInOven());"
}

You are completely free in how to implement this. Some options are:

1. Add metadata to a test that the test runner can then discover while running the tests (e.g. an attribute or annotation)
2. Define a test name/task id mapping (e.g. in the exercise's .meta/config.json file)
3. Any other option you can think of...

As this test runner currently implements version 1 of the test runner specification, it might be good to first upgrade to version 2 of the test runner specification before upgrading to version 3.

Let me know if there are any questions.

Add matrix dependency

Add matrix dependency which can be helpful for creative solutions to Crypto Square, for instance.

I tried pushing up a branch for this, but I'm getting this:

ERROR: Permission to exercism/haskell-test-runner.git denied to Pilatch.