muter-mutation-testing/muter

🔎 Automated mutation testing for Swift 🕳️

muter's Issues

Muter reports build errors as successful mutation test runs

If a user misconfigures Muter, and that misconfiguration prevents their test suite from being run (e.g because xcodebuild is unable to locate a scheme), Muter will incorrectly report that is a successful mutation test. This issue is to improve Muter's detection of tooling errors, and to prevent them from becoming successful mutation test runs.

Muter provides a JSON output of the test report

This is part of #46

Reduce the amount of file I/O in the unit tests so they run faster

Update how Muter determines whether or not a particular file is a swift file

Right now, Muter doesn't anchor it search for a file name to the very end of a string, end instead simply does a contains check. This causes files like .swiftlint.yml to be discovered by Muter as a file that could potentially contain mutable source code.

This story would update Muter so it starts looking for a file extension at the very end of a file path or name. This would remove burden from a user who would currently need to explicitly ignore things like a .swiftversion or .swiftlint.yml file

Config should be YAML, not JSON

Or it could even be Swift? JSON isn't well suited to things like having list of command line arguments. YAML is nice because it has bare words, so you don't have quote marks mucking up the works.

Muter copies the codebase it's testing into a temporary directory

Falling out of a conversation and some research done on #3, it makes sense to have Muter copy the code base it's testing into a temporary directory - /tmp is most likely.

Muter will still need to restore files after mutating them, but prior to running the test suite again, so the existing code will probably stay the same. This issue adds a step prior to source file discovery or mutation discovery which moves the entire code base that will be analyzed into a new directory. Additionally, there's the potential that there will need to be some (likely very) small rework in associating this new directory with the existing code that uses file paths.

Muter can be ran on linux

Right now, the only thing getting in the way of this is using macOS specific FileManager functions. If we moved away from that, that it should just work on linux

Muter reports the average number of test failures per single change in a code base

Muter should create a config file on first run

It should populate it with reasonable defaults or, for bonus points, ask you for input and then build it. At the very least, the error message when the config file is missing should include a URL to the config section in the readme.

Muter supports mutating logic operators

Determine if using xcpretty, Fastlane or Buck requires additional logic for determining if a test suite has failed

Muter doesn't support removing void function calls that span multiple lines

Muter doesn't mutate logic that's within a guard statement

A main problem any tool that modifies source code has is knowing whether or not source code changes are going to cause static or dynamic errors within the program.

I have a hypothesis that modifying conditional logic inside a guard statement will nearly always lead to a static or dynamic error. This is partially proven by running Muter on a few code bases and noticing this tendency.

It could be valuable to have Muter avoid mutating logic that's inside a guard statement. This would speed up Muter by not having it make source code changes that would only lead to a compilation or fatal error.

Use excluded instead of blacklist

Makes it more clear what it does. Also matches naming from .swiftlint.yml.

Muter performs an initial test suite run which will prevent mutation testing if it fails

Create a separate acceptance test target so the unit tests can be run independently without false negatives

Muter supports inverting compound assignment operators

see how stryker defines it for an example: https://github.com/stryker-mutator/stryker-handbook/blob/master/mutator-types.md#assignment-expression

Migrate Muter's test suite to use Quick and Nimble

Muter doesn't mutate operator declarations

Right now, Muter will mutate operator declarations. This is an issue because it causes a compilation failure, which is needless and only slows Muter down.

Take this code as an example:

extension AbsolutePosition: Equatable {
    public static func == (lhs: AbsolutePosition, rhs: AbsolutePosition) -> Bool {
        return (lhs.column, lhs.line, lhs.utf8Offset) == (rhs.column, rhs.line, rhs.utf8Offset)
    }
}

This is an operator that is necessary for AbsolutePosition to be equatable; mutating it is only going to break the code, which isn't a very meaningful test failure to report on, especially given how long Muter takes to run

Fix the acceptance tests

Ejecting out of xcode broke the CLITests.swift code. These are valuable acceptance tests, and the only way that main.swift gets exercised in Muter's test suite. Fix this!

Muter reports how many test failures one mutation operator caused

Muter can show a mutation score for every file in a code base

Reduce the amount of text in Muter's output

Right now, Muter produces a lot of text as a result of running xctest so much. To make the output more approachable and easy to digest, this output should be suppressed somehow, potentially written to disk so it can be reviewed later to diagnose issues.

Muter pre-fills a generated config file with its best guesses for the options

Muter supports mutating statements which cause side effects

It's important to know if a test suite is aware and testing all of the side effects that are in a code base. Muter should have a way of detecting when a line of code is intended to cause a side effect, and have a way of causing the code to not cause that side effect (probably by deleting the line).

Create a model for the report that Muter creates

This is part of #54.

Muter tells the user when there's a new version available at the start of a run

Exclude deinit blocks from being mutated

Presently, Muter doesn't explicitly test for this case but should. This story is really just about adding a test for these cases with the Remove Side Effect Operator.

Lint code prior to any mutation testing so it can be operated on by Muter

Right now, Muter has a limitation that requires there to be spaces around operators so it can mutate them. The story is supposed to add a step that eliminates this limitation.

We can probably use swiftlint to do some very basic formatting of code Muter is operating on after it copies it to a temporary directory, but before it begins doing any file or mutation operator discovery.

Muter supports mutating integer literals

Muter stops early when it detects a certain number of consecutive build errors

Muter report its version within the header

Muter has a verbose flag which reveals more information about what's happening while it's running

Muter reports the number of tests that failed per mutation test run

Muter includes runtime errors, test failures, and passing tests in its mutation score calculations

Muter creates a config file if it detects there isn’t one

Muter will incorrectly report a passing test suite as a passing mutation test

When running Muter on BonMot, I observed this behavior across several applied mutations.

StylisticAlternates.swift                   Line: 204, Column: 41    Side Effects                passed
Tracking.swift                              Line: 23, Column: 21     Negate Conditionals         passed
StyleableUIElement.swift                    Line: 84, Column: 38     Negate Conditionals         passed

It's possible that this can be fixed by #8. However, this speaks to the need to improve how Muter determines whether or not a test suite is failing.

Interestingly enough, this behavior is inconsistent - there are times when Muter accurately determines that a test suite is failing or passing. Perhaps this bug is caused by an overreliance on the status code returned by xcodebuild.

Muter applies color to the scores it generates to convey additional meaning

Extend the Remove Side Effect operator so it removes side effects contained inside for/while loops

Muter cleans up after itself in response to UNIX signals

Right now, if you interrupt Muter while it's running, it's most likely that a user's codebase will still contain a mutant. This issue is to improve Muter's cleanup process by registering handlers for UNIX signals.

Create a regression test suite

Create a regression test suite which determines if the number of mutation operators, or the reported mutation scores, change unexpectedly when a new commit is added to Muter.

Update the path handling to always use urls instead of strings

Muter has a more legible output

Right now, the logs produced by Muter are pretty noisy. Scattered amongst all of the output of xcodebuild are the various actions that Muter performs. There's probably a way by using FileHandle to improve the legibility of the output while not losing the constant feedback which let's the user know that Muter is actually running.

Muter lists the total number of applied mutation operators in its test report

Muter supports removing the implementation of a void function

Automate the release process for new versions of Muter

Muters doesn't attempt to perform mutation testing when it discovers no mutation operators

Improve Muter's usage of domain language in its code

I think there are some subtle discrepancies between how muter refers to things and how the wider community refers to certain concepts and mutation testing. Go through and make sure it's consistent.

Surface the temporary files muter generates for the test logs so a user can leverage them to diagnose issues

Muter has friendly error messages for the few steps that can go wrong prior to mutation testing