sckott/mutant

mutation testing for R

mutant

mutant - mutation testing

(wikipedia: mutation testing vs. fuzzing)

info

• mutaters: in mutaters.R
  • currently only have:
    • boolean replacement (TRUE -> FALSE, and vice versa)
    • binary operator replacement (e.g., > to >=)

install

remotes::install_github("sckott/astr", "sckott/mutant")

current workflow

As of this writing (2020-05-18) ...

# path to an R package with working tests in tests/
path <- "../randgeo/" 
## collect fxns into an environment
env <- collect_fxns(path)
ls.str(env)
## make pkg map for later
pkgmap <- make_pkg_map(path)
## parse fxns with getParseData
# fxns <- parse_fxns(env)
## mutate something
mut_fxns <- mutate(as.list(env))
# what fxn was mutated?
which(vapply(mut_fxns, function(x) attr(x, "mutated"), logical(1)))
## write a new package with test suite to a tempdir
new_fxns <- make_fxns(mut_fxns)
newpath <- write_mutated_pkg(pkg_path = path, fxns = new_fxns, map = pkgmap)
## run test suite & collect diagnostics
mutout <- mutation_test(newpath)
# mutout
dplyr::select(data.frame(mutout), file, context, test, nb, failed, skipped, error, warning, passed)

This will all be internal code however - only exposing probably a few functions to users to run mutation testing, do something with results, etc.

To do

brainstorming high level steps:

1. map input package api
   • optionally map what test lines are linked to what code lines (#10)
2. generate mutants
   • each of these are full packages, which with a different mutation
3. put all mutants in a queue (#2)
4. test all mutants - pull jobs from the queue until all are done
5. collate results, write to disk

Meta

• Please report any issues or bugs.
• License: MIT
• Get citation information for mutant in R doing citation(package = 'mutant')
• Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.

mutant's Issues

Mutant reporting improvements

We need a one line summary telling us how many mutants survived and how many were killed.

A command will list the IDs of the surviving mutants. Another command will inspect a surviving mutant by its ID.

It appears that the last line of the example workflow gives us information about the tests. We should change the example to get information about the mutants. Any information about tests is secondary.

I'm guessing the whole workflow represents a single mutant, and the passing tests tell us that the mutant survived. If you confirm this, I can update the example.

pkg API

...

queueing options

write_mutated_pkg: replacing lines instead of writing new lines?

Right now write_mutated_pkg makes completely new R files, and writes the mutated fxns into those files. Maybe copy over R files from original location, and use line number/column position to replace lines - so that roxygen tags/constants/etc are retained as those things could affect outcomes

Framework issues

First of all, congratulations on doing what I think is the first and only mutation test package for R.

The thing is, I tried testing the package using the workflow proposed, but I can't even use the collect_fxns function. It's not an export object. Am I doing something wrong? Is this package even working or..?

Thank you in advance.

libs in other langs

python https://cosmic-ray.readthedocs.io/en/latest/index.html#
ruby https://github.com/mbj/mutant

`R CMD check` fails to find `pkgapi` in R 4.0

In this workflow, R CMD check fails to find pkgapi

R CMD check 
* using R version 4.0.0 (2020-04-24)
##[error]Package suggested but not available: ‘pkgapi’

eventually edit wikipedia page if this pkg is a thing

https://en.wikipedia.org/wiki/Mutation_testing

adding to list of mutation testing pkgs

Making pkg AST's

possibly https://github.com/r-lib/pkgapi

[Security] Workflow R-CMD-check.yaml is using vulnerable action actions/checkout

The workflow R-CMD-check.yaml is referencing action actions/checkout using references v1. However this reference is missing the commit a6747255bd19d7a757dbdda8c654a9f84db19839 which may contain fix to the some vulnerability.
The vulnerability fix that is missing by actions version could be related to:
(1) CVE fix
(2) upgrade of vulnerable dependency
(3) fix to secret leak and others.
Please consider to update the reference to the action.

starting R sessions

https://github.com/r-lib/callr/
tests for examples https://github.com/r-lib/callr/blob/feature/session-improvements/tests/testthat/test-r-session.R

library(callr)
# start session
rs <- r_session$new()
# poll - for session to be ready to accept commands
rs$poll_io(-1)
#>   output    error  process
#> "nopipe" "nopipe"  "ready"

Install fail

We can't build the package because:

Package required but not available: 'functionMap'

Looks like functionMap is neither a CRAN nor ropensci package?

mapping tests to code lines

covr does not have the ability to produce this mapping. if desired, we'd need to build this ourselves. possibly like

test_file = "tests/testthat/test-geo_linestring.R"
x = covr::file_coverage("R/geo_linestring.R", test_file)
df = covr::tally_coverage(x)
df$test_file = basename(test_file)
# some lines below deleted for brevity
#>              filename       functions line value             test_file
#> 1  R/geo_linestring.R  geo_linestring   23     1 test-geo_linestring.R
#> 6  R/geo_linestring.R  geo_linestring   28     0 test-geo_linestring.R
#> 11 R/geo_linestring.R  geo_linestring   34     1 test-geo_linestring.R
#> 12 R/geo_linestring.R  geo_linestring   36     1 test-geo_linestring.R
#> 13 R/geo_linestring.R  geo_linestring   37     9 test-geo_linestring.R
#> 19 R/geo_linestring.R initial_bearing   49     9 test-geo_linestring.R
#> 21 R/geo_linestring.R initial_bearing   51     9 test-geo_linestring.R
#> 25 R/geo_linestring.R   final_bearing   58     9 test-geo_linestring.R
#> 26 R/geo_linestring.R   final_bearing   59     9 test-geo_linestring.R

then at least we can only run certain test files (e.g., test-geo_linestring.R) we know are testing lines in which a mutation is located

don't think there's any way to just run specific test_that blocks within test files, and probably not a good idea anyway since there could be setup stuff we miss