sixty-north/cosmic-ray

Mutation testing for Python

cosmic-ray's Issues

Setup continuous integration

drone, landscape, etc. Look into the options.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/1
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T12:56:32.874

Need system for comparing ast nodes

Our tests need a reliable way to compare two ast nodes for value equality. For example, do two Num nodes hold the same numeric value? I'm sure there are devilish details here, but this is necessary for more realistic tests.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/13
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T21:05:56.143

Investigate how to detect incompetent mutants that enter infinite loops

This is the kind of wacky thing that mutants might do.

One example of bad behavior is when we run cosmic ray over bottle.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/12
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T20:23:52.927

Create command via setup.py

We could install a cosmic-ray executable as part of setup.py. This can use the same underlying mechanism as the executable package, of course.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/8
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:01:40.393

Support more test systems

Right now we only really support unittest-based test discovery. There are other systems out there that we might reasonably support, so consider a way to at least abstract the discovery concept.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/5
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:00:01.487

Create command via setup.py

We could install a cosmic-ray executable as part of setup.py. This can use the same underlying mechanism as the executable package, of course.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/8
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:01:40.393

Support more versions of python

Right now we only support Python 3 (and probably really only Python 3.3+ or something like that.) It may not be easy to support multiple versions of Python in a single code base; for example, the finder/loader system may not be available very far back. In this case perhaps we should just have two version of the tool.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/3
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T12:59:08.661

Support more exotic forms of modules

Right now we're only known to work with normal Python source-based modules. We may not work with C-extension modules, zip modules, or anything else, but in principle I think we can work with a lot of these. And if we can't, we need to be robust (and informative) in the face of things we can't handle.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/4
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T12:59:33.506

Support multiple test roots

Some users might want to be able to specify more than one test root. This should be very straightforward to implement.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/6
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:00:37.172

More mutation operators

We obviously need more operations. This task really is "make a list of operators and make issues for creating them."

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/11
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:03:15.254

More mutation operators

We obviously need more operations. This task really is "make a list of operators and make issues for creating them."

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/11
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:03:15.254

Figure out what's happening when you run CR over bottle

Something screwy happens with modules or importing or something. Just go to the bottle source directory and run python -m cosmic_ray.app bottle test.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/14
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T21:45:42.611

Investigate parallelization via multiprocessing

It stands to reason that we could parallelize the heck out of Cosmic Ray with the multiprocessing module. In fact, considering how simple it should be, we would be remiss if we didn't.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/2
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T12:58:22.957

Support more versions of python

Right now we only support Python 3 (and probably really only Python 3.3+ or something like that.) It may not be easy to support multiple versions of Python in a single code base; for example, the finder/loader system may not be available very far back. In this case perhaps we should just have two version of the tool.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/3
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T12:59:08.661

System for exceptions

In some cases we'll find that surviving mutations are completely acceptable. Consider ways to allow users to add exceptions.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/7
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:01:05.851

Support more test systems

Right now we only really support unittest-based test discovery. There are other systems out there that we might reasonably support, so consider a way to at least abstract the discovery concept.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/5
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:00:01.487

Support more exotic forms of modules

Right now we're only known to work with normal Python source-based modules. We may not work with C-extension modules, zip modules, or anything else, but in principle I think we can work with a lot of these. And if we can't, we need to be robust (and informative) in the face of things we can't handle.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/4
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T12:59:33.506

Setup continuous integration

drone, landscape, etc. Look into the options.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/1
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T12:56:32.874

More mutation operators

We obviously need more operations. This task really is "make a list of operators and make issues for creating them."

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/11
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:03:15.254

System for exceptions

In some cases we'll find that surviving mutations are completely acceptable. Consider ways to allow users to add exceptions.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/7
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:01:05.851

Create executable package

Just a simple main.py to kick off main processing.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/9
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:02:06.457

Support more exotic forms of modules

Right now we're only known to work with normal Python source-based modules. We may not work with C-extension modules, zip modules, or anything else, but in principle I think we can work with a lot of these. And if we can't, we need to be robust (and informative) in the face of things we can't handle.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/4
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T12:59:33.506

Improved logging and messaging

We currently pipe all logging and results to stdout via the logging module. We should consider a system whereby mutation results are piped to a place where different formatters can be applied. For example, it would be nice to be able to get emacs-formatted output for simple clicking in a compilation-mode buffer.

Perhaps the logging module gives us everything we need. Probably so. But at a minimum it should be possible to see just the results – incompetent, killed, or survived + location information + operation info – without seeing everything else.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/10
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:02:29.870

Create command via setup.py

We could install a cosmic-ray executable as part of setup.py. This can use the same underlying mechanism as the executable package, of course.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/8
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:01:40.393

Need system for comparing ast nodes

Our tests need a reliable way to compare two ast nodes for value equality. For example, do two Num nodes hold the same numeric value? I'm sure there are devilish details here, but this is necessary for more realistic tests.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/13
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T21:05:56.143

System for exceptions

In some cases we'll find that surviving mutations are completely acceptable. Consider ways to allow users to add exceptions.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/7
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:01:05.851

Investigate how to detect incompetent mutants that enter infinite loops

This is the kind of wacky thing that mutants might do.

One example of bad behavior is when we run cosmic ray over bottle.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/12
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T20:23:52.927

Support multiple test roots

Some users might want to be able to specify more than one test root. This should be very straightforward to implement.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/6
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:00:37.172

Support more exotic forms of modules

Right now we're only known to work with normal Python source-based modules. We may not work with C-extension modules, zip modules, or anything else, but in principle I think we can work with a lot of these. And if we can't, we need to be robust (and informative) in the face of things we can't handle.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/4
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T12:59:33.506

Create executable package

Just a simple main.py to kick off main processing.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/9
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:02:06.457

Improved logging and messaging

We currently pipe all logging and results to stdout via the logging module. We should consider a system whereby mutation results are piped to a place where different formatters can be applied. For example, it would be nice to be able to get emacs-formatted output for simple clicking in a compilation-mode buffer.

Perhaps the logging module gives us everything we need. Probably so. But at a minimum it should be possible to see just the results – incompetent, killed, or survived + location information + operation info – without seeing everything else.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/10
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:02:29.870

Support more test systems

Right now we only really support unittest-based test discovery. There are other systems out there that we might reasonably support, so consider a way to at least abstract the discovery concept.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/5
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:00:01.487

Support multiple test roots

Some users might want to be able to specify more than one test root. This should be very straightforward to implement.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/6
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:00:37.172

Investigate parallelization via multiprocessing

It stands to reason that we could parallelize the heck out of Cosmic Ray with the multiprocessing module. In fact, considering how simple it should be, we would be remiss if we didn't.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/2
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T12:58:22.957

Investigate parallelization via multiprocessing

It stands to reason that we could parallelize the heck out of Cosmic Ray with the multiprocessing module. In fact, considering how simple it should be, we would be remiss if we didn't.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/2
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T12:58:22.957

Improved logging and messaging

We currently pipe all logging and results to stdout via the logging module. We should consider a system whereby mutation results are piped to a place where different formatters can be applied. For example, it would be nice to be able to get emacs-formatted output for simple clicking in a compilation-mode buffer.

Perhaps the logging module gives us everything we need. Probably so. But at a minimum it should be possible to see just the results – incompetent, killed, or survived + location information + operation info – without seeing everything else.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/10
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:02:29.870

Support more versions of python

Right now we only support Python 3 (and probably really only Python 3.3+ or something like that.) It may not be easy to support multiple versions of Python in a single code base; for example, the finder/loader system may not be available very far back. In this case perhaps we should just have two version of the tool.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/3
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T12:59:08.661

Create executable package

Just a simple main.py to kick off main processing.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/9
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:02:06.457

Setup continuous integration

drone, landscape, etc. Look into the options.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/1
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T12:56:32.874

Setup continuous integration

drone, landscape, etc. Look into the options.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/1
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T12:56:32.874

Support more test systems

Right now we only really support unittest-based test discovery. There are other systems out there that we might reasonably support, so consider a way to at least abstract the discovery concept.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/5
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:00:01.487

Need system for comparing ast nodes

Our tests need a reliable way to compare two ast nodes for value equality. For example, do two Num nodes hold the same numeric value? I'm sure there are devilish details here, but this is necessary for more realistic tests.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/13
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T21:05:56.143

Investigate how to detect incompetent mutants that enter infinite loops

This is the kind of wacky thing that mutants might do.

One example of bad behavior is when we run cosmic ray over bottle.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/12
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T20:23:52.927

Figure out what's happening when you run CR over bottle

Something screwy happens with modules or importing or something. Just go to the bottle source directory and run python -m cosmic_ray.app bottle test.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/14
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T21:45:42.611

System for exceptions

In some cases we'll find that surviving mutations are completely acceptable. Consider ways to allow users to add exceptions.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/7
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:01:05.851

Create command via setup.py

We could install a cosmic-ray executable as part of setup.py. This can use the same underlying mechanism as the executable package, of course.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/8
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:01:40.393

Support multiple test roots

Some users might want to be able to specify more than one test root. This should be very straightforward to implement.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/6
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T13:00:37.172

Support more versions of python

Right now we only support Python 3 (and probably really only Python 3.3+ or something like that.) It may not be easy to support multiple versions of Python in a single code base; for example, the finder/loader system may not be available very far back. In this case perhaps we should just have two version of the tool.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/3
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T12:59:08.661

Investigate parallelization via multiprocessing

It stands to reason that we could parallelize the heck out of Cosmic Ray with the multiprocessing module. In fact, considering how simple it should be, we would be remiss if we didn't.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/2
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T12:58:22.957

Figure out what's happening when you run CR over bottle

Something screwy happens with modules or importing or something. Just go to the bottle source directory and run python -m cosmic_ray.app bottle test.

---

• Bitbucket: https://bitbucket.org/austin_bingham/cosmic-ray/issue/14
• Originally reported by: austin_bingham
• Originally created at: 2015-02-04T21:45:42.611