jmvrbanac/specter

Specter is a Python BDD testing library which takes inspiration from RSpec and Jasmine

Specter

Specter is a Python testing framework inspired from RSpec and Jasmine. The library was created out of a desire to have a relatively flexible Python testing framework that adopted a more code-centric approach to BDD.

Specter is open-source and is available on GitHub. We love contributions!

Getting Started

• Specter Documentation
• Problems or Questions? Ask us on Freenode on the #specterframework channel

Continuous Integration

Release Notes

• Release notes are available in the documentation

specter's Issues

Add numbered source output for expect failures

Currently when an exception is thrown from a test method, the numbered source output for that exception traceback is output, which helps in locating in which file and line number the error occurred.

This issue adds similar numbered source output for expect failures as well, as it could be difficult to locate which file and line number such failures occurred, especially if a lot of tests are run.

Implement independent text content

We really need to execute a test under a clean state.

Refactor reporting system to be plugin-based

To facilitate the creation of other reporters Specter needs to dynamically load reporters.

Coverage reports empty in parallel mode

When running Specter in parallel mode, the results of the coverage seems to be getting cleared rather than combined.

specter --coverage --parallel

Issue with cross-module fixtures

Attempting to use a fixture from another module is causing an odd traceback. However, if I move the fixture into the module I want to use it in, the error goes away and it works just fine.

 Error Traceback:
          - /home/john/Repositories/github/requests-cloud/spec/keystone.pyc
          ------------------
          ------------------
          - /home/john/.virtualenvs/cloudauth/lib/python2.7/site-packages/specter/expect.py
          ------------------
            161:     """
            162:     src_line = get_called_src_line(use_child_attr='__spec__')
        --> 163:     src_params = get_expect_param_strs(src_line)
            164:     expect_obj = ExpectAssert(obj, src_params=src_params,
          ------------------
          - /home/john/.virtualenvs/cloudauth/lib/python2.7/site-packages/specter/util.py
          ------------------
            38: def get_expect_param_strs(src_line):
        --> 39:     matches = re.search('\((.*?)\)\..*\((.*?)\)', src_line)
            40:     return (matches.group(1), matches.group(2)) if matches else None
          ------------------
          - /home/john/.virtualenvs/cloudauth/lib/python2.7/re.py
          ------------------
            140:     """Scan through string looking for a match to the pattern, returning
            141:     a match object, or None if no match was found."""
        --> 142:     return _compile(pattern, flags).search(string)
          ------------------
          - Error | TypeError: expected string or buffer

Fix exit code

Currently, Specter isn't returning an exit code depending on the test results. For this to work in a CI process, Specter needs to return proper exit codes.

Add support for skipping a test in the middle of execution

The primary use case for this is if you would like to have a helper function that get run-time information needed to run a test.

Add a no-color option

Often in CI systems, you cannot see the ascii color output. Considering Specter's default reporter outputs in color, we need to provide an option to disable color.

Fix LaTeX Sphinx doc generation

PyPI package doesn't retain the version number correctly

The PyPI package version turns to 0.0.0 after install. This needs to reflect the actual installed version number.

Coverage data isn't properly reported for Python 3+

I'm wondering if this issue is due to an issue with Pynsive's module loading process as this resembles the issue that I fixed in Pynsive to support coverage in Python 2.7.

Coverage report for Python 3.4.x

py34 runtests: commands[1] | coverage report -m
Name               Stmts   Miss  Cover   Missing
------------------------------------------------
lplight/__init__       0      0   100%   
lplight/client        54     54     0%   1-101
lplight/models       124    124     0%   1-143
------------------------------------------------
TOTAL                178    178     0%   
_________________________________________

Coverage report for Python 2.7.x

py27 runtests: commands[1] | coverage report -m
Name               Stmts   Miss Branch BrMiss  Cover   Missing
--------------------------------------------------------------
lplight/__init__       0      0      0      0   100%   
lplight/client        54     54     12     12     0%   1-101
lplight/models       124      0      4      1    99%   8->7
--------------------------------------------------------------
TOTAL                178     54     16     13    65%   

Create reporter for the Specter json format

Output a format something like:

{
    "specs": [
        "ExampleSpec": {
            "docstring": "",
            "metadata": {},
            "tests": [
                "it_can_create_an_object": {
                    "docstring": "",
                    "metadata": {},
                    "success": false,
                    "error": Null,
                    "incomplete": true
                }
            ],
            "specs": []
        }
    ]
}

Utility to identify which tests hit a line of code

Adding a test category for tests that are incomplete

There needs to be a separator between what is skipped and what is just not implemented.

Lets add a decorator for "incomplete" which will skip the test, but it gets charted on a different metric.

Add except(x).to.raise(Exception)

Spec/Describe: Names with CamelCase words aren't separated by spaces

Optional kwarg output on data-driven tests

Feature requested by @reaperhulk

Summary always outputs red even when all tests pass

Add more data around failed expects and requires

As Specter grabs the parameters as written the reporter needs to output the actual values as well.

Add settings manager

We need a way get and store configuration information across the specter platform.

Clean up odd traceback when mistyping the qualifier not_to

If you accidentally use to_not instead of not_to, you get the correct error, but it then pushes out a second traceback that doesn't make much sense. Should be cleaned up.

      --> 40:         expect(inst).to_not.be_none()
        ------------------
        - Error | AttributeError: 'ExpectAssert' object has no attribute 'to_not'
Traceback (most recent call last):
  File ".tox/py27/bin/specter", line 9, in <module>
    load_entry_point('Specter==0.1.15', 'console_scripts', 'specter')()
  File "/home/john/Repositories/github/Rift/.tox/py27/lib/python2.7/site-packages/specter/runner.py", line 163, in activate
    runner.run(args)
  File "/home/john/Repositories/github/Rift/.tox/py27/lib/python2.7/site-packages/specter/runner.py", line 137, in run
    parallel_manager=self.parallel_manager)
  File "/home/john/Repositories/github/Rift/.tox/py27/lib/python2.7/site-packages/specter/spec.py", line 316, in execute
    self.standard_execution(select_metadata)
  File "/home/john/Repositories/github/Rift/.tox/py27/lib/python2.7/site-packages/specter/spec.py", line 294, in standard_execution
    self.top_parent.dispatch(TestEvent(case))
  File "/home/john/Repositories/github/Rift/.tox/py27/lib/python2.7/site-packages/pyevents/manager.py", line 58, in dispatch
    listener.callback(event)
  File "/home/john/Repositories/github/Rift/.tox/py27/lib/python2.7/site-packages/specter/reporting/console.py", line 93, in event_received
    print_expects(test_case, level)
  File "/home/john/Repositories/github/Rift/.tox/py27/lib/python2.7/site-packages/specter/reporting/console.py", line 219, in print_expects
    expect_msg = u'{mark} {msg}'.format(mark=mark, msg=expect)
  File "/home/john/Repositories/github/Rift/.tox/py27/lib/python2.7/site-packages/specter/expect.py", line 109, in __str__
    action_list[-1] = self.expected_src_param or str(self.expected)
AttributeError: 'ExpectAssert' object has no attribute 'expected'

Add error handling around expect and require

In certain circumstances, expect and require are exempt from some error catching. This should be fixed.

Incomplete tests do not show up as skipped in xunit output

Literal expect messages aren't captured across lines

If someone has a line break in their expect call, specter will pull the value of the parameter instead of outputting the literal code that the person used.

Code:
expect(
    bug_attr).to.equal(sample_dict[key])

Results:
✔ Test id to equal Test id

Instead of the expected:
✔ bug_attr to equal sample_dict[key]

Filter tests by module path

It would be good if we could run only specific tests based on a given module path.

Switch to only showing error/failed expectations by default

It would be good to have an optional cli argument to turn them all on if the user wants. I'm thinking --show-all-expects

Parallel testing

We need to be able to run our tests in parallel. This will cause a bit of pain around reporting and data-driven describes.

A way to attach release test outputs to git tags

It would be cool to view test outputs for every release. This isn't a Specter issue, but maybe a hook/plugin that can be used with Specter and GitHub so that after every release, tests can be saved for documentation/artifact purposes. This would allow us to diff on tests for every release to check on bugs introduced/deprecated features/ etc.

Capture stdout/stderr that occurs during the execution of a test

Currently, Specter doesn't capture the output for code that occurs during a test. It would be nice to have the option to suppress it.

This is was it currently looks like (kind of ugly):

          ___
        _/ @@\
    ~- ( \  O/__     Specter
    ~-  \    \__)   ~~~~~~~~~~
    ~-  /     \     Keeping the Bogeyman away from your code!
    ~- /      _\
       ~~~~~~~~~

Some actions Actions
  ∟ should not contain duplicates
(None, None)
GOT HERE!
  ∟ can disconnect

Refactor all reporting into a cleaner more uniform system

Right now it is just a hodgepodge of garbage.

Reporter: Log output

Redirect all print statements

To make specter easier to test, we need to unify how specter prints out to the console into a separate function. After it's unified, we can redirect print statements and test the output.

Add expectations for checking types and isinstance

I'm thinking adding support for the following expectations would be good:

Checking Object Type:

expect(obj).to.be_a(object_type)

Checking object is an instance:

expect(obj).to.be_an_instance_of(object_type)

Difficult to trace error messages during runtime when evaluating the expected parameter

If for some reason you get a KeyError in the expected parameter such as:

d = {}
expect('thing').to.equal(d['bam'])

Specter will give you a very odd error message that just says "bam" and that it couldn't find "expected" on the ExpectAssert class.

This should be cleaned up to output the name of the error and it's message.

Expect raises an error when attempting to catch an exception type that don't contain __name__

Specter cannot format the name as it is expecting __name__ to be there. This causes a passing test to fail.

        Error Traceback:
          - /home/john/Repositories/github/requests-cloud/spec/keystone.py
          ------------------
            15: 
            16:         def get_token_should_raise_not_implemented(self):
        --> 17:             expect(self.auth.get_token).to.raise_a(NotImplemented)
            18: 
          ------------------
          - /home/john/.virtualenvs/cloudauth/lib/python2.7/site-packages/specter/expect.pyc
          ------------------
            121:             msg = _('Function {func_name} {was} expected to raise "{excpt}"'
            122:                     ''.format(func_name=self.target_src_param,
        --> 123:                               excpt=self.expected.__name__,
            124:                               was=was))
          ------------------
          - Error | AttributeError: 'NotImplementedType' object has no attribute '__name__'

Add error handling for raised exceptions in external module constructors

Right now, if an error is raised in the constructor of an external module it is not safely caught and handled. If possible, we want to handle this with a pretty numbered source trace.

Add error test metric

Currently if a test throws an exception it gets counted as a failed test. We should have a separate category for errors.
Requested by: @samu4924

num-processes: integer end argument expected, got str

Looks like --num-processes argument doesn't get converted to a number.

Be able to remove duplicates from combined data-driven datasets

While we current handle duplication of names, we don't check if a test has duplicated kwargs. The use case is if we are loading a large dataset and want to make sure we don't have data-driven tests that are identical except for the test name; essentially making a duplicated test.

Issue mentioned by: @reaperhulk

Add basic xUnit output for CI systems

This won't handle Specter's concept of incomplete tests, but Specter should support xUnit output.

Mark incomplete as skipped.

Fix Jenkins Unicode error

This happened with using --no-color.

          ___
        _/ @@\
    ~- ( \  O/__     Specter
    ~-  \    \__)   ~~~~~~~~~~
    ~-  /     \     Keeping the boogy man away from your code!
    ~- /      _\
       ~~~~~~~~~

Example Spec
Traceback (most recent call last):
  File "specter/runner.py", line 128, in <module>
    activate()
  File "specter/runner.py", line 121, in activate
    runner.run(args)
  File "specter/runner.py", line 100, in run
    suite.execute(select_metadata=select_meta)
  File "/var/lib/jenkins/jobs/Specter Test/workspace/specter/spec.py", line 217, in execute
    self.top_parent.dispatch(TestEvent(case))
  File "/var/lib/jenkins/.virtualenvs/Specter/lib/python2.7/site-packages/pyevents/manager.py", line 58, in dispatch
    listener.callback(event)
  File "/var/lib/jenkins/jobs/Specter Test/workspace/specter/reporting/console.py", line 129, in event_received
    self.print_test_msg(name, level, status)
  File "/var/lib/jenkins/jobs/Specter Test/workspace/specter/reporting/console.py", line 65, in print_test_msg
    self.print_indent_msg(msg=msg, level=level, color=color)
  File "/var/lib/jenkins/jobs/Specter Test/workspace/specter/reporting/console.py", line 92, in print_indent_msg
    self.print_colored(msg=msg, color=color)
  File "/var/lib/jenkins/jobs/Specter Test/workspace/specter/reporting/console.py", line 98, in print_colored
    print(msg)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u221f' in position 2: ordinal not in range(128)

Fix summary report colors

Currently the summary report outputs the color green at all times. If there is an error or failure during the test run, it should change the summary color to red.

Split out test events

Currently there is only a generic TestEvent when a test completes. This should be split up into start and complete test events.

Having a parameter named "name" in before_all raises an exception

I noticed that when I had a parameter with the name "name" in my before_all, it would cause an exception to be raised. It appears to be trying to overwrite a specter specific variable:

Traceback (most recent call last):
File "/usr/local/bin/specter", line 9, in
load_entry_point('Specter==0.1.12', 'console_scripts', 'specter')()
File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/specter/runner.py", line 121, in activate
runner.run(args)
File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/specter/runner.py", line 100, in run
suite.execute(select_metadata=select_meta)
File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/specter/spec.py", line 209, in execute
self.before_all()
File "", line 120, in before_all
AttributeError: can't set attribute

Coverage doesn't seem to be catching some executed lines

Not sure what is causing this, but on a sample project I created the other day, coverage.py doesn't seem to be catching executed lines. This could be a coverage.py bug as it detects the test code, but just not the source code. This needs to be looked into.

Add coverage.py run support

Support integration of specter and coverage.py by allowing the following:

coverage run --source=blah/,tests/ -m specter

metadata decorator does not re-raise assertions

Remove duplicated traceback information

Traceback information seems to be getting duplicated between errors. In the following example, the first traceback is bleeding into the second's results:

Target Model
  ∟ Deserialization
    ∟ can deserialize from a dictionary
  ∟ Serialization
    ∟ can serialize to a summary dictionary
        Error Traceback:
          - /home/john/Repositories/github/Rift/spec/rift/data/models/target.pyc
          ------------------
            28:             summary_dict = self.target.summary_dict()
            29: 
        --> 30:             expect(summary_dict['id']).to.equal(self.example_dict['id'])
            31:             expect(summary_dict['name']).to.equal(self.example_dict['name'])
          ------------------
          - Error | KeyError: 'id'
      ✘ summary_dict['id'] self.example_dict['id']
          summary_dict['id']: fc619f5c-0844-4013-8ff0-86e0782c9978
          self.example_dict['id']: ERROR - Couldn't evaluate expected value
    ∟ can serialize to a dictionary
        Error Traceback:
          - /home/john/Repositories/github/Rift/spec/rift/data/models/target.pyc
          ------------------
            28:             summary_dict = self.target.summary_dict()
            29: 
        --> 30:             expect(summary_dict['id']).to.equal(self.example_dict['id'])
            31:             expect(summary_dict['name']).to.equal(self.example_dict['name'])
          ------------------
          - /home/john/.virtualenvs/Rift/lib/python2.7/site-packages/specter/spec.pyc
          ------------------
            54:         self.start()
            55:         try:
        --> 56:             MethodType(self.case_func, context or self)(**kwargs)
            57:         except TestIncompleteException as e:
          ------------------
          - /home/john/Repositories/github/Rift/spec/rift/data/models/target.pyc
          ------------------
            14:             target_dict = self.target.as_dict()
            15: 
        --> 16:             expect(target_dict['id']).to.equal(self.example_dict['id'])
            17:             expect(target_dict['name']).to.equal(self.example_dict['name'])
          ------------------
          - Error | KeyError: 'id'
      ✘ target_dict['id'] self.example_dict['id']

Simple Coverage.py integration

Assertion be_in() is reversed