masukomi/rtest

An RSpec wrapper to cut down on noise and simplify rerunning tests.

rtest

An RSpec wrapper that cuts down on the noisy output, and makes it easier to re-run failing tests.

Why Use It?

I'll answer that with 2 pictures. I've added 5 failures into the multipart-post gem and then run the full test suite. The 1st image is rtest output. The 2nd image is what you'll have to wade through if you keep using RSpec. Under the covers, rtest is still running the same rspec command. It's just trimming the fat from the output, making it more readable, and easier to understand.

Click the images to see a full-size readable version.

rtest output

This is typical rtest's output.

RSpec output

This is what the exact same test run looks like in RSpec.

Things you can do with RTest:

• easily see the line of code that failed and the line of your spec that failed.
  • more focused test output
  • radically more focused test output
  • no more wading through hundreds of lines of stuff that doesn't help you.
• quickly see a numbered list of what failed on your last run
  • rtest
• trivially rerun a past failure
  • rtest <n> where n is the number of the failed test
• trivially rerun all the past failures (and nothing else)
  • rtest rerun
• get a list of files that contained failing tests
  • rtest files
• get the path to the file that contained a specific failure
  • rtest file <n> where n is the number of the failed test.

Run rtest --help for all the options.

Or view the full usage docs here in the source.

Installation

macOS with Homebrew

brew tap masukomi/homebrew-apps
brew install rtest

everything else

Clone this repo.
Add the rtest executable to your path.

Usage

To start the process you invoke rtest in one of the following ways:

• rtest path/to/spec.rb
• rtest directory/of/tests
• rtest all

The first two work in exactly the same way as rspec. The last one just runs bundle exec rspec with no parameters, to invoke all of your tests.

rtest will give you the summarized output with each test numbered. After that you can run rtest <n> to rerun a specific failed test. Or, run rtest with no arguments to see the full list of failures and their details. This happens immediately, without rerunning the tests.

Run rtest --help to see the full list of available commands.

Notes

To do its job rtest creates a hidden file in the current directory named .rtest.json which contains the relevant information from whatever RSpec tests you had it run last.

The line numbers of failed tests are the lines they were, when it ran, but you're probably about to change that as you fix them, and reruning the old line number could cause the wrong test to be run, or maybe multiple tests. To compensate for this, and still just run 1 specific failed test you can give rtest the offset. For example, let's say you're working on failure # 3 and your changes moved the test down 5 lines. You'd say rtest 3 +5 If it was 5 lines higher you'd say rtest 3 -5 Alternately you just start the process over by invoking it with a specific path or running rtest all if you want to run all available tests.

rtest's Issues

running for a specific file / line shouldn't eliminate all past failures

if i have failures for four files, and then rerun it for 1 of them manually (e.g. by specifying rtest path/to/foo_spec.rb) then the new results should be added to the old results, not replace the old results which contained other file's failures.

at the moment i'm unclear what this means if the specified file is also in the results. I think...

• if the file is specified without a line number we delete all results for that file before running
• if the file is specified with a line number AND that line number matches an existing one (compensating for offset if needed) then we delete that entry before running.

We may want to punt on this until v3 when we have the database available.

[v2] Gem::LoadError results in misleading failure

rtest output

(rtest is at commit 5b08e32 )

❯ rtest <path/to/test>:53
Running specs in <path/to/test>:53
..........................................
1: [REDACTED CLASS NAME]#call when <rspec context stuff>
    complains that the account doesn't exist should... do something
Traceback (most recent call last):
	7711: from /Users/kayrhodes/bin/rtest:298:in `<main>'
	7710: from /Users/kayrhodes/workspace/rtest/lib/rtest/runner.rb:14:in `run_and_display_this'
	7709: from /Users/kayrhodes/workspace/rtest/lib/rtest/persisters.rb:13:in `update_record_and_display'
	7708: from /Users/kayrhodes/workspace/rtest/lib/rtest/failure_printer.rb:18:in `display_failures'
	7707: from /Users/kayrhodes/workspace/rtest/lib/rtest/failure_printer.rb:18:in `each_with_index'
	7706: from /Users/kayrhodes/workspace/rtest/lib/rtest/failure_printer.rb:18:in `each'
	7705: from /Users/kayrhodes/workspace/rtest/lib/rtest/failure_printer.rb:19:in `block in display_failures'
	7704: from /Users/kayrhodes/workspace/rtest/lib/rtest/failure_printer.rb:13:in `print'
	 ... 7699 levels...
	   4: from /Users/kayrhodes/workspace/rtest/lib/rtest/print_helpers.rb:52:in `next_split_char'
	   3: from /Users/kayrhodes/workspace/rtest/lib/rtest/print_helpers.rb:52:in `find_all'
	   2: from /Users/kayrhodes/workspace/rtest/lib/rtest/print_helpers.rb:52:in `each'
	   1: from /Users/kayrhodes/workspace/rtest/lib/rtest/print_helpers.rb:52:in `block in next_split_char'
/Users/kayrhodes/workspace/rtest/lib/rtest/print_helpers.rb:52:in `match': stack level too deep (SystemStackError)

rspec output

❯ rspec <path/to/test>:53
WARN: Unresolved or ambiguous specs during Gem::Specification.reset:
      rake (~> 13.0)
      Available/installed versions of this gem:
      - 13.0.6
      - 13.0.1
      - 12.3.3
      unf_ext (>= 0)
      Available/installed versions of this gem:
      - 0.0.8.2
      - 0.0.8
      - 0.0.7.7
      diff-lcs (>= 1.2.0, < 2.0)
      Available/installed versions of this gem:
      - 1.5.0
      - 1.4.4
WARN: Clearing out unresolved specs. Try 'gem cleanup <gem>'
Please report a bug if this causes problems.

An error occurred while loading ./engines/payment_processing/spec/services/payment_processing/payment_methods/bank_accounts/operations/list_spec.rb.
Failure/Error: require File.expand_path("../config/environment", __dir__)

Gem::LoadError:
  You have already activated ffi 1.15.5, but your Gemfile requires ffi 1.15.4. Prepending `bundle exec` to your command may solve this.
# ./config/boot.rb:5:in `<top (required)>'
# ./config/application.rb:3:in `require_relative'
# ./config/application.rb:3:in `<top (required)>'
# ./config/environment.rb:4:in `require_relative'
# ./config/environment.rb:4:in `<top (required)>'
# ./spec/rails_helper.rb:7:in `<top (required)>'
# <path/to/spec>:3:in `<top (required)>'
Run options: include {:locations=>{"<path/to/spec>"=>[53]}}

All examples were filtered out

Top 0 slowest examples (0 seconds, 0.0% of total time):

Finished in 0.00002 seconds (files took 0.94179 seconds to load)
0 examples, 0 failures, 1 error occurred outside of examples

[v2] exiting pry-byebug with "exit!" doesn't exit.

Not sure what exactly it's doing. I think it may be just doing what it does without the exclamation point.

^c will get you out, but i need explicit handling for that too so that it doesn't present you with a stack trace.

Definitely present in the v2 branch but may also be an issue in v1.

[v2] failure_notes sometimes contain newlines

[next branch work]

in FailurePrinter#printable_failure_notes(failure)

it calls failure.failure_notes

this is expected to produce an array of lines. Sometimes they have a literal "\n" (backslash n) in them, which is fine (bad output from rtest), but sometimes they have an actual newline in them, which violates the assumption that elements of failure notes contain no newline characters.

i've plastered over that bug ( commit 5b08e32 ) , by splitting on literal newlines as well as backslash n BUT the the failure_notes array containing newlines within individual elements is a bug that should be corrected at its source.

average time check

• we should track the run time for every repo and file path.
• when beginning a run we should see if we've got recorded runs for it and provide the script with the average run time for this repo/file.
• while running a file the loop that spits out the marching ants should compare current run time against the average. If we deviate by X%
  • print out a note "Average run length exceeded by 20%"
  • change dot color to red.
• run length should be recorded in seconds

gotchas:

• We need to ignore any data from a run that involved stopping at a breakpoint.
• the loop that does the check needs to know if a breakpoint has been hit earlier so that it can not provide run length warnings for something that's almost guaranteed to be beyond the expected delta.

I think this implies that we have a separate table for run times

project	file	runtime_length
my_repo	foo/bar_spec.rb	33
my_repo	null	999

rtest next...

This isn't a todo item / bug so much as it is a place to hold ideas for the next iteration presuming 1 key difference: the incorporation of sqlite

if we replace the .rtest.json file with a central sqlite file (presumably somewhere like ~/.config/rtest/database.db ) then it opens up a number of possibilities:

Benefits

• reporting on the files that had the most problems
• reporting on the spec tests (and/or files) that took the most reruns (rtest rerun or rtest <n>). Tests that took a lot of reruns to fix are generally tests that were harder to debug (retesting to learn info about the problem) and/or resulted in more failed attempts at a fix. This helps raise awareness of areas of your system that should probably be refactored, either for clarity of understanding or for simplifying the code itself.
• sometimes after you fix a bug (and it disappears from rtest) you do a little cleanup and want to rerun the test again... but now it's gone from rtest so it becomes harder, especially if you've cleared your terminal history.
• other things i haven't thought of yet.

Complications

I don't have a problem adding sqlite as a dependency, but...

Assuming we don't shell out to the sqlite command line executable, it introduces a code dependency which is not within the standard library.

Because this is a project specifically for ruby people it's not a big ask to say "hey, run bundle install". They're familiar with it. However, most ruby devs i know have a ruby version manager like rbenv or asdf or whatever on their system to swap between different rubies. As soon as you swap into a new ruby executables like rtest that aren't part of your project break if they have any dependencies outside of the Standard Library, so now you have to cd to the rtest repo, make sure you're in the appropriate ruby, and install the dependencies.

Alternately i package this as a gem, which addresses the dependencies, but you'd still have to do a gem install rtest every time you install a new ruby and want to use it.

I've been hit by that on projects like rubocop and I think it sucks. I want my tools to "just work". So, now I'm thinking about rewriting this in a language that lets me generate an executable (Janet, Chicken Scheme, Racket, Go, etc.)

There are 2 problems with that.

1. Not many languages have support for PTY (pseudoterminal) stuff (required for interactive debugging), and I don't want to write that from scratch.
2. Now we've got a tool for ruby folks that I want ruby folks to feel comfortable contributing to (hah... ever receiving public contributions... hahahhahah 🤣) but is in a language that they prolly don't use. Maybe that doesn't matter though, because so few folks ever contribute back to open source projects anyway... and this is unlikely to ever become a wildly popular project.

Langs (other than ruby) with PTY libraries that I'd consider writing in (that I've found libs for so far): Chicken Scheme, Go.

provide mechanism for rerunning a specific file from the list

like rtest last_file but a list of files is presented and you can choose one.
I'm thinking it provides a numbered list, since that's already an existing UX pattern in rtest. Then the user enters the number of the file they want to rerun.

This will alleviate some of the problems raised in Issue #11

Control+c should not generate a stack trace

We should be explicitly handling ^c rather than letting it just die all ugly with a stack trace.

I/O error before results are shown (running in dockerized environment)

I've tried this in two docker projects:

1. move the rtest file directly into the root of the project
2. make the rtest file executable
3. docker compose run --rm my-project ./rtest all

In both projects, I get a sequence of dots ending with an I/O error. The number of dots are different between runs.

Running ALL specs...
..................................................................................................Traceback (most recent call last):
	8: from ./rtest:479:in `<main>'
	7: from ./rtest:336:in `run_this'
	6: from ./rtest:301:in `interactive_run_handler'
	5: from ./rtest:301:in `spawn'
	4: from ./rtest:323:in `block in interactive_run_handler'
	3: from ./rtest:181:in `read_into_buffer'
	2: from ./rtest:181:in `loop'
	1: from ./rtest:183:in `block in read_into_buffer'
./rtest:183:in `read': Input/output error @ io_fread - /dev/pts/1 (Errno::EIO)

An .rtest.json file is NOT generated. If I touch .rtest.json that doesn't help except I get an initial JSON parsing error at the beginning of the test run:

problem parsing .rtest.json: 766: unexpected token at ''
Running ALL specs...

Both projects have a couple of failing tests when run with docker compose run --rm my-project bundle exec rspec.

add vision impaired mode

DO NOT BEGIN WORK ON THIS UNTIL 2.0 IS RELEASED.
( everything is changing under the covers with the addition of rspec_formatter.rb )

reading rspec output via a screen reader would absolutely suck.

reading rtest output would be better.

listening to rtest output specifically designed to be read by a screen reader would be awesome.

To implement this i think it requires breaking the presentation stuff out into a class that you initialize with an instance of RunLog. Then we can have a StandardPresenter and a BlindPresenter class.

I think BlindPresenter should be invoked with a flag because otherwise the --rspec=NUMBER and files and file <number> inputs become hobbled.

I think the blind presenter should:

• tell you how many failures are present
• wait for input (assuming there are any)
  • hitting enter reads the 1st failure
  • entering a number reads that failure or tells you that it's not a valid number
• read one & wait for input.
  • hitting enter tells it to continue on to the next one
  • hitting a (for "again") tells it to read again. Yes, I know this is english-centric but one problem at a time.
  • hitting n (for "next") tells it to proceed to the next one.
  • entering a number reads that failure or tells you that it's not a valid number

Question: should it avoid using characters that are normally not spoken, or should we assume that the developers will have some form of text to speech enabled that reads all the characters? I assume the latter, but this should be confirmed with an actual blind dev.

last file should work even when no known failures

this is just a task to modify the current behavior once we have the db so that we can just reference the last file run in this project and rerun rtest for it.

redo screenshots with better example

the examples in the current screenshot show the FAILED HERE and ERROR HERE being in the same file. It happens to be true, in that case, but there really should be an example that shows what happens when the failure is actually some other file.

[v2] capture compilation errors

as you can see in the attached screenshot, rspec is capturing and formatting compilation errors (and noting "no examples found" as a result) but they aren't passed to the example_failed endpoint because it's not a failed example

We can get this in the message method

RSpec::Core::Formatters.register self,
                                   :example_failed,
                                   :close,
                                   :message
# ...

  def message(notification)
    notification.message 
    # has an error string 
    # with "Failure/Error: ..." and maybe "NameError" and a backtrace
    # (see screenshot) so, yet another custom string to parse. 
  end

UNFORTUNATELY
this kinda breaks the whole "there's a number you can reference to run it again" and probably breaks failure display.

I think the formatter is going to need to subclass Failure as Message to handle the completely different data structure.

I think rtest is going to have to subclass Failure as Message for a similar reason and....maybe also FailurePrinter ?

maybe the number is 0 ?

provide more detailed stack trace option

right now stack traces are intentionally very minimal.

However, sometimes you really want more info than is provided. At the same time RSpec's are excessively verbose and contain tons of info about stuff in gems that really doesn't help you.

Task:
provide an option that allows users to get a stack trace that includes all of the lines in the RSpec stack trace that reference files under the root of the project (as defined by git rev-parse --show-toplevel) and nothing else.

modify colors to look good on normal people's terminals

Looks great on mine, but apparently my colors are so whack it looks terrible on more "normal" terminals. ;)

record & report run times for v2

[based on "next" branch for impending v2]

record and display last run time for time, file, specific line run

recording

run_times: {
  total: 666,
  per_file: {
    "<file_path>": 123,
    "<other_file_path>": [132]
  },
  per_specific: {
    "<file_path>:<line_number>": [22, 33, 23]
  }
}

• per_specific and per_file won't be available unless the run was initiated with the same or unless it's a specific rerun.
  • per_specific should be seconds for a specific line number specified test
  • per_file should be seconds for a run of a whole file without specifying any specific test.
• times must not be recorded if a debugger pause is encountered
• times must be in seconds in an appended list.

displaying

when beginning a rerun for a file or a specific failure note the average run length for the specified file/test if known.