Covered code shown as skipped

[simplecov-ruby]

Covered code shown as skipped

[dieb]

I'm also using simplecov on a rails app. with spork and I managed to get it working using a initializer:

# config/initializers/01_coverage.rb
if ENV['RAILS_ENV'] == 'test' and ENV['COVERAGE'] and !ENV['DRB']
  require 'simplecov'
  SimpleCov.start 'rails'
end

For spork I tried doing the same as @relaxdiego, which does load simplecov but does not generate the coverage at the end. Still figuring that out.

[JeanMertz]

I've also noticed something similar.

For me though, it's a hit or miss thing. When I am working on a specific feature file and I run the test for that feature, it shows as everything 100% tested. Then, when I run all the features and specs, a method which was previously shown as 100% tested, now has some parts shown as untested.

Then, when I rerun that specific feature again, it switches back to 100%.

Weird behavior.

[colszowka]

I suspect this has to do with the way the test stack is loaded - if some part of your code is loaded before SimpleCov.start, it will not end up in the coverage report.

What test frameworks are you using and how do you fire up the test suites (Rake task, specific command, ...)?

[bikashp]

I'm using rspec. I've SimpleCov.start 'rails' right after require statements in spec_helper.rb. So, loading order is not a problem i guess. I use bundle exec rspec spec/models to fire up model test suites.

[btedev]

I'm seeing the same issue with ruby 1.9.2-p180, Rails 3.0.9, and simplecov 0.4.2.

[colszowka]

That's really odd. Guys, could you please try:

1. Update to latest dev release

Update your Gemfile with this:

gem 'simplecov', :require => false, :git => 'https://github.com/colszowka/simplecov.git'
gem 'simplecov-html', require => false, :git => 'https://github.com/colszowka/simplecov-html.git'

The latest dev version sports quite a few fixes and refactorings and I'm curious what the results are on that version.

2. Check out cached resultset

Please find a line that is clearly executed in your tests but reported as uncovered in the HTML report. Then check out coverage/.resultset.json (if you're not using the DEV release, that will be coverage/resultset.yml instead) and find the cached coverage result for your file and the line number. It's an array with a record for each line of source, so if line 4 is shown as skipped in the report, the result for it should be a 0 in the array like [1, null, null, 0]. Please let me know if that's the case.

Further things:

• If any of you happened to run into this with a piece of open source software, please post the repo of it here. It would be much easier if I could fiddle with the problem myself
• Please post your Platform (OS X, Linux, ...), ruby version, test framework (possibly also including version) so we can see if there is some common ground where this occurs (it seems to me that it only happens with RSpec?). I don't think that this has to do with any
• It's also possible this is a problem with the Ruby Coverage library SimpleCov uses. Could you please clone the simplecov git repo and run the tests to see whether they fail on your machine? Since the test suite normally passes, that would be a clear hint that something else is wrong. After clone, do a bundle install. To test run bundle exec rake test && bundle exec cucumber features

Thanks!

[ncri]

Well, I tested a bit more and my problem seems that when I type 'rake test' coverage results from the unit tests are overwritten by the ones from the functional and integration tests instead of accumulated. I guess also integration test results overwrite functional test results.

[colszowka]

What does the HTML report state in the footer as the sources for the report? Normally it should be "Unit Tests, Functional Tests, Integration Tests".

Are you using Test::Unit 2?

To fix that issue temporarily please see the section on customizing the command_name in the readme. You'll have to do SimpleCov.command_name 'Functional Tests' (Unit Tests, Integration Tests...) for each test suite in one of the _test.rb files in that given suite (it's not important which one, just be sure to have that for each of the test suites)

[ncri]

It only says

Generated by simplecov v0.5.0 and simplecov-html v0.4.5
using /Users/nico/.rvm/gems/ruby-1.9.2-p290/gems/rake-0.9.2/lib/rake/rake_test_loader.rb

in the footer.

Using Test unit 2.3.2

[colszowka]

Ok, so this obviously results from the problems described in #45 - test unit 2 messes with ARGV and thus SimpleCov cannot detect the test suite name properly. Please use the workaround mentioned above until I find a way to handle this automatically, sorry for the trouble!

[ncri]

Yeah, no problem, I will try the workaround, doesn't seem so complicated. ;-)

[ncri]

The workaround seems to work, the coverage percentage looks much more realistic now, however it says "Coverage report generated for test:functionals, test:integration" So it seems it still ignores the unit tests, although I put SimpleCov.command_name 'test:units' in one of my unit test files.

[ncri]

Any updates on this? Wonder if my unit tests are included. but it seems so. although it only says "generated for test:functionals, test:integration" in the coverage report. I think it is because unit tests are somehow included/recognized as integration tests. The output from rake:test after running the unit tests is: Coverage report generated for Integration Tests, test:functionals, test:integration, test:units to ...

[pjammer]

I have the same issue. Rails 3.0.10 && Ruby 1.9.2p180, simplecov 0.4.2.

Using autotest I see just the method declaration highlighted in green and no code executed (red). However, i tried rake just now and it works as expected.

require 'simplecov'
SimpleCov.start 'rails' do 
  add_filter 'vendor'
end
ENV["RAILS_ENV"] = "test"
require File.expand_path('../../config/environment', __FILE__)
require 'rails/test_help'
require 'mocha'
require 'factory_girl'
require 'active_resource/http_mock'
require 'webmock/test_unit'

class ActiveSupport::TestCase

I guess i could update autotest too and such.

my final coverage report using rake:

Test run options: --seed 13796
Coverage report generated for -e , Unit Tests, Functional Tests, Integration Tests to /Users/whoami/current_project/project/coverage. 1153 / 1673 LOC (68.92%) covered.

[klauern]

I had this problem, too, until I realized that it could be something related to the Rakefile itself. Instead of putting my

require 'simplecov'
SimpleCov.start

in my spec/spec_helper.rb file, I put it in my Rakefile at the very top. That seemed to fix things. I was also getting a coverage report generated before my specs even ran (Running minitest/spec FWIW), so I thought it could be due to Rake doing something behind the scenes before my specs run.

[timhabermaas]

I also came across the same weird behavior (using rspec btw.). In my case none of the workarounds seem to help. In fact: Adding one line of Capybara code breaks the code coverage of an (almost) completely unrelated code.

To reproduce the bug:

git clone git://github.com/timhabermaas/simplecov-fail.git
bundle install
RAILS_ENV=test rake db:setup
rspec spec

At this point you'll notice, that the model competition.rb has 100% code coverage (as expected). But if you add the line

visit competition_path(@c)

in spec/requests/competitions_spec.rb and run rspec spec again, my custom validation in my Competition model isn't covered anymore.

I'm on Mac OS X 10.6.8 and my Ruby version is:

ruby 1.9.2p0 (2010-08-18 revision 29036) [x86_64-darwin10.4.0]

BTW. The name of the repository might be a bit insulting. I don't mean it, it's still a great gem! :)

[timhabermaas]

I spent some more time figuring out what's going on and it seems that FactoryGirl is somehow involved in this issue. I don't get that bug if I use ActiveRecord directly to create records. If I use FactoryGirl, though, it breaks the code coverage of my model.
I've reduced my sample application to a minimum, have a look at https://github.com/timhabermaas/simplecov-fail/blob/master/spec/requests/competitions_spec.rb for further instructions on how to reproduce the bug.

All simplecov tests pass on my machine, btw.

[colszowka]

Ah, finally some code to try out. Thanks @timhabermaas - I'll check it out over the weekend.

[colszowka]

Ok, I checked out @timhabermaas' sample project, and the problem clearly lies in Ruby's Coverage module itself. When ditching SimpleCov and just using the raw coverage lib like this, the problem seen in competitions_spec still appears:

require 'coverage'
Coverage.start
at_exit do
  require 'pp'
  pp Coverage.result.find{|file, cov| file =~ /competition\.rb/ }
end

$ rake spec # using factory_girl
=> [1, 1, nil, 1, 1, nil, 1, 1, 0, 0, nil, nil, nil]
$ rake spec # using ActiveRecord.create
=> [1, 1, nil, 1, 1, nil, 1, 1, 1, 1, nil, nil, nil]

I'm not sure why this happens, but I'll investigate this further and will file a bug for Ruby once I have narrowed this down.

[colszowka]

Ok, I filed a bug on Ruby 1.9 here: http://redmine.ruby-lang.org/issues/5333

I also forked and documented the project submitted by @timhabermaas at https://github.com/colszowka/coverage-bug

I guess now we'll have to wait and see where this brings us. Thanks for the patience folks :)

[TravisTheTechie]

Is this related to FactoryGirl? I converted my rails app from using fixtures to FactoryGirl and coverage has been 0/0 since then. I changed a bunch of stuff so I can't say that exactly it though.

I guess let's all the useful information: rails (3.1.0), mac os lion, rspec (2.6.0), simplecov (0.4.2), ruby 1.9.2

[colszowka]

@LegoMaster: Not sure, but in the sample provided by @timhabermaas it is. I'm not sure why this happens as from what I know FactoryGirl does not do any black magic, so it's weird. Also, I always use factory girl on my projects and haven't had this issue myself yet - but I mostly use Test::Unit, so it might be a problem with RSpec in conjunction with FactoryGirl. Still, it seems awkward the Coverage lib doesn't report code that clearly has been executed.

[TravisTheTechie]

It's very well I've messed up the configuration or something in my stuff, I'll keep digging. @timhabermaas seems to be getting coverage, just not correct coverage. I'm not getting anything at all currently. Thanks!

[mame]

Hello, thank you all for tracking this issue. I'm an author of coverage.so.
I tried both @timhabermaas's and @colszowka's simple projects, but I cannot reproduce this issue on my Ubuntu.
Are you all using os x? Anyone could succeed to reproduce the bug on Linux?

[colszowka]

Hi @mame, oh, that's really interesting, didn't figure yet this might be a problem on OS X only... I just tried it out on OS X so far, but I will try it on Ubuntu too to approve it works there.

[timhabermaas]

Thanks @mame for looking into it.
I'm also using OS X. Unfortunately I don't have any other OS installed, though.

[mame]

I might do something wrong to reproduce the bug because I'm not familiar with rails, rake and rspec :-)
So I've written what I did: http://redmine.ruby-lang.org/issues/5333
Please let me know if you find a mistake in my reproducing process.

[mame]

After investigation with @nagachika and @nobu, the same file is reloaded twice, which forces to reset the coverage. So this is not a bug of coverage.so. I wrote the detail in http://redmine.ruby-lang.org/issues/5333
I'm sorry I can't help you. This is just my guess, but the cause may be active_support's bad design. I recommend contacting active_support's maintainers.

[jsgarvin]

Having the same problem. I rolled back through my commits to find where this started happening and it very clearly started when I changed from Rails 3.1.0.rc1 to the released version of 3.1.0. The only change in that commit was updating the Gemfile to the newer version of Rails (and the related changes to Gemfile.lock), but there's a very stark contrast to what's being "covered" before and after that commit. Now on 3.1.1 and the problem persists. So, it might be a Rails issue with excessive reloading of files.

[colszowka]

@jsgarvin Are you using RSpec? I suspect this only happens with Rails + RSpec, for whatever reason.

I bugged @josevalim and @tenderlove about this on Twitter since I have no idea how to further investigate this issue :/ Hopefully they'll find the time to give some directions :)

[jsgarvin]

Nope, just reliable old Test::Unit

[tarsolya]

Any news on this, I've been hit by the exact same problem.

For me, this was an issue with the way rake test was running.

I am using Padrino, and a test.rake file is automatically generated in my app. It dynamically creates rake tasks based on the directories directly below test/ (e.g. rake test:models). Running a plain rake test actually just runs those smaller tasks in sequence. Each individual task reloads test_config.rb (where my SimpleCov directives are). So, basically, my integration test counts were overwriting my model test counts were overwriting my controller test counts.

I adjusted the rake test task to simply run every _test.rb in my tests directory. As result, test_config.rb was only loaded once and my coverage statistics are cumulative.

My advice is to verify that the file containing your SimpleCov directives is only being loaded once during the entire test task. I caught this by placing a simple puts at the top.

EDIT: I now see this is due to the SimpleCov.command_name, in my case it uses the same default for each rake task, overwriting the stats instead of accumulating them.

[tenderlove]

If you add this diff to the coverage repo, you can see that the file is being loaded multiple times:

diff --git a/app/models/competition.rb b/app/models/competition.rb
index 9c0ee55..ccd64e8 100644
--- a/app/models/competition.rb
+++ b/app/models/competition.rb
@@ -1,3 +1,7 @@
+puts "#" * 90
+puts caller
+puts "#" * 90
+
 class Competition < ActiveRecord::Base
   has_many :news

Loading a file multiple times will break the coverage report. This constant is loaded via const_missing and managed by the class reloading in active support. Since this class is automatically loaded, it is considered to be an "auto loaded constant". The remove_unloadable_constants! method will remove this auto loaded constant.

The Reloader middleware will call the cleanup callbacks when the request finishes. This triggers the clear_dependencies_hook set up in railties. That eventually calls our remove_unloadable_constants! method, which removes the code and messes up the coverage report. That explains why you must perform a request in order to see the coverage be wrong.

I haven't had time to dive in to the FactoryGirl code, but if you add a require in your test for the model you're testing, then active support will not unload your constant. In other words, if you apply this diff to the coverage bug repo:

diff --git a/spec/models/competition_spec.rb b/spec/models/competition_spec.rb
index f0987c8..08ab3fe 100644
--- a/spec/models/competition_spec.rb
+++ b/spec/models/competition_spec.rb
@@ -1,4 +1,5 @@
 require 'spec_helper'
+require 'competition'

 describe Competition do
   describe "validations" do
diff --git a/spec/requests/competitions_spec.rb b/spec/requests/competitions_spec.rb
index 38188b5..cd38c2e 100644
--- a/spec/requests/competitions_spec.rb
+++ b/spec/requests/competitions_spec.rb
@@ -1,4 +1,5 @@
 require 'spec_helper'
+require 'competition'

 describe "Competition" do
   describe "GET /competitions/:id" do
@@ -16,4 +17,4 @@ describe "Competition" do
       Competition.find(c.id)
     end
   end
-end
\ No newline at end of file
+end

Your output will be correct:

$ rake spec
/Users/aaron/.local/bin/ruby -S bundle exec rspec ./spec/models/competition_spec.rb ./spec/requests/competitions_spec.rb
No DRb server is running. Running in local process instead ...

Competition
  validations
    should require ends_at >= starts_at

Competition
  GET /competitions/:id
    should display basic information

Finished in 0.28892 seconds
2 examples, 0 failures
["/Users/aaron/git/coverage-bug/app/models/competition.rb",
 [1, 1, nil, 1, 1, nil, 1, 1, 2, 2, nil, nil, nil]]
$

[jsgarvin]

I tried applying Aaron's suggestion above to one of my projects (which uses Test::Unit, not Rspec) with the following results.

I have an Event model and an EventsController. When running just the unit tests before applying Aaron's suggestion, my Event model shows 100% coverage, but when running the complete suite, coverage on the Event model drops to 88.24%.

So, I added a require for the event model to the top of both the event_test.rb and the events_controller_test.rb, directly beneath the require for test_helper...

require 'test_helper'
require 'event'

class EventsControllerTest < ActionController::TestCase
  ...

However, after making this change and running the full suite of tests, the Event model still shows only 88.24% coverage.

Maybe Aaron's suggestion only works with Rspec? Or, maybe I misunderstood his suggestion?

[smathson]

I'm having the same issue with some helper tests of mine and the workaround suggested by tenderlove doesn't seem to be working (although this may be because his suggestion was for a model and not a helper). I'm also using RSpec. I added an explicit require to both of my tests and they both still report 100% coverage when run independently, but missed when run together with the entire rspec spec/.

require 'spec_helper'
require 'home_helper'

describe HomeHelper do
    ...

and

require 'spec_helper'
require 'application_helper'

describe ApplicationHelper do
    ...

New to Rails so I'm not sure what else I can try, but it is interesting that this seems to happen with both of my helper tests.

I'm having the same issue Rails + Rspec + FactoryGirl on OS X. I was getting practically 0% coverage on a controller, only method definitions were being counted. I followed @tenderlove's suggestion and put

require 'my_model'
require 'my_models_controller'

at the top of my_models_controller_spec.rb and coverage is now being reported accurately. Until a proper fix is found, that solution works well :)

[jaryl]

@smathson how are you running your rspec suite? I do this inside my project directory:

rspec .

[cameronhickey]

Has there been any movement on this issue, or an alternate workaround? I tried requiring my model in my spec:

require 'spec_helper'
require 'user'

describe User do

but this didn't affect the output of the entire suite. However, I can confirm that when only running one spec:

rspec spec/models/user_spec.rb

The generated coverage report is accurate.

Rails 3.1, Rspec, FactoryGirl, OSX and with/without Guard & Spork.

[smathson]

@jaryl No difference for me whether I run my rspec suite with:

rspec .

or

rspec spec

[colszowka]

First of all thanks a lot @tenderlove for taking the time and investigating this! Hopefully now that the root cause of this is known figuring out how to avoid it shouldn't be too far away.

What I don't really understand is why Rails would need to do class reloads in the test environment. While auto-loading missing constants makes sense to me in regards to optimizing load time when running individual tests, having them reloaded once they're in the stack doesn't seem required to me. Anyone got an idea why this would be required?

I'll have to dig further into this for further conclusions as I didn't have the time to look into this in the last weeks due to christmas/new year madness, real life and fracturing my shoulder. I'm sure not far from now we'll have a working solution for this big problem though.

Next steps/ideas include:

• Figure out why Rails does the reload in test env
• Figure out if it's possible to deactivate it, either by force-removing the reloader middleware, by setting some config, tricking the class list that is to be reloaded (evil...) or ...
• Figure out whether this has changed with Rails 3.2 as the reloading stuff has seen major changes there (anyone got some experience on this?)

[colszowka]

@smathson I think in line with what Aaron said you'll need to require your controllers since at some point they might be reloaded. Try to add the puts caller line bit from Aaron's post to the top of your helper if you can't seem to figure out which controller to require. If you force-require all the source files mentioned there you should be able to see coverage for it.

[colszowka]

Ok, had a look at the Rails 3.1 source files Aaron mentioned (https://github.com/rails/rails/blob/3-1-stable/actionpack/lib/action_dispatch/middleware/reloader.rb#L58-74 and https://github.com/rails/rails/blob/3-1-stable/railties/lib/rails/application/bootstrap.rb#L54-59) and compared them against Rails 3.2-stable - As I expected, there were quite a few changes to this. Most importantly, the clear_dependencies_hook seems to be gone from bootstrap.

I then went on to try the test repo (https://github.com/colszowka/coverage-bug) against Rails 3.2, and obviously this made things work! Both the Factory Girl and the ActiveRecord examples (see the README in the project there) now give me correct results for Coverage. This means that on Rails 3.2 at least, this problem should be resolved. It would be great if all of you who bumped into this problem could check whether this is the case for you.

Still not sure what to do about Rails 3.0 / 3.1 though as probably a lot of people will be on those versions still.

[colszowka]

Turns out, forcing config.cache_classes = true in your config/environments/test.rb makes this work on Rails 3.0 and 3.1 as well (at least on the demo repo mentioned above). This probably does mean a performance hit when running just individual spec files because from what I understand this will load all classes on load, but for the (small!) demo app it actually even improved spec run time as reported by the shell time tool.

TL;DR: How to fix it

On Rails 3.0/3.1

# config/environments/test.rb
config.cache_classes = true

On Rails 3.2

Should work out of the box

All Rails versions: Please report back as to whether this really works :)

Awesome! Thanks very much @colszowka! That fix works brilliantly :)

Nice that the problem has gone in 3.2 as well!

[smathson]

@colszowka config.cache_classes = true fixes the issue for me as well. Thanks!

[jsgarvin]

Still seem to be having the same problem under 3.2.1, with config.cache_classes = true. Using Test::Unit, not Rspec.

[jsgarvin]

Putting SimpleCov.command_name in the test_helper.rb immediately after SimpleCov.start 'rails' worked for me. So, top of my test_helper.rb looks now looks like so...

require 'simplecov'
SimpleCov.start 'rails'
SimpleCov.command_name
#etc. etc. etc.

[relaxdiego]

The above workaround that @jsgarvin suggested also worked for me. I'm using SimpleCov with Rspec and Spork on OS X Lion. This is how part of my spec_helper.rb looks like:

Spork.each_run do
  require 'simplecov'
  SimpleCov.start 'rails'
  SimpleCov.command_name
end

[relaxdiego]

I take what I said above back. :-)

Using @colszowka's suggestion above doesn't work for me either because I use Spork which preloads Rails and my objects in the prefork block which means whenever I edit a model or any of my app classes, the tests will never use them.

I'll try looking for other means...

[colszowka]

@relaxdiego For spork there appears to be a working solution at #42 now, please check that out. I'm not using it myself, but a couple people reported it as working.

@jsgarvin You can safely remove the SimpleCov.command_name bit. Without an argument giving a custom test suite name that statement is useless, see the docs.

As I haven't heard new complaints on this and everyone who reported back got it to work I think this can finally be closed. For resolving problems with Spork please check out #42.

[alexdowad]

I'm trying to use SimpleCov to measure unit test coverage for my Ruby code, and the results are clearly only counting lines which are run during initialization. For a method def, it will count the def as executed, but not the body of the method. The funny thing is, there are some lines with puts which are clearly being executed (because I can see the output in the console), but it still says they are not being executed!

I'm not using Rake, Rails, or anything like that: just running a Test::Unit test suite directly from the command line! I'm using Ruby 1.9.2, on Windows Vista.

[alexdowad]

To anyone who runs into the same problem when running Test::Unit directly with ruby... add Test::Unit::AutoRunner.run to the bottom of your test suite file!

[enderahmetyurt]

Thank you @klauern you are the best, mate!!

[tomasmetroy]

To anyone still struggling with this (using test/unit library and running the tests with rake):

• at the very beginning of your Rakefile: require 'rake/testtask'

• right underneath that:
  Rake::TestTask.new("test:all") do |t|
  t.libs = ["lib"]
  t.warning = true
  t.test_files = FileList['test/**/*_test.rb']
  end

• then run the tests with: rake test:all