knapsackpro / knapsack Goto Github PK
View Code? Open in Web Editor NEWKnapsack splits tests evenly across parallel CI nodes to run fast CI build and save you time.
Home Page: https://knapsackpro.com
License: MIT License
Knapsack splits tests evenly across parallel CI nodes to run fast CI build and save you time.
Home Page: https://knapsackpro.com
License: MIT License
I will fix that soon.
I see in the README that you can provide custom configuration in spec_helper.rb
, test_helper.rb
, etc.
This affects the configuration when running tests and generating a report.
However, how do you provide configuration when running the knapsack rake tasks?
For example, I'm using rspec, and have a custom configuration to change the report_path
.
This correctly saves the report where I want it to when I run rspec
to generate the report, but how do I tell rake knapsack:rspec
to use the alternate report location?
Any idea about having Jenkins support?
when I use knapsack with rspec. When try to run it with command rspec
, it give errors:
/Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/activesupport-4.1.14.1/lib/active_support/dependencies.rb:247:in `require': cannot load such file -- knapsack (LoadError)
from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/activesupport-4.1.14.1/lib/active_support/dependencies.rb:247:in `block in require'
from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/activesupport-4.1.14.1/lib/active_support/dependencies.rb:232:in `load_dependency'
from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/activesupport-4.1.14.1/lib/active_support/dependencies.rb:247:in `require'
from /Users/fakeuser/Sites/turbo/spec/spec_helper.rb:4:in `<top (required)>'
from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/activesupport-4.1.14.1/lib/active_support/dependencies.rb:247:in `require'
from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/activesupport-4.1.14.1/lib/active_support/dependencies.rb:247:in `block in require'
from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/activesupport-4.1.14.1/lib/active_support/dependencies.rb:232:in `load_dependency'
from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/activesupport-4.1.14.1/lib/active_support/dependencies.rb:247:in `require'
from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/rspec-core-3.3.2/lib/rspec/core/configuration.rb:1280:in `block in requires='
from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/rspec-core-3.3.2/lib/rspec/core/configuration.rb:1280:in `each'
from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/rspec-core-3.3.2/lib/rspec/core/configuration.rb:1280:in `requires='
from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/rspec-core-3.3.2/lib/rspec/core/configuration_options.rb:109:in `block in process_options_into'
from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/rspec-core-3.3.2/lib/rspec/core/configuration_options.rb:108:in `each'
from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/rspec-core-3.3.2/lib/rspec/core/configuration_options.rb:108:in `process_options_into'
from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/rspec-core-3.3.2/lib/rspec/core/configuration_options.rb:21:in `configure'
from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/rspec-core-3.3.2/lib/rspec/core/runner.rb:101:in `setup'
from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/rspec-core-3.3.2/lib/rspec/core/runner.rb:88:in `run'
from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/rspec-core-3.3.2/lib/rspec/core/runner.rb:73:in `run'
from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/rspec-core-3.3.2/lib/rspec/core/runner.rb:41:in `invoke'
from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/rspec-core-3.3.2/exe/rspec:4:in `<top (required)>'
from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/activesupport-4.1.14.1/lib/active_support/dependencies.rb:241:in `load'
from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/activesupport-4.1.14.1/lib/active_support/dependencies.rb:241:in `block in load'
from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/activesupport-4.1.14.1/lib/active_support/dependencies.rb:232:in `load_dependency'
from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/activesupport-4.1.14.1/lib/active_support/dependencies.rb:241:in `load'
from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/spring-commands-rspec-1.0.4/lib/spring/commands/rspec.rb:18:in `call'
from /Users/fakeuser/.rvm/rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
from /Users/fakeuser/.rvm/rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
from -e:1:in `<main>'
but when run it with bundle exec rspec
, it works like a charm.
have anyone know the reason?
Currently upgrading an app from Rails 5.1 to Rails 5.2 RC1 and have started to run into some issues that don't exist on our primary/production branches. It appears that there are some issues with routes, as several tests are failing with No route matches /whatever
although the routes are working fine (when looking at them in development environment), and Rails 5.1 branches pass these tests with no issues.
Something to note, when running the tests locally (they have some basic assert_routing statements) using bin/rails test test_name
, they pass, however they fail when the same tests are run using bundle exec ruby -Itest
. How is Minitest within Knapsack running the tests? Is there an internal issue with Knapsack and Rails 5.2?
Hoping you can help, thanks!
I tried to use with Cucumber but everytime I try, I get the following error:
undefined method `scenario_outline' for #<Cucumber::RunningTestCase::Scenario:0x0000000667af60> (NoMethodError)
I checked the Cucumber docs and scenario_outline
doesn't exist on Cucumber::RunningTestCase::Scenario.
http://www.rubydoc.info/gems/cucumber/Cucumber/RunningTestCase/Scenario
And also, my report is being generated as
Knapsack report was generated. Preview:
{
}
https://github.com/ArturT/knapsack#how-can-i-run-tests-from-multiple-directories using the approach described here bundle exec rake knapsack:rspec
fails with require': cannot load such file -- spec_helper (LoadError)
when KNAPSACK_TEST_FILE_PATTERN
contains other directories.
Apologies if this is not the right place to post this request.
It appears as though knapsack sequences tests in a particular order. The issue with this is it can hide caching issues an dependencies. A more ideal solution would be to randomise test ordering on each node and, potentially, across nodes. This would help catch cross test dependencies.
Cheers
I would love to be able to setup knapsack to with a series of tags. This would allow me to skip certain specs.
The reason I want this is because I have a series of specs that are testing an interface with an external service. These specs take 20 minutes to run and I don't want to run them on every push (I use travis).
Basically I want to know if the best way to do this would be to setup knapsack to pass arguments to it through to rspec.
So if I called bundle exec rake knapsack:rspec -t ~really_slow_running_specs
it would then call rspec with bundle exec rspec --default-path #{allocator.spec_dir} -- #{allocator.stringify_node_specs} -t ~really_slow_running_specs
Or would it be better to set it in the config?
Our app has binstubs generated by rails/spring in the application root/bin folder. So for example, here is Rails.root/bin/rspec
#!/usr/bin/env ruby
begin
load File.expand_path('../spring', __FILE__)
rescue LoadError => e
raise unless e.message.include?('spring')
end
require 'bundler/setup'
load Gem.bin_path('rspec-core', 'rspec')
So, we execute all commands with the bin/ prefix.
bin/rspec spec
bin/rails c
When we run knapsack
bundle exec knapsack:rspec
We get loads of errors, including undefined_method
errors
How can we get Knapsack to play nice with our binstubs?
I'm using simplecov with my rspec suite, so I want to eager load all my application code so that my coverage stats actually include all the code in my application. Knapsack worked fine before I added the eager loading to my spec_helper, but once I added this, my test suite crashes because Rails is not defined:
in spec_helper.rb
if ENV['UPLOAD_CODECOV']
require 'simplecov'
require 'codecov'
SimpleCov.start 'rails' do
formatter SimpleCov::Formatter::Codecov
end
Rails.application.eager_load!
end
I get this error: uninitialized constant Rails (NameError)
.
I'm running Knapsack like this: bundle exec rake knapsack:rspec
. Also tried with knapsack_pro, with the same results.
Any idea why Rails would not be defined here? Without this line, the specs will run, so at some point in the process Rails does get loaded, just not in my spec_helper.rb.
Thanks!
I need to know if anyone has a documented solution for using this with the Kubernetes plugin for Jenkins.
Hi -
I'm getting the below error running bundle exec rake knapsack:minitest
in both my CI and development environments:
Traceback (most recent call last):
4: from /Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/rake_test_loader.rb:5:in `<main>'
3: from /Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/rake_test_loader.rb:5:in `select'
2: from /Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/rake_test_loader.rb:17:in `block in <main>'
1: from /Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/rake_test_loader.rb:17:in `require'
/Users/ruben/projects/budgit/test/models/article_test.rb:1:in `<top (required)>': uninitialized constant ActiveSupport (NameError)
rake aborted!
Command failed with status (1): [ruby -I"lib:test" -I"/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib" "/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/rake_test_loader.rb" "test/models/article_test.rb" ]
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/testtask.rb:130:in `block (3 levels) in define'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/file_utils.rb:57:in `sh'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/file_utils.rb:105:in `ruby'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/testtask.rb:117:in `block (2 levels) in define'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/file_utils_ext.rb:59:in `verbose'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/testtask.rb:111:in `block in define'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `block in execute'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `each'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `execute'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:182:in `invoke'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/knapsack-1.16.0/lib/knapsack/runners/minitest_runner.rb:28:in `run'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/knapsack-1.16.0/lib/tasks/knapsack_minitest.rake:5:in `block (2 levels) in <main>'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `block in execute'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `each'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `execute'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:182:in `invoke'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:160:in `invoke_task'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `each'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `block in top_level'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:125:in `run_with_threads'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:110:in `top_level'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:83:in `block in run'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:186:in `standard_exception_handling'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:80:in `run'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
/Users/ruben/.rbenv/versions/2.5.1/bin/rake:23:in `load'
/Users/ruben/.rbenv/versions/2.5.1/bin/rake:23:in `<top (required)>'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/lib/bundler/cli/exec.rb:74:in `load'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/lib/bundler/cli/exec.rb:74:in `kernel_load'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/lib/bundler/cli/exec.rb:28:in `run'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/lib/bundler/cli.rb:424:in `exec'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/lib/bundler/cli.rb:27:in `dispatch'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/lib/bundler/cli.rb:18:in `start'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/exe/bundle:30:in `block in <top (required)>'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/exe/bundle:22:in `<top (required)>'
/Users/ruben/.rbenv/versions/2.5.1/bin/bundle:23:in `load'
/Users/ruben/.rbenv/versions/2.5.1/bin/bundle:23:in `<main>'
Tasks: TOP => knapsack:minitest_run
I am on Rails 5.2.1, knapsack 1.16.0 and Ruby 2.5.1.
Here is my Minitest configuration:
#test_helper.rb
ENV['RAILS_ENV'] ||= 'test'
require_relative '../config/environment'
require 'rails/test_help'
require 'knapsack'
knapsack_adapter = Knapsack::Adapters::MinitestAdapter.bind
knapsack_adapter.set_test_helper_path(__FILE__)
class ActiveSupport::TestCase
end
I am not running into any problems with the Rspec runner.
I'm getting uninitialized constant Cucumber::RbSupport (NameError)
I would like to start coding and contribute to the project the mentioned feature.
Take this scenarios as example:
Having 5 parallel CI containers running the same commit at first time, I would expect that every container generates execution for the files they are executing, and store in a specific folder, that will be aggregated at the end of the proccess.
At the next execution, instead of loading a single report file, it will load every file it can locates on that folder, and aggregate all values, so even the first run will be fast enough.
We have been using Knapsack to split RSpec on our builds on Travis, and have experienced an issue where some leftover specs would not be run on any of the jobs of a single build. Here is an example of the issue that we are having.
Assume we have the following specs:
When running, we might get something like the following on run one...
$ export CI_NODE_TOTAL=2
$ export CI_NODE_INDEX=0
$ bundle exec rake "knapsack:rspec[--profile 10]"
...
Leftover specs:
spec/a.rb
spec/c.rb
spec/e.rb
However, on job two, we get something like this...
$ export CI_NODE_TOTAL=2
$ export CI_NODE_INDEX=1
$ bundle exec rake "knapsack:rspec[--profile 10]"
...
Leftover specs:
spec/c.rb
spec/a.rb
Note that "b" is not run on either spec, and that "a" is run on both. Also, note that "a" comes before "c" in the first run, but after "c" in the second run.
The best reason I can come up with is the fact that when Knapsack globs for tests, it does not then order them. This doesn't matter for the tests with times reported, since the report distributor appears to use the test run time to sort before assigning to nodes. However, the order in which files are returned would matter when assigning leftover tests.
I still am not sure exactly why in two separate jobs, Travis is returning the list of files with different orders (I would think that both running on the same operating system, they would return the same order). However, this can all be avoided by sorting the test files in a consistent way, either when they are first retrieved or when determining the leftover tests. That way, Knapsack is not relying on the Dir
method's ordering (which according to the documentation "depends on your system") for proper distribution.
Hey guys,
I'm trying to use knapsack and semaphoreapp CI to spread the tests across multiple nodes and it looks like the before :each
callback from spec_helper.rb
is not running before the specs.
localhost
knapsack rspec โนchore/knapsack-split-specs โน2.4.0โบ
Report specs:
[SOME IRRELEVANT SPEC PATHS]
Leftover specs:
[SOME IRRELEVANT SPEC PATHS]
Knapsack time offset warning enabled!
################## Rspec configure
bundler: failed to load command: rspec (/Users/raducroitoru/.rbenv/versions/2.4.0/bin/rspec)
CountryManager::CountryNotSet: The country_code is not set!
/Users/raducroitoru/Coding/Work/A/a/lib/country_manager.rb:88:in `block in country'
/Users/raducroitoru/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/request_store-1.3.2/lib/request_store.rb:47:in `fetch'
/Users/raducroitoru/Coding/Work/A/a/lib/country_manager.rb:88:in `country'
/Users/raducroitoru/Coding/Work/A/a/app/models/api/v5/search_criteria_adapter.rb:57:in `country_locale'
/Users/raducroitoru/Coding/Work/A/a/app/models/api/v5/search_criteria_adapter.rb:11:in `criteria'
spec_helper.rb
require 'rubygems'
ENV["RAILS_ENV"] ||= 'test'
require File.expand_path("../../config/environment", __FILE__)
require 'knapsack'
require 'rspec/rails'
require 'email_spec'
require 'vcr'
require 'sidekiq/testing'
require "fakeredis/rspec"
require 'webmock/rspec'
require 'paper_trail/frameworks/rspec'
require 'shoulda/matchers'
# Requires supporting ruby files with custom matchers and macros, etc,
# in spec/support/ and its subdirectories.
Dir[Rails.root.join("spec/support/**/*.rb")].each {|f| require f}
Knapsack::Adapters::RSpecAdapter.bind
Shoulda::Matchers.configure do |config|
config.integrate do |with|
with.test_framework :rspec
with.library :rails
end
end
RSpec.configure do |config|
puts "################## Rspec configure"
config.infer_spec_type_from_file_location!
config.mock_with :rspec
config.use_transactional_fixtures = false
config.include RequestHelpers
config.include EmailSpec::Helpers
config.include EmailSpec::Matchers
config.include CountryHelper
config.extend WithModel
# For using focus tags
config.filter_run :focus => true
config.run_all_when_everything_filtered = true
config.tty = true
config.before :each do
puts "################## Before each: setting country"
CountryManager.country = :da
end
config.after :each, except: 'zipcode de' do
puts "################## After each"
# TODO Sanity stabilization experiment (2015-01, remove again soon)
expect(CountryManager.country).to eq(:da)
end
end
Knapsack: 1.13.2
Rbenv with ruby 2.4.0
Rails: 4.2.8
Rspec: 3.5.0
Rspec-rails: 3.5.2
It's working fine with normal rspec command. Is this supported?
Hi,
Thanks for creating this gem, it's very useful. I have a questions regarding what happens when tests are not specified in the report. For example if I have an rspec report that contains:
{
"spec/A_spec.rb": 3.047823905944824,
"spec/B_spec.rb": 3.047823905944824
}
And if I add a new test file spec/C_spec.rb
but I don't add it to the report will the tests in spec/C_spec.rb
be run or will they be ignored and not run?
Thanks
In the gemspec the Timecop gem is listed as a regular dependency:
spec.add_dependency 'timecop', '>= 0.1.0'
You probably want to include it only as a development dependency since it's only being used in spec/knapsack/tracker_spec.rb
:
https://github.com/ArturT/knapsack/search?utf8=%E2%9C%93&q=Timecop&type=
As our test suite becomes increasingly large and increasingly parallel, generating a full knapsack report becomes increasingly time-consuming and difficult.
It would be great to be able to run a subset of the tests and to only update their entries in the knapsack report without clearing the rest.
This gem looks awesome and solve the problem of balancing tests but on our project we only run tests with minitest. I will be possible to get a minitest adapter?
Hey @ArturT , thanks much for this, it's a lovely library and a huge time-saver for us at DCAF Engineering (and by extension, me personally).
We recently ported some integration tests over to system test (in minitest), and it took a little bit of poking around to get them to show up in the report -- KNAPSACK_GENERATE_REPORT=true bundle exec rake test test:system
in CI wound up doing the trick.
I thought I'd make a PR to the docs describing how to get systemtests to show up in the report to pay it forward, but wanted to check with you first and make sure this wouldn't be intrusive. Let me know if this is something that would be helpful.
Hi,
We are using knapsack on Semaphore CI and loves it. When our dev push their branches, we ran parallel tests on the suite, but for each test, we also want it to have --fail-fast option, stop running test on first failing test. How can I pass in that option?
Thanks.
we have "spec/requests/api/artists_spec.rb": 5.043759346008301,
in the report but its not actually executed in our CI. I was wondering is there a way to check what's the files being executed?(is it only possible to look through the output in the CI pipeline) And we do find that the problem is fixed after we regenerate the report again so not sure what's wrong(the line in the report was updated). By the way we are using minitest.
Hey Artur!
I've read the README section on running specs with tags but I'm not 100% sure this fully describes our use case. Let me tell you how we roll ๐
Before configuring Knapsack we were already using rspec tags to parallelize our build. We have 4 different tags:
What I did when configuring Knapsack was I generated a separate report file for each tagged subset of specs. So we have 4 different files: knapsack_ordered.json
, knapsack_hive.json
, etc. Based on them we further parallelize each tagged rake task.
Is this a good setup for Knapsack or would you suggest something else? One of my concerns is that such a setup creates a lot of leftover specs and I'm not sure how Knapsack deals with them in mix with tags.
I got some errors when I was using dis Gem. The rake tasks is not being loaded.
I found ou that on the example project you're loading the tasks there: https://github.com/KnapsackPro/rails-app-with-knapsack/blob/master/Rakefile
If that's required, you should update the Documentation.
I've got following errors:
$ KNAPSACK_GENERATE_REPORT=true bundle exec rake knapsack:minitest
rake aborted!
Knapsack report file knapsack_minitest_report.json doesn't exist. Please generate report first!
/Users/hong/.rbenv/versions/2.5.3/bin/bundle:23:in `load'
/Users/hong/.rbenv/versions/2.5.3/bin/bundle:23:in `<main>'
Caused by:
Errno::ENOENT: No such file or directory @ rb_sysopen - knapsack_minitest_report.json
/Users/hong/.rbenv/versions/2.5.3/bin/bundle:23:in `load'
/Users/hong/.rbenv/versions/2.5.3/bin/bundle:23:in `<main>'
Tasks: TOP => knapsack:minitest
(See full trace by running task with --trace)
It seemed there is no creating the report file at the first time, just open/write it, in codes. But this doc says like it should be created:
https://docs.knapsackpro.com/ruby/knapsack#common-step
Is it gonna fixed?
I solved this problem with creating an empty json report file manually
Docs are helpful for circleci 1.0, but 2.0 has a lot of changes and weak documentation. This is just a request to update docs for those of us migrating to 2.0!
Am I correct in assuming that If I add or delete a spec, I need to re-create the knapsack report by running the entire test suite?
Whenever we run our cukes (roughly 3 minutes to run), we get this (anonymized) output in Travis:
554 scenarios (554 passed)
4545 steps (4545 passed)
2m6.315s
Knapsack report was generated. Preview:
{
"features/**/*.feature": 1.003159761428833,
"features/**/*.feature": 1.196033239364624,
"features/**/recommend.feature": 57018183.49009848,
"features/**/*.feature": 0.2130286693572998,
"features/**/*.feature": 0.2160334587097168,
"features/**/*.feature": 0.45173168182373047,
"features/**/*.feature": 0.04832601547241211,
"features/**/*.feature": 1.230623483657837,
"features/**/*.feature": 0.7682235240936279,
"features/**/*.feature": 0.10379314422607422,
"features/**/*.feature": 0.5373373031616211,
"features/**/*.feature": 2.7034032344818115,
"features/**/*.feature": 0.32158589363098145,
"features/**/*.feature": 0.19092035293579102,
"features/**/*.feature": 0.6193909645080566,
"features/**/*.feature": 0.5318620204925537,
"features/**/*.feature": 0.7692575454711914,
"features/**/*.feature": 0.7230222225189209,
"features/**/*.feature": 0.495070219039917,
"features/**/*.feature": 0.4870891571044922,
"features/**/*.feature": 0.20952224731445312,
"features/**/*.feature": 0.7473139762878418,
"features/**/*.feature": 0.47346997261047363,
"features/**/*.feature": 1.0329043865203857,
"features/**/*.feature": 1.8328258991241455,
"features/**/*.feature": 0.8722712993621826,
"features/**/*.feature": 1.2147345542907715,
"features/**/*.feature": 0.48812413215637207,
"features/**/*.feature": 0.42723989486694336,
"features/**/*.feature": 0.9765002727508545,
"features/**/*.feature": 0.8365063667297363,
"features/**/*.feature": 1.4077143669128418,
"features/**/*.feature": 0.029282331466674805,
"features/**/*.feature": 0.11276030540466309,
"features/**/*.feature": 0.5681333541870117,
"features/**/*.feature": 0.38565659523010254,
"features/**/*.feature": 1.343261480331421,
"features/**/*.feature": 0.6901326179504395,
"features/**/*.feature": 1.1266040802001953,
"features/**/*.feature": 2.198932647705078,
"features/**/*.feature": 0.11878585815429688,
"features/**/*.feature": 0.27373504638671875,
"features/**/*.feature": 3.770233631134033,
"features/**/*.feature": 0.2858238220214844,
"features/**/*.feature": 0.5811190605163574,
"features/**/*.feature": 0.6995368003845215,
"features/**/*.feature": 0.6289098262786865,
"features/**/*.feature": 0.4094107151031494,
"features/**/*.feature": 0.74979567527771,
"features/**/email.feature": 5391999.136254549,
"features/**/*.feature": 0.33169007301330566,
"features/**/*.feature": 0.2166881561279297,
"features/**/*.feature": 0.03311800956726074,
"features/**/*.feature": 0.030640840530395508,
"features/**/*.feature": 0.17110943794250488,
"features/**/*.feature": 0.48764514923095703,
"features/**/*.feature": 0.8722720146179199,
"features/**/*.feature": 0.555567741394043,
"features/**/*.feature": 0.9950838088989258,
"features/**/*.feature": 0.25239014625549316,
"features/**/*.feature": 0.12649965286254883,
"features/**/*.feature": 0.1031959056854248,
"features/**/*.feature": 0.029646635055541992,
"features/**/*.feature": 0.1288614273071289,
"features/**/*.feature": 4.771716833114624,
"features/**/export.feature": 26731273.147525072,
"features/**/*.feature": 0.10891866683959961,
"features/**/*.feature": 0.6129484176635742,
"features/**/*.feature": 0.5339069366455078,
"features/**/*.feature": 2.2400553226470947,
"features/**/*.feature": 3.9317946434020996,
"features/**/*.feature": 1.2024469375610352,
"features/**/*.feature": 1.5093786716461182,
"features/**/*.feature": 0.39945054054260254,
"features/**/*.feature": 0.529149055480957,
"features/**/*.feature": 0.4336683750152588,
"features/**/*.feature": 1.2973487377166748,
"features/**/*.feature": 0.11217498779296875,
"features/**/*.feature": 1.6423816680908203,
"features/**/*.feature": 0.3878803253173828,
"features/**/*.feature": 0.5852077007293701,
"features/**/*.feature": 1.1968224048614502,
"features/**/*.feature": 1.0070815086364746,
"features/**/*.feature": 1.324880599975586,
"features/**/*.feature": 0.7847650051116943,
"features/**/*.feature": 0.15419507026672363,
"features/**/*.feature": 0.1375415325164795,
"features/**/*.feature": 0.2437891960144043,
"features/**/*.feature": 0.038895606994628906,
"features/**/*.feature": 0.07857513427734375,
"features/**/*.feature": 1.3422677516937256,
"features/**/*.feature": 0.472412109375,
"features/**/*.feature": 2.7329916954040527,
"features/**/*.feature": 0.029615402221679688,
"features/**/*.feature": 0.7670252323150635,
"features/**/*.feature": 0.21507048606872559,
"features/**/search.feature": 7807642.681318045,
"features/**/*.feature": 3.499208450317383,
"features/**/*.feature": 0.235215425491333,
"features/**/*.feature": 0.5279958248138428,
"features/**/*.feature": 0.21576786041259766,
"features/**/*.feature": 0.09094381332397461,
"features/**/*.feature": 0.37752389907836914,
"features/**/*.feature": 1.363354206085205,
"features/**/*.feature": 3.0626842975616455,
"features/**/*.feature": 0.1384115219116211,
"features/**/*.feature": 7.06137752532959,
"features/**/*.feature": 1.7372627258300781,
"features/**/*.feature": 1.0493583679199219,
"features/**/*.feature": 0.25408387184143066,
"features/**/*.feature": 4.52888560295105,
"features/**/*.feature": 0.5256309509277344,
"features/**/*.feature": 0.2931356430053711,
"features/**/*.feature": 1.2082080841064453,
"features/**/*.feature": 0.5562317371368408,
"features/**/*.feature": 1.2475011348724365,
"features/**/*.feature": 0.3939666748046875,
"features/**/*.feature": 0.13701748847961426,
"features/**/*.feature": 0.28044581413269043,
"features/**/*.feature": 0.10454559326171875,
"features/**/*.feature": 7.583114147186279,
"features/**/*.feature": 3.2411201000213623,
"features/**/*.feature": 0.423907995223999,
"features/**/widget.feature": 7807685.043151617
}
Knapsack global time execution for tests: 11h 08m 15s
While some of those features take a while longer, they don't take that much longer, and also the "global time execution" reported by Knapsack is a long way off compared to the real Travis run.
We recently upgraded to rails 5 which no longer uses rake and it seems that the reports are no longer generating. We're using minitest so we would run:
# Step for Minitest
$ KNAPSACK_GENERATE_REPORT=true bundle exec rake test
Now when we run the updated command the reports are not generated:
# Step for Minitest
$ KNAPSACK_GENERATE_REPORT=true bundle exec rails test
We cannot use rake because it no longer works with our tests. Is there another way to generate a report?
One of our longest specs has this line:
before(:all) do
Timecop.freeze(Time.new(2014, 6, 10, 11, 11, 11))
...
end
However, Knapsack them believes that the entire file takes 0 seconds to run.
Would it make sense (and be relatively easy?) for Knapsack to simply bypass Timecop completely when computing metrics? i.e. maybe shell out to the system to get the "real" time, regardless of the shenanigans Timecop is playing w/ Ruby's clock?
Hi, we moved our app into Docker and are now running the tests inside the container via a compose file. Is it possible to still parallelize the execution? Currently we are doing something like: @docker-compose -f ./docker-compose.test.yml run -e RAILS_ENV=test platform bash -c "bundle exec rake db:create && bundle exec rake db:migrate && bundle exec rake knapsack:rspec
Previously we were able to run with 4 worker nodes on CirceCI 2.0, would love to get back to that.
Coverage report generated for Unit Tests to /app/coverage. 5422 / 26782 LOC (20.24%) covered.
rake aborted!
Command failed with status (1)
/usr/local/bundle/gems/knapsack-1.15.0/lib/knapsack/runners/minitest_runner.rb:28:in `run'
/usr/local/bundle/gems/knapsack-1.15.0/lib/tasks/knapsack_minitest.rake:5:in `block (2 levels) in <top (required)>'
/usr/local/bundle/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
/usr/local/bundle/bin/bundle:104:in `load'
/usr/local/bundle/bin/bundle:104:in `<main>'
Tasks: TOP => knapsack:minitest_run
(See full trace by running task with --trace)
Is there something wrong here?
Would Knapsack work with CircleCI & parallel_tests gem?
I have a test suite that is a mixture of spec files in spec/**/*_spec.rb
and Turnip features in spec/**/*.feature
. I'm trying to generate the report on TravisCI and with the pattern spec/**/*{_spec.rb,.feature}
it correctly runs all of the specs and turnip features; however the generated report doesn't include any of the features. I've looked through the source of this project and didn't spot anything obvious that I thought could be to blame. Do you have any thoughts on how I could go about troubleshooting this?
This is the command I'm using to run the tests:
KNAPSACK_TEST_FILE_PATTERN="spec/**/*{_spec.rb,.feature}" KNAPSACK_GENERATE_REPORT=true bundle exec rspec spec
We, at GitLab, have an issue where the --features_path
option passed to Spinach is set to features/
: https://github.com/ArturT/knapsack/blob/e2e71d3974653bd3333fbf707fa20a8f0d95e209/lib/knapsack/runners/spinach_runner.rb#L15
The problem is that Spinach is not expecting a trailing slash for this option: https://github.com/codegram/spinach/blob/f561ec621dc309c87a60dd2c6ccca48a818ab298/lib/spinach/config.rb#L88
This ends in Spinach failing to require features/support/env.rb
first: https://github.com/codegram/spinach/blob/f561ec621dc309c87a60dd2c6ccca48a818ab298/lib/spinach/runner.rb#L136-L139
I believe we should just call .chomp('/')
on allocator.test_dir
in Knapsack::Runners::SpinachRunner
and move on with it. :)
For more detail about the issue we had, see https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10395#note_26752554.
Can you please update the README to explain the difference between Report specs:
and leftover specs:
thanks
In earlier version of minitest, Minitest.after_run
isn't available. Instead, the method for doing that was Minitest::Unit.after_tests
. I will submit a PR soon with a fix for this.
I have a machine with 8+ threads. How can I run my tests in parallel with these reports on my local system?
First of all, thank you very much for knapsack and maintaining it!
Since a few days we are getting crashes from knapsack in one of our builds because the exitstatus is nil:
rake aborted!
TypeError: no implicit conversion from nil to integer
/home/travis/build/the/project/vendor/bundle/ruby/2.3.0/gems/knapsack-1.13.1/lib/knapsack/runners/rspec_runner.rb:18:in `exit'
It crashes in rspec_runner.rb#L18 because exit
can not handle nil
itself.
I have yet to find the reason why the exitstatus is nil, but you have any ideas, that would be great.
The reason for the exitcode being nil
was an infinite recursion in another dependency.
I also tried looking into writing a patch, but I am not sure how knapsack should behave if the exitstatus is nil
.
pry(Knapsack::Presenter)> seconds
=> 700.0878098011017
time = Time.at(seconds).gmtime.strftime('%Hh %Mm %Ss') gives me error
gems/knapsack-1.16.0/lib/knapsack/presenter.rb:82: [BUG] Segmentation fault at 0x0000000000004001
Followed the install instructions. I'm able to run regular rspec and generate the knapsack report, but when I try to run the knapsack rake task I get the load error:
Rails 4.2
LoadError:
cannot load such file -- knapsack
Gemfile
group :development, :test do
...
gem 'knapsack'
...
Rakefile
require File.expand_path('../config/application', __FILE__)
Myapp::Application.load_tasks
Knapsack.load_tasks if defined?(Knapsack)
rails_helper.rb
ENV['RAILS_ENV'] = 'test'
require File.expand_path("../../config/environment", __FILE__)
require 'knapsack'
Knapsack::Adapters::RSpecAdapter.bind
Has anyone else experienced this?
Hiya!
Knapsack looks great, thanks!
On small problem - we're using the shoulda-context gem, and it seems like this is interfering with the test timing reporter. The JSON that's being generated is just:
{
"/home/travis/.rvm/gems/ruby-2.1.4/gems/shoulda-context-1.2.1/lib/shoulda/context/context.rb": 183.92807483673096
}
Is there anything we can do to fix this up? Or is that what we're expecting?
Thanks!
Hello, I want to send my complete test coverate report to codeclimate, but with knapsak, the tests are splitted, and the report is not complete.
Hava anyway to resolve this problem? ๐ค
When i start knapsack rake task rake knapsack:rspec
and finish it via ctrl-c capybara webserver is not terminated properly
Are there plans to add a merge_report
feature so knapsack will update the report file instead of overwriting it?
I'd be happy to put together a PR is the feature is wanted :)
Hey there!
You can selectively run some tests by specifying their directory, but is there a way to deselect some tests and keep the others?
Thanks!
We are using the Knapsack rake task to start our RSpec test suite:
rake knapsack:rspec
Knapsack is putting a lot of information into our build output, due to the Knapsack.logger.level
begin set to INFO
. I've tried changing this level to a lower setting, but because Knapsack is calling the logger from the rake task, there is no way to influence the logger level.
Is there any way to lower the logger level for the rake task? Maybe an environment variable can be introduced to influence the log level?
Hello,
I know this is probably not the best place but I couldn't find a link to a discussion in the readme.
I can't get knapsack to work in my cucumber file with parralalism and custom arguments on a multiline yml
test:
override:
- mkdir -p $CIRCLE_TEST_REPORTS/cucumber
# Run critical tests first
- >
KNAPSACK_REPORT_PATH=".circleci/knapsack_cucumber_critical_report.json"
RAILS_ENV=test bundle exec rake "knapsack:cucumber[
-t @critical
--format junit --out $CIRCLE_TEST_REPORTS/cucumber/junit.xml
--format json --out $CIRCLE_TEST_REPORTS/cucumber/tests.cucumber
]":
parallel: true
Gives me errors when parsing on circleci. I've also tried
test:
override:
- mkdir -p $CIRCLE_TEST_REPORTS/cucumber
# Run critical tests first
- "KNAPSACK_REPORT_PATH=\".circleci/knapsack_cucumber_critical_report.json\"
RAILS_ENV=test bundle exec rake \"knapsack:cucumber[
-t @critical
--format junit --out $CIRCLE_TEST_REPORTS/cucumber/junit.xml
--format json --out $CIRCLE_TEST_REPORTS/cucumber/tests.cucumber
]\"":
parallel: true
But I run into parsing errors. How did you manage to make multiline declarations with the parallel syntax for circleci ? I saw examples of custom arguments, and how to make the build parallel, but nos how to write that nicely in multiline.
Best error I got was
Don't know how to build task 'knapsack:cucumber[
-t @critical
--format junit --out /tmp/circle-junit.syU98Vs/cucumber/junit.xml
--format json --out /tmp/circle-junit.syU98Vs/cucumber/tests.cucumber
]'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.