sage/fudge

A tool for enforcing code coverage and documentation in continuous integration.

fudge's Issues

Factor out tasks which do just a single command line running thing

Taking options, and arguments etc, and using a CommandRunner to run the command, could be factored into a base class for Tasks.

Coverage testing issue when you have a multiple of 10 rspec tests.

If you have rspec tests which total a multiple of 10, then the coverage checker will not work if the tests are passing:

be fudge build
Running build default
Running task rspec {:coverage=>100}
Run options: include {:focus=>true}

All examples were filtered out; ignoring {:focus=>true}
......................................................................

Finished in 0.06867 seconds
70 examples, 0 failures

Randomized with seed 4963

Coverage report generated for RSpec to /Volumes/Macintosh HD 2/development/my_repo/coverage. 384 / 700 LOC (54.86%) covered.

Skipping callbacks...
Build SUCCEEDED!

The reason being that the regex to check the coverage, matches on 0 examples 0 failures which of course 70 examples 0 failures also matches!

Issue seems to be with this line: https://github.com/Sage/fudge/blob/master/lib/fudge/tasks/rspec.rb#L11

# Preconditions to check for coverage, that if not met make the test pass
# for example, if no tests exist, no need to fail
def pre_conditions_regex
    /(0 examples, 0 failures)/ # no tests exist
end

I believe this can be fixed by adjusting the regex as follows:

def pre_conditions_regex
    /^(0 examples, 0 failures)/ # no tests exist
end

flog hangs with multiple :exclude directories

The :flog directive works with a single :exclude option but not more than one. If more than one :exclude option is specified for :flog the process will hang.

This works with :flay, but not with :flog.

• This works:
  task :flay, :exclude => ['spec/','db/']
• These also work:

task :flog, :exclude => 'spec/', :max => 20, :methods => true
task :flog, :exclude => 'db/', :max => 20, :methods => true

• This doesn't work:
  task :flog, :exclude => ['spec/','db/'], :max => 20, :methods => true

Build directories in parallel

Extend each_directory to implement each_directory_parallel, which will allow the build process to use multiple cores. This would be explicit because if a build process has order dependencies it would break.

As a developer, I am tired of waiting for continuous integration to run. I want the build process to use all of the cores on my machine.

colorize mutates source string

I couldn't replicate it in the fudge specs (maybe different versions of colorize or something else) but the following happens in my project that uses fudge:

str = 'asdf'
str.bright
str #=> "\e[1masdf\e[0m"

Causing the builds to break.

Changing str.bright to str.dup.bright fixes the problem

Task rspec doesn't load the whole environment

Hi Steven,

I try to use fudge to implement a rake ci task. I got the following error after running fudge build:

/home/tobias/Documents/dev/sage/einfach_buero/germanify/lib/germanify/engine.rb:67:in `block in <class:Engine>': uninitialized constant Germanify::Engine::AccountsEngine (NameError)

So the accounts_engine is not loaded.

When I cd germanify and run bundle exec rspec spec/ all works fine.

It only works partly when I add all my dependencies to my top level Gemfile. This is my Gemfile:

source :rubygems
source '[internal gem server redacted]'

gemspec :path => 'germanify/'
gemspec :path => 'germanify_ui_components/'

gem 'fudge', :git => '[email protected]:Sage/fudge.git'

How can I solve this problem?

Add autoloading

I thought getting rid of autoloading would lead to neater code, turns out it doesn't - requires all over the shop.

Lets get autoloading back.

After build success/failure hooks

Like email, git tag etc.

Get rid of unnessecary exceptions

Though this was a good idea at the time, but exceptions whouldn't be used for command line stuff like no arg given and no command found. Just print a message and print the usage.

Add support for Rubocop

Reports on violations of the Ruby Style Guide.

https://github.com/bbatsov/rubocop

Running task... output should say the args given to the task aswell

in_directory task should print out the directory it is in when running

Add support for Flog and Flay

http://sadi.st/Ruby_Sadist.html

Could mostly replace Code Climate

Upgrade rainbow to 2.0.0

Fudge doesn't work with Rainbow 2.0.0.

See #82 for temporary workaround (requiring ~> 1.1.4).

License missing from gemspec

RubyGems.org doesn't report a license for your gem. This is because it is not specified in the gemspec of your last release.

via e.g.

spec.license = 'MIT'
# or
spec.licenses = ['MIT', 'GPL-2']

Including a license in your gemspec is an easy way for rubygems.org and other tools to check how your gem is licensed. As you can imagine, scanning your repository for a LICENSE file or parsing the README, and then attempting to identify the license or licenses is much more difficult and more error prone. So, even for projects that already specify a license, including a license in your gemspec is a good practice. See, for example, how rubygems.org uses the gemspec to display the rails gem license.

There is even a License Finder gem to help companies/individuals ensure all gems they use meet their licensing needs. This tool depends on license information being available in the gemspec. This is an important enough issue that even Bundler now generates gems with a default 'MIT' license.

I hope you'll consider specifying a license in your gemspec. If not, please just close the issue with a nice message. In either case, I'll follow up. Thanks for your time!

Appendix:

If you need help choosing a license (sorry, I haven't checked your readme or looked for a license file), GitHub has created a license picker tool. Code without a license specified defaults to 'All rights reserved'-- denying others all rights to use of the code.
Here's a list of the license names I've found and their frequencies

p.s. In case you're wondering how I found you and why I made this issue, it's because I'm collecting stats on gems (I was originally looking for download data) and decided to collect license metadata,too, and make issues for gemspecs not specifying a license as a public service :). See the previous link or my blog post about this project for more information.

fudge init generates invalid Fudgefile

Generated Fudgefile:

build :default do |b|
  b.task :rspec
end

Should be:

build :default do |b|
  task :rspec
end

Provide a way to specify per-directory metrics for flog, flay, etc.

Repos with multiple subdirectories need to have different thresholds for errors. In the example below one sub-gem is very small and has very low flog/flay scores, but another is very large and has much higher scores.

task_group :analysis do
  in_directory 'meta_addresses' do
    task :flay, :exclude => '^\.\/(db|factories|spec)\/'
    task :flog, :exclude => '^\.\/(db|factories|spec)\/', :max => 20, :average => 5, :methods => true
  end
  in_directory 'meta_banks' do
    task :flay, :exclude => '^\.\/(db|factories|spec)\/', :max => 172
    task :flog, :exclude => '^\.\/(db|factories|spec)\/', :max => 74.9, :average => 9.1, :methods => true
  end
  in_directory 'meta_invoices' do
    task :flay, :exclude => '^\.\/(db|factories|spec)\/', :max => 2117
    task :flog, :exclude => '^\.\/(db|factories|spec)\/', :max => 46.8, :average => 6.5, :methods => true
  end
end

We need higher granularity, and these metrics probably don't all belong in the Fudgefile in the root directory.

Proposal: in each subdirectory, include fudge_settings.yml:

flay:
  max: 2117
flog:
  max: 46.8
  average: 6.5

Fudge can read these files when operating inside a given directory (with in_directory or each_directory) and provide those settings to the specified task when called.

Task to just run a command line command

New projects with no code fail Yard coverage

A new project with no files will fail yard because Yard calculates coverage at 0% instead of 100%.

Running task yard stats --list-undoc, {:coverage=>100}
Files:           0
Modules:         0 (    0 undocumented)
Classes:         0 (    0 undocumented)
Constants:       0 (    0 undocumented)
Methods:         0 (    0 undocumented)
 0.00% documented
Insufficient Documentation.

This should pass.

Collaboration with rubycritic

Hi,

just stumbled upon your repository. I will look into more, from my seconds of expose especially the completeness (brakeman etc. ) is interesting. But I couldn't see the report outcome on first sight.

Do you know of https://github.com/whitesmith/rubycritic, it's scope is very similar and it offers nice reports.

Maybe you should discuss some collaboration to make both projects better or to have one even better project.

Kind regards

RSpec tests are incompatible with RSpec 3

Fudge has a development dependency on RSpec >= 2.8.0. Version 3.0 was released recently, which meets the dependency, but removes or deprecates features that our tests use:

• its has been removed
• the should syntax is deprecated and must be explicitly enabled

Available resolutions include:

• its is available in a separate gem we can depend on, and the should syntax can be enabled via a config option
• locking RSpec to >= 2.11 and < 3.0
• rewriting the tests to eliminate its and use the expect syntax

The last option is probably best, because the should syntax will eventually be removed altogether.

Documentation for running custom build name

Fudge readme is good on describing how you can define a build with a custom name (as opposed to default), but I don't think it actually says anywhere how to run a non-default build...

Move exceptions to have multiple classes in one file

Like the accounts_engine - a lot neater

Rename to cake (CI make)

Readme

Needs a nice readme on the current status, and how to define your own tasks etc.

Specs for utils (command runner and coverage checker)

At the moment these are implicitly tested

Randomly order rspec to detect order dependencies

sop_ui_components has some order-dependent specs as well. These show up in Ubuntu because the OS returns files in file creation order instead of file update order in globbing, while (I believe) in OSX it always returns in alphabetical order.

I'd like to propose adding "rspec --order random" to Fudge so that it helps find these order dependency problems sooner. I feel conflicted about this because that would make each build a shiny unique sparklepony instead of a methodically reproduced process. It also dents the Principle of Least Surprise.

Fudge::Tasks::Task#initialize doesn't extract its options correctly.

Fudge::Tasks::Task#initialize copies and stores incoming arguments in the instance variable @args, and then extracts the options hash from the arguments and stores those in @options.

However, it extracts the options from the incoming arguments, not the copy in @args. Aside from the fact that #initialize alters its parameters—which is bad form—@args still retains the options hash, which is probably unintended.

This could affect a subclass that uses @args expecting—not unreasonably—that it doesn't include the options hash that was stored in @options.

Example (admittedly a little contrived):

# file: my_shell.rb
module Fudge
  module Tasks
    class MyShell
      def initialize(*args)
        super
        self.arguments = @args.join(' ')
      end
    end
    register MyShell
  end
end

# file: Fudgefile
build :default do
  task :my_shell, 'ls -l', foo: 'bar'
end

Running fudge with the above Fudgefile would result in

Running build default
Running task my_shell ls -l, {:foo=>"bar"}
ls: cannot access {:foo=: No such file or directory
Skipping callbacks...
Build FAILED!

Complex regex not supported in exclude

You can't specify "complex" regex in the :exclude directories for file_find. This means you can't specify more than one subdirectory.

:exclude => 'spec' will properly exclude that word; :exclude => '(spec|db)' will include both.

fudge_settings leak across directories

If I specify a value in fudge_settings.yml in a directory that same value will carry over to the next directory unless overridden by a fudge_settings.yml file with that same key.

e.g.:

  each_directory ['meta_one', 'meta_too'] do
    task :torture
  end

meta_one/fudge_settings.yml:

torture:
  threshold: 8
  average: 4

meta_two/fudge_settings.yml:

torture:
  threshold: 8

In this case, the torture task will still apply the average of 4 in meta_two.

Fuji 0.3.3 raises an error with Foreground error

After upgrading to this fudge version, this error is raised:

...lib/fudge/runner.rb:23:in output_start': undefined methodforeground' for "Running build ":String (NoMethodError)

After researches, seems that rainbow requires to add:
require 'rainbow/ext/string'

I tried to add it in the lib/fudge.rb:

require 'rainbow'
require 'rainbow/ext/string' # -> this fixes the issue
require 'active_support/all'

and it works

Don't use the color red for info output

There are some outputs like:

--> In directory germanify:
Skipping callbacks...

They shouldn't be red as the color is usually used for error messages.

Should exit with the correct exit status

Improve code documentation

Spin out none-core tasks to own gems

Fudge should supply:

• shell
• compound
• bundle_aware
• subprocess

Everything else should be a separate gem:

• fudge-rspec
• fudge-yard
• fudge-cane
• fudge-flog
• fugde-flay

this would allow #47 to be created as fudge-rubocop and then people can pick and choose which analysis they want to include.

Fudge build fails with no tests

If you don't have any test, the fudge build fails.

No examples found.


Finished in 0.00006 seconds
0 examples, 0 failures
Output didn't match (?-mix:(?-mix:((\d+\.\d+)%\) covered))|(?-mix:^(0 examples, 0 failures))).
Skipping callbacks...
Build FAILED!

The issue seems to be on the pre_conditions_regex method:
https://github.com/Sage/fudge/blob/master/lib/fudge/tasks/rspec.rb#L12

def pre_conditions_regex
    /^(0 examples, 0 failures)/ # no tests exist
end

The output string for no tests is something like this:
"No examples found.\n\n\nFinished in 0.00005 seconds\n\e[32m0 examples, 0 failures\e[0m\n"

The line don't start with "0 examples…" but with the console color code "\e[32m" so we could change the method to

def pre_conditions_regex
    /[^\d](0 examples, 0 failures)/ # no tests exist
end

Need to look at formatting of output code

As I've been tidying up a lot of the code to pass the newer quality metrics, it's apparent that the formatting of output is quite common, but is distributed throughout tasks and perhaps some abstraction would be in order.

Perhaps we could pass a formatting STDOUT or similar into the tasks so that they can be given a string and an abstract 'class' for it, and the way it looks is then done by the formatter.

This would lead nicely to the fudge_server possibly being able to format into HTML for example

Add support for Brakeman for security testing

https://github.com/presidentbeef/brakeman

Analysis plugin support

There should be a way for task gems to register analytics data into some adapter. There should be a null adapter, and flags to run with a chosen adapter (local builds would use null by default).

Then we could write, for example, an elastic search adapter which can put the task analysis into elastic search for something to produce nice trend reports.

We really just need to basic framework in place for the stable 1.0 release, that way the adapters can be written and improved later..

Changes should affect the version number

Hi Steven,

you should reflect your changes by bumping the version.

Tobias