thirtysixthspan/descriptive_statistics

Laravel PHP Framework

Laravel is a web application framework with expressive, elegant syntax. We believe development must be an enjoyable, creative experience to be truly fulfilling. Laravel attempts to take the pain out of development by easing common tasks used in the majority of web projects, such as authentication, routing, sessions, and caching.

Laravel aims to make the development process a pleasing one for the developer without sacrificing application functionality. Happy developers make the best code. To this end, we've attempted to combine the very best of what we have seen in other web frameworks, including frameworks implemented in other languages, such as Ruby on Rails, ASP.NET MVC, and Sinatra.

Laravel is accessible, yet powerful, providing powerful tools needed for large, robust applications. A superb inversion of control container, expressive migration system, and tightly integrated unit testing support give you the tools you need to build any application with which you are tasked.

Official Documentation

Documentation for the entire framework can be found on the Laravel website.

Contributing To Laravel

All issues and pull requests should be filed on the laravel/framework repository.

License

The Laravel framework is open-sourced software licensed under the MIT license

descriptive_statistics's Issues

Overriding Array Sum method causes issues with active support's Enumerable Sum method.

If an enumerable object contains a block or a any sort of non numerical values, this could cause unexpected behaviour.

Before Adding Gem
{:test => 1}.sum{|k,v| "#{k} #{v}"} # "test 1"

After adding Gem
{:test => 1}.sum{|k,v| "#{k} #{v}"} # [:test, 1]

Active Support is a very popular gem. It might be wise to implement it similar to active support. I can provide a patch if needed.

Variance & STDEV => population not sample

The variance & standard deviation methods return the population var/st. deviation, not the sample method which is usually what one would want and what users would be likely to expect.

variance.rb ought to read:

module DescriptiveStatistics
  def variance
    mean = self.mean
    self.map{ |sample| (mean - sample) ** 2 }.inject(:+) / (self.number - 1)
  end
end

I've submitted a pull request for this change

Interested in a patch to make this library less invasive?

Hi,

If you're interested in a patch that will make this library a little bit less dependent on direct monkey patching, I'd be happy to prepare one.

The steps to build it would basically be this:

1. Move all your method definitions into a DescriptiveStatistics module rather than directly into Enumerable
2. Add to lib/descriptive_statistics.rb something like module Enumerable; include DescriptiveStatistics; end
3. Add a file called lib/descriptive_statistics/safe.rb which will require all the files but NOT mix DescriptiveStatistics into Enumerable.

This will not change the behavior in the default use case of the library, it will only improve the architecture somewhat and give a few more options as to how these features can be used. If this sounds useful to you, let me know and I'll prepare a pull request.

Remove the gem from git

Seems not useful keeping it in repo

Compatibility Inquiry: descriptive_statistics 2.5.1 with Ruby 3.3.0

Hi,

I'm seeking confirmation regarding the compatibility of the descriptive_statistics 2.5.1 with the recently released Ruby 3.3.0 (December 25, 2023).

While I reviewed the gem's documentation on RubyGems and table mentioned it only specifies a minimum required Ruby version of > 0. Unfortunately, I haven't found any explicit confirmation on compatibility with newer versions like 3.3.0.

Given your expertise in this area, could you offer any insights or leads on this matter? Any information you can provide would be greatly appreciated.

Thank you for your time and assistance!

Please add changelog

I have no idea what is breaking in 2.0.0

requiring fails in Ruby 1.9.3

You'll get the following stack on a require:

/Users/doug/.rbenv/versions/1.9.3-p448/lib/ruby/gems/1.9.1/gems/descriptive_statistics-2.3.0/lib/descriptive_statistics/class_methods.rb:24:in `define_method': bind argument must be a subclass of DescriptiveStatistics (TypeError)
    from /Users/doug/.rbenv/versions/1.9.3-p448/lib/ruby/gems/1.9.1/gems/descriptive_statistics-2.3.0/lib/descriptive_statistics/class_methods.rb:24:in `block in singletonclass'
    from /Users/doug/.rbenv/versions/1.9.3-p448/lib/ruby/gems/1.9.1/gems/descriptive_statistics-2.3.0/lib/descriptive_statistics/class_methods.rb:23:in `each'
    from /Users/doug/.rbenv/versions/1.9.3-p448/lib/ruby/gems/1.9.1/gems/descriptive_statistics-2.3.0/lib/descriptive_statistics/class_methods.rb:23:in `singletonclass'
    from /Users/doug/.rbenv/versions/1.9.3-p448/lib/ruby/gems/1.9.1/gems/descriptive_statistics-2.3.0/lib/descriptive_statistics/class_methods.rb:3:in `<module:DescriptiveStatistics>'
    from /Users/doug/.rbenv/versions/1.9.3-p448/lib/ruby/gems/1.9.1/gems/descriptive_statistics-2.3.0/lib/descriptive_statistics/class_methods.rb:1:in `<top (required)>'
    from /Users/doug/.rbenv/versions/1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:54:in `require'
    from /Users/doug/.rbenv/versions/1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:54:in `require'
    from /Users/doug/.rbenv/versions/1.9.3-p448/lib/ruby/gems/1.9.1/gems/descriptive_statistics-2.3.0/lib/descriptive_statistics/safe.rb:14:in `<top (required)>'
    from /Users/doug/.rbenv/versions/1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:54:in `require'
    from /Users/doug/.rbenv/versions/1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:54:in `require'
    from /Users/doug/.rbenv/versions/1.9.3-p448/lib/ruby/gems/1.9.1/gems/descriptive_statistics-2.3.0/lib/descriptive_statistics.rb:1:in `<top (required)>'
    from /Users/doug/.rbenv/versions/1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:128:in `require'
    from /Users/doug/.rbenv/versions/1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:128:in `rescue in require'
    from /Users/doug/.rbenv/versions/1.9.3-p448/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:39:in `require'
    from ./graphite_tools.rb:6:in `<main>'

I upgraded to Ruby 2.0 and it worked fine.

"TypeError: bind argument must be an instance of DescriptiveStatistics" in JRuby 1.7.0

Get the above error with following repro:

require 'descriptive_statistics'
[1,2,3,4,5].standard_deviation

Full error output:

TypeError: bind argument must be an instance of DescriptiveStatistics
bind at org/jruby/RubyUnboundMethod.java:114
(root) at descriptive_statistics_bug.rb:2

why rename not pull-request?

As a user I'm confused by your renaming and republication of the descriptive-statistics gem. Why not collaborate on that instead of creating a new gem and repo? Did you try contacting @jtescher ?

What's the difference between this and that? It'd be useful to have an explanation at the top or bottom of the readme about why you've forked it and what the difference in vision is.

Thanks!

is this gem still active?

Hi, I'm looking at the last commit and it was a long time ago, is this project still on use or not?

thanks

(Object doesn't support #inspect) caused by descriptive_statistics

I have a Portfolio model, which has_many AssetWeight models. Upon inclusion of the descriptive_statistics gem, I get a strange error, (Object doesn't support #inspect)

Models:

class Portfolio < ActiveRecord::Base
  belongs_to :investment_case
  has_many :asset_weights, :dependent => :destroy
  attr_accessible :name
end

class AssetWeight < ActiveRecord::Base
  belongs_to :portfolios
  belongs_to :asset
  attr_accessible :name, :weight
end

Code to cause the error

1.9.3-p392 :001 > p = Portfolio.first
1.9.3-p392 :002 > p.asset_weights
(Object doesn't support #inspect)
 => 

It worked fine before adding the gem to my Gemfile. Removing it after stopped the problem also.

PG:Error when gem is included in Gemfile

Rails 3.2.12
Gemfile line:
gem 'descriptive_statistics', :github =>'thirtysixthspan/descriptive_statistics'

When I include this gem in my gemfile I am unable to run my tests due to the error below. If I remove the gem from my Gemfile the error is fixed.

rake db:test:load

PG::Error: ERROR:  syntax error at or near "["
LINE 1: CREATE DATABASE "xxxxx_test"[:encoding, "utf-8", :adap...
                                    ^
: CREATE DATABASE "xxxxx_test"[:encoding, "utf-8", :adapter, "postgresql", :host, "localhost", :pool, 5, :password, "qwerty", :username, "xxxxx", :database, "xxxxx_test"]

/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/activerecord-3.2.12/lib/active_record/connection_adapters/postgresql_adapter.rb:652:in `async_exec'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/activerecord-3.2.12/lib/active_record/connection_adapters/postgresql_adapter.rb:652:in `block in execute'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/activerecord-3.2.12/lib/active_record/connection_adapters/abstract_adapter.rb:280:in `block in log'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-3.2.12/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/activerecord-3.2.12/lib/active_record/connection_adapters/abstract_adapter.rb:275:in `log'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/activerecord-3.2.12/lib/active_record/connection_adapters/postgresql_adapter.rb:651:in `execute'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/activerecord-3.2.12/lib/active_record/connection_adapters/postgresql_adapter.rb:766:in `create_database'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/activerecord-3.2.12/lib/active_record/railties/databases.rake:130:in `rescue in create_database'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/activerecord-3.2.12/lib/active_record/railties/databases.rake:74:in `create_database'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/activerecord-3.2.12/lib/active_record/railties/databases.rake:518:in `block (3 levels) in <top (required)>'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/rake-10.0.3/lib/rake/task.rb:228:in `call'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/rake-10.0.3/lib/rake/task.rb:228:in `block in execute'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/rake-10.0.3/lib/rake/task.rb:223:in `each'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/rake-10.0.3/lib/rake/task.rb:223:in `execute'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/rake-10.0.3/lib/rake/task.rb:166:in `block in invoke_with_call_chain'
/home/fkumro/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/rake-10.0.3/lib/rake/task.rb:159:in `invoke_with_call_chain'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/rake-10.0.3/lib/rake/task.rb:187:in `block in invoke_prerequisites'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/rake-10.0.3/lib/rake/task.rb:185:in `each'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/rake-10.0.3/lib/rake/task.rb:185:in `invoke_prerequisites'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/rake-10.0.3/lib/rake/task.rb:165:in `block in invoke_with_call_chain'
/home/fkumro/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/rake-10.0.3/lib/rake/task.rb:159:in `invoke_with_call_chain'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/rake-10.0.3/lib/rake/task.rb:152:in `invoke'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/rake-10.0.3/lib/rake/application.rb:143:in `invoke_task'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/rake-10.0.3/lib/rake/application.rb:101:in `block (2 levels) in top_level'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/rake-10.0.3/lib/rake/application.rb:101:in `each'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/rake-10.0.3/lib/rake/application.rb:101:in `block in top_level'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/rake-10.0.3/lib/rake/application.rb:110:in `run_with_threads'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/rake-10.0.3/lib/rake/application.rb:95:in `top_level'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/rake-10.0.3/lib/rake/application.rb:73:in `block in run'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/rake-10.0.3/lib/rake/application.rb:160:in `standard_exception_handling'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/rake-10.0.3/lib/rake/application.rb:70:in `run'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/gems/rake-10.0.3/bin/rake:33:in `<top (required)>'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/bin/rake:23:in `load'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/bin/rake:23:in `<main>'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/bin/ruby_noexec_wrapper:14:in `eval'
/home/fkumro/.rvm/gems/ruby-2.0.0-p247/bin/ruby_noexec_wrapper:14:in `<main>'

Any idea if there is a work around for this?

Variance denominator should be (values.number - 1) - thx

descriptive_statistics/lib/descriptive_statistics/variance.rb

Line 7 in a54e313

values.map { |sample| (mean - sample) ** 2 }.reduce(:+) / values.number

cannot calculate standard_deviation without global monkey patch

Following the example for extending an individual object, it falls over when you try and calculate standard_deviation. Adding the global monkey path works around the problem (but I don't want a global monkey patch):

$ irb
irb(main):001:0> require 'descriptive_statistics/safe'
=> true
irb(main):002:0> data = [2,6,9,3,5,1,8,3,6,9,2]
=> [2, 6, 9, 3, 5, 1, 8, 3, 6, 9, 2]
irb(main):003:0> data.extend(DescriptiveStatistics)
=> [2, 6, 9, 3, 5, 1, 8, 3, 6, 9, 2]
irb(main):004:0> data.number
=> 11.0
irb(main):005:0> data.sum
=> 54
irb(main):006:0> data.standard_deviation
NoMethodError: undefined method `sum' for #<Array:0x007fea6799c970>
        from /home/ianh/.gem/ruby/2.1.2/gems/descriptive_statistics-1.1.5/lib/descriptive_statistics/variance.rb:4:in `variance'
        from /home/ianh/.gem/ruby/2.1.2/gems/descriptive_statistics-1.1.5/lib/descriptive_statistics/standard_deviation.rb:3:in `standard_deviation'
        from (irb):6
        from /home/ianh/.rubies/2.1.2/bin/irb:11:in `<main>'
irb(main):007:0> require 'descriptive_statistics'
=> true
irb(main):008:0> data.standard_deviation
=> 2.778310325442932
irb(main):009:0> 

mean code

I think there is a problem with the code when using integers
try calculating mean of [3,4]

problem is also shown below
irb(main):008:0> 7/2
=> 3
irb(main):009:0> 7.to_f/2
=> 3.5

i did not verify it with your gem as I do not use it but I might be right
if not, sorry :-)

#sum method breaks with Ruby 2.4

Rails does some gymnastics to use the Enumerable#sum method built into Ruby 2.4+, and it falls back to the ActiveSupport Enumerable#sum method to handle situations in which the Ruby method doesn't handle it, such as an Array of Strings. Unfortunately, DescriptiveStatistics' #sum method gets in the way and breaks things when the Array is empty.

Using Ruby 2.4.2 and Rails 5.1.4 with descriptive_statistics 2.5.1:

>> [].sum
#> NoMethodError: undefined method `each' for nil:NilClass

See this post on StackOverflow for details.

rake db:create fails

Looks like there's an overriding method which causes this to fail.

Rails 4.1.4 and descriptive_statistics 2.5.1

Gem is inconsistent with Rails methods which lead to really bad behavior.

Defining all the DescriptiveStatistics methods on enumerable without namespacing whatsoever can lead to incompatibility with the Rails framework. Given that it is the most used framework in the community, I think this can lead to really nasty bugs.

For instance Rails already define a method sum on Enumerable, which is:

# File activesupport/lib/active_support/core_ext/enumerable.rb, line 20
def sum(identity = 0, &block)
  if block_given?
    map(&block).sum(identity)
  else
    inject { |sum, element| sum + element } || identity
  end
end

and DescriptiveStatistics#sum is:

  def sum(collection = self, &block)
    values = Support::convert(collection, &block)
    return DescriptiveStatistics.sum_empty_collection_default_value if values.empty?

    return values.reduce(:+)
  end

As you can see the behaviour AND the signature is not the same.
Most notably, in rails:

[].sum # -> 0
[].sum(3) # -> 3

in this gem:

[].sum # -> nil
[].sum(3) # -> NoMethodError: undefined method `each' for 3:Fixnum

the 0 becoming nil is the most concerning for those using both Rails and this gem, since the bug can go undetected for quite a while.

In my opinion, the best approach would be either to namespace/prefix the methods, (or instead of including it as a mixin in Enumerable, having module functions but that's way more restrictive).

What do you think of it? what do you think would be the best solution to it?

It would be cool to have a method to get the number of unique values

Return value for an empty set

Hi Derrick,

Thank you for an awesome gem!

Zero is only the identity under sum, i.e. for all xs: sum(xs) = sum(xs + [0]).

Therefore, sum([] + [0]) = sum([]). Since [] + [0] = [0], it follows that sum([0]) = sum([]) = 0, so sum([]) should always return 0.

For all the other statistics, the opposite is true: f(xs + [0]) != f(xs), so they should not return 0.

Would you like me to open a pull request?

Median of an array with only one element fails

Calculating the median of an array with only a single element fails. The median code uses percentile(50) and that crashes if there are not al least two elements.

I know that it makes little sense to calculate the median of a single element, however, I use the desctiptive_statistics gem to do batch processing of large amounts of data and process arrays with lengths from 1 to n.

While I can handle that case in my code, I think that others would also benefit if median would work for one element.

Max and Min Gives Misleading Data

require 'descriptive_statistics'
hash = {:a=>2,:b=>6,:c=>9,:d=>3,:e=>5,:f=>1,:g=>8,:h=>3,:i=>6,:j=>9,:k=>2}
hash.min
#=> [:a, 2]
hash.min_by { | key, value | value }
#=> [:f, 1]

hash.max
#=> [:k, 2]
hash.max_by { | key, value | value }
#=> [:c, 9]

This gem is using the native "Ruby" max and min functions, which don't really work well for Hashes (it defaults to finding the maximum and minimum keys, which is a fairly useless piece of trivia...most of the time, we don't care about alphabetical order). While you can get the correct answers by using the max_by or min_by methods that is also within native Ruby, this little "feature" can get pretty annoying when you decide to use this Ruby gem's descriptive_statistics method.

hash.descriptive_statistics
#=>
{:number=>11.0, 
:sum=>54.0, 
:variance=>7.7190082644628095, 
:standard_deviation=>2.778310325442932, 
:min=>[:a, 2],
:max=>[:k, 2],
:mean=>4.909090909090909,
:mode=>2,
:median=>5.0,
:range=>8.0,
:q1=>2.5,
:q2=>5.0,
:q3=>7.0}

I don't really know of a good way to fix this annoyance, which is why I am reporting this issue to you. At the very least, having this issue here is a great way to prevent future people from getting annoyed.

you are using incorrect formula for kurt.

its
n_M4 / (M2_M2) - 3.0;
not
double(n)_M4 / (M2_M2 - 3.0);

Missing tests for .gem file

Hey ,
I'm planning to package descriptive_statistics to be available in debain.Unfortunately the gem file downloaded from rubygems.org does not come with tests. Can you include tests to be available for the gem uploaded in rubygems.org ?

anyone thought to ignore nulls, as an option

Often I have holes in my data, and I want to get some stats but ignore nulls.
example:
data = [1, 2, 3, 4, 5, nil] # => [1, 2, 3, 4, 5, nil]
stats = DescriptiveStatistics::Stats.new(data)
stats.mean # => 2.5

because the nil is treated like a zero. If instead this "data hole" is ignored, the mean should be 3.0, as in
stats = DescriptiveStatistics::Stats.new(data.compact)
stats.mean # => 3.0

Has there been any discussion of an optional parameter to ignore nulls? Perhaps something like:
stats = DescriptiveStatistics::State.new(data, ignore_nulls=true)