cardmagic / classifier Goto Github PK
View Code? Open in Web Editor NEWA general classifier module to allow Bayesian and LSI classifications.
License: Other
A general classifier module to allow Bayesian and LSI classifications.
License: Other
There is a lot of info about the quality of a classification guess
in LSI which is hidden from the caller.
Sometimes, the guess is of very high quality and sometimes it's a
toss-up. The caller has no way to distinguish them though.
It would be nice if there was an option to get not only the classification
but a measure of the system's confidence.
I propose adding a method classify_with_confidence(text) which returns
both the classification AND a measure of the confidence as a number
between 0 and 1.0
i.e. guess, confidence = lsi.classify_with_confidence(text)
Then a caller could choose not to use any classification guesses whose
confidence is below some threshold.
It looks like the prior probability for each classification is ignored in the Naive Bayes classifier. You should keep track of the number of times "train" is called with each class, and then use that to calculate the prior for each class: P(class) = # class trainings / # all trainings. You should then add Math.log(P(class)) to your log sum (score[category.to_s]) in your "classifications" method.
Without the addition of that prior you do not have a Naive Bayes classifier.
I am using sourceclassifier gem, but whenever i want to identify a piece of code an error pops out. From the stacktrace I understood that this happens in classifier 'classification' method.
Here is the full stacktrace:
from /Users/olegkiviljov/.rvm/gems/ruby-2.0.0-p247/gems/classifier-1.3.4/lib/classifier/bayes.rb:88:in classify' from /Users/olegkiviljov/.rvm/gems/ruby-2.0.0-p247/gems/chrislo-sourceclassifier-0.2.3/lib/sourceclassifier.rb:19:in
identify'
from experiment.rb:21:in `
I posted the same issue in sourceclassifier issues but with no avail. If someone could atleast tell me if it is a problem with sourceclassifier or with classifier, this would give me enough information to try and fix it myself.
I'm using Jekyll, which uses the Classifier gem to index related posts. I have GSL 1.14 installed via Homebrew, and it's in my path. I have the GSL gem 1.14.7 installed (using rbenv, but I can load it in irb so that's not the problem). When Classifier runs, it raises this error:
Notice: for 10x faster LSI support, please install http://rb-gsl.rubyforge.org/
What should I try to get Classifier to recognize the GSL gem?
I'm trying to create a classifier using the cardmagic classifier gem. This is my code:
require 'classifier'
classifications = '1007.09', '1006.03'
traindata = Hash["1007.09" => "ADAPTER- SCREENING FOR VALVES VBS", "1006.03" => "ACTUATOR- LINEAR"]
b = Classifier::Bayes.new classifications
traindata.each do |key, value|
b.train(key, value)
end
But when I run this i get the following error:
Notice: for 10x faster LSI support, please install http://rb-gsl.rubyforge.org/
c:/Ruby192/lib/ruby/gems/1.9.1/gems/classifier-1.3.3/lib/classifier/bayes.rb:27:in block in train': undefined method
[]' for nil:NilClass (NoMethodError)
from c:/Ruby192/lib/ruby/gems/1.9.1/gems/classifier-1.3.3/lib/classifier/bayes.rb:26:in each' from c:/Ruby192/lib/ruby/gems/1.9.1/gems/classifier-1.3.3/lib/classifier/bayes.rb:26:in
train'
from C:/_Chris/Code/classifier/smdclasser.rb:13:in block in <main>' from C:/_Chris/Code/classifier/smdclasser.rb:11:in
each'
from C:/_Chris/Code/classifier/smdclasser.rb:11:in `
This is the source from the gem code:
Provides a general training method for all categories specified in Bayes#new
For example:
b = Classifier::Bayes.new 'This', 'That', 'the_other'
b.train :this, "This text"
b.train "that", "That text"
b.train "The other", "The other text"
def train(category, text)
category = category.prepare_category_name
text.word_hash.each do |word, count|
@categories[category][word] ||= 0
@categories[category][word] += count
@total_words += count
end
end
I am lost where to go to troubleshoot this error, what is the next step i should take?
Using active_record 2.3.8 and classifier 1.3.1, it appears that requiring classifier causes an error with active_record.
You can see what I'm talking about (hopefully) by running the test Ruby script at http://gist.github.com/463836 (and creating a test database with the included mysql dump). Works fine if you don't require classifier, errors if you do.
The trace is also included in the gist.
Hi! Not sure if you can help with this? I see you had some recent fixes/commits related to mathn.
Thank you!
gem install classifier
Building native extensions. This could take a while...
ERROR: Error installing classifier:
ERROR: Failed to build gem native extension.
current directory: /Users/dpetruha/.gem/ruby/3.1.2/gems/mathn-0.1.0/ext/mathn/complex
/Users/dpetruha/.rubies/ruby-3.1.2/bin/ruby extconf.rb
creating Makefile
current directory: /Users/dpetruha/.gem/ruby/3.1.2/gems/mathn-0.1.0/ext/mathn/complex
make DESTDIR\= sitearchdir\=./.gem.20240731-20135-dhnesj sitelibdir\=./.gem.20240731-20135-dhnesj clean
current directory: /Users/dpetruha/.gem/ruby/3.1.2/gems/mathn-0.1.0/ext/mathn/complex
make DESTDIR\= sitearchdir\=./.gem.20240731-20135-dhnesj sitelibdir\=./.gem.20240731-20135-dhnesj
compiling complex.c
linking shared-object mathn/complex.bundle
ld: warning: -multiply_defined is obsolete
Undefined symbols for architecture arm64:
"_nucomp_canonicalization", referenced from:
_Init_complex in complex.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [complex.bundle] Error 1
make failed, exit code 2
Gem files will remain installed in /Users/dpetruha/.gem/ruby/3.1.2/gems/mathn-0.1.0 for inspection.
Results logged to /Users/dpetruha/.gem/ruby/3.1.2/extensions/arm64-darwin-21/3.1.0/mathn-0.1.0/gem_make.out
Unfortunately the problem I was having before still isn't solved...here's the error I'm getting now: /Library/Ruby/Gems/1.8/gems/classifier-1.3.3/lib/classifier/extensions/vector.rb:16:in
sum': undefined method to_f' for []:Array (NoMethodError)
b = Classifier::Bayes.new "yes", "no"
b.train_yes "yes"
b.train_yes "ok"
b.train_no "no"
b.train_no "nope"
b.train_no "stop"
b.classify "ok"
Hi,
I noticed that 1.3.5 gem version doesn't include the tests, whereas they were present in the 1.3.4 version. Could you please continue to include them? For Debian packages, we try to use the gem as the base of source package, and it is useful for our quality assurance that tests are present directly there.
Thanks a lot
Cedric
I think that's a good idea to make a function that let us to save the current state of training and able to continue training with new content in the future.
I know that we can use marshaling, but it would be nice to make a method 'save'.
require 'mathn' breaks calculations:
(10 / 23).class
=> Fixnum
require 'classifier'
(10 / 23).class
=> Rational
This means e.g. date calculations will break:
Date.civil(year, month, day)
=> ArgumentError: invalid date
See:
http://blog.brightredglow.com/2008/1/17/evil-can-be-dangerous/
We included your gem to http://gems.sciruby.com/ and we use the source information from github for synchronization. Just log in at rubygems.org, edit the gem and add a source link. Thx!
There's a consistently unacceptable behavior of choosing the first category as default when it cannot determine the proper category.
Will confirm later with examples.
If anyone is interested in continuing work on this project, we've forked it over at https://github.com/jekyll/classifier-reborn. We just released a "v2.0.0" which contains many of the fixes addressed in the PR's and issues here.
I've noticed the LSI classifier fails on certain input strings. It appears to be related to repeating words, but I haven't figured out the exact pattern. As an example:
require 'classifier'
lsi = Classifier::LSI.new
strings = [
["This text deals with dogs. dogs.", "dog"],
["This text involves dogs too. Dogs! ", "dog"],
["Remind me to get milk on monday", "reminder"]
]
strings.each {|x| lsi.add_item x.first, x.last}
puts lsi.classify "Remind me to Remind"
results in
/Users/willstrinz/.rvm/gems/ruby-2.0.0-p353/gems/classifier-1.3.4/lib/classifier/lsi.rb:190:in `sort_by': comparison of Float with Float failed (ArgumentError)
from /Users/willstrinz/.rvm/gems/ruby-2.0.0-p353/gems/classifier-1.3.4/lib/classifier/lsi.rb:190:in `proximity_array_for_content'
from /Users/willstrinz/.rvm/gems/ruby-2.0.0-p353/gems/classifier-1.3.4/lib/classifier/lsi.rb:255:in `classify'
from lsi.rb:11:in `<main>'
but if I change the repeat of "Remind" it works fine"
puts lsi.classify "Remind me to zzRemind"
#=> remind
It looks like the vectorization process is somehow getting NaNs in the ContentNode creation process, which is the direct source of the error, but I haven't been able to track down what's causing that yet.
It'd be awesome if you could use that when available, because in my limited tests, it runs faster, even with rb-gsl installed.
Hi,
Could you please tag commits corrsponding to releases in the repositories? The last gem version is 1.3.5 but it is not easy to spot which commit it corresponds to. This would allow also automatic creation of tarballs from the tag, useful for packagers for distributions.
Thanks.
Cédric
I see a method to add a category from the bayesian filter... but is there a way to remove one?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.