sztheory/tf-idf

tf-idf elixir

#Tfidf An Elixir implementation of tf-idf

Based on the blog post by Steven Loria

##What is tf-idf?

tf–idf, short for term frequency–inverse document frequency, is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus. It is often used as a weighting factor in information retrieval and text mining.

tf-idf on Wikipedia

Installation

defp deps do
  [{:tfidf, "~> 0.1.0"}]
end

Usage

Tfidf.calculate(word, text, corpus, tokenize_fn \\ &tokenize(&1))

Calculates the tf-idf for a given word within a text and a corpus (List) of texts.

iex> Tfidf.calculate("dog", "nice dog dog", ["dog hat", "dog", "cat mat", "duck"])
0.19178804830118723

An optional tokenizer function can be passed as the last argument to replace the default tokenizer:

iex> Tfidf.calculate("dog", "nice,dog,dog", ["dog,hat", "dog", "cat,mat", "duck"], &String.split(&1, ","))
0.19178804830118723

=====

Tfidf.calculate(word, tokenized_text, corpus)

Calculates the tf-idf for a given word within a pre-tokenized list and a corpus comprised of pre-tokenized lists.

iex> Tfidf.calculate("dog", ["nice", "dog", "dog"], [["dog", "hat"], ["dog"], ["cat", "mat"], ["duck"]])
0.19178804830118723

=====

Tfidf.calculate_all(text, corpus, tokenize_fn \\ &tokenize(&1))

Calculates the tf-idf for all words in a given text, returns a list of {word, score} tuples.

iex> Tfidf.calculate_all("nice dog", ["dog hat", "dog", "cat mat", "duck"])
[{"nice", 0.6931471805599453}, {"dog", 0.14384103622589042}]

As with Tfidf.calculate/4 an optional tokenizer function can be passed as the last argument. This will be used in place of the default tokenizer.

iex> Tfidf.calculate_all("nice,dog", ["dog,hat", "dog", "cat,mat", "duck"], &String.split(&1, ","))
[{"nice", 0.6931471805599453}, {"dog", 0.14384103622589042}]