rotrixx/sptm

Software Project

Description

The goal of this project is to classify books in a genre using a maschine learning algorithm.

Software

Dependencies

• SciKit-Learn
• Pandas
• SpaCy
• NLTK

Features

• Words per sentence
• Number of Sentences
• Relative frequency of nouns, adjectives and verbs
• Number of Symbols
• Degree of correspondence with created wordbooks of each genre

sptm's Issues

Implement Ensemble

The numbers in the matrix are too damn high

output with missing feature to find most important feature

split nSym into sperate features(n$,n§etc.)

improve wordbooks

-make dictionary add weight based on in how many other wordbooks the word is

read multiple categories

Make current code functional

write everything in functions

Extend stopwordlist

Improve Ensemble

Dicts split words.

Dictionaries split words. maybe caused by linebreak

Dict unwanted words

bsp "nur "schon" in dictAG

Theory: dictLU is too important, pls fix

Handel Anomalies

Dict unicode may cause problems

add title to blurb

Comma and symbols in percentageOfNounsVerbsAdjectivesAndNumberCommasSymbols

Function returns "wrong" numbers

Add file for TrainingDataFrame to look at(extra parameter)

lemmatize

add spacy lemmatize for dicts

improving performance

improve performance by

• making reading test/train data in a function

Problem with lemmatization

Textblob-de dosent support lemmatization yet or its not working correctly

• use textblob (english) and translate between them (may be performance heavy)
  -write own lemmanizer with big databse (may be also performance heavy)

add author to text

add author as "one word"

Fixing Radius Neighbor

The confusion matrix for Radius Neighbor Classification shows that almost all the books are classified as Literatur/Unterhaltung.
The way this classifier works: It looks at a certain radius around the specific book, that is to be classified. Now when calling the function we have set a fixed radius and we set a "default class" (L/U), which the book is assigned to, in case no other neighbor is found inside of that radius.

PROBLEM:

EITHER: The radius is wayyy too small, so that every time we check for neighbors, no other neighbors are found and we just assign the default class "Literatur und Unterhaltung".

OR: The radius is wayyy too big, so that every time we check for neighbors, hundreds or thousands of neighbors of other classes are inside of that radius. "L/U" being the most represented class in the training set, would always be the majority inside of that big radius.

ERGO: Make the fucking radius bigger or smaller and check if we get other results :)

Maybe I find the motivation to download all of that shit and try it myself. Maybe, but not today :)

In case we already tried that... ignore everything above.