joclement/master_thesis

Implementation of my master's thesis

master_thesis's Issues

Implement fingerprint from TU Graz

Related to #3.

Check for other datasets on GIL DC PD

Ask Moritz questions

Questions:

1. Is it a good idea to check the PDDC Analyzer code? If yes, where is it?
2. Which defect types should I classify? Should I differentiate normal particle from spiral particle? (Refer to what Lukas did)
3. Is there a class noise? Noise it mentioned in research from TU Graz.
4. On p. 82 it is written: "Fuer die Spitze an ... [HS] ist kein Zusammenhang ersichtlich". Wieso nicht? Die sind doch aehnlich.
5. Warum fehlt bei der Zielbereichsanalyse eine Messung beim Testen des Verfahrens? Ich sehe nur 4, bei den anderen beiden werden 5 benutzt. Ich glaube, dass die Messung mit der Klasse Spitze an Erde fehlt.
6. Should I look for other GIL DC PD datasets to have more data or see whether learning can be transferred?
7. Should I compare my methods to Lukas mean difference method?
8. Is using pictures with references okay?
9. Why should I be careful with the direct translation of DC and AC to Gleich- and Wechselspannung?
10. Is the misclassification of every class equally important? Probably not, right? The reason is that some defect types cause more harm than others? Should I respect this when training a classifier or just accuracy as the training metric?

Build dendogram of fingerprints

Remove 1st value to make access to TIMEDIFF easier

Thereby think whether the access to PD diff via the function pd_diff is correct. Currently thought of solutions:

• Have column PD_DIFF in measurement
• Drop column TIMEDIFF from measurement

Check for numerical problems in fingerprint skewness and kurtosis

The values are 0 for a lot of datasets. Investigate this.

Define which metrics I will use

• I will definitely use the metric accuracy.
• Which other metrics should I use?
• Find sources describing the metrics, if necessary

Good sources to answer this question:

• https://towardsdatascience.com/multi-class-metrics-made-simple-part-i-precision-and-recall-9250280bddc2
• https://towardsdatascience.com/multi-class-metrics-made-simple-part-ii-the-f1-score-ebe8b2c2ca1

Compare different ways of treating negative PD values

Currently all negative PD values are converted to 0. Try converting them to their corresponding positive value and see whether this will improve the accuracy.

Read "Data Mining - Practical Machine Learning Tools and Techniques"

Delete measurement data in LFS

Make sure that the data is deleted from the history as well.

Repeat ANN

Repeat SVM

• Find good paper
• Do tutorials
• Check what parameters there are
• What is a Gaussian RBF kernel?

Compare Lukas fingerprint to TU Graz

Implement fingerprint sequence classification

Improve data read functions

• Drop 2nd argument path from recursive read
• Maybe unify read_recursive and read

Check that Weibull fit looks correct

It was implemented from Lukas and TU Graz using this Matlab function. Check for some known values that my function is doing something similar.

Do confusion matrix and score cross validation calculation at the same time

See this function.

Add visualization plots

• Check out seaborn
• Make a plot describing the different time lengths of the data and different data point lengths
• Make nice 3d Plots describing the data Not done

Add polarity to own fingerprint

Add top k metric

See here.

Convert mat files to csv files

Moritz sent me .mat files. I have to convert these to .csv files in order to work with them. Here is a good source for doing that.

Understand Weibull distribution

• Read again about it in Lukas thesis
• Play around with different parameters and distributions to understand it
• Related to #3

Check again what classifier wants what scale

• MLP classifier
• k-NN classifier
• SVM classifier

Add DecisionTree for comparison

https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html

Look how to organize results better

• Look for frameworks how to organize results better
• Check if this will also influence what config for runs I have to use

Add rolling to tsfresh

Try 5 folds for cross validation

I think it shouldn't be possible currently with the amount of data.

Add file loading option for fingerprints as well

This will speed up the processing a lot.

Make number of epochs configurable for MLPs

Add config to organize results better

• Depends on #31
• Check if config library from bachelor thesis is suitable
• Ask Kai for a good config library

Use normalized Weibull for own fingerprint

Read measurements recursively

Implement initial k-NN DTW

Should I normalize the actual data or the fingerprints?

I think the actual data, but I should an expert to be sure.

Replace CLASS column by attrs field for DataFrames

Split each measurement file into samples of equal duration

Do this instead of #46.

Normalize sequence fingerprint

Building upon #37 I want to normalize the time series. I am not able to do that currently with the TimeSeriesMinMaxScaler from tslearn. So the goal of this issue is to either make it work or implement an adaption myself. The adaption is preferred, because I want to have a scaler which does a global scaling and not a scaling per time series.

Implement fingerprint from Lukas

Lukas develops a fingerprint from the same data I am using. It might be a good idea to compare my approach to his and start with implementing his approach, also for practice.

Do literature research on sequence classification

• Good start already done is here.

• Repeat LSTM

• Understand how RNNs work

• Understand how they or other networks can be used for sequence classification

Add PD_DIFF to have better calculation for fingerprint sequence

The skew and kurt become NaN currently sometimes.

• Add a `PD_DIFF`` column and use it where needed

Related to #37 .

Implement plots from Moritz

I want to replicate the plots from the data to get a better understanding of the data.

Repeat k-NN

• Find good paper describing different k-NN methods
• How should I choose k?

Write people for 2nd reviewer

Implement Bag of Words classifier

Add a classifier combining the bag of words approach with the MLP classifier over a particular duration e.g. 30 seconds.

Find out which defect types are more harmful

• Related to #7, #6

Get sparse implementation for 1D sequence classification

This is necessary to do that because otherwise too much data needs to be allocated crashing the program.

Check if using bins for Lukas correlation is better than not using bins

Lukuas bins the PD values into 100 bins. Is this a good idea before doing the correlation?

Check that using the images like this is okay

Add simple k-NN classifier

Read class label from measurement file

The class label needs to be added to the measurement. The class label is written in the filename.