philipp-sc/llm-fraud-detection

Robust semi-supervised spam detection using Rust native NLP pipelines.

llm-fraud-detection's Issues

Use a Neural Network instead of the Random Forest Regressor

Replacing the Random Forest Regressor with a better model (Neural Network) should improve the performance.

(Random Forests can not extrapolate, that means it has difficulties to generalize and handle unseen data.)

Re-generate topics and re-train fraud detection

• Re-generate topics and re-train fraud detection with bigger dataset of governance proposals.

governance_proposal_spam_ham.csv 
---------------
count spam: 172
count ham: 2551

Note: This will be great to reduce false positives, since the model has not yet seen many ham (and spam) data for governance proposals.

Note: consider reducing the ham dataset by filtering some of the rejected proposals with high votes against. To make sure not to train likely spam as ham.

improve inference speed

• BERT
• KNN Regressor

gov prpsl spam_likelihood dataset improved

re-train using the improved crypto governance proposal dataset.

• now considering the final state (rejected, passed, failed) as well
• more proposals added

feature selection

#1

access data quality: inspect entries where the prediction != label

add phishing website dataset

add K-Means Clustering step before KNN

to reduce the KNN model size.

Model Evaluation

Instead of testing the model performance on the same data it was trained on, generate a training and test dataset.

90% training data
10% test data

make sure to sample spam and ham.

use a Llama 2 based model to generate the embeddings

https://huggingface.co/AlekseyKorshuk/vicuna-7b

add phishing url detection prediction

inspiration / dataset: https://github.com/ebubekirbbr/phishing_url_detection

add feature url-fraud-likelihood

KNN and text embedding

worth a try

add DAO governance proposals dataset

#1

experiment with embeddings

right now one text document is embedded as whole, experiment with partitioning the text first (by sentences, paragraphs, etc)

Improve engineered features for even better accuracy

The current engineered 'hard-coded' features are very basic, while they provide useful information there is room for improvement.

src/build/feature_engineering/mod.rs

Instead of hard-coded conditions, create / augment with Bag Of Words vector that is derived from the training dataset.

E.g then using a frequency encoding of common words that often occur within spam but not in ham and vice versa.
Resulting in two vectors that together contain the most important/common words for/against a spam classification.

refactor and cleanup code

lots of testing resulted in ugly code