In this case study, we address the detection of malicious URLs as a multi-class classification problem. We classify raw URLs into different classes such as benign or safe URL, phishing URL, malware URL, or defacement URL.
Machine learning algorithms require numeric inputs, so we will create numeric lexical features from the input URLs. The input to the machine learning algorithms will be these numeric lexical features rather than the actual raw URLs. If you are unfamiliar with lexical features, you can refer to this discussion on StackOverflow.
In this case study, we will be using three well-known boosting machine learning classifiers: XGBoost, Light GBM, and Gradient Boosting Machines.
For training and testing machine learning algorithms, we have used a substantial dataset of 651,191 URLs, comprising:
- 428,103 benign or safe URLs
- 96,457 defacement URLs
- 94,111 phishing URLs
- 32,520 malware URLs
The dataset is curated from five different sources, which are combined and pre-processed to retain only URLs and their class type.
Sources:
- URL dataset (ISCX-URL-2016)
- Malware domain black list dataset
- Faizan Git Repo
- Phishtank dataset
- PhishStorm dataset
The final pre-processed dataset is available on Kaggle: Malicious URL Dataset
Detection of Malicious URLs Using Lexical Features | Machine Learning, Cybersecurity, NLP, Boosting, Sklearn, Pandas, Regex
Developed a multi-class classification system to detect malicious URLs using lexical features and boosting algorithms (XGBoost, Light GBM, Gradient Boosting Machines), leveraging Pandas and Sklearn for data processing and model training.
Dataset: Malicious URL Dataset
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent