rohanphadnis/alphamine

Data-mining framework for compiling a machine learning dataset.

---

About

AlphaMine was created in October 2022 as a submission for HackGT 9. It won 1st place for the Cheat Code Hacks category.

As part of the hackathon submission, we had to create a video describing and demonstrating the project. It can be found here: AlphaMine Video

---

Problem Statement

"Data is the new Oil." - Clive Humby
Machine Learning has gained traction in recent years. However, training and validating machine learning models requires the availability of labeled data at scale. The new demands of data ensure that data collection will always be a consistent struggle.

Enter the AlphaMine. Our framework allows users to simply enter keywords and our alpha software will automatically generate a dataset, cleaned and prepared for training.

---

Text Mining

Text Mining can be extremely useful for Natural Language Processing applications. With easy dataset generation, sophisticated models like GPT-3 will become more accessible to a wider audience.

• Parameters
  • Number of Classes
  • Number of Samples (Webpages Traversed)

---

Image Mining

AlphaMine also supports Computer Vision dataset generation.

• Parameters
  • Number of Classes
  • Number of Samples (Images Traversed)
  • Size - A Tuple. Optional parameter, default will not resize the images
  • Number of Grayscale. Optional parameter, defaulted to no Grayscale
  • Boundary boxes, generated from yolo model on 80 common classes. Box locations are stored in meta_images.json

---

Installation Instructions

Clone the repository using git clone https://github.com/Shikhar2929/HackGT

---

Command-Line Usage

Create a classes.txt file listing all the different classes that you want, separated by the newline character. Simply move into the code directory and then run the following script
cd HackGT/AlphaMine/src/alpha_mine
python3 mine_data.py

1. When prompted to enter classes, simply enter the path of the classes.txt
2. When prompted to enter the type of data, enter "text" or "image"
3. Follow the input instructions for defining additional parameters.

---

Python Usage

import alpha_mine
mine()