ammarlodhi255/image-captioning-system-to-assist-the-blind

An image captioning system that is able to predict and speak out a caption of an image taken by visually impaired.

Image Captioning System to Assist The Blind

Table of Contents

• About
• Getting Started
• Screenshots
  • Dataset Split
  • Model Anatomy
  • Project Workflow
  • Results
  • Final Outcome
  • Additional Outputs

About

The goal of the project is to develop a system using deep learning techniques to assist visually impaired individuals in obtaining information by describing images taken by them. The system uses a CNN model and an NLP model to create a single image captioning system that takes image features as input and generates a text sequence describing the image.

Incorporated state-of-the-art pre-trained models, such as ResNet50, VGG16, and VGG19, for image feature extraction and LSTM and Bidirectional LSTM for text generation. Evaluated various models to determine the best-performing model with a BLEU-score of 0.61 and deployed it using Flask and pyttsx3 for web and text-to-speech functionality in the app.

Getting Started

These instructions will get you a copy of the project up and running on your local machine.

1. Clone the project repository from GitHub:

git clone https://github.com/ammarlodhi255/image-captioning-system-to-assist-the-blind.git

2. Navigate to the project directory:

cd image-captioning-system-to-assist-the-blind

3. Create a virtual environment for the project:

python3 -m venv env

4. Activate the virtual environment:

source env/bin/activate

5. Export the Flask app:

export FLASK_APP=app.py

6. Run the Flask app:

flask run

Screenshots

Dataset Split

Model Anatomy

Project Workflow

Results

Final Outcome

Additional Outputs