rishysp/visual-question-answering

Visual Question Answering

The aim is to develop an“AI Pathologist”, where an AI agent is provided a pathology image and can give the correct response to a question that is provided by the user in that context. The questions can be related to different tasks of computer vision such as image classification - is the image provided of a white blood cell or a red blood cell? Or object detection - where is increased fibrosis? For developing this AI agent we will be using the PathVQA and VQA-RAD dataset.

Table of Contents

1. About The Project
   • Built With
2. Usage

About The Project

In context of our problem statement, the clinicians' confidence in interpreting complex medical images could also be enhanced by a “second opinion” provided by an automated system. In addition, patients may be interested in the morphology/physiology and disease-status of anatomical structures around a lesion that has been well characterized by their healthcare providers – and they may not necessarily be willing to pay significant amounts for a separate office- or hospital visit just to address such questions. Although patients often turn to search engines (e.g. Google) to disambiguate complex terms or obtain answers to confusing aspects of a medical image, results from search engines may be nonspecific, erroneous and misleading, or overwhelming in terms of the volume of information.

Built With

• Pytorch
• Google Visual Transformer
• BERT

Usage

The PACMAN problem environment helps us in understanding the Reinforcement algorithm concepts. This game environment can be related to real world scenarios and the model built can be used to solve using the deep reinforcement algorithms. Eg: This model can be used to navigate people past dynamic obstacles in an optimistic path.

Output