cleophasmogeni/equation-solver

Solving handwritten mathematical equations using OCR and CNN

Handwritten Equation solver

The application will be able to predict and solve handwritten mathematical equations from the given image. The system should be capable of solving expression involving arithmetic operations (addition, subtraction, multiplication, division) and solve equations of any degree (linear, quadratic, cubic and so on).

Key AI concepts used include OCR (Optical Character Recognition) and CNN (Convolutional Neural Networks). OCR is used to preprocess the image and segment characters, while CNN is used to predict the characters.

Download the repo, move to the folder and run

Installation

The repository can either be cloned or downloaded as a zip.

Run npm install inside the project directory to install all dependencies.

Execution

Both ReactJS and Flask have to be executed :

npm start
cd Equation-Solver
python -m flask run

Result

When image is written through sketchpad

When image is uploaded

Overview

• The Frontend part has been developed using ReactJS. Here the user enters the image either by uploading or by using the sketchpad. The image is encoded to base64 format and sent to the REST-API as a POST request.

• The REST-API has been implemented using Flask. The request data is decoded and saved as an image locally and this image is sent to the backend where the equation is predicted and solved.

• The Backend has been implemented using Python, Tensorflow and OpenCV. The backend can be seen as two separate modules : Equation Prediction and Equation Solver.

  • OpenCV is used to perform binarization and line and character segmentation. A Tensorflow model trained using the EMNIST (Extended MNIST) dataset is used to predict each of the segmented characters and the equation generated is passed as a string to the Equation Solver.

  • The Equation Solver solves the mathematical equation and passes it back to the Frontend where it can be viewed.

Character Segmentation

The major steps include : Noise Removal, Binarization, Thresholding and Image Segmentation.

The Binarized image and the segmented images can be viewed below :

Solving the Equation

After each of the character in the image is detected, the string containing the equation is passed to this final module which solves the equation or mathematical expression.

The equation can be of two types :

• A mathematical string such as ‘5+3’ or ‘66x3+2’ (String that is input to this module is of this format). This string can either be evaluated using a custom-built function or the eval( ) function in python.

• A mathematical equation of any degree. The string ‘X2+5=0’ is interpreted as X**2 + 5 since the 2 appears after the variable. Whereas 2X+5=0 is interpreted as 2*X + 5 = 0. Since prediction of even a single character leads to incorrect results/failure, simple replacements are performed on the given string to increase accuracy. These include Z -> 2, G -> 6, B -> 8 and D -> 0. The equation is solved using the SymPy library, which is a python library for symbolic computation.

The 2 types of equations are distinguished by checking if the equation contains ‘=‘. If the equation contains ‘=‘, it is interpreted as the 2nd type, otherwise it is interpreted as the 1st type.

Links

Link to the presentation

References

Introduction to CNN Keras

Character Segmentation

ReactJS - Getting Started

React - P5 Wrapper

Flask Restful

React bootstrap