roujack/mathai

一个拍照做题程序。输入一张包含数学计算题的图片，输出识别出的数学计算式以及计算结果。This is a mathematic expression recognition project.

mathAI

一个拍照做题程序。输入一张包含数学计算题的图片，输出识别出的数学计算式以及计算结果。 请查看系统文档说明来运行程序。注意，这是一个半开源的项目，目前上传的版本只能处理简单的一维加减乘除算术表达式（如果想要识别更加复杂的表达式，可以参考数学公式识别的论文）。可以参考的代码是前面字符识别部分以及整个算法处理框架。

整个程序使用python实现，具体处理流程包括了图像预处理、字符识别、数学公式识别、数学公式语义理解、结果输出。

本程序使用opencv对输入的图像进行预处理，并将字符裁剪出来再归一化成固定大小的矩阵。我在TensorFlow上实现了一个lenet5 的卷积神经网络用来识别数学字符，训练使用CHROME数据集。对于数学公式的识别，主要是将识别出的独立的字符组织成计算机能够 理解的数学公式（这里的数学公式就是纯字符的可求解的数学计算题）。大概的方法是使用编译原理的算符优先法和递归下降法进行实现。 然后根据属性文法的值传递**，将数学公式的值计算出来。最后使用python的matlibplot库把计算过程和答案打印出来。

优点：这是一整套拍照做题的算法框架，同时能够处理多种多样的计算题，目前市面上还没有看到实现。OCR技术如此成熟的今天字符识别 已经不算有挑战的东西了。 缺点：字符空间关系判断只用了人类启发式规则，图像预处理不够鲁棒，数学公式的结构识别算法不够完美（可以考虑使用二维文法来做）。 系统还有很大的提升空间。

mathai's Issues

module 'parser' has no attribute 'characters_to_nodes

windows 平台，test.py 跑自带的图片验证，报出一个错误：AttributeError: module 'parser' has no attribute 'characters_to_nodes'
有人知道怎么解么？

问题

你好，对印刷体二次根式的图片识别，在运行程序时会报错：AttributeError: module 'tools' has no attribute 'extract_img'，请问您有什么好的建议吗

程序运行异常

您好！
我对这个项目很感兴趣，采用文档中的接口模式运行，结果有如下报错：
INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_model_dir': './my_cnn_model_config5', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x1c2d075748>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
Traceback (most recent call last):
File "/Users/wuziyan/tensorflow/mathAI-master/系统代码(code)/test.py", line 3, in
save_filename = solver.solve('./testImgs/easy +/1.jpg')
File "/Users/wuziyan/tensorflow/mathAI-master/系统代码(code)/solver/init.py", line 21, in solve
symbols = binary_img_segment(binary_img, original_img)
File "/Users/wuziyan/tensorflow/mathAI-master/系统代码(code)/tools/img_preprocess.py", line 68, in binary_img_segment
img, contours, hierarchy = cv2.findContours(binary_img, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)#cv2.RETR_TREE
ValueError: not enough values to unpack (expected 3, got 2)

Process finished with exit code 1

请教应如何配置才能正确运行？

多个算术题识别

请问该怎么实现多个算术题在同一张图片上都可以识别出来呢

Program KeyError

您好，当我输入例子，例如example/2.jpg，example/5.jpg，和hard下的例子，都会报一个status错误。
` 68 # print(parser_tree)
69 set_forward_step(0)
---> 70 post_order(parser_tree)
71 y_start = 0.9
72 y_stride = 0.2

~/Desktop/dino/mathAI/code/calculator/init.py in post_order(node)
275 elif node['type'] == NODE_TYPE['e']:
276 # print('post_order e')
--> 277 t = post_order(child[0])
278 node['status'] = child[0]['status']
279 node['value'] = child[0]['value']

~/Desktop/dino/mathAI/code/calculator/init.py in post_order(node)
199 # print('post_order t',child[0])
200 f = post_order(child[0])
--> 201 node['status'] = child[0]['status']
202 node['value'] = child[0]['value']
203 latex_str = f

KeyError: 'status'
`

TypeError: data must be either a numpy array or pandas DataFrame if pandas is installed; got dict

环境都已经建好

因为没有您说的opencv版本号
所以用pip install opencv-python==3.4.17.63下载
但在run main.py时出现以下的错误

TypeError: data must be either a numpy array or pandas DataFrame if pandas is installed; got dict

错误是发生在main.py的
for i,p in enumerate(predictions):

estimator.py的
features, input_hooks = self._get_features_from_input_fn(input_fn, ModeKeys.PREDICT)

test.py 运行，在执行 img_preprocess.py 中报错

错误描述：

在 pycharm 中运行 test.py ，报错 ，信息如下：

Traceback (most recent call last):
File "F:/git_code/ai/mathAI/系统代码(code)/test.py", line 3, in
save_filename = solver.solve('./testImgs/easy +/1.jpg')
File "F:\git_code\ai\mathAI\系统代码(code)\solver_init_.py", line 20, in solve
symbols = binary_img_segment(binary_img, original_img)
File "F:\git_code\ai\mathAI\系统代码(code)\tools\img_preprocess.py", line 66, in binary_img_segment
img, contours, hierarchy = cv2.findContours(binary_img, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)#cv2.RETR_TREE
ValueError: not enough values to unpack (expected 3, got 2)

我的运行环境
py 3.7.3
cv 4.1.0
tensorflow 2.0.0beta1

是不是因为环境不一样导致的呢
请您帮忙看一下

问题

你好，我在求数的“平方根”时会报如下错误：
Traceback (most recent call last):
File "E:/AI_exercise/mathAI/系统代码(code)/test.py", line 3, in
save_filename = solver.solve('./testImgs/easy sqrt/61.jpg')
File "E:\AI_exercise\mathAI\系统代码(code)\solver_init_.py", line 70, in solve
post_order(parser_tree)
File "E:\AI_exercise\mathAI\系统代码(code)\calculator_init_.py", line 278, in post_order
t = post_order(child[0])
File "E:\AI_exercise\mathAI\系统代码(code)\calculator_init_.py", line 202, in post_order
node['status'] = child[0]['status']
KeyError: 'status'
请问时什么原因，应该怎么修改？

无法打开dataset1中的jpg文件

请教一下：我想用dataset1文件夹里面的图片训练一个识别器，请问这些图片为什么打不开

while clicking upload button getting the below error

[2020-07-01 14:43:17,926] ERROR in app: Exception on / [POST]
Traceback (most recent call last):
File "/Users/jeethusingh/PycharmProjects/aiMath/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/Users/jeethusingh/PycharmProjects/aiMath/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Users/jeethusingh/PycharmProjects/aiMath/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/Users/jeethusingh/PycharmProjects/aiMath/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/Users/jeethusingh/PycharmProjects/aiMath/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/Users/jeethusingh/PycharmProjects/aiMath/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functionsrule.endpoint
File "/Users/jeethusingh/PycharmProjects/aiMath/welcome.py", line 41, in upload_file
result_file = solver.solve(save_file_path)
File "/Users/jeethusingh/PycharmProjects/aiMath/solver/init.py", line 73, in solve
if parser_tree['status'] == STATUS['solved']:
KeyError: 'status'
127.0.0.1 - - [01/Jul/2020 14:43:17] "POST / HTTP/1.1" 500 -

小白请教一个问题，大佬勿喷...

我刚刚接触python，下载了你的程序，为什么test在运行的时候会报错呢

配置问题

按照文档配置好之后，127.0.0.1:5000 能进入，但是点击upload的时候显示内部服务器错误
错误信息如下：
WARNING: Logging before flag parsing goes to stderr.
W0704 22:22:49.066677 14804 deprecation_wrapper.py:119] From C:\Users\Nicho\Desktop\master\mathAI-master\系统代码(code)\tools\cnn_model.py:9: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

W0704 22:22:49.066677 14804 deprecation_wrapper.py:119] From C:\Users\Nicho\Desktop\master\mathAI-master\系统代码(code)\tools\cnn_model.py:9: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.

I0704 22:22:49.067677 14804 estimator.py:1790] Using default config.
I0704 22:22:49.067677 14804 estimator.py:209] Using config: {'_model_dir': './my_cnn_model_config5', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x000002574B928BA8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}

杠一下。

iOS上有一个手写的计算器。MyScript Calculator。
https://apps.apple.com/us/app/myscript-calculator/id1304488725
README.md,要不要改一下。

wanna contact with the author

我想要这个项目的部分源码 作者可以联系一下我吗

为什么会找不到这个模块的属性，明明已经引入？

Traceback (most recent call last):
File "e:\python_project\mathai-master\mainai\lib\site-packages\flask\app.py", line 2311, in wsgi_app
response = self.full_dispatch_request()
File "e:\python_project\mathai-master\mainai\lib\site-packages\flask\app.py", line 1834, in full_dispatch_request
rv = self.handle_user_exception(e)
File "e:\python_project\mathai-master\mainai\lib\site-packages\flask\app.py", line 1737, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "e:\python_project\mathai-master\mainai\lib\site-packages\flask_compat.py", line 36, in reraise
raise value
File "e:\python_project\mathai-master\mainai\lib\site-packages\flask\app.py", line 1832, in full_dispatch_request
rv = self.dispatch_request()
File "e:\python_project\mathai-master\mainai\lib\site-packages\flask\app.py", line 1818, in dispatch_request
return self.view_functionsrule.endpoint
File "E:\python_project\mathAI-master\mainai\welcome.py", line 49, in upload_file
result_file = solver.solve(save_file_path)
File "E:\python_project\mathAI-master\mainai\solver_init_.py", line 68, in solve
node_list = parser.characters_to_nodes(characters)
AttributeError: module 'parser' has no attribute 'characters_to_nodes'

提交了很多临时文件

.DS_Store
.idea

有一个疑问想请教

请问你是采用什么方法进行图像分割的，比如等于号、根号这种你是如何处理的？期待你的回答，谢谢

自己拍摄的照片无法计算？

请问用项目自己提供的图片可以计算，换成自己手机拍摄的照片无法计算是怎么回事

无法运行简单除法测试

能运行加减测试 但是除法不能测试
修改main.py
将第22行original_img, binary_img = read_img_and_convert_to_binary('./testImgs/easy div/25.jpg')后无法通过测试
错误提示
(cv3) G:\mathAI\系统代码(code)>python main.py C:\Users\Administrator\.conda\envs\cv3\lib\site-packages\tensorflow\python\framework\dtypes.py:458: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) C:\Users\Administrator\.conda\envs\cv3\lib\site-packages\tensorflow\python\framework\dtypes.py:459: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) C:\Users\Administrator\.conda\envs\cv3\lib\site-packages\tensorflow\python\framework\dtypes.py:460: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) C:\Users\Administrator\.conda\envs\cv3\lib\site-packages\tensorflow\python\framework\dtypes.py:461: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) C:\Users\Administrator\.conda\envs\cv3\lib\site-packages\tensorflow\python\framework\dtypes.py:462: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) C:\Users\Administrator\.conda\envs\cv3\lib\site-packages\tensorflow\python\framework\dtypes.py:465: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) INFO:tensorflow:Using default config. INFO:tensorflow:Using config: {'_model_dir': './my_cnn_model_config5', '_tf_random_seed': 1, '_save_summary_steps': 100, '_save_checkpoints_secs': 600, '_save_checkpoints_steps': None, '_session_config': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000} 2020-07-20 17:55:06.593325: W c:\l\tensorflow_1501918863922\work\tensorflow-1.2.1\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE instructions, but these are available on your machine and could speed up CPU computations. 2020-07-20 17:55:06.604402: W c:\l\tensorflow_1501918863922\work\tensorflow-1.2.1\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE2 instructions, but these are available on your machine and could speed up CPU computations. 2020-07-20 17:55:06.614630: W c:\l\tensorflow_1501918863922\work\tensorflow-1.2.1\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations. 2020-07-20 17:55:06.634000: W c:\l\tensorflow_1501918863922\work\tensorflow-1.2.1\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. 2020-07-20 17:55:06.645301: W c:\l\tensorflow_1501918863922\work\tensorflow-1.2.1\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. 2020-07-20 17:55:06.656658: W c:\l\tensorflow_1501918863922\work\tensorflow-1.2.1\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. 2020-07-20 17:55:06.684921: W c:\l\tensorflow_1501918863922\work\tensorflow-1.2.1\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. 2020-07-20 17:55:06.697061: W c:\l\tensorflow_1501918863922\work\tensorflow-1.2.1\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. INFO:tensorflow:Restoring parameters from ./my_cnn_model_config5\model.ckpt-16000 <generator object Estimator.predict at 0x000001F0EC80BD58> INFO:tensorflow:Restoring parameters from ./my_cnn_model_config5\model.ckpt-16000 排序前的字符列表 [{'location': (54, 42, 84, 105), 'candidates': [{'symbol': '4', 'probability': 0.9998342}]}, {'location': (197, 47, 18, 23), 'candidates': [{'symbol': 'times', 'probability': 0.98560405}]}, {'location': (162, 84, 81, 13), 'candidates': [{'symbol': '-', 'probability': 0.9998073}]}, {'location': (198, 105, 23, 26), 'candidates': [{'symbol': 'times', 'probability': 0.9965108}]}, {'location': (260, 56, 53, 75), 'candidates': [{'symbol': '2', 'probability': 0.99996924}]}] 排序后的字符序列 [[(54, 42, 84, 105), [{'symbol': '4', 'probability': 0.9998342}]], [(197, 47, 18, 23), [{'symbol': 'times', 'probability': 0.98560405}]], [(162, 84, 81, 13), [{'symbol': '-', 'probability': 0.9998073}]], [(198, 105, 23, 26), [{'symbol': 'times', 'probability': 0.9965108}]], [(260, 56, 53, 75), [{'symbol': '2', 'probability': 0.99996924}]]] 识别出的token [{'location': [54, 42, 84, 105], 'token_string': '4', 'token_type': 1}, {'location': [197, 47, 18, 23], 'token_string': 'times', 'token_type': 0}, {'location': [162, 84, 81, 13], 'token_string': 'f', 'token_type': 0}, {'location': [198, 105, 23, 26], 'token_string': 'times', 'token_type': 0}, {'location': [260, 56, 53, 75], 'token_string': '2', 'token_type': 1}] {'structure': [{'structure': [{'structure': 4, 'type': 2, 'location': [54, 42, 84, 105]}, {'structure': ['times', {'structure': 'f', 'type': 15, 'location': [162, 84, 81, 13]}, {'structure': ['times', {'structure': 2, 'type': 2, 'location': [260, 56, 53, 75]}], 'type': 5}], 'type': 5}], 'type': 6}], 'type': 8} Traceback (most recent call last): File "main.py", line 73, in <module> latex_str = post_order(parser_tree) File "G:\mathAI\系统代码(code)\calculator\__init__.py", line 278, in post_order t = post_order(child[0]) File "G:\mathAI\系统代码(code)\calculator\__init__.py", line 206, in post_order t_pi = post_order(child[1]) File "G:\mathAI\系统代码(code)\calculator\__init__.py", line 170, in post_order node['status'] = max(child[1]['status'],child[2]['status']) KeyError: 'status'

请问未开源的部分有推荐的论文吗

作者你好，我是一名非计算机专业的本科生，我已跑通你的代码，并把lenet换成了其他识别率更高的网络，。如果我想实现你未开源的部分，你有推荐的论文吗？我比较菜，一直没找到合适的论文。

程序异常

您好我对您的项目很感兴趣 也在学习您的项目 为什么我下载的项目运行减法运算会抛错

分数的辨别或运算

请问如何才能正确将某图片中的分数值 5/6 + 3/7 显示出来或计算结果?
需要注意那些要项?

.py函数调用出错

试运行，发现parser的characters_to_nodes函数无法调用到，也无法按照from xx import xx，抑或import xx.xx，删除该文件夹并另存__init__为.py仍旧没解决问题，不只是为何？

Math ai

import re

def math_ai(question):
# Use regular expressions to extract the numbers and operator from the question
numbers = re.findall(r'\d+', question)
operator = re.search(r'[+-*/]', question)

if not operator or not numbers:
    return "I'm sorry, I didn't understand the question."

# Convert the numbers to integers
numbers = [int(x) for x in numbers]

# Perform the calculation
if operator.group() == '+':
    result = sum(numbers)
elif operator.group() == '-':
    result = numbers[0] - sum(numbers[1:])
elif operator.group() == '*':
    result = 1
    for x in numbers:
        result *= x
elif operator.group() == '/':
    result = numbers[0]
    for x in numbers[1:]:
        result /= x

return result

print(math_ai("What is 5 + 3?"))

Output: 8

Mathpix

这个是个不错的app，可以借鉴参考