testdotai/classifier-builder

test.ai Classifier Builder

This is the classifier and training/running scripts for the test.ai elements classifier. This is used to build a model for the testai appium-classifier-plugin(https://github.com/testdotai/appium-classifier-plugin).

System setup

You will need to install Tensorflow(www.tensorflow.org), Google's open source AI framework.

pip install tensorflow

Build Classifier

To build the classifier from the provided images run the following.

python retrain.py --image_dir training_images/ --output_graph output/saved_model.pb --output_labels output/saved_model.pbtxt --how_many_training_steps 4000 --learning_rate 0.30 --testing_percentage 25 --validation_percentage 25 --eval_step_interval 50 --train_batch_size 2000 --test_batch_size -1 --validation_batch_size -1 --bottleneck_dir /tmp/bottleneck

Run Classifier

To use the classifier to classify images, run the script under sample_run/ directory.

python run_model.py --image cart.png

classifier-builder's Issues

How to add my own image and train it? How to use it in a test?

Failed to build on macOS 10.14.5 when Final test accuracy = 67.2%

INFO:tensorflow:2019-05-27 12:54:22.651062: Step 3999: Train accuracy = 77.1%
I0527 12:54:22.651128 4364408256 retrain.py:1099] 2019-05-27 12:54:22.651062: Step 3999: Train accuracy = 77.1%
INFO:tensorflow:2019-05-27 12:54:22.651270: Step 3999: Cross entropy = 1.030875
I0527 12:54:22.651283 4364408256 retrain.py:1101] 2019-05-27 12:54:22.651270: Step 3999: Cross entropy = 1.030875
INFO:tensorflow:2019-05-27 12:54:51.107501: Step 3999: Validation accuracy = 66.9% (N=38485)
I0527 12:54:51.107573 4364408256 retrain.py:1120] 2019-05-27 12:54:51.107501: Step 3999: Validation accuracy = 66.9% (N=38485)
2019-05-27 12:55:26.988836: W tensorflow/core/graph/graph_constructor.cc:1272] Importing a graph with a lower producer version 26 into an existing graph with producer version 27. Shape inference will have run different parts of the graph with different producer versions.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
I0527 12:55:29.890471 4364408256 saver.py:1483] Saver not created because there are no variables in the graph to restore
WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
W0527 12:55:30.225637 4364408256 deprecation.py:323] From /usr/local/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from /tmp/_retrain_checkpoint
I0527 12:55:30.231771 4364408256 saver.py:1270] Restoring parameters from /tmp/_retrain_checkpoint
INFO:tensorflow:Final test accuracy = 67.2% (N=38201)
I0527 12:55:31.543164 4364408256 retrain.py:856] Final test accuracy = 67.2% (N=38201)
INFO:tensorflow:Save final result to : output/saved_model.pb
I0527 12:55:31.774097 4364408256 retrain.py:1148] Save final result to : output/saved_model.pb
2019-05-27 12:55:31.840187: W tensorflow/core/graph/graph_constructor.cc:1272] Importing a graph with a lower producer version 26 into an existing graph with producer version 27. Shape inference will have run different parts of the graph with different producer versions.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
I0527 12:55:33.283593 4364408256 saver.py:1483] Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Restoring parameters from /tmp/_retrain_checkpoint
I0527 12:55:33.602644 4364408256 saver.py:1270] Restoring parameters from /tmp/_retrain_checkpoint
WARNING:tensorflow:From retrain.py:906: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.convert_variables_to_constants
W0527 12:55:33.850260 4364408256 deprecation.py:323] From retrain.py:906: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.convert_variables_to_constants
WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/python/framework/graph_util_impl.py:245: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
W0527 12:55:33.850518 4364408256 deprecation.py:323] From /usr/local/lib/python3.6/site-packages/tensorflow/python/framework/graph_util_impl.py:245: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
INFO:tensorflow:Froze 137 variables.
I0527 12:55:34.327029 4364408256 graph_util_impl.py:268] Froze 137 variables.
INFO:tensorflow:Converted 137 variables to const ops.
I0527 12:55:34.352867 4364408256 graph_util_impl.py:301] Converted 137 variables to const ops.
Traceback (most recent call last):
  File "retrain.py", line 1333, in <module>
    tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "retrain.py", line 1151, in main
    save_graph_to_file(graph, FLAGS.output_graph, module_spec, class_count)
  File "retrain.py", line 909, in save_graph_to_file
    f.write(output_graph_def.SerializeToString())
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/lib/io/file_io.py", line 108, in write
    self._prewrite_check()
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/lib/io/file_io.py", line 94, in _prewrite_check
    compat.as_bytes(self.__name), compat.as_bytes(self.__mode), status)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: output/saved_model.pb; No such file or directory

Running the retrain.py script resulted in an error

Running
classifier-builder ericsavoie$ python retrain.py --image_dir training_images/ --output_graph output/saved_model.pb --output_labels output/saved_model.pbtxt --how_many_training_steps 4000 --learning_rate 0.30 --testing_percentage 25 --validation_percentage 25 --eval_step_interval 50 --train_batch_size 2000 --test_batch_size -1 --validation_batch_size -1 --bottleneck_dir /tmp/bottleneck

At the very end it produced an error:

INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Restoring parameters from /tmp/_retrain_checkpoint INFO:tensorflow:Final test accuracy = 67.0% (N=38201) INFO:tensorflow:Save final result to : output/saved_model.pb 2018-11-05 13:19:31.581472: W tensorflow/core/graph/graph_constructor.cc:1263] Importing a graph with a lower producer version 26 into an existing graph with producer version 27. Shape inference will have run different parts of the graph with different producer versions. INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Restoring parameters from /tmp/_retrain_checkpoint INFO:tensorflow:Froze 137 variables. INFO:tensorflow:Converted 137 variables to const ops. Traceback (most recent call last): File "retrain.py", line 1333, in <module> tf.app.run(main=main, argv=[sys.argv[0]] + unparsed) File "/usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run _sys.exit(main(argv)) File "retrain.py", line 1151, in main save_graph_to_file(graph, FLAGS.output_graph, module_spec, class_count) File "retrain.py", line 909, in save_graph_to_file f.write(output_graph_def.SerializeToString()) File "/usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/lib/io/file_io.py", line 108, in write self._prewrite_check() File "/usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/lib/io/file_io.py", line 94, in _prewrite_check compat.as_bytes(self.__name), compat.as_bytes(self.__mode), status) File "/usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/framework/errors_impl.py", line 526, in __exit__ c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.NotFoundError: output/saved_model.pb; No such file or directory

running run_model.py get error

AttributeError: module 'tensorflow' has no attribute 'GrGraphaph'

deprecated message running retrain.py

Execute python3 retrain.py --image_dir training_images/ --output_graph output/saved_model.pb --output_labels output/saved_model.pbtxt --how_many_training_steps 4000 --learning_rate 0.30 --testing_percentage 25 --validation_percentage 25 --eval_step_interval 50 --train_batch_size 2000 --test_batch_size -1 --validation_batch_size -1 --bottleneck_dir /tmp/bottleneck

Messages:

WARNING:tensorflow:From retrain.py:1333: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

WARNING:tensorflow:From retrain.py:989: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

W1004 12:24:52.106491 4313146816 deprecation_wrapper.py:119] From retrain.py:989: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

WARNING:tensorflow:From retrain.py:989: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.

W1004 12:24:52.106732 4313146816 deprecation_wrapper.py:119] From retrain.py:989: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.

WARNING:tensorflow:From retrain.py:914: The name tf.gfile.Exists is deprecated. Please use tf.io.gfile.exists instead.

W1004 12:24:52.106897 4313146816 deprecation_wrapper.py:119] From retrain.py:914: The name tf.gfile.Exists is deprecated. Please use tf.io.gfile.exists instead.

WARNING:tensorflow:From retrain.py:915: The name tf.gfile.DeleteRecursively is deprecated. Please use tf.io.gfile.rmtree instead.

W1004 12:24:52.107104 4313146816 deprecation_wrapper.py:119] From retrain.py:915: The name tf.gfile.DeleteRecursively is deprecated. Please use tf.io.gfile.rmtree instead.

WARNING:tensorflow:From retrain.py:916: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

W1004 12:24:52.108059 4313146816 deprecation_wrapper.py:119] From retrain.py:916: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

WARNING:tensorflow:From retrain.py:169: The name tf.gfile.Walk is deprecated. Please use tf.io.gfile.walk instead.

W1004 12:24:52.108767 4313146816 deprecation_wrapper.py:119] From retrain.py:169: The name tf.gfile.Walk is deprecated. Please use tf.io.gfile.walk instead.

WARNING:tensorflow:From retrain.py:181: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.

W1004 12:24:54.436023 4313146816 deprecation_wrapper.py:119] From retrain.py:181: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.

INFO:tensorflow:Looking for images in '_negative'
I1004 12:24:54.436197 4313146816 retrain.py:181] Looking for images in '_negative'
WARNING:tensorflow:From retrain.py:184: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.

W1004 12:24:54.436317 4313146816 deprecation_wrapper.py:119] From retrain.py:184: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.

[SSL: CERTIFICATE_VERIFY_FAILED]

ran the command python3 retrain.py --image_dir training_images/ --output_graph output/saved_model.pb --output_labels output/saved_model.pbtxt --how_many_training_steps 4000 --learning_rate 0.30 --testing_percentage 25 --validation_percentage 25 --eval_step_interval 50 --train_batch_size 2000 --test_batch_size -1 --validation_batch_size -1 --bottleneck_dir /tmp/bottleneck

The retraining stops with this error.

I1004 12:25:00.789146 4313146816 retrain.py:181] Looking for images in 'warning'
I1004 12:25:00.827511 4313146816 resolver.py:79] Using /var/folders/0c/vy2myj416hlbkzv58fdt7krh0001tr/T/tfhub_modules to cache modules.
I1004 12:25:00.828396 4313146816 resolver.py:400] Downloading TF-Hub Module 'https://tfhub.dev/google/imagenet/mobilenet_v1_050_224/feature_vector/1'.
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1317, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1016, in _send_output
self.send(msg)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 956, in send
self.connect()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1392, in connect
server_hostname=server_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 412, in wrap_socket
session=session
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 853, in _create
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1117, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "retrain.py", line 1333, in
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "retrain.py", line 1017, in main
module_spec = hub.load_module_spec(FLAGS.tfhub_module)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_hub/module.py", line 60, in load_module_spec
path = registry.resolver(path)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_hub/registry.py", line 42, in call
return impl(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_hub/compressed_module_resolver.py", line 103, in call
self._lock_file_timeout_sec())
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_hub/resolver.py", line 402, in atomic_download
download_fn(handle, tmp_dir)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_hub/compressed_module_resolver.py", line 98, in download
response = url_opener.open(request)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1360, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1319, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)>

when training classifier builder. It is not using gpu. giving unknown error after bottlenecking completed.

when training classifier builder. It is not using gpu. giving unknown error after bottlenecking part completed.
Failed to get device properties, error code: 30
but tf.test.is_gpu_available() is giving true.

Issue when trying to add new training data to classifier

After training the classifier on some "paper airplane" images I followed the steps to bring the training data over to the /usr/local/lib/node_modules/appium/node_modules/test-ai-classifier folder.

After following the steps in the doc I got this error: Encountered internal error running command: Error: Prediction result array had 107 elements but labels list had 106 elements. They need to match.

I tried updating the labels.js files with my new "paper airplane" tag but that didn't change either value in the error message.

Okay more information - I tried first replacing the contents of /usr/local/lib/node_modules/appium/node_modules/test-ai-classifier/model/ with the contents of the generated web_model folder that is what happened to get this error:

Encountered internal error running command: Error: Prediction result array had 107 elements but labels list had 106 elements. They need to match.

---

After the error above I deleted the test-ai-classifier and reinstalled it. Once reinstalled I pasted in the files from web_model. The shard file was a duplicate title so it asked me to replace and I did. The rest of the files pasted in no problem.

My folder structure looks like this:

Now I am getting:

driver.find_element(:custom => "ai:paper airplane") Selenium::WebDriver::Error::NoSuchElementError: An element could not be located on the page using the given search parameters. from NoSuchElementError: An element could not be located on the page using the given search parameters.

I can't tell if this is progress or not.

I have a feeling its not correct but the pre-packaged finds are still working.

driver.find_element(:custom => "ai:mail") => #<Selenium::WebDriver::Element:0x..fd9bd9f88f08d316 id="A0000000-0000-0000-0614-000000000000">

So while it's not finding the new elements that I was hoping I trained it for it is still working with the default data.

---

New information! I just started playing around with the sample_run folder to see if I had trained the classifier on paper airplanes at all. Turns out paper airplanes are actually held in the "send" training images folder. So maybe this explains my issues?

not bug

Traceback (most recent call last):
File "retrain.py", line 1333, in
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
AttributeError: 'module' object has no attribute 'app'

python2.7.16 macos 10.14.6 tensorflow 2.1.0

resolved : import tensorflow.compat.v1 as tf

sorry....