tensorspace-team/tensorspace-converter

🔨TensorSpace Standard Preprocess Tool for Pre-trained Models from TensorFlow, Keras, TensorFlow.js

tensorspace-converter's Issues

Keras model conversion

Convert a provided .h Keras model into TSP compatible tfjs model with multiple outputs.

python package setup

• set "tensorspace_converter" as the default entry point after installing tensorspacejs package
• config Python package dependencies
• config JavaScript dependencies

Version information

Add version information information, get version information by:

tensorspacejs_converter -v

or

tensorspacejs_converter -V

tensorspace-converter install workflow

• package all necessary libraries into one single pip install command
• validate workflow of pip install

Docker installation failing at RUN sudo add-apt-repository ppa:jonathonf/python-3.6:

When Installing docker file, it's failing at

[ 7/15] RUN sudo add-apt-repository ppa:jonathonf/python-3.6:
#10 4.929 Cannot add PPA: 'ppa:~jonathonf/ubuntu/python-3.6'.
#10 4.929 The user named '~jonathonf' has no PPA named 'ubuntu/python-3.6'
#10 4.929 Please choose from the following available PPAs:
#10 4.929 * 'ansible': Ansible
#10 4.929 * 'atslang': ATS2 programming language

Change tf_hdf5_model to tf_keras

API change:

• tf_hdf5_model -> tf_keras
• tf_hdf5_model_separated -> tf_keras_separated
• tf_frozen_model -> tf_frozen
• tf_saved_model -> tf_saved

Python argparser for script commands

• basic setup and configuration
• commands design
• command implementations

Build tf.keras converter

Similar to Keras, but using tf.keras library.

Keras converting for tye 'None' ?

I have saved a keras model using model.save('xxx.hdf5'), but when I try to convert it using:

tensorspacejs_converter \
    --input_model_from="keras" \
    --input_model_format="topology_weights_combined" \
    --output_node_names='conv2D_1,conv2D_2,conv2D_3,conv2D_4,conv2D_5,conv2D_6,conv2D_7,up_sampling2d_1,conv2D_8,up_sampling2d_2,conv2D_9,up_sampling2d_3,conv2D_10,up_sampling2d_4,conv2D_11,up_sampling2d_5,conv2D_12,up_sampling2d_6,conv2D_13,up_sampling2d_7,conv2D_14' \
    ../rawModel/combined/model_Encry_keras.hdf5 \
    ../convertedModel/

How can I fix it, because I don't know where goes wrong?

Traceback (most recent call last):
  File "c:\programdata\anaconda3\envs\tensorspace\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\programdata\anaconda3\envs\tensorspace\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\ProgramData\Anaconda3\envs\tensorspace\Scripts\tensorspacejs_converter.exe\__main__.py", line 9, in <module>
  File "c:\programdata\anaconda3\envs\tensorspace\lib\site-packages\tensorspacejs\tsp_converters.py", line 131, in main
    'but the current input model format is "%s".' % flags.input_model_format)
ValueError: For input_model_from == "keras", the --input_model_format flag can only be set to"topology_weights_combined" and "topology_weights_separated" but the current input model format is "None".

Add more console information

Add console information when converting the model.

Add js files as data files

Add input layer for converted Keras model

• Catch input layer from both Sequential() and Model()
• Encapsulate input layer as one of the output layers

Conflicting package versions

$ pip install tensorspacejs==0.2.0

(...)

ERROR: Cannot install tensorspacejs because these package versions have conflicting dependencies.

The conflict is caused by:
    keras 2.2.2 depends on keras-applications==1.0.4
    tensorflow 1.12.0 depends on keras-applications>=1.0.6

Without version requirement, pip installs version 0.0.2 which only output "Hello world from converter"

Create output folder if folder is not existed

frozen_model.pb Output names

I want to use the frozen_model.pb of MNIST model in LabVIEW. To do this I need the Output Node Names of the frozen graph. Could you help me how could I find the names of the nodes for output tensors.
I.e. the frozen graph of the Tensor Flow Object-Detection API (tensorflow/models/research/object_detection) the names of the nodes for output tensors are defined in the file "exporter.py". This works great together with LabVIEWs Deep Learning library for Tensor Flow. Sorry for my bad english! :-)
Best regards Peter

Installation issues

Hello together,

after having produced a .json file of my keras architecture I tried to load and visualize the architecture based on JavaScript code. Unfortuantely I only get a blank page. Same is the case for the examples provided on github.
Since Python 3.6 is required I started installing tensorspace again based on a new virtual environment suggested over here: https://tensorspace.org/converter/install.html
When using tensorspacejs_converter I get an error message that it cannot be found although tensorspace was installed with pip.

NPM Version: 6.14.10
Python Version: 3.6.12
Pip Version: 20.3.3

I would be very helpful for guidance.

Thank you in advance!

Dependencies error while installing through pip

Hi, I was trying to install TensorSpace Converter as the guide says, but I receive many errors about incompatible dependencies which I can't fix.

As it says:

TensorSpace-Converter requires to run under Python 3.6, Node 11.3+, NPM 6.5+

so I created a fresh virtual environment:

$ virtualenv venv36 --python=python3.6

Running virtualenv with interpreter /usr/bin/python3.6
Already using interpreter /usr/bin/python3.6
Using base prefix '/usr'
New python executable in /home/david/Git/project/venv36/bin/python3.6
Also creating executable in /home/david/Git/project/venv36/bin/python
Installing setuptools, pip, wheel...
done.

After activating it, everything looks correct:

$ source venv36/bin/activate

(venv36) $

Then, I pip as suggested:

(venv36) $ pip install tensorspacejs

receiving the output (errors in bold):

(venv36) $ pip install tensorspacejs
Collecting tensorspacejs
Using cached https://files.pythonhosted.org/packages/0a/68/4cd1d0dffe6a6f41ef128723e999991dbd822e92df28ea8b2d4b7e342ab6/tensorspacejs-0.2.0-py3-none-any.whl
Collecting tensorflow==1.12.0 (from tensorspacejs)
Using cached https://files.pythonhosted.org/packages/22/cc/ca70b78087015d21c5f3f93694107f34ebccb3be9624385a911d4b52ecef/tensorflow-1.12.0-cp36-cp36m-manylinux1_x86_64.whl
Collecting keras==2.2.2 (from tensorspacejs)
Using cached https://files.pythonhosted.org/packages/34/7d/b1dedde8af99bd82f20ed7e9697aac0597de3049b1f786aa2aac3b9bd4da/Keras-2.2.2-py2.py3-none-any.whl
Collecting tensorflowjs==0.8.0 (from tensorspacejs)
Using cached https://files.pythonhosted.org/packages/6b/0c/89f4b64a9c55a161287781f8c63e4492f6cb2faa01195c9f8990291c1404/tensorflowjs-0.8.0-py3-none-any.whl
Requirement already satisfied: six>=1.10.0 in /usr/lib/python3.7/site-packages (from tensorflow==1.12.0->tensorspacejs) (1.12.0)
Collecting grpcio>=1.8.6 (from tensorflow==1.12.0->tensorspacejs)
Using cached https://files.pythonhosted.org/packages/f2/5d/b434403adb2db8853a97828d3d19f2032e79d630e0d11a8e95d243103a11/grpcio-1.22.0-cp36-cp36m-manylinux1_x86_64.whl
Requirement already satisfied: wheel>=0.26 in ./venv36/lib/python3.6/site-packages (from tensorflow==1.12.0->tensorspacejs) (0.33.4)
Collecting keras-applications>=1.0.6 (from tensorflow==1.12.0->tensorspacejs)
Using cached https://files.pythonhosted.org/packages/71/e3/19762fdfc62877ae9102edf6342d71b28fbfd9dea3d2f96a882ce099b03f/Keras_Applications-1.0.8-py3-none-any.whl
Collecting termcolor>=1.1.0 (from tensorflow==1.12.0->tensorspacejs)
Collecting gast>=0.2.0 (from tensorflow==1.12.0->tensorspacejs)
Requirement already satisfied: numpy>=1.13.3 in /usr/lib/python3.7/site-packages (from tensorflow==1.12.0->tensorspacejs) (1.16.4)
Collecting protobuf>=3.6.1 (from tensorflow==1.12.0->tensorspacejs)
Using cached https://files.pythonhosted.org/packages/eb/f4/a27952733796330cd17c17ea1f974459f5fefbbad119c0f296a6d807fec3/protobuf-3.9.1-cp36-cp36m-manylinux1_x86_64.whl
Collecting absl-py>=0.1.6 (from tensorflow==1.12.0->tensorspacejs)
Collecting tensorboard<1.13.0,>=1.12.0 (from tensorflow==1.12.0->tensorspacejs)
Using cached https://files.pythonhosted.org/packages/07/53/8d32ce9471c18f8d99028b7cef2e5b39ea8765bd7ef250ca05b490880971/tensorboard-1.12.2-py3-none-any.whl
Collecting astor>=0.6.0 (from tensorflow==1.12.0->tensorspacejs)
Using cached https://files.pythonhosted.org/packages/d1/4f/950dfae467b384fc96bc6469de25d832534f6b4441033c39f914efd13418/astor-0.8.0-py2.py3-none-any.whl
Collecting keras-preprocessing>=1.0.5 (from tensorflow==1.12.0->tensorspacejs)
Using cached https://files.pythonhosted.org/packages/28/6a/8c1f62c37212d9fc441a7e26736df51ce6f0e38455816445471f10da4f0a/Keras_Preprocessing-1.1.0-py2.py3-none-any.whl
Collecting h5py (from keras==2.2.2->tensorspacejs)
Using cached https://files.pythonhosted.org/packages/30/99/d7d4fbf2d02bb30fb76179911a250074b55b852d34e98dd452a9f394ac06/h5py-2.9.0-cp36-cp36m-manylinux1_x86_64.whl
Requirement already satisfied: scipy>=0.14 in /usr/lib/python3.7/site-packages (from keras==2.2.2->tensorspacejs) (1.3.0)
Requirement already satisfied: pyyaml in /usr/lib/python3.7/site-packages (from keras==2.2.2->tensorspacejs) (5.1.1)
Collecting tensorflow-hub==0.1.1 (from tensorflowjs==0.8.0->tensorspacejs)
Using cached https://files.pythonhosted.org/packages/5f/22/64f246ef80e64b1a13b2f463cefa44f397a51c49a303294f5f3d04ac39ac/tensorflow_hub-0.1.1-py2.py3-none-any.whl
Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from protobuf>=3.6.1->tensorflow==1.12.0->tensorspacejs) (41.0.1)
Requirement already satisfied: markdown>=2.6.8 in /usr/lib/python3.7/site-packages (from tensorboard<1.13.0,>=1.12.0->tensorflow==1.12.0->tensorspacejs) (3.1.1)
Requirement already satisfied: werkzeug>=0.11.10 in /usr/lib/python3.7/site-packages (from tensorboard<1.13.0,>=1.12.0->tensorflow==1.12.0->tensorspacejs) (0.15.2)
keras 2.2.2 has requirement keras-applications==1.0.4, but you'll have keras-applications 1.0.8 which is incompatible.
keras 2.2.2 has requirement keras-preprocessing==1.0.2, but you'll have keras-preprocessing 1.1.0 which is incompatible.
tensorflowjs 0.8.0 has requirement h5py==2.8.0, but you'll have h5py 2.9.0 which is incompatible.
tensorflowjs 0.8.0 has requirement numpy==1.15.1, but you'll have numpy 1.16.4 which is incompatible.
tensorflowjs 0.8.0 has requirement six==1.11.0, but you'll have six 1.12.0 which is incompatible.
Installing collected packages: grpcio, h5py, keras-applications, termcolor, gast, protobuf, absl-py, tensorboard, astor, keras-preprocessing, tensorflow, keras, tensorflow-hub, tensorflowjs, tensorspacejs
Successfully installed absl-py-0.7.1 astor-0.8.0 gast-0.2.2 grpcio-1.22.0 h5py-2.9.0 keras-2.2.2 keras-applications-1.0.8 keras-preprocessing-1.1.0 protobuf-3.9.1 tensorboard-1.12.2 tensorflow-1.12.0 tensorflow-hub-0.1.1 tensorflowjs-0.8.0 tensorspacejs-0.2.0 termcolor-1.1.0

Running on:

Linux msi 5.2.6-arch1-1-ARCH #1 SMP PREEMPT Sun Aug 4 14:58:49 UTC 2019 x86_64 GNU/Linux

Error: browserHTTPRequest is not supported outside the web browser without a fetch polyfill.

TensorFlow.js version
tensorflow/tfjs: 1.0.0
tensorflow/tfjs-node: 1.2.11

tensorspacejs 0.2.0

Node version
node 11.11.0 on Centos7

I have used TensorSpace-Converter to preprocess pre-trained TensorFlow.js model:

Tensorspacejs_converter  \
   --input_model_from="tfjs" \
    --output_layer_names="myPadding,myConv1,myMaxPooling1,myConv2,myMaxPooling2,myDense1,myDense2,myDense3" \
    /root/mnist.json \
    /root/test/

Following error was displayed:

2019-10-23 11:16:14.243013: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-23 11:16:14.451468: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3600000000 Hz
2019-10-23 11:16:14.451808: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4821350 executing computations on platform Host. Devices:
2019-10-23 11:16:14.451837: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
Loading tfjs model...
(node:76897) UnhandledPromiseRejectionWarning: Error: browserHTTPRequest is not supported outside the web browser without a fetch polyfill.
    at new BrowserHTTPRequest (/root/anaconda2/envs/envname/lib/python3.6/site-packages/tensorspacejs/tfjs/node_modules/@tensorflow/tfjs-core/dist/io/browser_http.js:76:23)
    at Object.browserHTTPRequest (/root/anaconda2/envs/envname/lib/python3.6/site-packages/tensorspacejs/tfjs/node_modules/@tensorflow/tfjs-core/dist/io/browser_http.js:456:12)
    at Object.<anonymous> (/root/anaconda2/envs/envname/lib/python3.6/site-packages/tensorspacejs/tfjs/node_modules/@tensorflow/tfjs-layers/dist/models.js:245:50)
    at step (/root/anaconda2/envs/envname/lib/python3.6/site-packages/tensorspacejs/tfjs/node_modules/@tensorflow/tfjs-layers/dist/models.js:54:23)
    at Object.next (/root/anaconda2/envs/envname/lib/python3.6/site-packages/tensorspacejs/tfjs/node_modules/@tensorflow/tfjs-layers/dist/models.js:35:53)
    at /root/anaconda2/envs/envname/lib/python3.6/site-packages/tensorspacejs/tfjs/node_modules/@tensorflow/tfjs-layers/dist/models.js:29:71
    at new Promise (<anonymous>)
    at __awaiter (/root/anaconda2/envs/envname/lib/python3.6/site-packages/tensorspacejs/tfjs/node_modules/@tensorflow/tfjs-layers/dist/models.js:25:12)
    at Object.loadLayersModelInternal (/root/anaconda2/envs/envname/lib/python3.6/site-packages/tensorspacejs/tfjs/node_modules/@tensorflow/tfjs-layers/dist/models.js:232:12)
    at Object.loadLayersModel (/root/anaconda2/envs/envname/lib/python3.6/site-packages/tensorspacejs/tfjs/node_modules/@tensorflow/tfjs-layers/dist/exports.js:224:21)
(node:76897) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:76897) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Mission Complete!!!

How can I solve this error?

Convert tfjs model

To support convert tfjs model:

• include necessary JavaScript dependencies
• create JavaScript codes to complete the conversion
• python API to trigger JavaScript process

pip install tensorspacejs==0.1.0, Keras conflicts with TensorFlow dependencies

According to fig1, the following issue occurred when I installed pip install tensorspacejs=0.1.0.

I am using the Win11 system and have already created a new environment in Conda，pip use indexing https://pypi.tuna.tsinghua.edu.cn/simple.

$ conda create -n envname python=3.6 $ source activate envname $ pip install tensorspacejs==0.1.0

• Brief error message：

keras-applications==1.0.4 has a dependency conflict.
The conflict is caused by: keras 2.2.2 depends on keras-applications==1.0.4 tensorflow 1.12.0 depends on keras-applications>=1.0.6

• Complete error message：

`
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting tensorspacejs==0.1.0
Using cached https://pypi.tuna.tsinghua.edu.cn/packages/03/d3/de6fbbf49cef39726f28317d5c6bc3f25986fa24595fb06c450cc0b0a512/tensorspacejs-0.1.0-py3-none-any.whl (66 kB)
Requirement already satisfied: tensorflow==1.12.0 in c:\users\tsing.conda\envs\tensorspace\lib\site-packages (from tensorspacejs==0.1.0) (1.12.0)
Collecting keras==2.2.2
Using cached https://pypi.tuna.tsinghua.edu.cn/packages/34/7d/b1dedde8af99bd82f20ed7e9697aac0597de3049b1f786aa2aac3b9bd4da/Keras-2.2.2-py2.py3-none-any.whl (299 kB)
Requirement already satisfied: tensorflowjs==0.8.0 in c:\users\tsing.conda\envs\tensorspace\lib\site-packages (from tensorspacejs==0.1.0) (0.8.0)
Collecting keras-applications==1.0.4
Using cached https://pypi.tuna.tsinghua.edu.cn/packages/54/90/8f327deaa37a71caddb59b7b4aaa9d4b3e90c0e76f8c2d1572005278ddc5/Keras_Applications-1.0.4-py2.py3-none-any.whl (43 kB)
Requirement already satisfied: scipy>=0.14 in c:\users\tsing.conda\envs\tensorspace\lib\site-packages (from keras==2.2.2->tensorspacejs==0.1.0) (1.5.4)
Requirement already satisfied: six>=1.9.0 in c:\users\tsing.conda\envs\tensorspace\lib\site-packages (from keras==2.2.2->tensorspacejs==0.1.0) (1.16.0)
Requirement already satisfied: pyyaml in c:\users\tsing.conda\envs\tensorspace\lib\site-packages (from keras==2.2.2->tensorspacejs==0.1.0) (6.0.1)
Collecting keras-preprocessing==1.0.2
Using cached https://pypi.tuna.tsinghua.edu.cn/packages/71/26/1e778ebd737032749824d5cba7dbd3b0cf9234b87ab5ec79f5f0403ca7e9/Keras_Preprocessing-1.0.2-py2.py3-none-any.whl (26 kB)
Requirement already satisfied: numpy>=1.9.1 in c:\users\tsing.conda\envs\tensorspace\lib\site-packages (from keras==2.2.2->tensorspacejs==0.1.0) (1.19.5)
Requirement already satisfied: h5py in c:\users\tsing.conda\envs\tensorspace\lib\site-packages (from keras==2.2.2->tensorspacejs==0.1.0) (3.1.0)
Requirement already satisfied: absl-py>=0.1.6 in c:\users\tsing.conda\envs\tensorspace\lib\site-packages (from tensorflow==1.12.0->tensorspacejs==0.1.0) (1.4.0)
Requirement already satisfied: protobuf>=3.6.1 in c:\users\tsing.conda\envs\tensorspace\lib\site-packages (from tensorflow==1.12.0->tensorspacejs==0.1.0) (3.19.6)
Requirement already satisfied: tensorboard<1.13.0,>=1.12.0 in c:\users\tsing.conda\envs\tensorspace\lib\site-packages (from tensorflow==1.12.0->tensorspacejs==0.1.0) (1.12.2)
INFO: pip is looking at multiple versions of keras-preprocessing to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of keras-applications to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of keras to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of tensorspacejs to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install tensorflow==1.12.0 and tensorspacejs because these package versions have conflicting dependencies.

The conflict is caused by:
keras 2.2.2 depends on keras-applications==1.0.4
tensorflow 1.12.0 depends on keras-applications>=1.0.6

To fix this you could try to:

1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
`

Can you provide me with assistance.

Remove relative path defination

TensorSpace-Converter YAML configuration

Just a proposal, TensorSpace-Converter load a YAML configuration file and preprocess the model based on YAML configuration.

YAML configuration:

input_model_from: tensorflow
input_model_format: tf_keras
output_layer_names:
 - layer1Name
 - layer2Name
 - layer3Name
input_path: ./PATH/TO/MODEL/xxx.h5
output_path: ./PATH/TO/SAVE/DIR

Converter shell command:

tensorspacejs_converter \
  --configuration_yaml=PATH/TO/YAML/FILE/configuration.yaml

Establish TensorSpace-Converter website

Pip initialization

• Set pip name ("tensorspacejs")
• Set package management

convert image to json for prediction

how to convert jpeg image to json format

Convert TensorFlow model

To support conversion of different type of TensorFlow models:

• saved model (directory)
• frozen model (.pb)
• check point model (.ckpt)