DECIMER: Deep Learning for Chemical Image Recognition using Efficient-Net V2 + Transformer
License: MIT License
Check out what's playing on my Spotify right now!
`
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "github-actions[bot]")
Allowing people to drag-n-drop images from articles, like https://www.nature.com/articles/s41586-022-04828-5/figures/1, which often are .webp format nowadays:
Hi,
Thank you for sharing pre-trained model that works great. I am trying to train your model with my own data and have several questions. I am assuming the code base should be related to the article but feel free to correct.
I first had to modify the strategy so I could train on GPU only. I generated molecule images using CDK as outlined in the paper, created tfrecords.
I initially used selfies as captions and trained a model, packaged it using the code on github but for some reason it was insensitive to the input image hence producing the same output for all inputs. I poked around the deployed model and it seemed like tokenizer was a smiles tokenizer not selfies.
I switched to smiles captions to try and obviously still the same issue and I thought it must be something with the image encoder. I looked at the automl code included in the repo and seemed like it had some issues so switched to the recent automl efficientv2b3. Training shows loss getting lower, accuracy on training data increasing (didn't remove commented validation tracking but I will). No longer insensitivity to the input images. Another issue I realized that preprocessing for efficientnet should map the images to [-1,1] but preprocessing used in decode_image method was doing something else which eventually calling keras_applications.imagenet_utils with torch mode. Then I noticed that the code is actually training the efficientnet architecture and not using pretrained models to extract features. There is also this difference that current code is using an earlier layer as image feature as opposed to the layers as pointed out in the article.
TL;DR
Is there a way to just add data to the model you already trained, without retraining it completely?
It seems like something is fundamentally broken. I'm running it in the WSL on Windows 11 in a Python 3.8 environment.
When I run
from DECIMER import predict_SMILES
predict_SMILES("ethyl.png")
I get:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/otto/.local/lib/python3.8/site-packages/DECIMER/decimer.py", line 110, in predict_SMILES
predicted_tokens = DECIMER_V2(chemical_structure)
File "/home/otto/.local/lib/python3.8/site-packages/tensorflow/python/saved_model/load.py", line 704, in _call_attribute
return instance.__call__(*args, **kwargs)
File "/home/otto/.local/lib/python3.8/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/home/otto/.local/lib/python3.8/site-packages/tensorflow/python/saved_model/function_deserialization.py", line 295, in restored_function_body
raise ValueError(
ValueError: Could not find matching concrete function to call loaded from the SavedModel. Got:
Positional arguments (1 total):
* <tf.Tensor 'Decoded_Image:0' shape=(512, 512, 3) dtype=float32>
Keyword arguments: {}
Expected these arguments to match one of the following 1 option(s):
Option 1:
Positional arguments (1 total):
* TensorSpec(shape=(299, 299, 3), dtype=tf.float32, name='Decoded_Image')
Keyword arguments: {}
Questions
GitHub (source)
latest
windows
3.11
Hello, thank you for your model, I am doing research work and I would like to know how to generate only valid SMILES strings using your model, could you provide this code, if possible, I will be very grateful for your answer!
No response
-No response
Questions
GitHub (source)
Not relevant
No response
No response
Hello,
Have you ever thinked about converting your open-source from Tensorflow into Pytorch?
Best,
Khanh
No response
Not relevantNo response
Brand new install from pypi.
Using:
from DECIMER import predict_SMILES
from rdkit import Chem
from rdkit.Chem import Draw
smiles = "CC(=O)Nc1ccc2c(c1)C(=O)N(C)C(=O)c1ccc(O)cc1C2"
image_path = "mol.png"
mol = Chem.MolFromSmiles(smiles)
Draw.MolToFile(mol, image_path)
SMILES = predict_SMILES(image_path)
print(SMILES)
Causes:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[5], line 1
----> 1 SMILES = predict_SMILES(image_path)
2 print(SMILES)
File [~/miniconda3/envs/spresi_pubchem_decimer_comparison/lib/python3.9/site-packages/DECIMER/decimer.py:104](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/adam/projects/spresi-pubchem-decimer-comparison/notebooks/~/miniconda3/envs/spresi_pubchem_decimer_comparison/lib/python3.9/site-packages/DECIMER/decimer.py:104), in predict_SMILES(image_path)
93 """
94 This function takes an image path (str) and returns the SMILES
95 representation of the depicted molecule (str).
(...)
101 (str): SMILES representation of the molecule in the input image
102 """
103 chemical_structure = config.decode_image(image_path)
--> 104 predicted_tokens = DECIMER_V2(chemical_structure)
105 predicted_SMILES = utils.decoder(detokenize_output(predicted_tokens))
107 return predicted_SMILES
File [~/miniconda3/envs/spresi_pubchem_decimer_comparison/lib/python3.9/site-packages/tensorflow/python/saved_model/load.py:704](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/adam/projects/spresi-pubchem-decimer-comparison/notebooks/~/miniconda3/envs/spresi_pubchem_decimer_comparison/lib/python3.9/site-packages/tensorflow/python/saved_model/load.py:704), in _call_attribute(instance, *args, **kwargs)
703 def _call_attribute(instance, *args, **kwargs):
--> 704 return instance.__call__(*args, **kwargs)
File [~/miniconda3/envs/spresi_pubchem_decimer_comparison/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py:153](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/adam/projects/spresi-pubchem-decimer-comparison/notebooks/~/miniconda3/envs/spresi_pubchem_decimer_comparison/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py:153), in filter_traceback..error_handler(*args, **kwargs)
151 except Exception as e:
152 filtered_tb = _process_traceback_frames(e.__traceback__)
--> 153 raise e.with_traceback(filtered_tb) from None
154 finally:
155 del filtered_tb
File [~/miniconda3/envs/spresi_pubchem_decimer_comparison/lib/python3.9/site-packages/tensorflow/python/saved_model/function_deserialization.py:295](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/adam/projects/spresi-pubchem-decimer-comparison/notebooks/~/miniconda3/envs/spresi_pubchem_decimer_comparison/lib/python3.9/site-packages/tensorflow/python/saved_model/function_deserialization.py:295), in recreate_function..restored_function_body(*args, **kwargs)
291 positional, keyword = concrete_function.structured_input_signature
292 signature_descriptions.append(
293 "Option {}:\n {}\n Keyword arguments: {}".format(
294 index + 1, _pretty_format_positional(positional), keyword))
--> 295 raise ValueError(
296 "Could not find matching concrete function to call loaded from the "
297 f"SavedModel. Got:\n {_pretty_format_positional(args)}\n Keyword "
298 f"arguments: {kwargs}\n\n Expected these arguments to match one of the "
299 f"following {len(saved_function.concrete_functions)} option(s):\n\n"
300 f"{(chr(10)+chr(10)).join(signature_descriptions)}")
ValueError: Could not find matching concrete function to call loaded from the SavedModel. Got:
Positional arguments (1 total):
*
Keyword arguments: {}
Expected these arguments to match one of the following 1 option(s):
Option 1:
Positional arguments (1 total):
* TensorSpec(shape=(299, 299, 3), dtype=tf.float32, name='Decoded_Image')
Keyword arguments: {}
Downgrading to decimer v2.1.3 resolves the issue.
Questions
GitHub (source)
2.4.0
No response
No response
Hi!
Is there a way of running the SMILES prediction on a batch of images?
As of now, the predict_SMILES function only accepts filepaths of single images.
However, i also had no luck with loading the DECIMER model manually and trying to infer on a batch.
Is there any way to do batch processing of images with DECIMER?
Inference currently takes around 2sec/image (On Tesla T4) and i was hoping to speed up inference by using batch processing.
Thank you for your help!
No response
# attempt to manually batch images before processing
from DECIMER import predict_SMILES
from DECIMER import utils, config
from DECIMER.decimer import DECIMER_V2
import tensorflow as tf
testfiles = [
"image1.png",
"image2.png"
]
prepped = [config.decode_image(f) for f in testfiles]
batch = tf.stack(prepped, axis=0)
print('batch shape:', batch.shape)
# >>> shape image batch: (2, 512, 512, 3)
DECIMER_V2(batch)
# ValueError: Could not find matching concrete function to call loaded from the SavedModel. Got:
# Positional arguments (1 total):
# * <tf.Tensor 'Decoded_Image:0' shape=(2, 512, 512, 3) dtype=float32>
# Keyword arguments: {}
# Expected these arguments to match one of the following 1 option(s):
# Option 1:
# Positional arguments (1 total):
# * TensorSpec(shape=(512, 512, 3), dtype=tf.float32, name='Decoded_Image')
# Keyword arguments: {}No response
The code does not follow a consistent style. I would suggest applying the black tool, which automatically formats your code in a consistent way. This is important because it makes it easier for anyone who is reading your code to more quickly understand it. After, I would suggest setting up CI to ensure that all code is following the black format.
Hi,
This looks like a great resource. I'm trying to get an understanding of how to use this model. I installed from github source and ran the following code:
from decimer import DECIMER
model_name = 'Isomeric'
smiles = DECIMER.predict_SMILES('image.png',model_name)
print(smiles)where the image is
The output was:
CCCOCCOCCOCCCCCNCCCNCCNCCNCCNCCNCCCCCCCCNNNNNNN
which is quite far from correct. Do you have any idea what might be going on? Is there some per-processing of the image I need to do? Thank you!
It's not obvious from the documentation (which currently is only the README, as far as I can tell) which parts of this repository are responsible for training the model or how to use them. This would be something to work into a simple CLI that could be documented from the README, as well as more thorough technical documentation in the docs that should get added to address #4
Performance
GitHub (source)
2.2.0
No response
No response
The model cannot recognize pretty simple structures. I tried all of these and it gives wrong results
No response
Just use the provided images as inputNo response
Hi,
It would be appreciated if you could kindly share any suggestions with me for my issue below.
When I run the below decimer command below, I could not get a correct SMILES result, or sometimes, it runs into errors.
And I am running it on a Windows 10 machine.
PS C:\Git\DECIMER-Image_Transformer> decimer --model Canonical --image Sample_Images/caffeine.png
Traceback (most recent call last):
File "C:\Users\SS\AppData\Local\Programs\Python\Python37\Scripts\decimer-script.py", line 33, in
sys.exit(load_entry_point('decimer', 'console_scripts', 'decimer')())
File "c:\Git\decimer-image_transformer\src\decimer\cli.py", line 74, in main
get_SMILES = predict_SMILES(sys.argv[4])
File "c:\Git\decimer-image_transformer\src\decimer\cli.py", line 167, in predict_SMILES
predicted_SELFIES = evaluate(image_path)
File "c:\Git\decimer-image_transformer\src\decimer\cli.py", line 159, in evaluate
result.append(SELFIES_tokenizer.index_word[int(predicted_id)])
KeyError: 36
PS C:\Git\DECIMER-Image_Transformer> decimer --model Canonical --image Sample_Images/caffeine.png
Predicted SMILES for Sample_Images/caffeine.png :FF
PS C:\Git\DECIMER-Image_Transformer> decimer --model Canonical --image Sample_Images/caffeine.png
Traceback (most recent call last):
File "C:\Users\SS\AppData\Local\Programs\Python\Python37\Scripts\decimer-script.py", line 33, in
sys.exit(load_entry_point('decimer', 'console_scripts', 'decimer')())
File "c:\Git\decimer-image_transformer\src\decimer\cli.py", line 74, in main
get_SMILES = predict_SMILES(sys.argv[4])
File "c:\Git\decimer-image_transformer\src\decimer\cli.py", line 167, in predict_SMILES
predicted_SELFIES = evaluate(image_path)
File "c:\Git\decimer-image_transformer\src\decimer\cli.py", line 159, in evaluate
result.append(SELFIES_tokenizer.index_word[int(predicted_id)])
Since the package will be pip-installable after #2 and does not rely on any exotic dependencies, you should deploy it to PyPI. If you've never done this before, first, bump the version in setup.cfg from 0.0.1-dev to 0.0.1. Then, follow these steps in the bash in the root directory of the repository:
python3 setup.py -q sdist bdist_wheel
python3 -m pip install twine
twine upload --skip-existing dist/*You should do this step after you name a certain version as a "release" and upload to Zenodo (as outlined in #6) then immediately follow-up with a commit to bump the version in setup.cfg from 0.0.1 to 0.0.2-dev. Later, when you're done with version 0.0.2-dev (or whatever you want to call it), you take away the -dev suffix before releasing again. This suffix is important because it lets users know that they're not on stable versions if they're installing from github's master (main) branch
Also, please include in your manuscript that your code is available through PyPI! There are many potential users who won't be comfortable with cloning the repo or even using the tricky git+https:// way of installing through GitHub who would benefit from easy installation through PyPI.
Hi, Kohulan:
When I was testing this model, I met one problem:
I used train model but get wrong predictions: C(C1=[X2]=[X3]=NC2=C1N=C(N2)[R1])R(Cl)[R14] and CC(CC1=CN=C(C(=C1)N([R2])S(=O)(=O)C2=C(C(=C(C=C2Br)Br)[R20])[R12])[R1])C[X]
Do you have any advices to get high accuracy to test these pictures, we want to input these pictures and get right smiles?
In my personal opinion, it is because there is no such sample when the model is trained, so can I use this decimerV-2 model that has been trained?
There are two parts to this repo - the training and the usage. I think users would benefit from all of it being in a unified package that could be pip installed. I could make a PR if you agree this is a good idea.
the question is on the title. now I can run decimer.py to get the result of SMILES, but can't run the trainer and predictor---I can't find correlative doc.
Thanks much.
While GitHub is an excellent place to centralize your code and keep it under version control, it is not so appropriate for archiving the state of the repository. Zenodo is a proper alternative, and it will even make your life easy by connecting to GitHub. I'd recommend following this tutorial to:
I personally thing Zenodo is the best tool for code because of its integration with GitHub, but other services exist for archiving your research output that are also good, such as Figshare.
Hello, I found there's no embedding(Dense) layer on encoder part of transformer model.
class TransformerEncoder(tf.keras.layers.Layer):
def __init__(
self,
num_layers,
d_model,
num_heads,
dff,
maximum_position_encoding,
dropout_rate=0.1,
):
super(TransformerEncoder, self).__init__()
self.d_model = d_model
self.num_layers = num_layers
self.pos_encoding = positional_encoding_1d(
maximum_position_encoding, self.d_model
)
self.enc_layers = [
TransformerEncoderLayer(d_model, num_heads, dff, dropout_rate)
for _ in range(num_layers)
]
self.dropout = tf.keras.layers.Dropout(dropout_rate)
def call(self, x, training, mask=None):
# adding embedding and position encoding.
# – (batch_size, input_seq_len, d_model)
# -> Here you need embedding! ( I think you just deleted it updating the code)
x += self.pos_encoding
x = self.dropout(x, training=training)
for i in range(self.num_layers):
x = self.enc_layers[i](x, training, mask)
# – (batch_size, input_seq_len, d_model)
return xSupport
GitHub (source)
2.0.0
No response
No response
Can I get the trained model as a .h5 or tf model format that I can load? I did not find the model in the repo
No response
# Not code related question
# Can I get the trained model as a .h5 or tf model format that I can load? I did not find the model in the repoNo response
Feature Request
GitHub (source)
2.3.1
not relevant
not relevant
Predictor_EfficientNet2.py has not been updated for a year.
Why is this relevant?
The script is not used in the Python package, but it contains an example of how to load the model from a checkpoint. If we publish the checkpoints, we should also provide an example of how to use them. In its current state, the code does not work (see confusion in #65).
Two options:
No response
see #65
or just try running or importing the code in [Predictor_EfficientNet2.py](https://github.com/Kohulan/DECIMER-Image_Transformer/blob/master/DECIMER/Predictor_EfficientNet2.py)No response
Feature Request
GitHub (source)
2.3.0
No response
No response
N/A
No response
N/ANo response
Add preprint on DECIMER 1.0 to Literature section in README
Questions
GitHub (source)
2.0
No response
No response
from DECIMER import predict_SMILES
image_path = "path/to/imagefile"
SMILES = predict_SMILES(image_path)
print(SMILES)
The SMILES do not have the atom coordinates.
How to get the positional information?
Thanks!
No response
from DECIMER import predict_SMILES
# Chemical depiction to SMILES translation
image_path = "path/to/imagefile"
SMILES = predict_SMILES(image_path)
print(SMILES)No response
Hello,
Is it possible to train DECIMER on a new dataset consisting of pairs of images and SMILES, and without having access to TPUs?
Best,
Lucas
Hi, I've keep getting the attribute error that module decimer has no attribute <predict_SMILES>
After installing decimer by git clone, I've checked dir(decimer) and I've got
['builtins', 'cached', 'doc', 'file', 'loader', 'name', 'package', 'path', 'spec']
I wonder where is the <predict_SMILES> is located at.
Best
Taeuk
Questions
GitHub (source)
Not relevant
Not relevant
Not relevant
First, I want to thank for your hard-working with an awesome open-source. I just have a naive question that Why do you choose EfficientNet for backbone CNN model but not ResNet or others?
Thank for reading! Hope you reply soon.
No response
Not relevantNo response
I am having trouble getting this to work. I installed using "pip install decimer" in the anaconda prompt on Windows.
Running "decimer --model Canonical --image img.png" results first in
(DECIMER_V1) C:\Users\curtis.colwell\Desktop\Takeda Images\Img_Page_01\cells>decimer --model Canonical --image "C:\Users\curtis.colwell\Desktop\Takeda Images\Img_Page_01\cells\001-002.png"
Traceback (most recent call last):
File "C:\Users\curtis.colwell\Anaconda3\envs\DECIMER_V1\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "C:\Users\curtis.colwell\Anaconda3\envs\DECIMER_V1\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\curtis.colwell\Anaconda3\envs\DECIMER_V1\Scripts\decimer.exe\__main__.py", line 7, in <module>
File "C:\Users\curtis.colwell\Anaconda3\envs\DECIMER_V1\lib\site-packages\decimer\cli.py", line 35, in main
get_QUOTE = helper.get_quote()
File "C:\Users\curtis.colwell\Anaconda3\envs\DECIMER_V1\lib\site-packages\decimer\Network\helper.py", line 139, in get_quote
quote = random_line(open((HERE.joinpath("Quotes_database")), "r"))
File "C:\Users\curtis.colwell\Anaconda3\envs\DECIMER_V1\lib\site-packages\decimer\Network\helper.py", line 131, in random_line
for num, aline in enumerate(file_name, 2):
File "C:\Users\curtis.colwell\Anaconda3\envs\DECIMER_V1\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 7403: character maps to <undefined>
This is fixed by going to the helper.py and adding " encoding="utf8" " to line 139.
Then I get the following error and am stumped:
(DECIMER_V1) C:\Users\curtis.colwell\Desktop\Takeda Images\Img_Page_01\cells>decimer --model Canonical --image "C:\Users\curtis.colwell\Desktop\Takeda Images\Img_Page_01\cells\001-002.png"
Downloading trained model to C:\Users\curtis.colwell\.data\decimer\Trained_Models/Canonical/
C:\Users\curtis.colwell\.data\decimer\DECIMER_trained_models_v1.0.zip
... done downloading trained model!
Traceback (most recent call last):
File "C:\Users\curtis.colwell\Anaconda3\envs\DECIMER_V1\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "C:\Users\curtis.colwell\Anaconda3\envs\DECIMER_V1\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\curtis.colwell\Anaconda3\envs\DECIMER_V1\Scripts\decimer.exe\__main__.py", line 7, in <module>
File "C:\Users\curtis.colwell\Anaconda3\envs\DECIMER_V1\lib\site-packages\decimer\cli.py", line 74, in main
) = load_trained_model(model_id)
File "C:\Users\curtis.colwell\Anaconda3\envs\DECIMER_V1\lib\site-packages\decimer\cli.py", line 132, in load_trained_model
helper.download_trained_weights(model_url, checkpoint_path)
File "C:\Users\curtis.colwell\Anaconda3\envs\DECIMER_V1\lib\site-packages\decimer\Network\helper.py", line 104, in download_trained_weights
model_path.parent.as_posix(),
File "C:\Users\curtis.colwell\Anaconda3\envs\DECIMER_V1\lib\subprocess.py", line 488, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\curtis.colwell\Anaconda3\envs\DECIMER_V1\lib\subprocess.py", line 800, in __init__
restore_signals, start_new_session)
File "C:\Users\curtis.colwell\Anaconda3\envs\DECIMER_V1\lib\subprocess.py", line 1207, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
After #4, you can set up ReadTheDocs.org to automatically build and host the sphinx docs
How to judge the image is consistent with SMILES?
the molecular image to SMILES，how can i judge that they are consistent?
or how to prediction the output SMILES accurate?
When I ran decimer (after #2), it downloaded all of the assets into the current working directory. This isn't desired after the code is packaged, and it would be better to have it download to a consistent location, such as in the home directory. I've spent a lot of time thinking about this and written the PyStow package to support this. I could send a PR after #2 is done that demonstrates what I mean
Hi,
We are using the decimer tool for chemical structure extraction, Is there any option to get the predicted score/ accuracy after extraction of image?
Hi everyone,
I am currently trying to run the "run_decimer_save_results.py" script to reproduce benchmarks. Trying this I am encouting errors in different files importing packages.
An Example:
File "/data/anaconda3/envs/DECIMER/lib/python3.10/site-packages/DECIMER/Predictor_EfficientNet2.py", line 8, in
import Efficient_Net_encoder
ModuleNotFoundError: No module named 'Efficient_Net_encoder'
"import DECIMER.Efficient_Net_encoder" would fix it but this doen't work for example in the Efficient_Net_encoder.py file, if I put "DECIMER.efficientnetv2" for simply "efficinetnetv2" the import is functioning but later in the same script the function "fficientnetv2.effnetv2_model.EffNetV2Model(model_name=model_name)" is prompting an error that the module efficientnetv2 doesn' have a module effnetv2_model.
Also a question: in the Predictor_EfficientNet2.py script the tokenizer "tokenizer_Isomeric_SELFIES" and max length "max_length_Isomeric_SELFIES" are loaded. Are those just renamed or different objects than the two which are downloaded with the model? These are called "tokenizer_SMILES" and "max_length" respectively.
I encountered the same issue on two installation on a Windows machine as well as on a Unix machine.
I followed the ReadMe instructions and created an conda environment and running the code inside the environment.
I hope someone can help me. Let me know if you need any other informations.
Cheers,
Sebastian
If a BytesIO object is handed to the predict_SMILES function, the following error occurs. This used to work without any problems.
It seems like this behaviour has been introduced with the HEIC/HEIF compatibility. Can we check for the image type differently without checking for 'HEIC' in the name?
File "/home/otto/.local/lib/python3.8/site-packages/DECIMER/decimer.py", line 109, in predict_SMILES
chemical_structure = config.decode_image(image_path)
File "/home/otto/.local/lib/python3.8/site-packages/DECIMER/config.py", line 197, in decode_image
img = remove_transparent(image_path)
File "/home/otto/.local/lib/python3.8/site-packages/DECIMER/config.py", line 110, in remove_transparent
if pathlib.Path(image_path).suffix == ".HEIC":
File "/usr/lib/python3.8/pathlib.py", line 1042, in __new__
self = cls._from_parts(args, init=False)
File "/usr/lib/python3.8/pathlib.py", line 683, in _from_parts
drv, root, parts = self._parse_args(args)
File "/usr/lib/python3.8/pathlib.py", line 667, in _parse_args
a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not _io.BytesIO
The model performs badly on hand drawn images or pictures taken by phone.
Right now, the README is very confusing for any potential users. Typically I don't think the README of a code repository should read the same way as a scientific work. It should start with at most one or two lines about what the package does, a link to the relevant paper. Then, you should have the quick version of how to install the code. Then, minimal examples that showcase what the package does. After that, anything is fair game - this is where you could elaborate on some of the scientific aspects, though I would suggest cutting this completely from the README - you have a scientific paper for fluffy words. They can also be part of the first page of more thorough documentation, which I will suggest in an alternate issue
Bug
GitHub (source)
2
No response
No response
Hi, I tried to train this neural network on my dataset, but I don't quite understand how you used tokenizer when encoding the text. I didn't find any information about it in the training file. I use the following code to load the tokenizer:
tokenizer = pickle.load(
open(
default_path.as_posix() + "/DECIMER_model/assets/tokenizer_SMILES.pkl",
{ "rb",
)
)
But when I try to encode the text I get strange values:
tokenizer.texts_to_sequences(['O=C(O)C(CCCCn1cnc2c1NC=NCC2O)(Cc1ccccc1C(F)(F)F)C(=O)O', 'C1CCCCCCC1']) -> [[1], [1]]
Can you please tell me how tokenizer should be used correctly?
No response
# Set path
default_path = pystow.join("DECIMER-V2")
# model download location
checkpoint_url = "https://zenodo.org/record/8093783/files/DECIMER_512_checkpoints.zip"
checkpoint_path = str(default_path) + "/DECIMER_checkpoints/"
if not os.path.exists(checkpoint_path):
config.download_trained_weights(checkpoint_url, default_path)
# load assets
tokenizer = pickle.load(
open(
default_path.as_posix() + "/DECIMER_model/assets/tokenizer_SMILES.pkl",
"rb",
)
)
tokenizer.texts_to_sequences(['O=C(O)C(CCCCn1cnc2c1NC=NCC2O)(Cc1ccccc1C(F)(F)F)C(=O)O', 'C1CCCCC1'])No response
I see the selfies new version is 2.0.0 , This causes DECIMER fail to run, I tried 1.0.4 version and it worked.
pip install selfies==1.0.4
Compatibility issue
PyPi
2.3.0
Linux 4.18.0-348.el8.0.2.x86_64 x86_64
3.9.7
DNN library is not found when using predict_SMILES
It's not an image-related error but here it is anyway.
from DECIMER import predict_SMILES
image = 'CID89150000.png'
smiles = predict_SMILES(image)File "/nfs/image2structure/patent_image2structure_comparison_decimer_cluster_gpu.py", line 30, in image_to_smiles
SMILES = predict_SMILES(image_file)
File "/nfs/image2structure/decimer/lib/python3.9/site-packages/DECIMER/decimer.py", line 135, in predict_SMILES
predicted_tokens, _ = DECIMER_V2(chemical_structure)
File "/nfs/image2structure/decimer/lib/python3.9/site-packages/tensorflow/python/saved_model/load.py", line 740, in _call_attribute
return instance.__call__(*args, **kwargs)
File "/nfs/image2structure/decimer/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/nfs/image2structure/decimer/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 52, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.UnimplementedError: Graph execution error:
Detected at node 'encoder/eff_net_v2_model/stem/conv2d/Conv2D' defined at (most recent call last):
Node: 'encoder/eff_net_v2_model/stem/conv2d/Conv2D'
DNN library is not found.
[[{{node encoder/eff_net_v2_model/stem/conv2d/Conv2D}}]] [Op:__inference_restored_function_body_186354]The docs for a package shouldn't all live in the README - it makes sense to have a more full documentation that's living somewhere else. Most python packages use Sphinx to do this. After #2 is resolved, I could also send a PR to set up the structure for automatically building docs. This would be a place to put the more high level/fluffy text that I mentioned in #3
Questions
PyPi
2.3.0
MacBook Pro M1, 2020
3.10
Hey!
Is there a way to access the encoder output using decimer's loaded model?
I'm interested in the embedded representation that is fed to the decoder, not the smiles itself. I was wondering if it's possible to access them once the Transformer class calls the encoder and the decoder separately.
I could reproduce the predict_SMILES function by loading the model from the checkpoint available in zenodo, but since it's a TF model, I can only __call__ it.
Is there any possible way to load these weights into the Transformer class so I can call the t_encoder to access the enc_output?
Having an argument in the call to expose the hidden_states would also work fine.
Any suggestion is welcome!
Thanks!
Mayk
No response
N/A, but I'm using this code to test:
import pickle
import DECIMER
import tensorflow as tf
tokenizer = pickle.load(
open(
"DECIMER_model/assets/tokenizer_SMILES.pkl", # Downloaded from zenodo
"rb",
)
)
def detokenize_output(predicted_array: int) -> str:
"""
This function takes the predited tokens from the DECIMER model
and returns the decoded SMILES string.
Args:
predicted_array (int): Predicted tokens from DECIMER
Returns:
(str): SMILES representation of the molecule
"""
outputs = [tokenizer.index_word[i] for i in predicted_array[0].numpy()]
prediction = (
"".join([str(elem) for elem in outputs])
.replace("<start>", "")
.replace("<end>", "")
)
return prediction
DECIMER_V2 = tf.saved_model.load("DECIMER_model/") # Downloaded from zenodo
chem_struct = DECIMER.config.decode_image('sample.png')
pred_tkns = DECIMER_V2(chem_struct)
predicted_SMILES = DECIMER.utils.decoder(detokenize_output(pred_tkns))
print(predicted_SMILES)No response
Hello,
I know this is not a DECIMER problem but do you know how to solve the following error:
2022-08-13 11:49:01.216206: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2022-08-13 11:49:01.217252: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2022-08-13 11:49:10.677139: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'nvcuda.dll'; dlerror: nvcuda.dll not found
2022-08-13 11:49:10.677615: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2022-08-13 11:49:10.686175: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: LAPTOP-KE4CUS2E
2022-08-13 11:49:10.686322: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: LAPTOP-KE4CUS2E
2022-08-13 11:49:55.236936: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Currently, the package gives a command line tool. There should also be an example in the README on how to import the code from inside your own python script and use it (this will make more sense after #2)
File "/home/zhouhantao/miniconda3/envs/oldtf/lib/python3.7/site-packages/tensorflow/python/saved_model/load.py", line 890, in load_internal
ckpt_options, filters)
File "/home/zhouhantao/miniconda3/envs/oldtf/lib/python3.7/site-packages/tensorflow/python/saved_model/load.py", line 160, in init
self._load_all()
File "/home/zhouhantao/miniconda3/envs/oldtf/lib/python3.7/site-packages/tensorflow/python/saved_model/load.py", line 256, in _load_all
self._load_nodes()
File "/home/zhouhantao/miniconda3/envs/oldtf/lib/python3.7/site-packages/tensorflow/python/saved_model/load.py", line 434, in _load_nodes
node, setter = self._recreate(proto, node_id)
File "/home/zhouhantao/miniconda3/envs/oldtf/lib/python3.7/site-packages/tensorflow/python/saved_model/load.py", line 551, in _recreate
raise ValueError("Unknown SavedObject type: %r" % kind)
ValueError: Unknown SavedObject type: None
When trying to install DECIMER to my code I encountered the following error NameError: name 'shutil' is not defined. This could however be easily be solved by adding shutil to the imports.
just "AttributeError: module 'Transformer_decoder' has no attribute 'Transformer'"
Dose it mean Transformer_decoder.Decoder?
Feature Request
GitHub (source)
2.2.2
No response
No response
Integration with RDKIT for adding more features to it like providing structures in different formats like .sdf and .mol
No response
NANo response
Hi,
I tried to run Decimer2 from my script using:
from DECIMER import predict_SMILES
image_path = r"C:\Users........."
SMILES = predict_SMILES(image_path)
print(SMILES)
And I received to following error:
What can I do?
Downloading trained model to C:\Users\SAWACH.data\DECIMER-V2
C:\Users\SAWACH.data\DECIMER-V2\models.zip
done downloading trained model!
Traceback (most recent call last):
File "C:\Users\SAWACH\Desktop\RoseBoom2.2 Supersloth\untitled0.py", line 2, in
from DECIMER import predict_SMILES
File "C:\Users\SAWACH\anaconda3\lib\site-packages\DECIMER_init_.py", line 33, in
from .decimer import predict_SMILES
File "C:\Users\SAWACH\anaconda3\lib\site-packages\DECIMER\decimer.py", line 34, in
config.download_trained_weights(model_url, default_path)
File "C:\Users\SAWACH\anaconda3\lib\site-packages\DECIMER\config.py", line 296, in download_trained_weights
subprocess.run(
File "C:\Users\SAWACH\anaconda3\lib\subprocess.py", line 505, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\SAWACH\anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 108, in init
super(SubprocessPopen, self).init(*args, **kwargs)
File "C:\Users\SAWACH\anaconda3\lib\subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\SAWACH\anaconda3\lib\subprocess.py", line 1420, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] Das System kann die angegebene Datei nicht finden
Decimer1 worked by just installing it with pip.
As seen on #34
Feature Request
GitHub (source)
2.3.0
No response
No response
I would like to change the behavior of the model download (present in https://github.com/Kohulan/DECIMER-Image_Transformer/blob/master/DECIMER/decimer.py#L28-L52) to allow the definition of a global location that can be pre-downloaded and used by multiple users on the same system.
I can make a pull request with that feature. My intervention on the code would be minimal: I would not use any configuration library, I would decide a environment variable name (like DECIMER_MODEL_PATH), check its existence at runtime, and depending on it being defined, use it as the base path or default to the current system. I would also document that variable and usage on the README.
No response
There is no issue, this is a feature requestNo response
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.