Comments (12)
Hi @odinsbane,
we are happy to help with this, but would need a bit more information:
- Can you please post the exact error message you are getting?
- Can you try to put together a minimal example with the code and weights to reproduce this error?
from core-bioimage-io-python.
Thank you for the quick response. I can easily put together an example from what I have. Maybe you can help me before I start uploading things.
I have a tensorflow model. There is one input and three outputs. I made a version with one output to get it to work.
I don't mind changing the format, either .h5 or directory.
This is the code I used to build the model.
#!/usr/bin/env python3
from bioimageio.core.build_spec import build_model
build_model(
# the weight file and the type of the weights
weight_uri="simple",
weight_type="tensorflow_saved_model_bundle",
# the test input and output data as well as the description of the tensors
# these are passed as list because we support multiple inputs / outputs per model
test_inputs=["junk.npy"],
test_outputs=["output.npy"],
input_axes=["bczyx"],
output_axes=["bczyx"],
# where to save the model zip, how to call the model and a short description of it
output_path="simple-sample.zip",
name="OrganoidDnaModel",
description="For segemnting organoid dna in 3D",
# additional metadata about authors, licenses, citation etc.
authors=[{"name": "Odinsbane"}],
license="CC-BY-4.0",
documentation="organoids.md",
tags=["nucleus-segmentation"], # the tags are used to make models more findable on the website
cite=[{"text": "Smith et al.", "doi": "doi:10.1002/xyzacab123"}],
)
This is the output.
python build_package.py
/home/username/Desktop/bioimage-prep/bioimage-io-env/lib/python3.9/site-packages/bioimageio/spec/shared/_resolve_source.py:433: CacheWarning: found cached /tmp/username/bioimageio_cache/https/raw.githubusercontent.com/bioimage-io/bioimage.io/main/site.config.json. Skipping download of https://raw.githubusercontent.com/bioimage-io/bioimage.io/main/site.config.json.
warnings.warn(f"found cached {local_path}. Skipping download of {uri}.", category=CacheWarning)
/home/username/Desktop/bioimage-prep/bioimage-io-env/lib/python3.9/site-packages/bioimageio/spec/shared/_resolve_source.py:433: CacheWarning: found cached /tmp/username/bioimageio_cache/https/bioimage-io.github.io/collection-bioimage-io/collection.json. Skipping download of https://bioimage-io.github.io/collection-bioimage-io/collection.json.
warnings.warn(f"found cached {local_path}. Skipping download of {uri}.", category=CacheWarning)
Traceback (most recent call last):
File "/home/username/Desktop/bioimage-prep/basic/build_package.py", line 5, in
build_model(
File "/home/username/Desktop/bioimage-prep/bioimage-io-env/lib/python3.9/site-packages/bioimageio/core/build_spec/build_model.py", line 813, in build_model
weights, tmp_archtecture = _get_weights(
File "/home/username/Desktop/bioimage-prep/bioimage-io-env/lib/python3.9/site-packages/bioimageio/core/build_spec/build_model.py", line 103, in _get_weights
weight_hash = _get_hash(weight_path)
File "/home/username/Desktop/bioimage-prep/bioimage-io-env/lib/python3.9/site-packages/bioimageio/core/build_spec/build_model.py", line 32, in _get_hash
with open(path, "rb") as f:
IsADirectoryError: [Errno 21] Is a directory: '/home/username/Desktop/bioimage-prep/basic/simple'
If I try using a .h5 file (same mode, I load it in python, then save it with save_format="h5") I get a different error. What do you think is the best way to proceed?
from core-bioimage-io-python.
I tried a version of the build script using an h5 version of the tensorflow model, and everything seems to be working. It created a zip file and I am trying to upload the zip file to bioimage.io (or zenodo via bioimage.io) which suites my purpose well. So maybe the sole purpose of this issue is, why doesn't the directory format work?
from core-bioimage-io-python.
I tried a version of the build script using an h5 version of the tensorflow model, and everything seems to be working.
Ok, note that in this case you need to set the weight format to keras_hdf5
.
So maybe the sole purpose of this issue is, why doesn't the directory format work?
I think that we are expecting a zip file of the weights here. It should work if you zip the folder with the weights and then pass the filepath of the zipped file to the build_model
function.
from core-bioimage-io-python.
That worked! I think it worked because it created the bioimage-model zip file.
It could be helpful if the documentation said that the directory needs to be zipped, that they names in the weights spec page are the values that should passed to the weights_spec parameter. The "weights_spec" appears to be generated documentation (similar to Tensorflow doc's?)
from core-bioimage-io-python.
That worked! I think it worked because it created the bioimage-model zip file.
Ok, thanks for checking!
It could be helpful if the documentation said that the directory needs to be zipped, that they names in the weights spec page are the values that should passed to the weights_spec parameter. The "weights_spec" appears to be generated documentation (similar to Tensorflow doc's?)
I agree, it would be good to update this. (And indeed the weight spec documentation is autogenerated.) I am not so familiar with the doc generation procedure, maybe @FynnBe could have a look.
from core-bioimage-io-python.
I can close this issue, but I just want to make sure. Is there another way to test the model?
For me I had to work through fixing the data, then fixing the model, then fixing the data. Did you add the h5 support recently?
from core-bioimage-io-python.
I can close this issue, but I just want to make sure.
Let's leave it open until the documentation is updated.
Is there another way to test the model?
Once you have created the model you can test it via the bioimageio test-model
command. See also https://github.com/bioimage-io/core-bioimage-io-python#command-line.
Did you add the h5 support recently?
No, that has been there for a while.
from core-bioimage-io-python.
Okay, I finally got around to using the bioimageio test-model! If I use the h5 version of the saved tensorflow model, then the tests pass. When I use the .zip file of the folder saved model I get an error.
error: ('SavedModel file does not exist at: '
'/tmp/username/bioimageio_cache/extracted_packages/f5854334514ff19d36267b690c03df47182881e321ccfdb78b62eebe6f63d3e7/drosophila-crb-d5/{saved_model.pbtxt|saved_model.pb}')
from core-bioimage-io-python.
@odinsbane could you share the created zip folder, any additional files that should be included in it, but are not, and your updated build_model
call? With that we'll take a closer look at the issue.
from core-bioimage-io-python.
It's a large file, so I am looking at uploading it or making an example. I think though, looking at the temp folder the problem is the way the file has been zipped.
- save model as folder "drosophila-crb-d5"
- zip the folder:
zip drosophila-crb-d5.zip drosophila-crb-d5
Maybe I need to zip the contents and remove the top-level folder?
from core-bioimage-io-python.
Maybe I need to zip the contents and remove the top-level folder?
that's probably it. 👍
I can't improve this right now, but here are some notes from looking into it briefly, if someone else wants to tackle it already:
We should improve the TF Moder adapter here:
here weight_file
is a folder in which we should look for the expected saved_model.pb file and possibly add the nested folder to its path.
We potentially unzip nested zip files (calling self.require_unzipped twice, which we probably shouldn't)
We also implicitly assume the use of tensorflow 1:
Maybe https://www.tensorflow.org/api_docs/python/tf/saved_model/save is a relevant reference for updating this part.
from core-bioimage-io-python.
Related Issues (20)
- validate (citation) dois
- VALUE is not a is not a valid RESUNIT HOT 12
- Better default io names in build_model() HOT 12
- Implement sha256 check for the model
- MONAI translation to BioImage format HOT 1
- Image normalization causing test_model test failure HOT 3
- Provide our CI testing environment as docker containers
- Reproducibility test for model creation fails HOT 3
- Hackathon 2023/10 notes HOT 2
- programmatic access for datasets
- Add support for release candidates and automatic mail updates
- developer documentation
- Investigate integration of aicsimageio
- Add/improve weight converters
- Loading model resource gives assertion error HOT 2
- Link in example broken HOT 1
- Running `test_model` raises TypeError: list indices must be integers or slices, not str HOT 1
- Improve weights conversion to ONNX HOT 1
- bug when `add_deepimagej_config=True` in `build_model` HOT 5
- Error when importing HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from core-bioimage-io-python.