Http error: 404

[romainGuiet]

Http error: 404

[ksugar]

Hi @romainGuiet , thank you for trying it! The URL needs to be the complete endpoint URL including /sam/.
Could you try putting the following URL?

http://127.0.0.1:8000/sam/

[ksugar]

@romainGuiet please try qupath-extension-sam-0.2.0.jar.

@petebankhead RenderedImageServer works as I expected, thank you again for your help!

[romainGuiet]

it works on the RGB transmission ! thanks 🥳

but on my fluorescent image with 5 channels I get an error

INFO:     127.0.0.1:64372 - "POST /sam/ HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "C:\Users\guiet\.conda\envs\samapi\Lib\site-packages\uvicorn\protocols\http\httptools_impl.py", line 435, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\guiet\.conda\envs\samapi\Lib\site-packages\uvicorn\middleware\proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\guiet\.conda\envs\samapi\Lib\site-packages\fastapi\applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "C:\Users\guiet\.conda\envs\samapi\Lib\site-packages\starlette\applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "C:\Users\guiet\.conda\envs\samapi\Lib\site-packages\starlette\middleware\errors.py", line 184, in __call__
    raise exc
  File "C:\Users\guiet\.conda\envs\samapi\Lib\site-packages\starlette\middleware\errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "C:\Users\guiet\.conda\envs\samapi\Lib\site-packages\starlette\middleware\exceptions.py", line 79, in __call__
    raise exc
  File "C:\Users\guiet\.conda\envs\samapi\Lib\site-packages\starlette\middleware\exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "C:\Users\guiet\.conda\envs\samapi\Lib\site-packages\fastapi\middleware\asyncexitstack.py", line 21, in __call__
    raise e
  File "C:\Users\guiet\.conda\envs\samapi\Lib\site-packages\fastapi\middleware\asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "C:\Users\guiet\.conda\envs\samapi\Lib\site-packages\starlette\routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "C:\Users\guiet\.conda\envs\samapi\Lib\site-packages\starlette\routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "C:\Users\guiet\.conda\envs\samapi\Lib\site-packages\starlette\routing.py", line 66, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\guiet\.conda\envs\samapi\Lib\site-packages\fastapi\routing.py", line 237, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\guiet\.conda\envs\samapi\Lib\site-packages\fastapi\routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\guiet\.conda\envs\samapi\Lib\site-packages\samapi\main.py", line 80, in predict_sam
    image = decode_image(body.b64img)
            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\guiet\.conda\envs\samapi\Lib\site-packages\samapi\utils.py", line 13, in decode_image
    return np.array(Image.open(io.BytesIO(base64.b64decode(b64data))))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\guiet\.conda\envs\samapi\Lib\site-packages\PIL\Image.py", line 3298, in open
    raise UnidentifiedImageError(msg)
PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x000001B1DCD2FFB0>

I tried to reduce the number of displayed channels to 3 but without any positve effect 😅

Best

Romain

[petebankhead]

If using SAM as an 'annotation assistant', then I think RenderedImageServer and RGB makes most sense - QuPath's wand tool also uses the viewer rendering. It means that, for multiple channel images, the user can toggle on and off channels according to what they want to be visible, which could be really useful.

For training StarDist/CellPose, then passing the raw pixels could be useful - e.g. if you're able to train a single-channel StarDist model (like the DSB2018 model from StarDist's original creators). In that case, you might use TransformedServerBuilder along with the method I posted above to extract one or more channels and convert them to tiff bytes.

[ksugar]

Thanks @petebankhead , both totally make sense. I will use RenderedImageServer for the SAM module.
About the modules for training StarDist/CellPoseI, only grayscale images are supported for now, but I will update them using TransformedServerBuilder as you suggested.
I really appreciate all your help!

[ksugar]

That's great it works for your RGB data! 👍
At the moment, it assumes grayscale or RGB images. I will try to find a workaround.
@petebankhead do you have a readRegion method that returns an RGB BufferedImage as shown in QuPath?

[petebankhead]

Hi readRegion will return whichever kind of BufferedImage is required for the bit-depth and channels - so it's RGB where possible, but could be something else.

If it's something else, then there's a strong chance ImageIO won't be able to write the image.

I understand you're using base64 in the end with PNG. One way to get a useful byte array to encode is to convert to ImageJ TIFF bytes, e.g.

     public static byte[] getImageBytes(ImageServer<BufferedImage> server, RegionRequest request) throws IOException {
		if (server.isRGB()) {
                     // Do whatever you usually do (PNG should be ok)
                     var img = server.readRegion(request);
                     return // ImageIO code to PNG
                }
         	var imp = IJTools.convertToImagePlus(server, request).getImage();
		return new FileSaver(imp).serialize();
	}

where FileSaver is a class in ImageJ. I haven't completed or properly tested this code, but I think the idea works, and the (uncompressed) TIFF can be read fine in Python.

Alternatively, if you really want an RGB version (applying whatever settings are used in the viewer), then you can create a RenderedImageServer and get the pixels from that. Although I prefer the solution that sends the raw pixel values, without the rendering to RGB required (at least if it's possible to handle non-RGB meaningfully on the other side).

[ksugar]

Thank you @petebankhead for your kind reply and nice suggestion.

Because all pre-trained models provided in the original SAM repository require an input image with three channels, we need to somehow convert a multi-channel image to a three-channel image in either Java or Python.

https://github.com/facebookresearch/segment-anything/blob/6fdee8f2727f4506cfbbe553e23b895e27956588/segment_anything/build_sam.py#L67
https://github.com/facebookresearch/segment-anything/blob/6fdee8f2727f4506cfbbe553e23b895e27956588/segment_anything/modeling/image_encoder.py#L22

I think using RenderedImageServer is easy to achieve it.
Assuming that the pre-trained SAM model is good at working with the objects that human eyes recognize, using the rendering settings on QuPath might make sense.

If you have a good approach to converting a multi-channel raw image into a three-channel image, please let me know.
Thanks.