Comments (11)
I think it's related to the default max body size: https://docs.rs/axum/latest/axum/extract/struct.DefaultBodyLimit.html#method.max
By default, it's 2MB. This is usually enough for a normal web application. But may not be sufficient for an ML image process service. I think we can increase it to 5MB.
from mosec.
I re-check the code and I found that the size limit is 10MiB which should be sufficient.
Line 38 in 7fbdca0
As for your test code, you should specify the dtype
since numpy will use float64
by default. Image data can use np.uint8
.
Feel free to re-open this issue if you still have any questions.
from mosec.
Indeed, a dtype affects on size but it dosen't solve my issue in sclae, because when I want to send more then one image / numpy array I still face this limitation.
For example inputs = np.zeros((10, 3, 1024, 1024), np.uint8)
It'd be great if you add an ability to change that parameter when running a server.
from mosec.
BTW, how can I re-open the issue? I don't see a "Reopen" button under the comment field.
from mosec.
Indeed, a dtype affects on size but it dosen't solve my issue in sclae, because when I want to send more then one image / numpy array I still face this limitation. For example
inputs = np.zeros((10, 3, 1024, 1024), np.uint8)
It'd be great if you add an ability to change that parameter when running a server.
You should send multiple images (async or multi-thread) instead of one large batch of images to fully utilize dynamic batching. 10MiB is actually very large for most of the use cases.
from mosec.
@kemingy got you, thanks
from mosec.
I noticed that I get the "request body is too large" error even when actual file size is smaller than 10MB.
It's kind of blocker for me now, because I can't work with 50% of my images. I hope we will figure out why this happens.
Please, try to reproduce my issue using my code and data.
server.py
import pickle
from predictor import Predictor, BboxPredictor
from mosec import Server, Worker, Runtime
from mosec.errors import DecodingError, EncodingError
from typing import Any
import numpy as np
BATCH_SIZE = 2
class PickleMixin:
def serialize(self, data: np.ndarray) -> bytes:
try:
data_bytes = pickle.dumps(data, fix_imports=False, protocol=pickle.HIGHEST_PROTOCOL)
except Exception as err:
raise EncodingError from err
return data_bytes # type: ignore
def deserialize(self, data: bytearray) -> Any:
try:
data_msg = pickle.loads(data, fix_imports=False)
except Exception as err:
raise DecodingError from err
return data_msg
class HeatmapWorker(Predictor, PickleMixin, Worker):
def __init__(self) -> None:
super().__init__(
model_path="model.onnx",
batch_size=BATCH_SIZE
)
class BboxWorker(BboxPredictor, PickleMixin, Worker):
def __init__(self) -> None:
super().__init__()
if __name__ == "__main__":
server = Server()
heatmap_runtime = Runtime(
HeatmapWorker,
num=4,
max_batch_size=BATCH_SIZE,
timeout=10
)
bboxes_runtime = Runtime(
BboxWorker,
num=4,
max_batch_size=BATCH_SIZE,
timeout=10
)
server.register_runtime(
{
"/heatmap": [heatmap_runtime],
"/bboxes": [bboxes_runtime],
}
)
server.run()
client.py
from http import HTTPStatus
import os
import cv2
import httpx
import pickle
def pickle_serialize(data) -> bytes:
data_bytes = pickle.dumps(data, fix_imports=False, protocol=pickle.HIGHEST_PROTOCOL)
return data_bytes # type: ignore
def pickle_deserialize(data):
return pickle.loads(data, fix_imports=False)
img_path = "path/to/my/image"
print(f"Image size: {os.path.getsize(img_path) / (1024 * 1024):.4}MB")
img = cv2.imread(img_path)
heatmap_prediction = httpx.post(
"http://127.0.0.1:8000/heatmap",
content=pickle_serialize(img),
timeout=None,
)
if heatmap_prediction.status_code != HTTPStatus.OK:
raise Exception(heatmap_prediction.content)
heatmap = pickle_deserialize(heatmap_prediction.content)
model
images
request_body_too_large_images.zip
from mosec.
This is because the images are JPEG format which means they are compressed. The compress ratio is about 10:1 to 20:1. If you read it as numpy array then it's about 10x larger than the JPEG format.
I suggest you send the bytes of the JPEG and de-compress it on the server side.
from mosec.
@kemingy, unfortunately, your suggestion doesn't fit to my real application because I have multiple modules which precede. where I preprocess image. So I just cannot operate JPEG bytes. Of course, I could save the processed images to disk and load them as JPEG bytes, but this looks like a workaround. I don't want to go that way.
I really ask you to add the ability to specify the --max_req_size
parameter like you did for --timeout
when starting a server.
from mosec.
from mosec.
@kemingy I am just asking to add such parameter as optional for guys (like me) who need to work with larger objects. In this case, the responsibility for changing this parameter other than the default (10 MB) rests with the user. So I don't ask to increase max body size by default and keep it for each user but give the ability to change it at one's own risk.
I would make these changes to the code myself, but I'm not a Rust specialist so it might take me a long time.
Hope for understanding.
from mosec.
Related Issues (20)
- feat: how to reload like fastapi or flask HOT 1
- feat: support adaptive batching HOT 2
- feat: support iteration level scheduling HOT 1
- feat: add swagger for the openapi
- feat(docs): add advanced user guide for custom mixin
- unload model automatic HOT 3
- feat: support server side response cache HOT 5
- feat: Have a base image
- feat: Migration guide for triton python backend HOT 1
- Is mosec available on windows? HOT 1
- bug: log is not formatted HOT 1
- feat: support for llama(.cpp) HOT 2
- feat: add route-level metrics
- feat: support dynamic configuration for dynamic batching without service restart HOT 1
- bug: Issue running `echo.py`: Thread 'main' panicked at src/main.rs:118:46: local time offset: IndeterminateOffset HOT 4
- bug: 'site-packages/mosec/bin/mosec' not found on macOS M1 HOT 3
- feat: CORS support ? HOT 1
- bug: Exception(b'inference timeout') HOT 2
- bug: rerank example got an error HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mosec.