Failed to test the tutorial in the clipper source code version

[ucbrise]

Failed to test the tutorial in the clipper source code version

[dcrankshaw]

I believe this is happening because you haven't downloaded the TensorFlow model. Are there any files in the examples/tutorial/tf_cifar_model directory?

The model deployment itself was successful because the Clipper manager library doesn't know that you were missing files and it was able to start the container fine. But when the container started, it was missing the model files it was expecting so it immediately exited.

Here are the instructions from the tutorial (tutorial/tutorial_part_two.ipynb) on how to download the model:

There is a pre-trained TensorFlow model stored in the repo using git-lfs. Once you install git-lfs, you can download the model with the command git lfs pull.

Let me know if that solves the problem.

[frank2wang87]

@ dcrankshaw
hi, dcrankshaw. I checked the examples/tutorial/tf_cifar_model directory and I could find the tensorflow model. In fact, the tensorflow model has been successfully copied to /tmp/clipper-models/tf_cifar/1/ tf_cifar_model directory.

When I modified the CLIPPER_IP variable to the explicit IP address(192.168.0.3) in the add_container function of clipper_manager.py, this problem was solved. Why?

add_container_cmd = (
"docker run -d --network={nw} -v {path}:/model:ro "
"-e "CLIPPER_MODEL_NAME={mn}" -e "CLIPPER_MODEL_VERSION={mv}" "
#"-e "CLIPPER_IP=query_frontend" -e "CLIPPER_INPUT_TYPE={mip}" "
"-e "CLIPPER_IP=192.168.0.3" -e "CLIPPER_INPUT_TYPE={mip}" "
"{image}".format(
path=model_data_path,
nw=DOCKER_NW,
image=image_name,
mn=model_name,
mv=model_version,
mip=model_input_type))

Unfortunately, I came across another problem both in clipper docker version and source code version. When I deployed a new container which predicted more than two classes(e.g., 1000 classes for imagenet dataset, the prediction result is 0-999)，the container could predict the right class(i.e., class number 100), while the clipper always returned class number 1.
In other words, the input data path(REST prediction request->clipper->RPC->model container)worked well,but there was some problem in the output data path(model container-> RPC->clipper -> REST response).

I read #118 and set an long latency requirements(e.g, 150ms), but it remained the same problem.

`

[dcrankshaw]

The current selection policies are kind of broken and only support binary classification right now. We are removing them in #89 which will fix this problem.

[frank2wang87]

@dcrankshaw. Does the clipper current support any other AI application(e.g., object detection, semantic segmentation) besides binary image classification?

[dcrankshaw]

Conceptually, Clipper supports any type of AI application. In practice, models are currently constrained to returning a single float per prediction. We are in the process of relaxing this constraint to allow models to return arbitrary strings (with the recommendation that they return JSON). This feature is tracked in CLIPPER-143.

[frank2wang87]

@dcrankshaw. Sounds Great! When will the new version clipper be published to support to return arbitrary strings?

[dcrankshaw]

I'm not sure yet. @Corey-Zumar is going to spend some time on it this week, so hopefully early next week. For sure in the next week or two.

[chakpongchung]

@dcrankshaw

" In practice, models are currently constrained to returning a single float per prediction."

If I want to add support for object detection with tensorflow, then one single float return is not enough. Ideally it can return a json file. Where is this constraint from?

For sending images as input to the object detection, can Clipper support images uploading? So far I am thinking using gRPC to upload the image by chunking, not sure whether this is a recommended approach. The default message size for protobuf is 64kb only but the image is from 1MB to 6MB. Or I should use other ways to send in the image to be detected?

I would like to add this support with some guidance as we have this business need. In the end we want to have the classification and detection module ready to provide microservices for downstream application. We have our own trained model in tensorflow and caffe format.

[dcrankshaw]

Hi @chakpongchung. It looks like that documentation is out of date. Where are you quoting from?

In fact, models must now return predictions as strings. These are commonly JSON strings, but aren't constrained to be valid JSON. The hope is that this gives users a lot of flexibility in terms of what their models product. So this should already meet your needs.

As far as sending the images to Clipper, currently we support predictions via a REST interface, which supports both integer/float vectors (if you already have your images as pixels), or base64 encoded byte strings (if your images are JPEGs or some other binary format). We don't have a GRPC frontend right now, although we recently implemented a prototype GRPC frontend for some performance evaluation (if you're curious, here's the code). We actually didn't find GRPC to be all that much faster than the REST frontend in our benchmarks, which included uploading ~1MB images for prediction.

I would recommend starting by base64 encoding your messages and uploading them using the REST interface. If that ends up being too slow for your purposes, I'd be happy to guide you through implementing a GRPC frontend based on the prototype linked above. We actually already have #238 to track this, although with no immediate plans to work on it.

Let me know if you have more questions.

[chakpongchung]

@dcrankshaw Thanks for the feedback!

From the paper and here it seems you are using base64 or int/float vectors for file transfer first, waiting for batch construction and then send the query using RPC? The images are pre-uploaded first before the batch size is reached or uploaded as arguments along with the RPC call after the batch size is reached ?

For detection, my model is trained using tensorflow. It seems detection template is not implemented in Clipper yet. If I need to implement it for business need, could you point out where should get started?

A brief performance summary, on 1080TI, my detection code runs at most 15fps since it is cascading two detection models A and B together. If the things are not detected from A, there is no need to use the result from A as input to B. The result from detection model A will also go as input to classification model C, which runs faster than 30fps. The current goal is to prioritize the speed of detection A since the pipeline is expected to run for a video stream.

A->B
A ->C

What would be the right strategy to implement that in Clipper?