JSON response from prediction

[ucbrise]

JSON response from prediction

[dcrankshaw]

Ahh I closed the issue too early. We ended up splitting some of the work to address this into CLIPPER-143 which hasn't been completed yet.

[dcrankshaw]

You are correct. This is currently the case. We have an open JIRA issue CLIPPER-107 to address this.

It's a fairly high priority for us. I expect we will get to it sometime this week.

[rmdort]

Cool. Do you have an ETA for this feature? We are eager to use clipper in our product :)

[dcrankshaw]

JSON requests are currently supported. You can specify containers that expect strings as input, which can of course be JSON strings. And yes ultimately all inputs come to Clipper as JSON through the REST interface.

The issue under discussion was being able to return arbitrary JSON from containers, which is not currently supported but will be shortly.

[Corey-Zumar]

This is now under review: #134

[Corey-Zumar]

@frank2wang87 That's presently the case. y_hat_ will be a string (json-formatted or otherwise) after #134 is merged.

[dcrankshaw]

Fixed by #116

[rmdort]

How does this work? I still cant return a string from my container

def predict_strings(self, inputs):
       	print ('received input', inputs)
       	return 'Hello'

It seems like the output from query_frontend.hpp is casted as float

Line 289

clipper::json::add_double(json_response, PREDICTION_RESPONSE_KEY_OUTPUT,
                              query_response.output_.y_hat_);

Am i doing something wrong? I created an app which accepts strings and i want to output a custom json.

[AirAI]

It seems that the image should be translated to a list to generate a json request.(Line 123 in cifar_utils.py)

req_json = json.dumps({'uid': uid, 'input': list(x)})

The list is then translated to original image if the model container receive the true image as input. It is low efficiency when disposing large image or video. Correct me if I am wrong.

Does the clipper support REST input? i.e, does clipper receive original image(or video, or texture) as input and transfer it to the docker container?

[frank2wang87]

When will this feature be published? We are eager to use it :)

[dcrankshaw]

@Corey-Zumar what's the status of this?

[Corey-Zumar]

@dcrankshaw In progress - continuing to work on it tonight and tomorrow

[frank2wang87]

L389 in query_frontend.hpp
clipper::json::add_double(json_response, PREDICTION_RESPONSE_KEY_OUTPUT, query_response.output_.y_hat_);

Now it still returns a double output(double y_hat_). Am I right ?

[frank2wang87]

Cool.

[AirAI]

Has this issue been solved yet? When will the 0.1 alpha be released? We are looking forward to it.

[Corey-Zumar]

Waiting on #134 to be merged. IMO, the change is stable. You can merge the branch associated with #134 into your copy of the repo and start experimenting with the functionality.

[rmdort]

Any ETA on when this is going to be merged?

[dcrankshaw]

Just merged. I was out of town traveling for a few days which caused a delay.

[rmdort]

I am still not able to send a simple JSON using predict_strings.

The following code

def predict_strings(self, inputs):
    return json.dumps(['foo', {'bar': ('baz')}])

gives the output

{"query_id":13,"output":"\u0001","default":false}

Am i doing anything wrong here?

[dcrankshaw]

Hmm let me dig into this. I'll report back soon.

[dcrankshaw]

Ahh the problem is how you are formulating the JSON response. The input to the predict function is a list of strings and Clipper expects you to return a list of JSON responses, one for each input. Try this:

def predict_strings(self, inputs):
    outputs = []
    for i in inputs:
        outputs.append(json.dumps(['foo', {'bar': ('baz')}]))
    return outputs

I just added some more explicit type checking in #151 to make this easier to catch.