Clipper prediction doesn't seem to match the model prediction before deployment

[ucbrise]

Clipper prediction doesn't seem to match the model prediction before deployment

[dcrankshaw]

Your hypothesis is probably correct. If the model doesn't return a prediction in time, you will see a 1 as output. The selection policies are sort of broken right now and we are in the process of removing them in #89 until they can be fixed.

You can try setting the application latency requirement at 150ms and see if you get better results. That shouldn't slow down your predictions because they will be returned before the deadline if they are available.

These are great ideas for debugging. Adding more visibility to the system should be at least partially addressed by CLIPPER-114. Once #89 is completed, we will add metrics to report how many predictions returned in time and how many used the default value. Also, #89 makes the default prediction configurable on a per-application basis which should make it easier to debug as well.

[jegonzal]

Does this mean their container is running slowly? We might want to add a basic dashboard for Clipper to monitor this stuff.

[dcrankshaw]

"Slowly" is somewhat subjective. Their container is running more slowly than the latency objective for their application. The metrics already show this. I agree a monitoring dashboard would be good. I created CLIPPER-142 to track this for the 0.2 release.

[dcrankshaw]

#116 now explicitly marks predictions as being the result of a default prediction or a prediction from the model container as part of the response object. The metrics at http://<clipper-url>:1338/metrics also report the ratio of default predictions.