Behavior of published model

[allegroai]

Behavior of published model

[bmartinn]

Hi @okyspace

I tested, when i first run my training but did not publish. When i start triton, this version is still avail in triton for inference. Is this correct?

You absolutely correct, here should be only_published=True
Let me make sure we fix that (or at least add a flag to control it)

The triton polling indicates no change, thus did not pull the models over.

How did you register the endpoint, with project/mode-name ? Any chance you can attach a log?

[okyspace]

Hi @bmartinn

1. This is how I create my serving service and then register an endpoint.

clearml-serving triton --project "serving_engine_svs" --name "serving_engine_svs"

clearml-serving triton --project "serving_engine_svs" --name "serving_engine_svs" --endpoint "mnist_endpoint"  --model-project "mnist_example" --model-name "mnist_task - mnist_model"

2. Any idea where should I capture the useful logs? The triton logs basically shows this:

Skipping update, nothing changed

3. Adding on more observations from related issue:
   Once I add the model name to config file, the different versions (draft + published) are copied over to triton.
   And this happens every polling iteration. But the published state did not take effect as well, i.e. last published version is still not available for inference.

[bmartinn]

Once I add the model name to config file

What do you mean by that? Are you referring to the config.pbtxt ?

... last published version is still not available for inference.

I think I was able to confirm it. Can you test if the following works for you:

clearml-serving triton --model-tags deploy_me  --project "serving_engine_svs" --name "serving_engine_svs" --endpoint "mnist_endpoint"  --model-project "mnist_example" --model-name "mnist_task - mnist_model"

Then manually add the tag "deploy_me" to the selected models
(basically it should only get models with the selected project/name and the specified tag)

[qwaxys]

Closing this issue as inactive, feel free to open a new issue and link to this one.