Comments (5)
Hi @eladamittai,
Under the hood, Model Analyzer uses Perf Analyzer. You can find documentation for passing in input data here: https://github.com/triton-inference-server/client/blob/main/src/c%2B%2B/perf_analyzer/docs/input_data.md
When using Model Analyzer, any args that you want to be passed on to Perf analyzer would go after a --
. For example:
model_analyzer <model analyzer args> -- --input-data /path/to/file
edit: My recollection was wrong. You will need to pass Perf Analyzer args via the perf_analyzer_flags
section of the config yaml file. Let me know if you need help with this.
Now then, you had a number of specific asks. I'm trying to wrap my head around if they are possible.
You need binary data:
Is this actually true? I see you mentioning GRPC, but are you sure it doesn't work by supplying normal fp16 data? I believe everything should work under the hood. Perf Analyzer should convert the data as needed before sending it to triton.
If you do need binary data, then there are a few possible options, although I'm not sure they are all compatible with the rest of your asks.
You want different shaped requests
Go here and search for optional "shape"
. That paragraph and following example show how to provide data with different shapes.
You want a ratio of different shapes
We have stories in our backlog to try to support cases like this, but for now you would need to do it yourself. If you wanted a 2:1 ratio of shape X to shape Y to be sent, then your input data file would need 3 entries: 2 with shape X and 1 with shape Y.
from model_analyzer.
Hi @eladamittai,
Under the hood, Model Analyzer uses Perf Analyzer. You can find documentation for passing in input data here: https://github.com/triton-inference-server/client/blob/main/src/c%2B%2B/perf_analyzer/docs/input_data.md
When using Model Analyzer, any args that you want to be passed on to Perf analyzer would go after a--
. For example:
model_analyzer <model analyzer args> -- --input-data /path/to/file
edit: My recollection was wrong. You will need to pass Perf Analyzer args via theperf_analyzer_flags
section of the config yaml file. Let me know if you need help with this.Now then, you had a number of specific asks. I'm trying to wrap my head around if they are possible.
You need binary data:
Is this actually true? I see you mentioning GRPC, but are you sure it doesn't work by supplying normal fp16 data? I believe everything should work under the hood. Perf Analyzer should convert the data as needed before sending it to triton.If you do need binary data, then there are a few possible options, although I'm not sure they are all compatible with the rest of your asks.
You want different shaped requests
Go here and search foroptional "shape"
. That paragraph and following example show how to provide data with different shapes.You want a ratio of different shapes
We have stories in our backlog to try to support cases like this, but for now you would need to do it yourself. If you wanted a 2:1 ratio of shape X to shape Y to be sent, then your input data file would need 3 entries: 2 with shape X and 1 with shape Y.
Hey, thank you for answering. I checked the perf analyzer documentation, and I managed to send the requests in multiple shapes for a float32 compiled version I have of the model using a json file, but as you can see from this older issue I opened about sending float16 input using json, it's not possible when using grpc. Unless something changed in the later releases of the model/perf analyzer. From your response I didn't understand if I can send multiple shapes using a binary data. I tried to combine the binary dir with the json file, as such:
{ Data: [ Input name: { Content: binary_input_dir Shape: [16000] } ] }
But it didn't work. Is there a different way to send multiple binary input files in multiple shapes?
from model_analyzer.
Hey, is there an answer?
from model_analyzer.
Apologies for the delay. I'm looking into this.
from model_analyzer.
I believe you can use base64 for binary data. Then you can stick to the normal input_data format and provide shapes.
There is an example on this page, although I can't link directly to it. You'll have to scroll down. I've cut and pasted it here:
{
"data":
[
{
"INPUT":
{
"content": {"b64": "/9j/4AAQSkZ(...)"},
"shape": [7964]
}
},
{
"INPUT":
{
"content": {"b64": "/9j/4AAQSkZ(...)"},
"shape": [7964]
}
}
]
}
Using that as a basis, you could provide 3 inputs, 2 of one shape and 1 of another, to accomplish the goal of a 2:1 ratio of input shapes.
from model_analyzer.
Related Issues (20)
- triton_server_flags does not allow to add keys: response-cache-byte-size or cache-config HOT 4
- Input data doesn't work for fp16 HOT 2
- model_analyzer.model_analyzer_exceptions.TritonModelAnalyzerException: The first 2 attempts to acquire measurements have failed. Please examine the Tritonserver/PA error logs to determine what has gone wrong. HOT 3
- onnx runtime error 2: not enough space - while running model-analyzer HOT 4
- Encountered an error: "Cannot specify both concurrency and request rate as model parameters." while not specifying `concurrency` at all.` HOT 4
- Model Analyzer GPU Memory Usage Differences HOT 5
- use of time intervals in model analyzer HOT 2
- [Bug] failed to load 'maxvit_rmlp_base', failed to poll from model repository (model_output path is the same) HOT 4
- How can i use perf_analyzer with python backend ? HOT 3
- Ensemble Model search error when specify perf_analyzer_flags.input-data HOT 3
- Model Analyzer - NO Gpu Metrics collection HOT 2
- Model Analyzer - NO Gpu Metrics collection
- Understanding GPU utilization HOT 5
- --triton-launch-mode=remote HOT 4
- custom samples HOT 1
- model analyser stuck HOT 2
- detailed report not generating HOT 2
- Time intervals in model analyzer
- UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 0: invalid start byte HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from model_analyzer.