Triton Server - support of Unified Conformer model fails

[pjhool]

Triton Server - support of Unified Conformer model fails

[Slyne]

Could you attach your config.pbtxt from encoder and feature_extractor?
Looks like the configuration file is not quite right.

[pjhool]

attatched config.pbtxt files for encoder, feature_extractor.

1. encoder config.pbtxt
   config.pbtxt.txt

name: "encoder"
backend: "onnxruntime"
default_model_filename: "encoder.onnx"

max_batch_size: 512

sequence_batching{
max_sequence_idle_microseconds: 5000000
oldest {
max_candidate_sequences: 1024
max_queue_delay_microseconds: 5000
}
control_input [
]
state [
{
input_name: "offset"
output_name: "r_offset"
data_type: TYPE_INT64
dims: [ 1 ]
initial_state: {
data_type: TYPE_INT64
dims: [ 1 ]
zero_data: true
name: "initial state"
}
},
{
input_name: "att_cache"
output_name: "r_att_cache"
data_type: TYPE_FP16
dims: [ #num_layers, #num_head, #cache_size, #att_cache_output_size ]
initial_state: {
data_type: TYPE_FP16
dims: [ #num_layers, #num_head, #cache_size, #att_cache_output_size ]
zero_data: true
name: "initial state"
}
},
{
input_name: "cnn_cache"
output_name: "r_cnn_cache"
data_type: TYPE_FP16
dims: [#num_layers, 256, #cnn_module_cache]
initial_state: {
data_type: TYPE_FP16
dims: [#num_layers, 256, #cnn_module_cache]
zero_data: true
name: "initial state"
}
},
{
input_name: "cache_mask"
output_name: "r_cache_mask"
data_type: TYPE_FP16
dims: [1, #cache_size]
initial_state: {
data_type: TYPE_FP16
dims: [1, #cache_size]
zero_data: true
name: "initial state"
}
}
]
}
input [
{
name: "chunk_xs"
data_type: TYPE_FP16
dims: [#decoding_window, 80]
},
{
name: "chunk_lens"
data_type: TYPE_INT32
dims: [ 1 ]
reshape: { shape: [] }
}
]
output [
{
name: "log_probs"
data_type: TYPE_FP16
dims: [-1, 10] # [-1, beam_size]
},
{
name: "log_probs_idx"
data_type: TYPE_INT64
dims: [-1, 10] # [-1, beam_size]
},
{
name: "chunk_out"
data_type: TYPE_FP16
dims: [-1, -1]
},
{
name: "chunk_out_lens"
data_type: TYPE_INT32
dims: [1]
reshape: { shape: [] }
}
]
instance_group [
{
count: 2
kind: KIND_GPU
}
]

config_template.pbtxt.txt
config_template.pbtxt.txt

config_template2.pbtxt.txt
config_template2.pbtxt.txt

2. feature_extractor config.pbtxt
   config.pbtxt.txt

name: "feature_extractor"
backend: "python"
max_batch_size: 512

parameters [
{
key: "frame_length_ms",
value: { string_value: "#frame_length" }
},
{
key: "frame_shift_ms"
value: { string_value: "#frame_shift" }
},
{
key: "sample_rate"
value: { string_value: "#sample_rate" }
},
{
key: "chunk_size_s",
value: { string_value: "#chunk_size_in_seconds" }
}
]
sequence_batching{
max_sequence_idle_microseconds: 5000000
oldest {
max_candidate_sequences: 512
preferred_batch_size: [ 32, 64, 128, 256]
}
control_input [
{
name: "START",
control [
{
kind: CONTROL_SEQUENCE_START
fp32_false_true: [0, 1]
}
]
},
{
name: "READY"
control [
{
kind: CONTROL_SEQUENCE_READY
fp32_false_true: [0, 1]
}
]
},
{
name: "CORRID",
control [
{
kind: CONTROL_SEQUENCE_CORRID
data_type: TYPE_UINT64
}
]
},
{
name: "END",
control [
{
kind: CONTROL_SEQUENCE_END
fp32_false_true: [0, 1]
}
]
}
]
}
input [
{
name: "wav"
data_type: TYPE_FP32
dims: [-1]
},
{
name: "wav_lens"
data_type: TYPE_INT32
dims: [1]
}
]
output [
{
name: "speech"
data_type: TYPE_#DTYPE # FP32
dims: [#decoding_window, #num_mel_bins]
},
{
name: "speech_lengths"
data_type: TYPE_INT32
dims: [1]
}
]
instance_group [
{
count: 2
kind: KIND_GPU
}
]

model.py
model.py.txt

config_template.pbtxt.txt
config_template.pbtxt.txt

[Slyne]

It looks like this step is not successfully completed. python3 wenet/bin/export_onnx_gpu.py --config=$model_dir/train.yaml --checkpoint=$model_dir/final.pt --cmvn_file=$model_dir/global_cmvn --ctc_weight=0.5 --output_onnx_dir=$onnx_model_dir --fp16 cp $model_dir/words.txt $model_dir/train.yaml $onnx_model_dir/ Could you check this again and ensure all the fields with #xxx being filled.

…

On Mon, May 13, 2024 at 19:05 JungHwan Park ***@***.***> wrote: Reopened #2525 <#2525>. — Reply to this email directly, view it on GitHub <#2525 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABP63VFRNGNTZCZCX53AKOLZCFWPZAVCNFSM6AAAAABHTZO2COVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJSG44TIOJTGM2DINA> . You are receiving this because you commented.Message ID: ***@***.***>

[pjhool]

run the same command as above
-- python3 -m wenet.bin.export_onnx_gpu --config $EXP/train.yaml --checkpoint $EXP/final_10.pt --cmvn_file=$EXP/global_cmvn --ctc_weight=0.5 --output_onnx_dir $onnx_dir --fp16

[Slyne]

run the same command as above -- python3 -m wenet.bin.export_onnx_gpu --config $EXP/train.yaml --checkpoint $EXP/final_10.pt --cmvn_file=$EXP/global_cmvn --ctc_weight=0.5 --output_onnx_dir $onnx_dir --fp16

Please also add --streaming if you are using streaming model.

[pjhool]

It works!!