-
Notifications
You must be signed in to change notification settings - Fork 1.9k
ASR Benchmark
RTF = 处理语音总时长 / 语音总时长
Aishell-1 test 集作为测试集。
TODO:数据分布。
机器硬件：CPU：Intel(R) Xeon(R) Gold 6271C CPU @ 2.60GHz , GPU: V100-SXM2-32GB
测试脚本：CLI
| Acoustic Model | dedoding_method | ctc_weight | decoding_chunk_size | num_decoding_left_chunk | RTF |
|---|---|---|---|---|---|
| conformer_aishell | attention_rescoring | 0.5 | 16 | -1 | 0.0623 |
| conformer_wenetspeech | attention_rescoring | 0.5 | 16 | -1 | 0.0623 |
| deepspeech2offline_aishell | ctc_prefix_beam_search | - | 1 | - | 0.1787 |
| Acoustic Model | dedoding_method | ctc_weight | decoding_chunk_size | num_decoding_left_chunk | RTF |
|---|---|---|---|---|---|
| conformer_aishell | attention_rescoring | 0.5 | 16 | -1 | 0.3 |
| conformer_wenetspeech | attention_rescoring | 0.5 | 16 | -1 | 0.51539 |
| deepspeech2offline_aishell | ctc_prefix_beam_search | - | 1 | - | 0.3953 |
机器硬件：CPU：Intel(R) Xeon(R) Gold 6271C CPU @ 2.60GHz , GPU: V100-SXM2-32GB
测试脚本：Streaming Server
| Acoustic Model | enigne | dedoding_method | ctc_weight | decoding_chunk_size | num_decoding_left_chunk | RTF |
|---|---|---|---|---|---|---|
| conformer_online_multicn | python | attention_rescoring | 0.5 | 16 | -1 | 0.250782 |
| conformer_wenetspeech | python | attention_rescoring | 0.5 | 16 | -1 | 0.26339 |
| deepspeech2online_aishell | inference | ctc_prefix_beam_search | - | 1 | - | 0.351434 |
| Acoustic Model | Model Size | enigne | dedoding_method | ctc_weight | decoding_chunk_size | num_decoding_left_chunk | RTF | CER |
|---|---|---|---|---|---|---|---|---|
| conformer_online_multicn | - | python | attention_rescoring | 0.5 | 16 | -1 | 1.55706 | - |
| conformer_wenetspeech | - | python | attention_rescoring | 0.5 | 16 | -1 | 0.8712686765174202(utts=40) | - |
| conformer_wenetspeech（reduce attention cache） | - | python | attention_rescoring | 0.5 | 16 | -1 | 0.7847489114800089(utts=40) | - |
| deepspeech2online_aishell | - | infernece | ctc_prefix_beam_search | - | 1 | - | 0.874739 | - |
| deepspeech2online_wenetspeech | 659MB | infernece | ctc_prefix_beam_search | - | 1 | - | 1.9108175171428279(utts=40) | - |
| deepspeech2online_wenetspeech | 659MB | onnx | ctc_prefix_beam_search | - | 1 | - | 0.7067303276583338 (utts=40) | 1.76% |
| deepspeech2online_wenetspeech | 166MB | onnx quant | ctc_prefix_beam_search | - | 1 | - | 0.40434764591598454 (utts=40) | 1.95% |
量化（quant）CER会升高。量化和机器有关，不是所有机器都支持。ONNX quant测试机器指令集支持: Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 arat umip pku ospke avx512_vnni spec_ctrl