Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Correct flan-t5 output size #451

Merged
merged 6 commits into from
May 28, 2024
Merged

Correct flan-t5 output size #451

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
9b05747
correct flan-t5 output size
wgzintel May 17, 2024
e02fe24
Merge branch 'master' into guozhong/correct_flan-t5_output_size
peterchen-intel May 22, 2024
141a502
Merge branch 'master' of https://github.com/openvinotoolkit/openvino.…
wgzintel May 24, 2024
77fc95c
format code
wgzintel May 24, 2024
1a98078
Merge branch 'guozhong/correct_flan-t5_output_size' of https://github…
wgzintel May 24, 2024
d8ffa7f
output token size should be same as infer count in greedy search
wgzintel May 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions llm_bench/python/benchmark.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,10 @@ def run_text_generation(input_text, num, model, tokenizer, args, iter_data_list,
generated_text_len = len(result[bs_idx]) - input_tokens[bs_idx].numel()
else:
generated_text_len = len(result[bs_idx])
# Encoder-decoder models expect the `decoder_input_ids` to start with a special token
# When counting the output length, subtract 1. The last token does not participate in inference.
if model.config.is_encoder_decoder and result[bs_idx][0] == model.config.decoder_start_token_id:
generated_text_len = generated_text_len -1
num_tokens += generated_text_len
if generated_text_len > max_gen_tokens:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we compare generated_text_len to inference number? I suppose they should be the same.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@peterchen-intel , I am not sure, len(generated_text) maybe not same with inference count, some special language maybe exist 2 words map to a token.

log.error('Output token size is over max output token size!')
Expand Down
Loading