Add prompt lookup decoding #379
Conversation
|
@Wovchena , we have a proposal for optimizing kv cache trimm from @sammysun0711: sammysun0711@d7a24e5, based on parallel for. |
…as/prompt_lookup_decoding
|
|
||
| // cut redundant candidates on last iteration | ||
| size_t tokens_to_generate = max_sequence_length - seq_len; | ||
| if (candidates.size() > tokens_to_generate - 1) { |
There was a problem hiding this comment.
looks like we can unconditionally call resize, because in case of the same size it does nothing.
There was a problem hiding this comment.
Do you mean something like: candidates.resize(std::min(candidates.size(), tokens_to_generate - 1)); ?
| return new_tensor; | ||
| } | ||
|
|
||
| void update_kv_cache(ov::InferRequest request, uint64_t seq_len_axis, uint64_t new_seq_len) { |
There was a problem hiding this comment.
@Wovchena , we have a proposal for optimizing kv cache trimm from @sammysun0711: sammysun0711@d7a24e5, based on parallel for. It could give 3x speed up for cache update. Should we apply it as well?
Is there a way to link with tbb from openvino package? @ilya-lavrenov, do you know a way? If yes, feel free to apply.
There was a problem hiding this comment.
see custom operations as example https://github.com/openvinotoolkit/openvino_contrib/blob/master/modules/custom_operations/user_ie_extensions/CMakeLists.txt#L20
ov::parallel_for is used there
There was a problem hiding this comment.
@ilya-lavrenov , @Wovchena , Don't you mind if I address remaining comments in the next PRs?
- Apply
parallel_foroptimization for trim tensor - Apply optimized trim tensor implementation for
speculative_decoding - Investigate
candidates_size + 1inference forspeculative_decoding
Ticket: 138549