Getting IndexError when any of 'cands', 'refs' and 'sources' contains empty string

[terryyz]

Current code_bert_score is able to handle the case where any of 'cands', 'refs' contains empty strings and no 'sources' is passed to the score method. See the example below:

from code_bert_score import score
score([''],['a'], lang="python")

However, when 'sources' is provided, the method will raise IndexError.

from code_bert_score import score
score([''],['a'], sources=["a"], lang="python")

It would be great if this kind of cases can be handled.

[urialon]

Thanks @terryyz !
I just pushed a fix.

In case the cand or ref is an empty string, I am now setting both precision and recall to zero.
It is now working when sources are provided or not provided.

Please let us know if you have any questions or find any other issues!
Best,
Uri

[littlepopy]

Can I ask you the data provided under code-bert-score/evaluation come from where? The generation codes ### (humaneval_hyps.txt) of 13025 lines, with the corresponding NL**(humaneval_srcs.txt)** and reference code**(humaneval_refs.txt)** . And if we want to use the data, is it be promised?

[terryyz]

Hi @littlepopy, the details were discussed in Appendix B.2 in the paper, where the data was collected from HumanEval-X. Besides, you may also find our paper useful. This folder may provide all the data you need 😃

[urialon]

Hi @littlepopy ,
Thank you for your interest in our work.

Please see the paragraph "functional correctness experiments" in Section 3.2 in the paper https://arxiv.org/pdf/2302.05527.pdf

And also, Appendix B.2.

Let us know if you have any questions!
Uri

[littlepopy]

Thanks for your reply. It is very useful. Besides, thanks for all the work you have done. Wish you a good future.

…

------------------ 原始邮件 ------------------ 发件人: "Uri ***@***.***>; 发送时间: 2024年1月4日(星期四) 下午4:49 收件人: ***@***.***>; 抄送: ***@***.***>; ***@***.***>; 主题: Re: [neulab/code-bert-score] Getting IndexError when any of 'cands', 'refs' and 'sources' contains empty string (Issue #4) Hi @littlepopy , Thank you for your interest in our work. Please see the paragraph "functional correctness experiments" in Section 3.2 in the paper https://arxiv.org/pdf/2302.05527.pdf And also, Appendix B.2. Let us know if you have any questions! Uri — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[littlepopy]

Thanks for your reply. It is very useful. Besides, thanks for all the work you have done. Wish you a good future.

…

------------------ 原始邮件 ------------------ 发件人: "Terry ***@***.***>; 发送时间: 2024年1月4日(星期四) 下午4:48 收件人: ***@***.***>; 抄送: ***@***.***>; ***@***.***>; 主题: Re: [neulab/code-bert-score] Getting IndexError when any of 'cands', 'refs' and 'sources' contains empty string (Issue #4) Hi @littlepopy, the details were discussed in Appendix B.2 in the paper, where the data was collected from [HumanEval-X](https://huggingface.co/datasets/THUDM/humaneval-x]. Besides, you may also find our paper useful. This folder may provide all the data you need 😃 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>