Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add benchmarking for stateful hash based schemes: speed_sig_stfl #1952

Merged
merged 11 commits into from
Oct 22, 2024

Conversation

cr-marcstevens
Copy link
Contributor

@cr-marcstevens cr-marcstevens commented Oct 15, 2024

This pull requests adds an executable tests/speed_sig_stfl to be able to benchmark the stateful hash based schemes in liboqs.

The source code tests/speed_sig_stfl.c is directly copied from tests/speed_sig.c,
but making the relevant changes to use the SIG_STFL functions and types instead.

Here are some issues encountered specific to SIG_STFL code:

  • A secure store needs to be set for the secret key, the context must not be NULL
  • A stfl sig has a maximum number of signatures, potentially as low as 2^5=32. One needs to create a new secret key once this number of signatures has been reached.
  • key generation and signing can be extremely costly for some stfl sig algorithms. Automatic testing should focus only on small parameters.
  • You cannot create a new secret key over an old (used up) one. You need to free the old secret key, and create a new one.
  • Bugfix new code: be careful to actually return the new secret key pointer between functions. Otherwise there is a use-after-free bug.
  • for stfl sigs it is possible to disable key generation and signing, but enable verification. Benchmarking doesn't work in such a case, and it simply needs to abort and return success.

Signed-off-by: cr-marcstevens <[email protected]>
@SWilson4
Copy link
Member

Thanks for the contribution! Would you please add the new executable to the list in the README? It would also be nice to have it run in tests/test_speed.py, similarly to speed_kem and speed_sig.

Copy link
Member

@baentsch baentsch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks very much for the contribution @cr-marcstevens ! The code LGTM. One question, though: Would it be OK for you to add it to the test harness, e.g., at "scripts/noregress.sh" (or any other place where it could run regularly such as to help improve test coverage)? [Edit/add: Sorry for the kind of duplicate message -- for some reason the comment by @SWilson4 didn't show up in my timeline when I wrote this... weird]

…d secure keystore with dummy keystore

Signed-off-by: cr-marcstevens <[email protected]>
Signed-off-by: cr-marcstevens <[email protected]>
@cr-marcstevens
Copy link
Contributor Author

I'll look into the extra changes needed throughout the repo with the added executable.
But I'm not fully familiar with how all that infrastructure is meant to be used.

For now I found out that speed_sig_stfl.c needed 2 fixes:

  • signing didn't actually happen unless a secure keystore is set to the private key, so I provided a dummy one.
  • timing sig ops didn't take into account max signatures. So for 2^10 max sigs, it could run out and invalidate the benchmark.

…refresh key when out of sigs for sig benchmark

Signed-off-by: cr-marcstevens <[email protected]>
…ithms with 2^10 max sigs

Signed-off-by: cr-marcstevens <[email protected]>
Signed-off-by: cr-marcstevens <[email protected]>
…f keygen and sign are not enabled.

Signed-off-by: cr-marcstevens <[email protected]>
@cr-marcstevens
Copy link
Contributor Author

cr-marcstevens commented Oct 20, 2024

I've noticed two issues and resolved them:

  • Bugfix new code: in my solution to reset the secret key, be careful to actually return the new secret key pointer between functions. Otherwise there is a use-after-free bug.
  • for stfl sigs it is possible to disable key generation and signing, but enable verification. Benchmarking doesn't work in such a case, and it simply needs to abort and return success.

@cothan
Copy link
Contributor

cothan commented Oct 21, 2024

@cr-marcstevens All tests have passed. Feel free to merge it whenever you like.
Great work.

@cr-marcstevens
Copy link
Contributor Author

Indeed I think all issues are resolved now, so it is ready for merging.

@SWilson4 SWilson4 merged commit 90030a4 into open-quantum-safe:main Oct 22, 2024
72 checks passed
@SWilson4
Copy link
Member

Thanks for the contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants