Better separate internal and public APIs

[SWilson4]

We have some common code which is not intended for library users but for which .h files exist (for example, SHA2 and SHA3). These headers are documented as not being part of the public API, but are still placed in the system include directory upon install (at least on my machine running x86_64 / Linux). We ought to fix this so that only the truly public headers are installed. On the other hand, we would like this code to be available to not only the implementations in src but also the programs in tests. It seems to me that a good solution would be to separate some sort of internal API (probably via tweaking the CMake configuration) so that our implementations of SHA2 / SHA3 / etc can be made available to select test programs but not users of the library in general.

[dstebila]

We do have our common symmetric code in the public headers in CMakeLists. On the one hand that might be a mistake, although if memory serves we might have done it so that the test programs could access those functions? But maybe there's a way to do it so that the test programs can access them but they're still not installed for users.

[baentsch]

maybe there's a way to do it so that the test programs can access them but they're still not installed for users.

That should be possible tweaking the cmake install logic: Tests should rely on the build-env only while the install logic wouldn't reference those header files. It seems we need to be more selective in setting PUBLIC_HEADERS for the install logic to do the right thing as per this suggestion.

[SWilson4]

Here's an idea: what about building an oqs-internal library corresponding to the internal API (currently just common/aes.h, common/sha2.h, and common/sha3.h). We can toggle the visibility for this library to default so that test programs can link against it and access the common functions. Naturally, it wouldn't be installed at any point; it would just stay in build/lib.

I tinkered with this a little on the branch sw-experimentation and I think it's fairly clean and scalable. It would provide us with an easy way to add common code that we want to expose to test programs but not users of the library. No per-test configuration required; just link against both oqs and oqs-internal. It should (I believe) also resolve a previous linking issue when building shared libraries on MacOS, where the custom randombytes setting was overridden without a bespoke configuration.

In particular, I think it provides an elegant way of dealing with this problem for the 100-KAT test: #1560 (comment)

[dstebila]

Here's an idea: what about building an oqs-internal library corresponding to the internal API (currently just common/aes.h, common/sha2.h, and common/sha3.h). We can toggle the visibility for this library to default so that test programs can link against it and access the common functions. Naturally, it wouldn't be installed at any point; it would just stay in build/lib.

I tinkered with this a little on the branch sw-experimentation and I think it's fairly clean and scalable. It would provide us with an easy way to add common code that we want to expose to test programs but not users of the library. No per-test configuration required; just link against both oqs and oqs-internal. It should (I believe) also resolve a previous linking issue when building shared libraries on MacOS, where the custom randombytes setting was overridden without a bespoke configuration.

In particular, I think it provides an elegant way of dealing with this problem for the 100-KAT test: #1560 (comment)

Sounds like a good solution, thanks Spencer!