Support for FIPS compliance mode

[beanuwave]

Description

This PR makes FIPS mode available through the OPENSEARCH_CRYPTO_STANDARD=FIPS-140-3 environmental parameter instead of the tests.fips.enabled setting. It provides FIPS 140-3 support by replacing all BC dependencies with BCFIPS dependencies and making FIPS approved-only mode configurable at launch. Running this mode restricts the BCFIPS provider to rely solely on FIPS-certified ciphers.

• The fips.gradle build script is removed in order to support a single-build solution.
• All BC dependencies are replaced by BCFIPS.
• Fixed creation of password protected internal keystore for valuable settings at node start. The add-string command was made with additional --stdin option, which interferes with password input.
• The Password Matcher inside Identity-Shiro that relies on BC to check if hashed passwords match with OpenBSDBCrypt is replaced by the password4j implementation.
• Adds full support for the BCFKS format (*.bcfks) for key and trust stores, also making it the default.
• KeyStore instantiation was added to forbidden-apis in favour of KeyStoreFactory.
• Google's truststore is converted to the BCFKS format.
• Makes the best guess of which store type is provided based on the filename extension.
• Store types are strictly limited to JKS, PKCS12, PKCS11, and BCFKS.
• Refactors PemUtils to parse private keys in formats EC, PKCS8, PKCS1, and DSA, with or without encryption, and with or without parameters.
• dependency ':libs:opensearch-common' was added to rest-client build, to support strict keystore types. It's also the reason for JVM bump JAVA8 to JAVA11.
• allow ingest-attachment plugin to run in FIPS mode, since BC dependencies find no use.
• The java.security file is added to the build to distinguish between FIPS and non-FIPS environments.
• The fips_java.security file is altered due to evolving security standards.
• The security.policy file is altered to grant necessary security permissions.
• Increased security standards in KeyTabs and algorithms for Kerberos.
• SecureRandom gets instantiated in to different way, depending on if it runs with FIPS or not.
• Uninstalls SunJCE provider from security providers list at runtime when FIPS mode is enabled.

Runtime limitations (known so far) that come with enabling FIPS mode:

Admins can continue to manage their systems without being impacted by this change. However, for those keen on FIPS compliance, the most common obstacle will likely be the requirement to set a stronger password for the internal keystore and also convert key and truststores to *.bcfks format.

• Does not allow empty passwords for keystores or private keys (they need to be at least 112 bits in strength).
• The ssl.verification_mode=NONE setting is not permitted.
• JKS and PKCS12 key and trust store types are not supported at all.
• The internal keystore cannot be auto-migrated from versions 1 or 2.
• Azure Classic Discovery plugin -> deprecated.
• SQL-CLI client.
• HDFS plugin won't connect since it's using RC4 cipher for token authentication.

Reasons for refactoring PemUtils, which is used by the Reindex API in cases of migrating data from a remote cluster that is TLS protected:

• Lack of support for evolving standards like PKCS#8.
• Password-Based Key Derivation Functions such as PBKDF-OPENSSL are not supported in FIPS mode in favor of the PBKDF2 standard.
• Java type safety.
• It is generally a good idea to let ASN1 annotation parsing be done by external security libraries.

Related Issues

opensearch-project/security#3420

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 6016d5d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 8e8ed47: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 6016d5d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[dblock]

Could use some help maybe from @cwperks or @peternied reviewing this, please.

[github-actions]

❕ Gradle check result for 951e66d: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

❌ Gradle check result for af76585: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 20a3362: SUCCESS

[terryquigleysas]

@reta the BCFIPS dependencies are going to leak into (mostly) every single project in the ecosystem (through clients, build tooling or core itself), no matter if BC/FIPS is on their agenda or not

I agree—the BC libraries are essentially everywhere at this point, since they're our primary security provider. Reducing their footprint hasn’t been on our agenda so far, but if you have any suggestions, I’m happy to consider them.

I am just catching up on the status of this again, so apologies if some of this has already been covered.

Do we think it would be worthwhile replacing all the instances of the Bouncy Castle libraries with the FIPS equivalents?
Any functionality no longer available can be replaced using other libs, as we have done with the security plugin and has been done in this PR. This would ensure more of the code paths are covered by existing tests.
(Assumes OpenSAML breaking acceptable for now and that the version 5.x upgrade may not resolve)

Any awkwardness may be around coordinating the separate components, as having the non-FIPS and FIPS jars on the same classpath breaks things. In the interim, again as we've done in the security plugin and referenced by @cwperks above, the code could be abstracted to support both sets of libraries, if required.

In a standard deployment (versions from 2.17), for example, we see the following non-FIPS libs for other components.
/plugins/opensearch-sql/bcprov-ext-jdk18on-1.75.jar
/plugins/opensearch-ml/bcprov-jdk18on-1.78.1.jar
/plugins/opensearch-flow-framework/bcprov-jdk18on-1.78.jar
/plugins/opensearch-performance-analyzer/bcpkix-jdk18on-1.78.1.jar
/plugins/opensearch-performance-analyzer/bcprov-jdk15to18-1.78.1.jar
/plugins/opensearch-security/bcprov-jdk18on-1.78.jar
/plugins/opensearch-security/bcpkix-jdk18on-1.78.jar
/performance-analyzer-rca/lib/bcutil-jdk15to18-1.78.1.jar
/performance-analyzer-rca/lib/bcprov-jdk15to18-1.78.1.jar
/performance-analyzer-rca/lib/bcpkix-jdk15to18-1.78.1.jar

The work we have completed from opensearch-project/security#3420 has allowed us to take a 2.18 deployment, swap in the FIPS equivalents, and run in approved mode with success. (One caveat being that we do not actively use the ML or SQL plugins but heavily use Core functionality and Security.)

[reta]

Any awkwardness may be around coordinating the separate components, as having the non-FIPS and FIPS jars on the same classpath breaks things. In the interim, again as we've done in the security plugin and referenced by @cwperks above, the code could be abstracted to support both sets of libraries, if required.

This great point @terryquigleysas , basically if we expand on that, how could we deal with the mix of FIPS/non-FIPS dependencies outside the plugins managed by OpenSearch Project? Fe, if user wants to run the distribution in FIPS mode with third party plugins installed, could we catch or swap these dependencies?

[terryquigleysas]

Any awkwardness may be around coordinating the separate components, as having the non-FIPS and FIPS jars on the same classpath breaks things. In the interim, again as we've done in the security plugin and referenced by @cwperks above, the code could be abstracted to support both sets of libraries, if required.

This great point @terryquigleysas , basically if we expand on that, how could we deal with the mix of FIPS/non-FIPS dependencies outside the plugins managed by OpenSearch Project? Fe, if user wants to run the distribution in FIPS mode with third party plugins installed, could we catch or swap these dependencies?

@reta In our container we remove all the non-FIPS libs and include the FIPS versions in the /lib dir only. This works for all the functionality we require.

I would lean towards only using the FIPS libraries throughout and standardizing the versions of those by either shipping them in the lib dir or mandating that plugins should use the FIPS versions declared in Core.

[cwperks]

Fe, if user wants to run the distribution in FIPS mode with third party plugins installed, could we catch or swap these dependencies?

In this case should the cluster fail to bootup? If running in FIPS mode and non-FIPS dependencies are on the classpath then I would envision that the cluster would fail to bootup with an error message that lets the cluster administrator know that there are non-compliant dependencies on the classpath

[reta]

In this case should the cluster fail to bootup? If running in FIPS mode and non-FIPS dependencies are on the classpath then I would envision that the cluster would fail to bootup with an error message that lets the cluster administrator know that there are non-compliant dependencies on the classpath

I would expect something along these lines, but it could be too much?

[cwperks]

I would lean towards only using the FIPS libraries throughout and standardizing the versions of those by either shipping them in the lib dir or mandating that plugins should use the FIPS versions declared in Core.

If it is completely backward compatible for the default distribution than I think this should be considered as well. There is a mechanism for forbidding dependencies in the core, but I'm not sure if the same check is extended to plugins outside of the core repo.