Support Koblitz-based and Keccak256-based custom witness verification

[AnnaShaleva]

Description

This PR is a port of nspcc-dev/neo-go#3425 and is an alternative to #3205 that does not require significant core protocol changes. This PR doesn't require a hard-fork. In this PR:

• Extend native CryptoLib's verifyWithECDsa by supporting Keccak256 hasher;
• Add example of how to build a custom sig/multisig witness for Koblitz signatures and Keccak256 hasher and test it.
• Move custom witness builder from test to Contract.cs class (if needed, we need to decide about it). Some CreateKoblitzSigRedeemScript and CreateKoblitzMultiSigRedeemScript APIs may be added to this class. Discussed, not needed.
• Implement CalculateNetworkFee extension that allows to properly calculate network fee for custom transaction witnesses. Ref. Improve CalculateNetworkFee for contract signers #2805 (in progress). Won't be included into 3.7, ref. Neo v3.7.0 plan #2905 (comment).
• Add more unit-tests for native CryptoLib's verifyWithECDsa.

Rational

This PR exists because of #3205 (comment).

A note for reviewers

The main builder logic is located in the tests/Neo.UnitTests/SmartContract/Native/UT_CryptoLib.cs. Please, review this file carefully. We need to decide whether these builders have to be moved outside of the tests to some Contract.cs class. If yes, then I'll update the PR.

Simple signature builder is in its final state. Multisignature builder requires minor simplification (ref. nspcc-dev/neo-go#3425 (review) and nspcc-dev/neo-go#3425 (comment)), but it works and is also almost done. I'll update the PR once we resolve the original conversations.

Also, this PR changes CryptoLib's manifest (name of the verifyWithECDsa argument was changed), and thus requires node resync from the genesis. But it's a minor incompatibility that won't affect anything. Anyway, we are going to resync mainnet/testnet for 3.7.

A special request for review from @vang1ong7ang, you'll like it.

Results

Single signature verification

What user signs	Invocation script length	Verification script length	Witness verification cost*
keccak256([4-bytes-network-magic-LE, txHash-bytes-BE])	66 bytes	110 bytes	2154270 GAS

* Verification cost is based on the default execution fee value.
** See the script variations in nspcc-dev/neo-go#3425, this PR contains the best one from our POW.

Multisignature verification

What user signs	Multisig M out of N	Invocation script length	Verification script length	Witness verification cost*
keccak256([4-bytes-network-magic-LE, txHash-bytes-BE])	3 out of 4	198 bytes (= 66*3 bytes)	264 bytes	8390070 GAS

* Verification cost is based on the default execution fee value.
** See the script variations in nspcc-dev/neo-go#3425, this PR contains the best one from our POW.

Type of change

• New feature (non-breaking change which adds functionality)

How Has This Been Tested?

• Unit tests for verifyWithECDsa CryptoLib's API;
• TestVerifyWithECDsa_CustomTxWitness_SingleSig;
• TestVerifyWithECDsa_CustomTxWitness_MultiSig.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules:
  • neo-modules update: not needed;
  • neo-devpack-dotnet update: Native: extend CryptoLib's verifyWithECDsa with compatible NamedCurveHash argument neo-devpack-dotnet#1035, will be merged after Support Koblitz-based and Keccak256-based custom witness verification #3209.

[Jim8y]

@AnnaShaleva really like your comment style, very very detailed explaination on everything. Easy to follow.

[cschuchardt88]

I still think this should be marked as [Obsolete] and add creating of new method. We dont want to break wallets, dapps, etc.

[AnnaShaleva]

The behaviour is compatible. It was SHA256 before this change, and it's now SHA256 by default. Users don't have to change anything in their code, and nothing is broken by this change.

[AnnaShaleva]

@Jim8y, this must not be changed, it’s an automatic output.

[AnnaShaleva]

Revert, please.

[AnnaShaleva]

Please make the test I asked for testing Transaction.VerifyWitness

@cschuchardt88, all tests are already included. TestVerifyWithECDsa_CustomTxWitness_SingleSig and TestVerifyWithECDsa_CustomTxWitness_MultiSig do call VerifyStateDependent under the hood. And VerifyStateDependent calls Transaction.VerifyWitness.

[superboyiii]

Thanks for your explaination! @AnnaShaleva
I just add witness scope to ensure it could pass checkwitnessinternal of GAS, everything works well now!

        /// <summary>
        /// Process "test k1" command
        /// </summary>
        [ConsoleCommand("test k1", Category = "Wallet Commands")]
        private void OnTestK1Command()
        {
            byte[] privatekey = "f011e762cbaa7d9596ed97921a7653ceb66e852d9bffeed5a620a24bd8b0c147".HexToBytes(); ;
            string publichexK1 = "04ff3df16f92c2a6ebde21fbaa499417fe8c2d5108bcf708e5f1f7a4218c0dcec0b842ad407495ecd90137adeb3d784a89951aba65eb2a57a8c3d8c712e259ce81";
            ECPoint publickeyK1 = ECPoint.Parse(publichexK1, ECCurve.Secp256k1);
            using ScriptBuilder vrf = new();
            vrf.EmitPush((byte)NamedCurveHash.secp256k1Keccak256); // push Koblitz curve identifier and Keccak256 hasher.
            vrf.Emit(OpCode.SWAP); // swap curve identifier with the signature.
            vrf.EmitPush(publickeyK1.EncodePoint(true)); // emit the caller's public key.

            // Construct and push the signed message. The signed message is effectively the network-dependent transaction hash,
            // i.e. msg = [4-network-magic-bytes-LE, tx-hash-BE]
            // Firstly, retrieve network magic (it's uint32 wrapped into BigInteger and represented as Integer stackitem on stack).
            vrf.EmitSysCall(ApplicationEngine.System_Runtime_GetNetwork); // push network magic (Integer stackitem), can have 0-5 bytes length serialized.

            // Convert network magic to 4-bytes-length LE byte array representation.
            vrf.EmitPush(0x100000000); // push 0x100000000.
            vrf.Emit(OpCode.ADD, // the result is some new number that is 5 bytes at least when serialized, but first 4 bytes are intact network value (LE).
                    OpCode.PUSH4, OpCode.LEFT); // cut the first 4 bytes out of a number that is at least 5 bytes long, the result is 4-bytes-length LE network representation.

            // Retrieve executing transaction hash.
            vrf.EmitSysCall(ApplicationEngine.System_Runtime_GetScriptContainer); // push the script container (executing transaction, actually).
            vrf.Emit(OpCode.PUSH0, OpCode.PICKITEM); // pick 0-th transaction item (the transaction hash).

            // Concatenate network magic and transaction hash.
            vrf.Emit(OpCode.CAT); // this instruction will convert network magic to bytes using BigInteger rules of conversion.

            // Continue construction of 'verifyWithECDsa' call.
            vrf.Emit(OpCode.PUSH4, OpCode.PACK); // pack arguments for 'verifyWithECDsa' call.
            EmitAppCallNoArgs(vrf, CryptoLib.CryptoLib.Hash, "verifyWithECDsa", CallFlags.None); // emit the call to 'verifyWithECDsa' itself.

            // Account is a hash of verification script.
            var vrfScript = vrf.ToArray();
            var acc = vrfScript.ToScriptHash();
            ConsoleHelper.Info("K1 address:\n", $"{acc}");
            ScriptBuilder sb = new ScriptBuilder();
            sb.EmitDynamicCall(NativeContract.GAS.Hash, "transfer", acc, UInt160.Parse("0x10c823717e6c50782dd11c50e91945f6c1f51dce"), 100000000, "test");
            var snapshot = NeoSystem.StoreView;
            Random rand = new();
            var tx = new Transaction
            {
                Attributes = [],
                NetworkFee = 1_00_0000,
                Nonce = (uint)rand.Next(),
                Script = sb.ToArray(),
                ValidUntilBlock = NativeContract.Ledger.CurrentIndex(snapshot) + NeoSystem.Settings.MaxValidUntilBlockIncrement,

                Signers = [new Signer { Account = acc, Scopes = WitnessScope.CalledByEntry }],
                SystemFee = 0,
                Version = 0,
                Witnesses = []
            };

            using (ApplicationEngine engine = ApplicationEngine.Run(sb.ToArray(), snapshot.CreateSnapshot(), tx, settings: NeoSystem.Settings, gas: ApplicationEngine.TestModeGas))
            {
                if (engine.State == VMState.FAULT)
                {
                    throw new InvalidOperationException($"Failed execution for '{Convert.ToBase64String(sb.ToArray())}'", engine.FaultException);
                }
                tx.SystemFee = engine.GasConsumed;
            }

            var tx_signature = Crypto.Sign(tx.GetSignData(NeoSystem.Settings.Network), privatekey, ECCurve.Secp256k1, Hasher.Keccak256);

            // inv is a builder of witness invocation script corresponding to the public key.
            using ScriptBuilder inv = new();
            inv.EmitPush(tx_signature); // push signature.

            tx.Witnesses =
            [
                new Witness { InvocationScript = inv.ToArray(), VerificationScript = vrfScript }
            ];

            if (!ConsoleHelper.ReadUserInput("Relay tx? (no|yes)").IsYes())
            {
                ConsoleHelper.Info("txraw:\n", $"{Convert.ToBase64String(tx.ToArray())}");
                return;
            }
            NeoSystem.Blockchain.Tell(tx);
            ConsoleHelper.Info("Signed and relayed transaction with hash:\n", $"{tx.Hash}");
        }
    }

[shargon]

Good to merge for me

[superboyiii]

Good for me as well.

[cschuchardt88]

Still think the name NamedCurveHash enum is misleading. It a prefix identifier. NamedCurvePrefix would allow us to add other identifiers to this list in the future. Because you having something like this NamedCurveHash.secp256k1MutliSigurnatureKeccak256 instead of something more like.

NamedCurvePrefix.Keccak256
NamedCurvePrefix.MutliSigurnature
NamedCurvePrefix.secp256k1
NamedCurvePrefix.secp256r1
NamedCurvePrefix.Sha256
NamedCurvePrefix.RedeemSigurature

At least here you can mix and match and can ensure the values will always be different; not the same.

Also I think price for verification should be re-calulated and adjust it.

[superboyiii]

Merge?