Release 0.8

.gitignore

@@ -5,3 +5,6 @@ build
 htmlcov
 .coverage
 /venv37/
+.idea/
+neo_mamba.egg-info/
+tests/**/*.json

CHANGELOG.rst

@@ -3,6 +3,18 @@ Changelog
 
 All notable changes to this project are documented in this file.
 
+[0.8] 2021-06-04
+----------------
+- RC3 code parity updates
+- Add basic wallet support
+- Restrict to Python 3.8
+- Update docstrings
+- Add ``ExtensiblePayload` handler to network
+- Add ``to_json/from_json`` support to ``Transaction`` class
+- Fix removing vote
+- Fix payload for requesting headers
+- Automatically add inventory to relay cache when relaying
+
 [0.7] 2021-05-12
 ----------------
 - RC2 updates

README.rst

@@ -67,13 +67,13 @@ separate repositories but be included here by default (e.g. virtual machine impl
 - Storage (v0.1)
 - Virtual Machine
 - Smart contracts (v0.4)
-
-Up next: Wallet support
+- Wallet support (v0.8)
 
 Performance
 -----------
-At the time of releasing v0.7 there is no RC2 test net available for performance measuring. Very few changes in RC2
-should affect performance, therefore the RC1 measurements below should still be pretty accurate.
+At the time of releasing v0.7 there was no RC2 test net available for performance measuring. Very few changes in RC2
+should affect performance, therefore the RC1 measurements below should still be pretty accurate. RC3 (v0.8) has even
+less changes. Performance measurements are on hold until other priorities have been processed.
 
 RC1 measurements
 ~~~~~~~~~~~~~~~~

docs/source/library/network/payloads/block.rst

@@ -51,7 +51,7 @@ The GetBlocksByIndexPayload is also used as for requesting headers via the :cons
 
 Responding
 ^^^^^^^^^^
-The :class:`~neo3.network.payloads.block.HeadersPayload` is used to reply to a :const:`~neo3.network.message.MessageType.GETHEADERS` type message and returns :class:`~neo3.network.payloads.block.Header` objects. 
+The :class:`~neo3.network.payloads.block.HeadersPayload` is used to reply to a :const:`~neo3.network.message.MessageType.HEADERS` type message and returns :class:`~neo3.network.payloads.block.Header` objects.
 
 The :class:`~neo3.network.payloads.block.MerkleBlockPayload` is used to reply to a :const:`~neo3.network.message.MessageType.GETDATA` or :const:`~neo3.network.message.MessageType.GETBLOCKDATA` type message under the specific condition that a bloomfilter has been loaded (TODO: add ref to filter). Responding to the aforementioned message types under the condition that `no` bloomfilter is present is respectively done via an :const:`~neo3.network.message.MessageType.INV` type message with an InventoryPayload or a :const:`~neo3.network.message.MessageType.BLOCK` type message where the Block object is the message payload.
 

docs/source/whatsnew/0.8.rst

@@ -0,0 +1,25 @@
+.. _whatsnew-v08:
+
+*****************************
+  What's New in neo-mamba 0.8
+*****************************
+
+:Author: Erik van den Brink
+
+RC3 compatability, wallet support and end of Python 3.7 support
+===============================================================
+
+This release catches up with all RC3 changes done in the C# core project. It also adds basic wallet support allowing
+you to sign and relay transactions to the network. We decided to drop Python 3.7 support at the last bug fix release
+happened at the beginning of the year. Part of mamba (specifically the network) relies heavily on the ``asyncio`` module
+which has had its fair share of bugs and memory leaks discovered. We don't expect that the last one was found so we want
+to stay up to date. Expet a move to Python 3.9 in the near future.
+
+Further more doc strings have been polished, some new features have
+been added and a couple of bugs have been fixed. See the changelog for more details.
+
+There is a significant resource difference between neo-mamba and the core C# project. As such the mamba project had to
+choose a strategy that allowed faster catching up at the cost of lower code coverage and less documentation. The less
+core critical changes are made, the more time can be spend on updating documentation. For now compatibility is
+prioritized over the user experience.
+

docs/source/whatsnew/index.rst

@@ -16,4 +16,5 @@ This chapter describes the most important changes between the neo-mamba versions
    0.4.rst
    0.5.rst
    0.6.rst
-   0.7.rst
+   0.7.rst
+   0.8.rst

examples/basic_node.py

@@ -1,6 +1,6 @@
 """
-Neo-mamba v0.4 Basic node example
-NEO Preview 3 compatible
+Neo-mamba v0.7 Basic node example
+NEO-RC2 compatible
 """
 from __future__ import annotations
 import asyncio
@@ -21,9 +21,9 @@ def enable_network_logging():
 
 
 async def main():
-    # Configure network to Preview 3 TestNet
-    # Values are taken from https://github.com/neo-project/neo-node/blob/d2e4eb28f415b751267f23a0593e007a3292919e/neo-cli/protocol.testnet.json
-    settings.network.magic = 1951352142
+    # Configure network to RC2 TestNet
+    # Values are taken from config.json on the neo-cli github repo
+    settings.network.magic = 844378958
     settings.network.seedlist = ['seed1t.neo.org:20333']
     settings.network.standby_committee = [
       "023e9b32ea89b94d066e649b124fd50e396ee91369e8e2a6ae1b11c170d022256d",

examples/transaction_signing.py

@@ -0,0 +1,169 @@
+"""
+Neo-mamba v0.8 transaction signing example
+NEO RC3 compatible
+
+This example will show how to withdraw NEO from the genesis block. It will show among others
+- Wallet creation
+- Key import
+- Transaction building to initiate a "transfer" of NEO tokens
+- Multi-signature transaction signing
+- Relaying the transaction
+
+If you do not have a single-node private net setup, please follow the guide here
+https://docs.neo.org/docs/en-us/develop/network/private-chain/solo.html
+
+You can skip the steps in the section "withdrawing-neo-and-gas" and beyond as we will do that here
+"""
+import logging
+import asyncio
+import json
+from neo3 import settings, wallet, contracts, vm, blockchain
+from neo3.network import payloads, convenience
+
+# First configure the network settings. The information is taken from `config.json` in neo-cli
+settings.network.magic = 692804366  # `Network` number
+settings.network.standby_committee = ["03a60c1deaf147b10691c344c76e5f3dac83b555fdd5a3f8d9e2f623b3d1af8df6"]
+settings.network.validators_count = 1  # set to the same number as
+settings.network.seedlist = ['127.0.0.1:20333']
+
+
+# This initialises the local chain with the genesis block and allows us to get a snapshot of the database. This is
+# required for calculating network fees automatically. Always call this AFTER setting the network settings otherwise network
+# syncing and other parts of the system will fail.
+bc = blockchain.Blockchain()
+snapshot = bc.currentSnapshot
+
+# We start with adding a wallet with a multi-signature account that we will later need to sign the transaction (as well as
+# obtain the address for to specify as source funds).
+
+# There are different means of creating/importing a wallet, choose your flavour. 
+# This creates a wallet and add the consensus node account (address) where the key is protected by the password "123"
+w = wallet.Wallet()
+w.account_add(wallet.Account.from_wif("L2aFaQabd35NspvBzC9xPUzKP1if5WgaC2uw4SkviA58DGvccUEy", "123"))  # See also Account.from_* for alternative constructors
+
+# Alternatively import a wallet by uncommenting the 3 lines below
+# with open('wallet.json') as f:
+#     data = json.load(f)
+#     w = wallet.Wallet.from_json(data)
+
+# Next add the consensus node multisig address to the wallet.
+# Note that we only have the consensus node account in our wallet, therefore we can access the account via the
+# `account_default` property.
+#
+# In the single consensus node setup we only need to sign with one key, thus we have a threshold of 1 and just 1 public key.
+# In a multi-signature setup (e.g. the 4 nodes setup also described on docs.neo.org) we would enter 3 as threshold and
+# supply a list of 4 public keys.
+account = w.import_multisig_address(signing_threshold=1, public_keys=[w.account_default.public_key])
+
+# With the wallet ready for use we can start building the script that will transfer NEO from a source account to the
+# destination account. For this we need to call the "transfer" function on the NeoToken smart contract.
+#
+# Its signature on the contract side looks as follows:
+#    https://github.com/CityOfZion/neo-mamba/blob/873932c8cb25497b90a39b3e327572746764e699/neo3/contracts/native/fungible.py#L109
+#
+#    def transfer(self,
+#                 engine: contracts.ApplicationEngine,
+#                 account_from: types.UInt160,
+#                 account_to: types.UInt160,
+#                 amount: vm.BigInteger,
+#                 data: vm.StackItem
+#                 ) -> bool:
+
+# Note that if the first parameter of a native contract is `engine` or `snapshot` then it can be ignored as it will be supplied automatically. This thus
+# leaves us with 4 parameters to supply. We can validate this as well by looking at the ABI in the contract's manifest
+# via ``print(contracts.NeoToken().manifest)``.
+# That looks as follows for the `transfer` function
+#{
+#        "name": "transfer",
+#        "parameters": [
+#          {
+#            "name": "account_from",
+#            "type": "ByteArray"
+#          },
+#          {
+#            "name": "account_to",
+#            "type": "ByteArray"
+#          },
+#          {
+#            "name": "amount",
+#            "type": "Integer"
+#          },
+#          {
+#            "name": "data",
+#            "type": "Any"
+#          }
+#        ],
+#        "returntype": "Boolean",
+
+# Source account converted to byte array to match the ABI interface
+from_account = account.script_hash.to_array()
+# Destination account converted to byte array to match the ABI interface
+to_account = wallet.Account.address_to_script_hash("NU5unwNcWLqPM21cNCRP1LPuhxsTpYvNTf").to_array()
+# We multiply this amount with the contract factor (to adjust for tokens with decimals)
+amount = 10_000_000 * contracts.NeoToken().factor
+# Arbitrary additional data to supply that will be printed in the "transfer" notify event.
+data = None
+
+sb = vm.ScriptBuilder()
+sb.emit_dynamic_call_with_args(contracts.NeoToken().hash, "transfer", [from_account, to_account, amount, data])
+
+# With our script done we need to create a Transaction and add the script to it.
+tx = payloads.Transaction(version=0,
+                          nonce=123,
+                          system_fee=0,
+                          network_fee=0,
+                          valid_until_block=1500,  # Make sure this is higher than the current chain height!
+                          attributes=[],
+                          signers=[],
+                          script=sb.to_array())
+
+# Add the multisig address as the sender. Do this before signing because it is part of the signed data.
+account.add_as_sender(tx)
+
+# The last step before signing is to add the required fees. This can be done manually, but you'll likely be overpaying
+# or adding too little. On a private network this isn't a problem, but on the real chain you'd like to be precise.
+wallet.add_system_fee(tx, snapshot)  # the price for executing the transaction script
+wallet.add_network_fee(tx, snapshot, account)  # the price for validation and inclusion in a block by the consensus node
+
+# Finally sign with the multisig account. For this we need a signing context.
+ctx = wallet.MultiSigContext()
+# Once the signing threshold is met the required witness is automatically added to the tx.
+account.sign_multisig_tx(tx, "123", ctx)
+# If we had a 3 out of 4 multi-signature contract, we'd simply call `sign_multisig_tx` multiple times with each required account.
+# The signing context has properties/methods to query which keys still need to sign and whether sufficient accounts have signed
+# the tx.
+
+
+# Add this point we're done with signing and the transaction is ready to be relayed to the network. There are 2 options for this
+# 1.  Send the transaction via RPC using the `sendrawtransaction` method
+#     https://docs.neo.org/docs/en-us/reference/rpc/latest-version/api/sendrawtransaction.html
+#     You can find a public RPC server at https://dora.coz.io/monitor
+#     To obtain the RPC input parameter use: print(base64.b64encode(tx.to_array().decode())
+#
+# 2.  Relay the transaction directly over the P2P layer.
+#     This option requires being connected over P2P to the private network and is shown below.
+
+async def sync_network():
+    # Will automatically connect to the nodes in the network.seedlist specified above and attempt to maintain connection
+    node_mgr = convenience.NodeManager()
+    node_mgr.start()
+
+    # The following line will automatically sync the blocks from the network. Assuming that you did not change any
+    # settings in the Policy contract, this step can be skipped for a private network. Otherwise you might want to first
+    # sync your local chain before building and relaying the transaction. Failing to do so might lead to incorrect
+    # calculation of the transaction fees and can lead to rejection of the transaction.
+
+    # sync_mgr = convenience.SyncManager()
+    # await sync_mgr.start()
+
+    # Give the node manager time to connect to nodes
+    while len(node_mgr.nodes) == 0:
+        await asyncio.sleep(1)
+    n = node_mgr.nodes[0]
+    # Relay the transaction
+    await n.relay(tx)
+
+if __name__ == '__main__':
+    loop = asyncio.get_event_loop()
+    loop.create_task(sync_network())
+    loop.run_forever()

neo3/__init__.py

@@ -6,7 +6,7 @@
 from types import SimpleNamespace
 from neo3.core import cryptography
 
-version = '0.7'
+version = '0.8'
 
 core_logger = logging.getLogger('neo3.core')
 network_logger = logging.getLogger('neo3.network')
@@ -58,6 +58,7 @@ class Settings(IndexableNamespace):
     default_settings = {
         'network': {
             'magic': 5195086,
+            'account_version': 53,
             'seedlist': [],
             'validators_count': 1,
             'standby_committee': ['02158c4a4810fa2a6a12f7d33d835680429e1a68ae61161c5b3fbc98c7f1f17765']

neo3/contracts/abi.py

@@ -2,7 +2,7 @@
 import enum
 from typing import List, Optional, Type, Union, cast
 from enum import IntEnum
-from neo3.core import types, IJson, IInteroperable, serialization, cryptography
+from neo3.core import IJson, IInteroperable, serialization, cryptography
 from neo3 import contracts, vm
 
 

neo3/contracts/applicationengine.py

@@ -2,9 +2,8 @@
 from neo3 import contracts, storage, vm
 from neo3.network import payloads
 from neo3.core import types, cryptography, IInteroperable, serialization, to_script_hash
-from typing import Any, Dict, cast, List, Tuple, Type, Optional, Callable, Union
+from typing import Any, Dict, cast, List, Tuple, Type, Optional, Union
 import enum
-from dataclasses import dataclass
 from contextlib import suppress
 
 
@@ -343,21 +342,34 @@ def _contract_call_internal2(self,
 
     def _stackitem_to_native(self, stack_item: vm.StackItem, target_type: Type[object]):
         # checks for type annotations like `List[bytes]` (similar to byte[][] in C#)
-        if hasattr(target_type, '__origin__') and target_type.__origin__ == list:  # type: ignore
-            element_type = target_type.__args__[0]  # type: ignore
-            array = []
-            if isinstance(stack_item, vm.ArrayStackItem):
-                for e in stack_item:
-                    array.append(self._convert(e, element_type))
-            else:
-                count = stack_item.to_biginteger()
-                if count > self.MAX_STACK_SIZE:
-                    raise ValueError
-
-                # mypy bug: https://github.com/python/mypy/issues/9755
-                for e in range(count):  # type: ignore
-                    array.append(self._convert(self.pop(), element_type))
-            return array
+        if hasattr(target_type, '__origin__'):
+            if target_type.__origin__ == list:  # type: ignore
+                element_type = target_type.__args__[0]  # type: ignore
+                array = []
+                if isinstance(stack_item, vm.ArrayStackItem):
+                    for e in stack_item:
+                        array.append(self._convert(e, element_type))
+                else:
+                    count = stack_item.to_biginteger()
+                    if count > self.MAX_STACK_SIZE:
+                        raise ValueError
+
+                    # mypy bug: https://github.com/python/mypy/issues/9755
+                    for e in range(count):  # type: ignore
+                        array.append(self._convert(self.pop(), element_type))
+                return array
+            if target_type.__origin__ == Union:  # type: ignore
+                # handle typing.Optional[type], Optional is an alias for Union[x, None]
+                # only support specifying 1 type
+                if len(target_type.__args__) != 2:  # type: ignore
+                    raise ValueError(f"Don't know how to convert {target_type}")
+                if isinstance(stack_item, vm.NullStackItem):
+                    return None
+                else:
+                    for i in target_type.__args__:  # type: ignore
+                        if i is None:
+                            continue
+                        return self._convert(stack_item, i)
         else:
             try:
                 return self._convert(stack_item, target_type)