Merge pull request #117 from trilitech/palmer@functori@snapshots-for-…

…basic-tests-for-nano-devices

Tests: use snapshots for basic tests for nano devices

.github/workflows/build.yml

@@ -94,7 +94,7 @@ jobs:
               ./tests/integration/${{ matrix.device }}
           else
             ./tests/integration/run_test_local.sh -F -m ${{ matrix.device }} \
-              ./tests/integration/nano ./tests/integration/${{ matrix.device }}
+              ./tests/integration/nano ./tests/integration/nano/${{ matrix.device }}
           fi
 
       - name: Upload results

Makefile

@@ -94,8 +94,8 @@ integration_tests_basic_%:	app_%.tgz			\
 				app_%_dbg.tgz			\
 				tests/integration/*		\
 				tests/integration/nano/*	\
-				tests/integration/%/*
-	$(RUN_TEST_DOCKER) $* tests/integration/nano tests/integration/$*
+				tests/integration/nano/%/*
+	$(RUN_TEST_DOCKER) $* tests/integration/nano tests/integration/nano/$*
 
 integration_tests_basic:	integration_tests_basic_nanos	\
 				integration_tests_basic_nanosp	\

README.md

@@ -94,8 +94,8 @@ and
 :; make integration-tests
 ```
 
-NOTE: the full integration tests are [currently] only available for the Nano S. The basic tests can be run for all
-devices with:
+NOTE: the full integration tests are [currently] only available for
+the Nano device. The basic tests can be run for all devices with:
 
 ```
 :; make integration_tests_basic
@@ -127,12 +127,54 @@ Both of these commands take the following arguments:
 -x
 : executes the tests with shell tracing (-x)
 
-### STAX
-
-The stax tests rely on gold-images, rather than OCR. They are stored under [snapshots](./tests/integration/stax/snapshots).
+Basic tests rely on gold-images, rather than OCR. They are stored under [nano/snapshots](./tests/integration/nano/snapshots) and [stax/snapshots](./tests/integration/stax/snapshots).
 
 To generate/reset the snapshots, you can do so for individual tests.
 
+### Nano
+
+#### Preparation
+
+First, start a container for running individual tests:
+
+```sh
+docker run --rm -it --entrypoint /bin/bash -v $(pwd):/app --network host \
+  ledger-app-tezos-integration-tests
+
+cd /app
+
+git config --global --add safe.directory /app
+```
+
+Before running the test, build the nano app you want to test, for example the nanox app:
+
+```sh
+make app_nanox_dbg.tgz
+```
+
+#### Running
+
+You can run an individual test from the test container. You should see the app progress on the vnc viewer.
+
+```sh
+./tests/integration/nano/<test_name>.py \
+   --device $DEVICE \
+   --port $PORT \
+   --display headless \
+   --vnc-port 41000 \
+   --app app/bin/app.elf
+```
+
+#### Setting goldimages
+
+You can reset/set goldimages using the `--golden-run` option:
+
+You will be requested to press enter to take snapshots in term.
+**NB** make sure that the screen has updated to the screen you want to snapshot each time. It's also a good idea to
+re-run the test normally afterwards, to ensure the snapshots have been set correctly.
+
+### STAX
+
 #### Preparation
 
 First, start a container for running individual tests:

docker/Dockerfile.integration-tests

@@ -2,7 +2,7 @@ FROM ghcr.io/ledgerhq/speculos:sha-bced779
 
 RUN apt update &&                                               \
     apt install -y curl jq build-essential libsodium-dev git xxd &&	\
-    pip install base58 pytezos pytesseract &&                   \
+    pip install base58 pytezos pytesseract GitPython &&                   \
     git clone --depth 1 --branch v1.11.4 https://github.com/ledgerHQ/ragger && \
     cd ragger &&                                                \
     pip install --extra-index-url https://test.pypi.org/simple/ '.[all_backends]' && \

tests/integration/.gitignore

@@ -0,0 +1 @@
+/nano/snapshots-tmp

tests/integration/nano/nanos/regression_batched_ops.py

@@ -0,0 +1,55 @@
+#!/usr/bin/env python3
+# Copyright 2023 Functori <[email protected]>
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+
+# http://www.apache.org/licenses/LICENSE-2.0
+
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import sys
+
+file_path=os.path.abspath(__file__)
+dir_path=os.path.dirname(file_path)
+root_path=os.path.dirname(dir_path)
+sys.path.append(root_path)
+
+from utils.apdu import *
+from utils.app import *
+
+# Operation (0): Transaction
+# Fee: 0.39 tz
+# Storage limit: 6
+# Amount: 0.02 tz
+# Destination: tz1cfdVKpBb9VRBdny8BQ5RSK82UudAp2miM
+# Entrypoint: jean_bob
+# Parameter: {Pair {} (Right -76723569314251090535296646);Pair {Elt Unit (Pair {Left Unit} (Pair (Left 0x03F01167865DC63DFEE0E31251329CEAB660D94606) (Pair 0x0107B21FCA96C5763F67B286752C7AAEFC5931D15A Unit)))} (Right 3120123370638446806591421154959427514880865200209654970345);Pair {} (Left (Some Unit))}
+# Operation (1): Transaction
+# Fee: 0.65 tz
+# Storage limit: 2
+# Amount: 0.06 tz
+# Destination: KT1CYT8oACUcCSNTu2qfgB4fj5bD7szYrpti
+
+if __name__ == "__main__":
+    test_name = Path(__file__).stem
+    with nano_app() as app:
+
+        app.assert_screen(Screen.Home)
+
+        data = app.sign_with_hash(DEFAULT_ACCOUNT,
+                                  "0300000000000000000000000000000000000000000000000000000000000000006c001597c45b11b421bb806a0c56c5da5638bf4b1adbf0e617090006a09c010000bac799dfc7f6af2ff0b95f83d023e68c895020baffff086a65616e5f626f620000009a020000009507070200000000050800c6bab5ccc8d891cd8de4b6f7070707020000004b0704030b070702000000040505030b070705050a0000001503f01167865dc63dfee0e31251329ceab660d9460607070a000000150107b21fca96c5763f67b286752c7aaefc5931d15a030b050800a9df9fc1e7eaa7a9c1f7bd87a9ba9cadf5b5b2cd829deea2b7fef9070707020000000005050509030b6c01ee572f02e5be5d097ba17369789582882e8abb8790d627063202e0d403012b704944f5b5fd30eed2ab4385478488e09fe04a0000",
+                                  path=test_name)
+
+        app.check_signature_with_hash(
+            hash="e3d5453110fb40a802b56b4af7f0d5370769b8154b0e6447357cbc54cc319ed4",
+            signature="ff0b5c23a1a4ce3846ea35c135a3a5a4b8d3125c83a72a16f7a4fc0ffdc2306fbaead5a3559bf5974801055f505376bcd9abbe9bfec3d80f00c7161db0280e08",
+            data=data)
+
+        app.quit()

tests/integration/nano/nanosp/regression_potential_empty_screen.py

@@ -0,0 +1,56 @@
+#!/usr/bin/env python3
+# Copyright 2023 Functori <[email protected]>
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+
+# http://www.apache.org/licenses/LICENSE-2.0
+
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import sys
+
+file_path=os.path.abspath(__file__)
+dir_path=os.path.dirname(file_path)
+root_path=os.path.dirname(dir_path)
+sys.path.append(root_path)
+
+from utils.apdu import *
+from utils.app import *
+
+# Operation (0): Transfer ticket
+# Fee: 0.01 tz
+# Storage limit: 4
+# Contents: UNPAIR
+# Type: pair "1" 2
+# Ticketer: tz1ixvCiPJYyMjsp2nKBVaq54f6AdbV8hCKa
+# Amount: 1
+# Destination: KT18amZmM5W7qDWVt2pH6uj7sCEd3kbzLrHT
+# Entrypoint: S
+#
+# S
+# S
+# S
+
+if __name__ == "__main__":
+    test_name = Path(__file__).stem
+    with nano_app() as app:
+
+        app.assert_screen(Screen.Home)
+
+        data = app.sign_with_hash(DEFAULT_ACCOUNT,
+                                  "0300000000000000000000000000000000000000000000000000000000000000009e00ffdd6102321bc251e4a5190ad5b12b251069d9b4904e02030400000002037a0000000a076501000000013100020000ffdd6102321bc251e4a5190ad5b12b251069d9b4010100000000000000000000000000000000000000000000000008530a0a530a530a53",
+                                  path=test_name)
+
+        app.check_signature_with_hash(
+            hash="ba220e5b9af0fa350d127665049ef6dcc85304a6bd62fcc5e4f12752092af1f7",
+            signature="03399e1639e7884f86b83714e5eea2acdc56d3449f029e7258ef3bbbd35f449105d9545c3c62f7ffa088d3dfebcfa38cd316e2b4d4067cf288e9e275b8fe6901",
+            data=data)
+
+        app.quit()

tests/integration/nano/nanosp/regression_press_right_works_across_apdu_recieves.py

@@ -0,0 +1,44 @@
+#!/usr/bin/env python3
+# Copyright 2023 Functori <[email protected]>
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+
+# http://www.apache.org/licenses/LICENSE-2.0
+
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import sys
+
+file_path=os.path.abspath(__file__)
+dir_path=os.path.dirname(file_path)
+root_path=os.path.dirname(dir_path)
+sys.path.append(root_path)
+
+from utils.apdu import *
+from utils.app import *
+
+# Expression: {IF_NONE {{SWAP;IF {DIP {{DROP 1;PUSH unit Unit;PUSH bool True;PUSH string ";L\\S?p$-Fq)VDg\n]te\no4v0_8)\""}}} {{DROP 2;PUSH unit Unit;PUSH bool False;PUSH string "Li-%*edF6~?E[5Kmu?dyviwJ^2\"\\d$FyQ>>!>D$g(Qg";PUSH string "*Tx<E`SiG6Yf*A^kZ\\=7?H[mOlQ\n]Ehs"}}}} {IF_NONE {DUP} {{DROP 4;PUSH unit Unit;PUSH bool True;PUSH string "\"\\6_4\n$k%";PUSH string "c^1\"\\?Ey_1!EVb~9;EX;YU\n#Kj2ZT8h`U!X "}}};SIZE}
+
+if __name__ == "__main__":
+    test_name = Path(__file__).stem
+    with nano_app() as app:
+
+        app.assert_screen(Screen.Home)
+
+        data = app.sign_with_hash(DEFAULT_ACCOUNT,
+                                  "050200000135072f02000000c502000000c0034c072c0200000040051f02000000390200000034052000010743036c030b07430359030a07430368010000001b3b4c5c533f70242d4671295644670a5d74650a6f3476305f3829220200000072020000006d052000020743036c030b07430359030307430368010000002b4c692d252a656446367e3f455b354b6d753f64797669774a5e32225c64244679513e3e213e4424672851670743036801000000202a54783c45605369473659662a415e6b5a5c3d373f485b6d4f6c510a5d4568730200000062072f020000000203210200000054020000004f052000040743036c030b07430359030a074303680100000009225c365f340a246b25074303680100000024635e31225c3f45795f31214556627e393b45583b59550a234b6a325a54386860552158200345",
+                                  path=test_name)
+
+        app.check_signature_with_hash(
+            hash="dcba96786c40c7894527a85050bc0b75eaee37aa020a52c634275450018da478",
+            signature="22d552099146915f4c3280b3dbd4ab5d459b96b585247e58fda9bf53c9951dbe017811d074e688f11d8f47ce7f94c0548dca76be9ae9e87df794d3facdb4bb0a",
+            data=data)
+
+        app.quit()

tests/integration/nano/nanox/regression_batched_ops.py

@@ -0,0 +1,55 @@
+#!/usr/bin/env python3
+# Copyright 2023 Functori <[email protected]>
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+
+# http://www.apache.org/licenses/LICENSE-2.0
+
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import sys
+
+file_path=os.path.abspath(__file__)
+dir_path=os.path.dirname(file_path)
+root_path=os.path.dirname(dir_path)
+sys.path.append(root_path)
+
+from utils.apdu import *
+from utils.app import *
+
+# Operation (0): Transaction
+# Fee: 0.39 tz
+# Storage limit: 6
+# Amount: 0.02 tz
+# Destination: tz1cfdVKpBb9VRBdny8BQ5RSK82UudAp2miM
+# Entrypoint: jean_bob
+# Parameter: {Pair {} (Right -76723569314251090535296646);Pair {Elt Unit (Pair {Left Unit} (Pair (Left 0x03F01167865DC63DFEE0E31251329CEAB660D94606) (Pair 0x0107B21FCA96C5763F67B286752C7AAEFC5931D15A Unit)))} (Right 3120123370638446806591421154959427514880865200209654970345);Pair {} (Left (Some Unit))}
+# Operation (1): Transaction
+# Fee: 0.65 tz
+# Storage limit: 2
+# Amount: 0.06 tz
+# Destination: KT1CYT8oACUcCSNTu2qfgB4fj5bD7szYrpti
+
+if __name__ == "__main__":
+    test_name = Path(__file__).stem
+    with nano_app() as app:
+
+        app.assert_screen(Screen.Home)
+
+        data = app.sign_with_hash(DEFAULT_ACCOUNT,
+                                  "0300000000000000000000000000000000000000000000000000000000000000006c001597c45b11b421bb806a0c56c5da5638bf4b1adbf0e617090006a09c010000bac799dfc7f6af2ff0b95f83d023e68c895020baffff086a65616e5f626f620000009a020000009507070200000000050800c6bab5ccc8d891cd8de4b6f7070707020000004b0704030b070702000000040505030b070705050a0000001503f01167865dc63dfee0e31251329ceab660d9460607070a000000150107b21fca96c5763f67b286752c7aaefc5931d15a030b050800a9df9fc1e7eaa7a9c1f7bd87a9ba9cadf5b5b2cd829deea2b7fef9070707020000000005050509030b6c01ee572f02e5be5d097ba17369789582882e8abb8790d627063202e0d403012b704944f5b5fd30eed2ab4385478488e09fe04a0000",
+                                  path=test_name)
+
+        app.check_signature_with_hash(
+            hash="e3d5453110fb40a802b56b4af7f0d5370769b8154b0e6447357cbc54cc319ed4",
+            signature="ff0b5c23a1a4ce3846ea35c135a3a5a4b8d3125c83a72a16f7a4fc0ffdc2306fbaead5a3559bf5974801055f505376bcd9abbe9bfec3d80f00c7161db0280e08",
+            data=data)
+
+        app.quit()