Tests fail on some architectures (using unittest.mock)

[johanneskastl]

I am trying to package for openSUSE this as it is a dependency of pynitrokey.

As openSUSE tries to remove use of python-mock, I simply switched all tests to unittest.mock:

find tests -type f -exec sed -i 's/from mock/from unittest.mock/g' {} +                                                                                                                      
find tests -type f -exec sed -i 's/import mock/import unittest.mock as mock/g' {} +

Worked like charm (and might be usable for #149), as the tests pass successfully on x86_64.

However there are some that fail on non-x86:

armv7l

• tests/unit/test_library.py::TestLibrary::test_initialize_windows

i586

• tests/unit/test_library.py::TestLibrary::test_initialize_windows

ppc64

• tests/unit/unlockers/test_unlock_kinetis.py::TestUnlockKinetis::test_unlock_kinetis_read_fail
• tests/unit/unlockers/test_unlock_kinetis.py::TestUnlockKinetis::test_unlock_kinetis_success

s390x

• tests/unit/unlockers/test_unlock_kinetis.py::TestUnlockKinetis::test_unlock_kinetis_read_fail
• tests/unit/unlockers/test_unlock_kinetis.py::TestUnlockKinetis::test_unlock_kinetis_success

Errors

[   32s] =================================== FAILURES ===================================
[   32s] _____________________ TestLibrary.test_initialize_windows ______________________
[   32s] 
[   32s] self = <tests.unit.test_library.TestLibrary testMethod=test_initialize_windows>
[   32s] mock_ctypes = <MagicMock name='ctypes' id='4116548240'>
[   32s] mock_find_library = <MagicMock name='find_library' id='4116458680'>
[   32s] mock_open = <MagicMock name='open' id='4116527232'>
[   32s] 
[   32s]     @mock.patch('sys.platform', new='windows')
[   32s]     @mock.patch('pylink.library.open')
[   32s]     @mock.patch('os.remove', new=mock.Mock())
[   32s]     @mock.patch('tempfile.NamedTemporaryFile', new=mock.Mock())
[   32s]     @mock.patch('ctypes.util.find_library')
[   32s]     @mock.patch('pylink.library.ctypes')
[   32s]     def test_initialize_windows(self, mock_ctypes, mock_find_library, mock_open):
[   32s]         """Tests creating a library on a Windows machine.
[   32s]     
[   32s]         Args:
[   32s]           self (TestLibrary): the ``TestLibrary`` instance
[   32s]           mock_ctypes (Mock): a mocked version of the ctypes library
[   32s]           mock_find_library (Mock): mock for mocking the
[   32s]             ``ctypes.util.find_library()`` call
[   32s]           mock_open (Mock): mock for mocking the call to ``open()``
[   32s]     
[   32s]         Returns:
[   32s]           ``None``
[   32s]         """
[   32s]         mock_windll = mock.Mock()
[   32s]         mock_windll.__getitem__ = mock.Mock()
[   32s]     
[   32s]         mock_cdll = mock.Mock()
[   32s]         mock_cdll.__getitem__ = mock.Mock()
[   32s]     
[   32s]         mock_ctypes.windll = mock_windll
[   32s]         mock_ctypes.cdll = mock_cdll
[   32s]         mock_find_library.return_value = self.lib_path
[   32s]     
[   32s]         lib = library.Library()
[   32s]         lib.unload = mock.Mock()
[   32s]     
[   32s] >       mock_find_library.assert_called_once_with(library.Library.WINDOWS_64_JLINK_SDK_NAME)
[   32s] 
[   32s] tests/unit/test_library.py:250: 
[   32s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   32s] /usr/lib/python3.9/unittest/mock.py:919: in assert_called_once_with
[   32s]     return self.assert_called_with(*args, **kwargs)
[   32s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   32s] 
[   32s] self = <MagicMock name='find_library' id='4116458680'>, args = ('JLink_x64',)
[   32s] kwargs = {}, expected = call('JLink_x64'), actual = call('JLinkARM')
[   32s] _error_message = <function NonCallableMock.assert_called_with.<locals>._error_message at 0xf5f42928>
[   32s] cause = None
[   32s] 
[   32s]     def assert_called_with(self, /, *args, **kwargs):
[   32s]         """assert that the last call was made with the specified arguments.
[   32s]     
[   32s]         Raises an AssertionError if the args and keyword args passed in are
[   32s]         different to the last call to the mock."""
[   32s]         if self.call_args is None:
[   32s]             expected = self._format_mock_call_signature(args, kwargs)
[   32s]             actual = 'not called.'
[   32s]             error_message = ('expected call not found.\nExpected: %s\nActual: %s'
[   32s]                     % (expected, actual))
[   32s]             raise AssertionError(error_message)
[   32s]     
[   32s]         def _error_message():
[   32s]             msg = self._format_mock_failure_message(args, kwargs)
[   32s]             return msg
[   32s]         expected = self._call_matcher(_Call((args, kwargs), two=True))
[   32s]         actual = self._call_matcher(self.call_args)
[   32s]         if actual != expected:
[   32s]             cause = expected if isinstance(expected, Exception) else None
[   32s] >           raise AssertionError(_error_message()) from cause
[   32s] E           AssertionError: expected call not found.
[   32s] E           Expected: find_library('JLink_x64')
[   32s] E           Actual: find_library('JLinkARM')
[   32s] 
[   32s] /usr/lib/python3.9/unittest/mock.py:907: AssertionError

[   25s] =================================== FAILURES ===================================
[   25s] _______________ TestUnlockKinetis.test_unlock_kinetis_read_fail ________________
[   25s] 
[   25s] self = <tests.unit.unlockers.test_unlock_kinetis.TestUnlockKinetis testMethod=test_unlock_kinetis_read_fail>
[   25s] mock_sleep = <MagicMock name='sleep' id='140735395602928'>
[   25s] 
[   25s]     @mock.patch('time.sleep')
[   25s]     def test_unlock_kinetis_read_fail(self, mock_sleep):
[   25s]         """Tests unlock Kinetis failing during a read.
[   25s]     
[   25s]         At any point, the unlock can fail if it tries to read from the an
[   25s]         MDM-AP register and a fault occurs.
[   25s]     
[   25s]         Args:
[   25s]           self (TestUnlockKinetis): the `TestUnlockKinetis` instance
[   25s]           mock_sleep (Mock): mocked `time.sleep()` function
[   25s]     
[   25s]         Returns:
[   25s]           `None`
[   25s]         """
[   25s]         mock_jlink = mock.Mock()
[   25s]         mock_jlink.tif = enums.JLinkInterfaces.SWD
[   25s]     
[   25s]         flags = (0x2 << 28) | (0xBA01 << 12) | 1
[   25s]         mock_jlink.coresight_read.side_effect = [flags, 0xFF]
[   25s]     
[   25s]         mock_jlink.swd_write.return_value = 0   # ACK
[   25s]         mock_jlink.swd_read8.return_value = -1  # Invalid Read
[   25s]         mock_jlink.swd_read32.return_value = 2  # Data
[   25s]     
[   25s]         res = unlock.unlock_kinetis(mock_jlink)
[   25s]         self.assertFalse(res)
[   25s]     
[   25s] >       self.assertEqual(1, mock_jlink.set_reset_pin_low.call_count)
[   25s] E       AssertionError: 1 != 0
[   25s] 
[   25s] tests/unit/unlockers/test_unlock_kinetis.py:157: AssertionError
[   25s] ________________ TestUnlockKinetis.test_unlock_kinetis_success _________________
[   25s] 
[   25s] self = <tests.unit.unlockers.test_unlock_kinetis.TestUnlockKinetis testMethod=test_unlock_kinetis_success>
[   25s] mock_sleep = <MagicMock name='sleep' id='140735394886032'>
[   25s] 
[   25s]     @mock.patch('time.sleep')
[   25s]     def test_unlock_kinetis_success(self, mock_sleep):
[   25s]         """Tests unlock Kinetis succesfully unlocking the device.
[   25s]     
[   25s]         Args:
[   25s]           self (TestUnlockKinetis): the `TestUnlockKinetis` instance
[   25s]           mock_sleep (Mock): mocked `time.sleep()` function
[   25s]     
[   25s]         Returns:
[   25s]           `None`
[   25s]         """
[   25s]         mock_jlink = mock.Mock()
[   25s]         mock_jlink.tif = enums.JLinkInterfaces.SWD
[   25s]     
[   25s]         flags = (0x2 << 28) | (0xBA01 << 12) | 1
[   25s]         mock_jlink.coresight_read.side_effect = [flags, 0xFF]
[   25s]     
[   25s]         ack = swd.Response.STATUS_ACK
[   25s]         wait = swd.Response.STATUS_WAIT
[   25s]     
[   25s]         mocked_data = mock.Mock()
[   25s]         mocked_data.read_data = [
[   25s]             0x0,  # Wait
[   25s]             0x1,  # Ack
[   25s]             0x3,  # Flash Ready Bit Set
[   25s]             0x1,  # Ack
[   25s]             0x3,  # Erase Command Bit Set
[   25s]             0x1,  # Ack
[   25s]             0x0,  # Flash Erase In Progress Bit Cleared
[   25s]         ]
[   25s]     
[   25s]         mocked_data.status_and_parity_data = [
[   25s]             ack,   # Error clearing write request.
[   25s]             ack,   # Selecting the MDM-AP Status Register
[   25s]             wait,  # First poll, wait
[   25s]             ack,   # Second poll, continue
[   25s]             0x1,   # Parity
[   25s]             ack,   # Flash Ready ACK
[   25s]             0x0,   # Flash Ready Parity
[   25s]             0x1,   # Mass Erase Request
[   25s]             ack,   # Erase Command ACK
[   25s]             0x1,   # Erase Command Parity
[   25s]             ack,   # Erase Command ACK
[   25s]             0x0,   # Erase Command Parity
[   25s]             ack,   # Flash Erase Command ACK
[   25s]             0x1,   # Flash Erase Command Parity
[   25s]             ack,   # Flash Erase Command ACK
[   25s]             0x0,   # Flash Erase Command Parity
[   25s]         ]
[   25s]     
[   25s]         # ACK
[   25s]         mock_jlink.swd_write.return_value = 0
[   25s]     
[   25s]         # Status and Parity
[   25s]         mock_jlink.swd_read8.side_effect = mocked_data.status_and_parity_data
[   25s]     
[   25s]         # Data
[   25s]         mock_jlink.swd_read32.side_effect = mocked_data.read_data
[   25s]     
[   25s]         res = unlock.unlock_kinetis(mock_jlink)
[   25s] >       self.assertTrue(res)
[   25s] E       AssertionError: False is not true
[   25s] 
[   25s] tests/unit/unlockers/test_unlock_kinetis.py:221: AssertionError

[hkpeprah]

We can probably switch to unittest.mock in dropping support for Python 2. As for the other things, we've only tested on OSX (x86_64), Windows, and Ubuntu / CentOS (x86_64). I would have to look to see GH has support for ARM-based OSX, if so, we should probably support that given that is what Apple is moving towards. As for i586 et. al, unless we have a runner to validate against in an automated fashion, I don't think we would be able to support stability on those architectures.

[hkpeprah]

Looks like M1 runners are supported, so this is something we can do for the unit testing: github/roadmap#528