notebook/Unit7-1-Unittest.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unit testing\n",
    "\n",
    "## TESTING AND DEBUGGING\n",
    "\n",
    "<strong style=\"color:blue\">Testing</strong> is the process of running a program to try and ascertain whether or not  it <strong style=\"color:blue\">works as intended</strong>.\n",
    "\n",
    "<strong style=\"color:blue\">Debugging</strong> is the process of trying to <strong style=\"color:blue\">fix a program </strong>that you already know does not work as intended.\n",
    "\n",
    "Good programmers **design their programs** in ways that make them <strong style=\"color:blue\"><b style=\"color:red\">easy</b> to test and debug</strong>. \n",
    "\n",
    "The key to doing this is breaking the program up into <strong style=\"color:blue\">separate</strong> components </strong>that can be \n",
    "\n",
    "* <strong style=\"color:blue\">implemented </strong>，<strong style=\"color:blue\">tested </strong>，and <strong style=\"color:blue\">debugged</strong> **independently** of other components\n",
    "\n",
    "## 1 unittest framework\n",
    "\n",
    "The unittest unit testing framework was originally inspired by **JUnit** \n",
    "\n",
    "and has a similar flavor as major unit testing frameworks in other languages. \n",
    "\n",
    "It supports test automation, sharing of setup and shutdown code for tests, aggregation of tests into collections, and independence of the tests from the reporting framework.\n",
    "\n",
    "To achieve this, **unittest** supports some important concepts in an object-oriented way:\n",
    "\n",
    "### test fixture(测试夹具）: \n",
    "\n",
    "   A test fixture represents the **preparation** needed to perform one or more tests, and any associate **cleanup actions**. \n",
    "\n",
    "  This may involve, for example, creating temporary or proxy databases, directories, or starting a server process.\n",
    "\n",
    "> Fixtures是建立一个固定/已知的**环境状态**以确保测试可重复并且按照预期方式运行\n",
    "\n",
    "### test case(测试用例): \n",
    "\n",
    "A test case is the **individual** unit of testing. \n",
    "   \n",
    "It checks for a **specific** response to a **particular** set of inputs.\n",
    "   \n",
    "**unittest** provides a base class,　**TestCase**, which may be used to create new test cases.\n",
    "\n",
    "\n",
    "### test suite(测试用例集): \n",
    "\n",
    "A test suite is a **collection** of test cases, test suites, or both.\n",
    "  \n",
    "It is used to **aggregate tests** that should be executed together.\n",
    "\n",
    "\n",
    "### test runner(测试自动执行器): \n",
    "\n",
    "A test runner is a component which orchestrates the execution of tests and provides the outcome to the user. \n",
    " \n",
    "\n",
    "\n",
    "## 2 Basic Test Structure\n",
    "\n",
    "Tests, as defined by <b>unittest</b>, have two parts: \n",
    "\n",
    "* <b>code to manage test “fixtures”</b>\n",
    "\n",
    "* <b>the test itself</b>. \n",
    "\n",
    "**Individual tests** are created by \n",
    "\n",
    "* subclassing **TestCase** \n",
    "\n",
    "* overriding or adding appropriate methods \n",
    "\n",
    "For example: \n",
    "\n",
    "* **unittest_simple.py**        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file ./code/unittest/unittest_simple.py\n",
    "\n",
    "import unittest\n",
    "\n",
    "class SimplisticTest(unittest.TestCase):\n",
    "\n",
    "    def test_true(self):\n",
    "        self.assertTrue(True)\n",
    " \n",
    "    def test(self):\n",
    "        self.assertTrue(True)\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    unittest.main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the <b>SimplisticTest</b> have \n",
    "\n",
    "* <b>test_true()</b> \n",
    "\n",
    "* <b>test()</b>\n",
    "\n",
    "methods would <b>fail if True is ever False</b>.\n",
    "\n",
    "The methods are defined with name \n",
    "\n",
    "* **start** with the letters **test**. \n",
    "\n",
    "This naming convention informs the <b>test runner</b> about which methods represent tests.\n",
    "\n",
    "## 3 Running Tests\n",
    "\n",
    "The easiest way to run unittest tests is to include:\n",
    "```python\n",
    "    if __name__ == '__main__':\n",
    "        unittest.main()\n",
    "```\n",
    "at the bottom of each test file, \n",
    "\n",
    "then simply run the script directly from the **command line**:\n",
    "```    \n",
    "   >python unittest_simple.py\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python ./code/unittest/unittest_simple.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run ./code/unittest/unittest_simple.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "includes \n",
    "\n",
    "* <b>a status indicator for each test</b> \n",
    "\n",
    "   * **”.”** on the first line of output means that a test <b>passed<b>\n",
    "\n",
    "* <b>the amount of time the tests took</b>, \n",
    "\n",
    "\n",
    "For **more** detailed test</b> results, \n",
    "\n",
    "**-v** option:\n",
    "\n",
    "```\n",
    ">python unittest_simple.py -v\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run ./code/unittest/unittest_simple.py -v"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can run tests with more detailed information by passing in the verbosity argument:\n",
    "```python\n",
    "unittest.main(verbosity=2)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file ./code/unittest/unittest_simple_more_detailed_information.py\n",
    "\n",
    "import unittest\n",
    "\n",
    "class SimplisticTest(unittest.TestCase):\n",
    "\n",
    "    def test_true(self):\n",
    "        self.assertTrue(True)\n",
    " \n",
    "    def test(self):\n",
    "        self.assertTrue(True)\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    unittest.main(verbosity=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run ./code/unittest/unittest_simple_more_detailed_information.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4 Test Outcomes\n",
    "\n",
    "Tests have 3 possible outcomes, described in the table below.\n",
    "\n",
    "| Outcome  |  Mark |  Describe  |\n",
    "|:--------:|----------:|--------:|\n",
    "|  ok    | **.** | The test passes |\n",
    "|  FAIL    | **F** | The test does not pass, and raises an **AssertionError** exception |\n",
    "|  ERROR  |  **E** |The test raises an **exception** other than `AssertionError`.|\n",
    "\n",
    "\n",
    "For Example: \n",
    "\n",
    "* **unittest_outcomes.py**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file ./code/unittest/unittest_outcomes.py\n",
    "\n",
    "import unittest\n",
    "\n",
    "class OutcomesTest(unittest.TestCase):\n",
    "\n",
    "    #   ok\n",
    "    def test_Pass(self):\n",
    "        return\n",
    "\n",
    "    # FAIL\n",
    "    def test_Fail(self):\n",
    "        # AssertionError exception.\n",
    "        self.assertFalse(True)\n",
    "        \n",
    "    # ERROR\n",
    "    def test_Error(self):\n",
    "        # raises an exception other than AssertionError\n",
    "        raise RuntimeError('Test error!')\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    unittest.main()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run ./code/unittest/unittest_outcomes.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**When a test fails or generates an error** \n",
    "\n",
    "the **traceback** is included in the output.\n",
    "In the example above, \n",
    "\n",
    "<b>testFail()</b> fails \n",
    "\n",
    "the traceback <b>shows the line</b> with the failure code.\n",
    "\n",
    "It is up to the person reading the test output to look at the code to figure out the semantic meaning of the failed test, though. \n",
    "\n",
    "### 4.1 fail with message\n",
    "\n",
    "To make it <b>easier to understand the nature of a test failure</b>,\n",
    "\n",
    "the <b>fail*() and assert*()</b> methods all accept an argument <b>msg</b>,\n",
    "\n",
    "which can be used to produce <b>a more detailed error message</b>\n",
    "\n",
    "Example: \n",
    "\n",
    "* **unittest_failwithmessage.py**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file ./code/unittest/unittest_failwithmessage.py\n",
    "\n",
    "import unittest\n",
    "\n",
    "class FailureMessageTest(unittest.TestCase):\n",
    "\n",
    "    def test_Fail(self):\n",
    "        self.assertFalse(True,'Should be False')\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    unittest.main()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run ./code/unittest/unittest_failwithmessage.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5 Assert methods\n",
    "\n",
    "The **TestCase** class provides several assert methods to check for and report failures. \n",
    "\n",
    "* https://docs.python.org/3/library/unittest.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1 Asserting Truth\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file ./code/unittest/unittest_true.py\n",
    "\n",
    "import unittest\n",
    "\n",
    "class TruthTest(unittest.TestCase):\n",
    "\n",
    "    def testAssertTrue(self):\n",
    "        self.assertTrue(True)\n",
    "\n",
    "    def test_AssertFalse(self):\n",
    "        self.assertFalse(False)\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    unittest.main()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run ./code/unittest/unittest_true.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 Testing Equality\n",
    "\n",
    "As a special case, `unittest` includes methods for testing <b>the equality of two values</b>.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file ./code/unittest/unittest_equality.py\n",
    "\n",
    "import unittest\n",
    "\n",
    "class EqualityTest(unittest.TestCase):\n",
    "\n",
    "    def test_Equal(self):\n",
    "        self.assertEqual(1, 3)\n",
    "\n",
    "    def test_NotEqual(self):\n",
    "        self.assertNotEqual(2, 1)\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    unittest.main()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run ./code/unittest/unittest_equality.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These special tests are handy, since the values being <b>compared appear in the failure message</b> when a test fails."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file ./code/unittest/unittest_notequal.py\n",
    "import unittest\n",
    "\n",
    "class InequalityTest(unittest.TestCase):\n",
    "\n",
    "    def test_Equal(self):\n",
    "        self.assertNotEqual(1, 1)\n",
    "\n",
    "    def test_NotEqual(self):\n",
    "        self.assertEqual(2, 1)\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    unittest.main()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run ./code/unittest/unittest_notequal.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.3 Almost Equal\n",
    "\n",
    "In addition to strict equality, it is possible to test for\n",
    "\n",
    "**near equality of floating point numbers** using\n",
    "\n",
    "* assertNotAlmostEqual()\n",
    "\n",
    "* assertAlmostEqual()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file ./code/unittest/unittest_almostequal.py\n",
    "import unittest\n",
    "\n",
    "class AlmostEqualTest(unittest.TestCase):\n",
    "\n",
    "    def test_NotAlmostEqual(self):\n",
    "        self.assertNotAlmostEqual(1.11, 1.3, places=1)\n",
    "\n",
    "    def test_AlmostEqual(self):\n",
    "       # self.assertAlmostEqual(1.1, 1.3, places=0)\n",
    "        self.assertAlmostEqual(0.12345678, 0.12345679) \n",
    "\n",
    "if __name__ == '__main__':\n",
    "    unittest.main()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run ./code/unittest/unittest_almostequal.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The arguments are the values to be compared, and **the number of decimal places** to use for the test.\n",
    "\n",
    "`assertAlmostEquals()` and `assertNotAlmostEqual()`  have an optional parameter named\n",
    "\n",
    "**places** \n",
    "\n",
    "and the numbers are compared by **computing the difference rounded to number of decimal places**.\n",
    "\n",
    "default **places=7**,\n",
    "\n",
    "hence:\n",
    "\n",
    "\n",
    "```python\n",
    "self.assertAlmostEqual(0.12345678, 0.12345679) is True.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6 Test Fixtures\n",
    "\n",
    "<b>Fixtures are resources needed by a test</b> \n",
    "\n",
    "* if you are writing several tests for the same class, those tests all need **an instance of that class** to use for testing. \n",
    "\n",
    "\n",
    "* test fixtures include `database` connections and temporary `files ` (many people would argue that using external resources makes such tests not “unit” tests, but they are still tests and still useful). \n",
    "\n",
    "**TestCase** includes a special hook to **configure** and **clean up** any fixtures needed by your tests.\n",
    "\n",
    "* To configure the **fixtures**, override **setUp()**.\n",
    "\n",
    "* To clean up, override **tearDown()**.\n",
    "\n",
    "### 6.1 setUp()\n",
    "\n",
    "Method called to **prepare** the test fixture. This is called immediately **before** calling the test method; \n",
    "\n",
    "### 6.2 tearDown()\n",
    "\n",
    "Method called immediately **after** the test method has been called and the result recorded.\n",
    "\n",
    "This is `called` even if the test method `raised an exception`, so the implementation in subclasses may need to be particularly careful about checking internal state.\n",
    "\n",
    "Any exception, other than `AssertionError` or `SkipTest,` raised by this method will be considered an `error` rather than a test failure. \n",
    "\n",
    "This method will only be called if the `setUp()` succeeds,  whether the test method succeeded or not.\n",
    "\n",
    "\n",
    "* automatically call `setUp()`  and `tearDown()` \n",
    "\n",
    "   The testing framework will automatically call `setUp()`  and `tearDown()` for **every single test** we run.\n",
    "\n",
    "* any `exception` raised by this `setUp()`  and `tearDown()`  will be considered an `error` rather than a test failure. \n",
    "\n",
    "Such a working environment for the testing code is called a `fixture`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file ./code/unittest/unittest_fixtures.py\n",
    "\n",
    "import unittest\n",
    "\n",
    "class FixturesTest(unittest.TestCase):\n",
    "\n",
    "    def setUp(self):\n",
    "        print('In setUp()')\n",
    "        self.fixture = range(1, 10)\n",
    "\n",
    "    def tearDown(self):\n",
    "        print('In tearDown()')\n",
    "        del self.fixture\n",
    "\n",
    "    def test_fixture1(self):\n",
    "        print('in test1()')\n",
    "        self.assertEqual(self.fixture, range(1, 10))\n",
    "     \n",
    "    def test_fixture2(self):\n",
    "        print('in test2()')\n",
    "        self.assertEqual(self.fixture, range(2, 10))\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    unittest.main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When this sample test is run, you can see \n",
    "\n",
    "**the order of execution** of the `fixture` and `test` methods:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run ./code/unittest/unittest_fixtures.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.3  Any `exception` raised \n",
    "\n",
    "**Any `exception` raised by this `setUp()` or `tearDown()`**\n",
    " \n",
    "* This **tearDown()** method will **only** be called if the **setUp() succeeds**,  whether the test method succeeded or not.\n",
    "\n",
    "* Any **exception** raised by this **setUp()**  and **tearDown()**  will be considered an **error** rather than **a test failure.** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file ./code/unittest/unittest_fixtures_exception.py\n",
    "\n",
    "import unittest\n",
    "\n",
    "class FixturesTest(unittest.TestCase):\n",
    "\n",
    "    def setUp(self):\n",
    "        print('In setUp()')\n",
    "        r=1/0\n",
    "        self.fixture = range(1, 10)\n",
    "\n",
    "    def tearDown(self):\n",
    "        print('In tearDown()')\n",
    "        r=1/0\n",
    "        del self.fixture\n",
    "\n",
    "    def test_fixture1(self):\n",
    "        print('in test1()')\n",
    "        self.assertEqual(self.fixture, range(1, 10))\n",
    "     \n",
    "    def test_fixture2(self):\n",
    "        print('in test2()')\n",
    "        self.assertEqual(self.fixture, range(2, 10))\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    unittest.main()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run ./code/unittest/unittest_fixtures_exception.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7 Test Suites\n",
    "\n",
    "**Test case instances** are grouped together according to the features they test. \n",
    "\n",
    "`unittest` provides a mechanism for this: the **test suite**, represented by unittest‘s **TestSuite** class. \n",
    "\n",
    "In most cases, calling `unittest.main()` will do the right thing and collect all the module’s test cases for you, and then execute them.\n",
    "\n",
    "However, should you want to `customize` the building of your test suite, you can do it yourself:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file ./code/unittest/test_TestSuite.py\n",
    "\n",
    "import unittest\n",
    "\n",
    "class EqualityTest(unittest.TestCase):\n",
    "\n",
    "    def test_Equal(self):\n",
    "        self.assertEqual(1, 3)\n",
    "\n",
    "    def test_NotEqual(self):\n",
    "        self.assertNotEqual(3, 3)   \n",
    "\n",
    "class AlmostEqualTest(unittest.TestCase):\n",
    "\n",
    "    def test_NotAlmostEqual(self):\n",
    "        self.assertNotAlmostEqual(1.2, 1.1, places=1)\n",
    "\n",
    "    def test_AlmostEqual(self):\n",
    "        self.assertAlmostEqual(1.1, 1.3, places=0)\n",
    "        \n",
    "def suiteEqual():\n",
    "    suite = unittest.TestSuite()\n",
    "    suite.addTest(EqualityTest('test_Equal'))\n",
    "    suite.addTest(AlmostEqualTest('test_AlmostEqual'))\n",
    "    return suite\n",
    "\n",
    "def suiteNotEqual():\n",
    "    suite = unittest.TestSuite()\n",
    "    suite.addTest(EqualityTest('test_NotEqual'))\n",
    "    suite.addTest(AlmostEqualTest('test_NotAlmostEqual'))\n",
    "    return suite\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    unittest.main(defaultTest = 'suiteNotEqual')\n",
    "    #unittest.main(defaultTest = 'suiteEqual')\n",
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run ./code/unittest/test_TestSuite.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8 Test  Algorithms\n",
    "\n",
    "Test Sorting Algorithms \n",
    "\n",
    "* Sorting Algorithms Module\n",
    "* Test Module\n",
    "\n",
    "### 8.1 Sorting Algorithms Module"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file ./code/unittest/sort.py\n",
    "\n",
    "def select_sort(L):\n",
    "    length=len(L)\n",
    "    for i in range(length):\n",
    "        min_idx = i  # assume fist element is the smallest\n",
    "        for j in range(i+1, length):\n",
    "            if L[j]<L[min_idx] :\n",
    "                min_idx = j\n",
    "        if min_idx!=i:\n",
    "            L[i], L[min_idx] = L[min_idx], L[i]   \n",
    "\n",
    "def merge(left, right, compare = lambda x,y:x<y):\n",
    "    result = []  # the copy of the list.\n",
    "    i,j = 0, 0\n",
    "    while i < len(left) and j < len(right):\n",
    "        if compare(left[i], right[j]):\n",
    "            result.append(left[i])\n",
    "            i += 1\n",
    "        else:\n",
    "            result.append(right[j])\n",
    "            j += 1\n",
    "    while (i < len(left)):\n",
    "        result.append(left[i])\n",
    "        i += 1\n",
    "    while (j < len(right)):\n",
    "        result.append(right[j])\n",
    "        j += 1\n",
    "    return result\n",
    "\n",
    "def merge_sort(L, compare = lambda x,y:x<y):\n",
    "    if len(L) < 2:\n",
    "        return L[:] \n",
    "    else:\n",
    "        middle = len(L)//2\n",
    "        left = merge_sort(L[:middle], compare)\n",
    "        right = merge_sort(L[middle:], compare)\n",
    "        return merge(left, right, compare)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8.2 The Test  Module\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file ./code/unittest/test_sort.py\n",
    "import unittest\n",
    "from sort import *\n",
    "\n",
    "class sortTest(unittest.TestCase):\n",
    "\n",
    "    def setUp(self):\n",
    "        self.L=[7, 4, 5, 9, 8, 2, 1]\n",
    "        self.sortedL=[1, 2, 4, 5, 7, 8, 9]\n",
    "   \n",
    "    def test_select_sort(self):\n",
    "        select_sort(self.L)\n",
    "        self.assertEqual(self.L,self.sortedL)\n",
    "        \n",
    "    def test_merge_sort(self):\n",
    "        L1=merge_sort(self.L)\n",
    "        self.assertEqual(L1,self.sortedL) \n",
    "        \n",
    "if __name__ == '__main__':\n",
    "    unittest.main()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run ./code/unittest/test_sort.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The test code in  **separate**  module  have  several advantages:\n",
    "\n",
    "* The **test module** can be `run` **standalone** .\n",
    "\n",
    "* The **test code** can more easily be **separated from shipped code**.\n",
    "\n",
    "* If the **testing strategy** changes, there is **no need to change the source code**.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9 Black and Glass Box Testing\n",
    "\n",
    "\n",
    "### The Test Suite\n",
    "\n",
    "The **key** to testing is <b style=\"color:blue\">finding</b> a collection of inputs, called  <b style=\"color:blue\">a test suite</b>, that has a high likelihood of <b style=\"color:blue\">revealing bugs, yet does not take too long to run</b>.\n",
    "\n",
    "Typically, people rely on the `specifications`  and `code` to constructe **a test suite**\n",
    " \n",
    "* **Black-Box**(黑箱）testing:  Heuristics based on `exploring paths` through the `specification`（基于软件功能描述设计测试路径的方法）\n",
    "\n",
    "* **Glass-Box**(白箱 White-Box）testing: Heuristics based on `exploring paths` through the `code`（基于代码分析设计测试路径的方法）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.1 Black-Box Testing\n",
    "\n",
    "**Black-box** testing tests the functionality of an application by looking its **specifications.**\n",
    "\n",
    "In principle,black-box tests are constructed `without looking at the code` to be tested.\n",
    "\n",
    "Black-box testing allows **testers** and **implementers** to be drawn from `separate populations`.\n",
    "\n",
    ">  **testers** -- |separate| -- **implementers**\n",
    "\n",
    "**Positive feature of black-box**\n",
    "\n",
    "1. This independence **reduces** the likelihood of generating **test suites** that exhibit **mistakes** that are correlated with **`mistakes` in the code**. \n",
    "\n",
    "2. black-box testing is **robust** with respect to `implementation` **changes**.\n",
    "\n",
    "\n",
    "**The good ways to generate black-box test cases**\n",
    "\n",
    "* 1 `explore paths` through a  `specification`（基于软件功能描述设计测试数据)\n",
    "\n",
    "* 2 `Boundary conditions` should be tested.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Consider the following code specification: \n",
    "```python\n",
    "def union(set1, set2):\n",
    "   \"\"\"\n",
    "   set1 and set2 are collections of objects, each of which might be empty.\n",
    "   Each set has no duplicates within itself, but there may be objects that\n",
    "   are in both sets. Objects are assumed to be of the same type.\n",
    "\n",
    "   This function returns one set containing all elements from\n",
    "   both input sets, but with no duplicates.\n",
    "   \"\"\"\n",
    "```\n",
    "\n",
    "According to the **specifications**, the possibilities for set1 and set2 are as follows: \n",
    "\n",
    "\n",
    "* both sets are empty; \n",
    "\n",
    "  * set1 is an empty set; set2 is an empty set - ```union('','')```\n",
    "\n",
    "* one of the sets is empty and one has at least one object;\n",
    "\n",
    "  * set1 is an empty set; set2 is of size greater than or equal to 1 -  ```union('','a')```\n",
    "\n",
    "  * set1 is of size greater than or equal to 1; set2 is an empty set - ```union('b','')```\n",
    "\n",
    "* both sets are not empty. \n",
    "\n",
    "   * set1 and set2 are both nonempty sets which do not contain any objects in common - ```union('a','b')```\n",
    " \n",
    "   * set1 and set2 are both nonempty sets which contain objects in common - ```union('ab','bcd')```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file ./code/unittest/unionsets.py\n",
    "def union(set1, set2):\n",
    "    \"\"\"\n",
    "     set1 and set2 are collections of objects, each of which might be empty.\n",
    "     Each set has no duplicates within itself, but there may be objects that\n",
    "     are in both sets. Objects are assumed to be of the same type.\n",
    "\n",
    "     This function returns one set containing all elements from\n",
    "     both input sets, but with no duplicates.\n",
    "    \"\"\"\n",
    "    if len(set1) == 0:\n",
    "        return set2\n",
    "    elif set1[0] in set2:\n",
    "        return union(set1[1:], set2)\n",
    "    else:\n",
    "        return set1[0] + union(set1[1:], set2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file ./code/unittest/test_union_black_box.py\n",
    "import unittest\n",
    "from unionsets import union\n",
    "\n",
    "class union_black_box(unittest.TestCase):\n",
    "\n",
    "    def setUp(self):\n",
    "        self.inputsets=[(\"\",\"\"),\n",
    "                         (\"\",\"a\"),\n",
    "                         (\"b\",\"\"),\n",
    "                         (\"a\",\"b\"),\n",
    "                         (\"ab\",\"bcd\")]\n",
    "        self.unionsets=[\"\",\"a\",\"b\",\"ab\",\"abcd\"]\n",
    "    \n",
    "   \n",
    "    def test_union(self):\n",
    "        for i,item in enumerate(self.inputsets):\n",
    "            set1,set2=item\n",
    "            self.assertEqual(self.unionsets[i],union(set1,set2))\n",
    "        \n",
    "        \n",
    "if __name__ == '__main__':\n",
    "    unittest.main()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run ./code/unittest/test_union_black_box.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.2 Glass-Box Testing\n",
    "\n",
    "**`Black-box` testing should never be skipped, but it is `rarely` sufficient**. \n",
    "\n",
    "Without looking at the internal structure of the code,\n",
    "\n",
    "* it is **impossible** to know `which test cases` are likely to `provide new information`. \n",
    "\n",
    "The **Glass-Box** method to constructing the test cases from the **code** written\n",
    "\n",
    "\n",
    "#### 9.2.1 A path-complete glass box test suite \n",
    "\n",
    "A path-complete glass box test suite would find test cases that go through **every possible path** in the code.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file ./code/unittest/max_three.py\n",
    "def max_three(a,b,c) :\n",
    "    \"\"\"\n",
    "      a, b, and c are numbers\n",
    "    returns: the maximum of a, b, and c        \n",
    "    \"\"\"\n",
    "    if a > b:\n",
    "        bigger = a\n",
    "    else:\n",
    "        bigger = b\n",
    "    if c > bigger:\n",
    "        bigger = c\n",
    "    return bigger\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, that means finding all possibilities for the conditional tests $a > b$ and $c > bigger$. \n",
    " \n",
    "So, we end up with four possible paths :\n",
    " \n",
    "* Case 1: a > b and c > bigger - maxOfThree(2, -10, 100).\n",
    "* Case 2: a > b and c <= bigger - maxOfThree(6, 1, 5)\n",
    "* Case 3: a <= b and c > bigger - maxOfThree(7, 9, 10).\n",
    "* Case 4: a <= b and c <= bigger - maxOfThree(0, 40, 20) \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file ./code/unittest/test_max_three_glass_box.py\n",
    "import unittest\n",
    "from max_three import *\n",
    "\n",
    "class max_three_glass_box(unittest.TestCase):\n",
    "\n",
    "    def setUp(self):\n",
    "        self.testinputs=[(2, -10, 100),\n",
    "                         (6, 1, 5),\n",
    "                         (7, 9, 10),\n",
    "                         (0, 40, 20) ]\n",
    "        self.okvalues=[100,6,10,40]\n",
    "    \n",
    "   \n",
    "    def test_max_three(self):\n",
    "        for i,item in enumerate(self.testinputs):\n",
    "            a,b,c=item\n",
    "            self.assertEqual(self.okvalues[i],max_three(a,b,c))\n",
    "        \n",
    "        \n",
    "if __name__ == '__main__':\n",
    "    unittest.main()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run ./code/unittest/test_max_three_glass_box.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 9.2.2 A good glass-box test suite\n",
    "\n",
    "A good glass-box test suite would try to test a good sample of **all the possible paths** through the code.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file ./code/unittest/unionsets.py\n",
    "def union(set1, set2):\n",
    "    \"\"\"\n",
    "     set1 and set2 are collections of objects, each of which might be empty.\n",
    "     Each set has no duplicates within itself, but there may be objects that\n",
    "     are in both sets. Objects are assumed to be of the same type.\n",
    "\n",
    "     This function returns one set containing all elements from\n",
    "     both input sets, but with no duplicates.\n",
    "    \"\"\"\n",
    "    if len(set1) == 0:\n",
    "        return set2\n",
    "    elif set1[0] in set2:\n",
    "        return union(set1[1:], set2)\n",
    "    else:\n",
    "        return set1[0] + union(set1[1:], set2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, it should contain tests that test \n",
    "\n",
    "* `if len(set1) == 0` : when set1 is empty,  ```union('', 'abc')```\n",
    "\n",
    "* `elif set1[0] in set2` :when set1[0] is in set2 ```union('a', 'abc')```\n",
    " ``\n",
    "* `else`: when set1[0] is not in set2. ```union('d', 'abc')```\n",
    "\n",
    "* The test suite should also test when the `recursion depth` is 0, 1, and greater than 1.\n",
    "\n",
    "  * Recursion depth = 0 - ```union('', 'abc')```\n",
    "  * Recursion depth = 1 - ```union('a', 'abc'), union('d', 'abc')```\n",
    "  * Recursion depth > 1 - ```union('ab', 'abc')```\n",
    "\n",
    "> Note that this test suite is **NOT path complete** because it would take essentially infinite time to test all possible **recursive depths**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file ./code/unittest/test_union_glass_box.py\n",
    "import unittest\n",
    "from unionsets import union\n",
    "\n",
    "class union_black_box(unittest.TestCase):\n",
    "\n",
    "    def setUp(self):\n",
    "        self.inputsets=[(\"\",\"abc\"),\n",
    "                         (\"a\",\"abc\"),\n",
    "                         (\"d\",\"abc\"),\n",
    "                         (\"ab\",\"abc\")]\n",
    "        self.unionsets=[\"abc\",\"abc\",\"dabc\",\"abc\"]\n",
    "    \n",
    "   \n",
    "    def test_union(self):\n",
    "        for i,item in enumerate(self.inputsets):\n",
    "            set1,set2=item\n",
    "            self.assertEqual(self.unionsets[i],union(set1,set2))\n",
    "        \n",
    "        \n",
    "if __name__ == '__main__':\n",
    "    unittest.main()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run ./code/unittest/test_union_glass_box.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10 Code Coverage\n",
    "\n",
    "Coverage.py measures code coverage, typically during test execution.\n",
    "\n",
    "It uses the code analysis tools and tracing hooks provided in the Python standard library to determine which lines are executable, and which have been executed.\n",
    "\n",
    ">在测试中，**代码覆盖率**被作为一种测试评判依据，特定的测试套件覆盖被侧源代码的程度来度量\n",
    ">\n",
    ">在Python代码中用来分析代码覆盖率的工具有很多，Coverage是其中之个。\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install coverage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!coverage run ./code/unittest/test_union_glass_box.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use coverage report to report on the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!coverage report -m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Stmts: 执行的行数\n",
    " \n",
    "* Miss: 没有执行的行数\n",
    " \n",
    "* Cover: 覆盖百分比"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further Reading\n",
    "\n",
    "### Python：unittest\n",
    "\n",
    "* [Python：unittest — Unit testing framework](https://docs.python.org/3/library/unittest.html)\n",
    "\n",
    "* [Pymotw：unittest — Automated Testing Framework](https://pymotw.com/3/unittest/index.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.5 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "165px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  },
  "vscode": {
   "interpreter": {
    "hash": "369f2c481f4da34e4445cda3fffd2e751bd1c4d706f27375911949ba6bb62e1c"
   }
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}