Unittest framework : Automatic test discovery not working if test has a relative import

[Dr-Irv]

This is with Python Language Server version 0.3.43.0 on Windows 10, Anaconda python 3.7.3.

Have a folder with the following structure:

mypkg
  --> foo.py
  --> __init__.py
mypkg_test
  --> __init__.py
  --> base.py
  --> test_foo.py

The files base.py, and both __init__.py are empty.

Here is foo.py:

def goo():
    return 5

Here is test_foo.py:

import unittest
from . import base
from mypkg.foo import goo


class FooTest2(unittest.TestCase):
    def test_foo(self):
        x = goo()
        self.assertTrue(5 == x)
        
if __name__ == '__main__':
    unittest.main()

Here is settings.json in the .vscode folder in that directory:

{
    "python.testing.unittestArgs": [
        "-v",
        "-s",
        "./mypkg_test",
        "-p",
        "test*.py"
    ],
    "python.testing.pytestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.unittestEnabled": true
}

If you comment out from . import base in test_foo.py, then the test is discovered. Commenting and uncommenting that line makes the test discovered versus not discovered, as evidenced by seeing the buttons Run Test|Debug Test in the editor.

If you rename mypkg_test to test, update the test settings to point to the newly named directory, and leave the source with the line from . import base present, then the test is discovered.

So the name of the test folder seems to affect test discovery when doing a relative import inside one of the tests.

We have a big source code with this naming structure (and I don't control the naming) and base.py has a lot more stuff in it, and we can run tests just fine from the command line, but not from VS Code.

[karrtikr]

If you rename mypkg_test to test, update the test settings to point to the newly named directory, and leave the source with the line from . import base present, then the test is discovered.

I am not able to confirm this. For me, tests are still not discovered if I rename mypkg_test to test. But having a relative import breaking discovery is still a bug.

As a workaround, I would ask you to try the pytest framework instead of unittest. Use Python: Configure Tests from command palette to configure pytest, I can confirm it works.

settings.json would look something like this

{
    "python.testing.pytestEnabled": true,
    "python.testing.nosetestsEnabled": false,
    "python.testing.unittestEnabled": false,
    "python.testing.pytestArgs": [
        "<Test_directory_name>"
    ]
}

[Dr-Irv]

@karrtikr We are working with an existing large code base that already uses unittest, so switching to pytest really isn't an option. Hopefully, this bug will get fixed

[karrtikr]

I see. Just FYI the pytest framework also discovers tests discovered by unittest framework. (i.e the tests using unittest module)

[ayagodinsc]

Having same problem with vscode not discovering tests if relative import is used :(

[C-nit]

By default the unittests top-level-directory is the start-directory -s

  -t TOP, --top-level-directory TOP
                        Top level directory of project (defaults to start
                        directory)

So I guess it's the "relative import beyond top-level package" problem, see https://stackoverflow.com/a/47030746/5955544, and should only happen with top level imports.

Try

    "python.testing.unittestArgs": [
        "-v",
        "-t",
        ".",
        "-s",
        "./mypkg_test",
        "-p",
        "test*.py"
    ],

[Dr-Irv]

@C-nit I tried what you said with the example provided above and see no change. No tests discovered.

[thalesaguiar21]

I found some answers for a similar problem here.

The problem is not the same, however, yours may have the same origin.

[jordanperr]

I think I'm running in to this bug with VSCode 1.41.1, using remote-ssh 0.48.0 and Python 2020.1.57204 modules. I'm running it through Python 3.6.9 on CentOS 7.

One of my test files is not discovered by VSCode, although it can be discovered just fine using the command line:
python -m unittest discover -s ./test -p *test.py

I can get VSCode to discover the test by commenting out one of the import statements in the test file. Interestingly, the offending line is not itself a relative import. There could be a relative import hidden somewhere in that module.

Hope this is fixed soon.

[markkuleinio]

Vscode 1.44.2 (user setup), Vscode Python module version 2020.4.74986, Python 3.7.3 (Win10).

Testingtest/

• module1.py
• tests/
  • test_module1.py

Vscode says: "No tests discovered, please check the configuration settings for the tests."
Config is:

{
    "python.pythonPath": "C:\\Users\\MarkkuLeiniö\\AppData\\Local\\Programs\\Python\\Python37\\python.exe",
    "python.testing.unittestArgs": [
        "-v",
        "-s",
        "./tests",
        "-p",
        "test_*.py"
    ],
    "python.testing.pytestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.unittestEnabled": true
}

Unittest discovery is working from terminal window:

PS C:\Users\MarkkuLeiniö\Desktop\Testingtest> python -m unittest discover -v -s ./tests -p test_*.py
test_func1_correct (test_module1.MyTests) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.001s

OK
PS C:\Users\MarkkuLeiniö\Desktop\Testingtest> 

Is this an unsupported organization of *.py files? Is the only supported way to test filename.py to have the test files in the same directory?

Testingtest.zip

[markkuleinio]

In my case the problem seems to be solved by:

1. Creating an empty __init__.py in tests directory
2. Configuring vscode-python to use the ". Root directory", not the tests directory

Weird but works...

[Green-Alex]

Hi markkuleinio,

I set the configuration you said:

{
    "python.pythonPath": ".venv\\Scripts\\python.exe",
    "python.testing.unittestArgs": [
        "-v",
        "-s",
        ".",
        "-p",
        "test_*.py"
    ],
    "python.testing.pytestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.unittestEnabled": true
}

But still no lucky with that. Also, I've added the empyt init.py file in tests folder :(

Is the configuration correct, or may I be forgetting something else...

[markkuleinio]

Also, I've added the empyt init.py file in tests folder

Do note that the correct filename is __init__.py (total of four underscores), not init.py.

[markkuleinio]

For the record: With the new Python extension version 2020.4.76186 (27-Apr-2020) there are no test discovery problems anymore in my projects. Thanks!

[Green-Alex]

Oh! Thank you for the update markkuleinio. I've checked that I'm using this version, but unittests are still not found :/

(also, I've created the __init__.py files)

[kylemcmearty]

running Python extension v2020.6.91350 and having issues with unittest discovery. I tried all of markkuleinio's suggestions and I could not get unittest working. Have to switch to "python.testing.pytestEnabled": true in setting.json to discover my test files.

[antoniovs1029]

I'm also facing this issue with Python extension v2020.7.96456 . At first it was working just well, and out of a sudden it stopped finding the unittests in files that included a relative import.

[Guddi11]

Same here with 2020.9.114305.
I've tried all of the above mentioned tricks without success. Also disabled all other extensions.
Is there any news on this matter - would really appreciate an automated unittest.

[Pabosik]

Hi,

still encountering this bug in:

Version: 1.52.0
Commit: 940b5f4bb5fa47866a54529ed759d95d09ee80be
Date: 2020-12-10T22:43:50.597Z
Electron: 9.3.5
Chrome: 83.0.4103.122
Node.js: 12.14.1
V8: 8.3.110.13-electron.0
OS: Linux x64 5.8.0-31-generic snap

Im on Ubuntu 20.10.

And Python version v2020.12.422005962.

Unit test is NOT discovered when there is a relative import. If I comment the import out, it gets discovered. Unfortunate :(.

[tobbedann]

I've ran into this issue and tried to narrow it down. What cause the bug for me is if I wrongly typehint and argument to a function, inside a file that im importing in the test. I.e.

dummy.py

from typing import Dict

def dummy_function(self, dummy_dict: Dict[int]):
    pass
    

test_dummy.py

import dummy

class TestDummy(unittest.TestCase):
    def test_if_true(self):
        pass

Will not work, meaning the test will not be discovered. If i however make the change (so i correctly typehint a Dict)

def dummy_function(self, dummy_dict: Dict[int]): -> def dummy_function(self, dummy_dict: Dict[int, str]):

The test will be discovered

[whompyjaw]

Using 2021.1.0 Python extension I had same issue as described initially (relative imports preventing vscode from detecting tests). I was able to run the tests using py -m unittest discover but vscode didn't detect them.

Using absolute imports fixed my issue.
from .constants import CONFIG (tests not detected)
from tests.constants import CONFIG (tests were detected)

I don't think that is ideal, though. But maybe that is the requirement?

[2bdkid]

Tests work fine from command line, python -m unittest discover -v -s ./transform_text/ -p "test_*.py" -t . but with the same settings in vscode it doesn't discover anything.

[kkaja123]

For those still running into this issue, here's an explanation from what I've found.

In the original issue description, the start directory is set to the test directory. That means that unittest uses the inside of that directory as the starting location. By using from . import ..., the Python file is trying to access the test folder itself, but since unittest started inside the folder, Python doesn't know about the folder itself. To resolve this, unittest provides a top level directory option that redirects Python to a different directory when starting relative import resolution. For the original issue, that would be -t ., the project's root directory.

The extension parses unittestArgs and extracts the start directory and the test pattern. Every other argument is completely ignored (when running unit test discovery). The typescript code that handles discovery would need to add the top level directory argument when parsing the args. Additionally, <ms-python.python>/pythonFiles/testing_tools/unittest_discovery.py would need to be able to accept that top level directory argument, since that file is responsible for invoking unittest's loader discovery.

I am trying to see if I can fix it on my fork of the repo. I will create a PR if I am successful.

[eleanorjboyd]

Hello! We have just finished our testing rewrite and are beginning the roll out to users. If you are able, it would be very helpful to know if your issue still exists on the rewrite! To use the rewrite yourself just add ”python.experiments.optInto": ["pythonTestAdapter"] to your user settings. You can confirm you have the rewrite enabled by setting your log level to trace, via the Developer: Set Log Level command in the command palette. Then check to see if Experiment 'pythonTestAdapter' is active is in your python logs.

Thanks!

[Dr-Irv]

Hello! We have just finished our testing rewrite and are beginning the roll out to users. If you are able, it would be very helpful to know if your issue still exists on the rewrite! To use the rewrite yourself just add ”python.experiments.optInto": ["pythonTestAdapter"] to your user settings. You can confirm you have the rewrite enabled by setting your log level to trace, via the Developer: Set Log Level command in the command palette. Then check to see if Experiment 'pythonTestAdapter' is active is in your python logs.

Thanks!
@eleanorjboyd

I don't see this as being fixed. The output in the python window is showing an error, and in the "testing" panel, it says "Unittest Discovery Error". Python output shows:

2023-11-21 10:25:30.092 [info] Discover tests for workspace name: vscode_utbug2 - uri: c:\Code\Misc\vscode_utbug2
2023-11-21 10:25:30.092 [info] Running discovery for unittest using the new test adapter.
2023-11-21 10:25:30.100 [info] Discovering unittest tests with arguments: c:\Users\irv\.vscode\extensions\ms-python.python-2023.20.0\pythonFiles\unittestadapter\discovery.py,--udiscovery,-v,-s,./mypkg_test,-p,test_*.py

2023-11-21 10:25:30.101 [info] > conda run -n base310 --no-capture-output python ~\.vscode\extensions\ms-python.python-2023.20.0\pythonFiles\get_output_via_markers.py ~\.vscode\extensions\ms-python.python-2023.20.0\pythonFiles\unittestadapter\discovery.py --udiscovery -v -s ./mypkg_test -p test_*.py
2023-11-21 10:25:30.101 [info] cwd: .
2023-11-21 10:25:31.802 [info] >>>PYTHON-EXEC-OUTPUT

2023-11-21 10:25:31.879 [info] Test server connected to a client.
2023-11-21 10:25:31.893 [error] Unittest test discovery error 
 Failed to import test module: test_foo
Traceback (most recent call last):
  File "C:\Anaconda3\envs\base310\lib\unittest\loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "C:\Anaconda3\envs\base310\lib\unittest\loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "c:\Code\Misc\vscode_utbug2\mypkg_test\test_foo.py", line 2, in <module>
    from . import base
ImportError: attempted relative import with no known parent package

2023-11-21 10:25:31.894 [info] Test server connected to a client.
2023-11-21 10:25:31.894 [info] <<<PYTHON-EXEC-OUTPUT

2023-11-21 10:25:31.894 [info] ResultResolver EOT received for discovery.

With that setting, if I uncomment from . import base, then the tests are discovered.

[eleanorjboyd]

@Dr-Irv, this might help: 9a5363c. It was just merged in today therefore you could test it on insiders when the new release comes out tomorrow or next week when stable comes out. Thanks!

[Dr-Irv]

@Dr-Irv, this might help: 9a5363c. It was just merged in today therefore you could test it on insiders when the new release comes out tomorrow or next week when stable comes out. Thanks!

If I opt into the ”python.experiments.optInto": ["pythonTestAdapter"] as you described, the little icon for test discovery disappears in VS Code.

I'm not sure which extension (pylance? python?) I should look at to determine if I have the right version installed or not.

[eleanorjboyd]

@Dr-Irv, it is the python extension. You should check to see if the extension is enabled (sometimes i have to just open a python file in the editor then it will enable), you can see the python interpreter selected in the bottom right of the screen.

You can also try using the command palette to look for run test / discover test there.

Let me know if that brings the icon back!

[github-actions]

Because we have not heard back with the information we requested, we are closing this issue for now. If you are able to provide the info later on, then we will be happy to re-open this issue to pick up where we left off.

Happy Coding!