Output from test explorer is different from terminal output

[amunger]

Testing #22290

From the terminal run python -m pytest test_logging.py and make sure it looks the same as it looks when you run just test_logging.py from the test explorer

output differs:

Terminal

PS C:\src\test\inc_dec_example_repo\inc_dec_example_test_suite> python -m pytest test_logging.py
======================================================================= test session starts =======================================================================
platform win32 -- Python 3.11.6, pytest-7.4.2, pluggy-1.3.0
rootdir: C:\src\test\inc_dec_example_repo\inc_dec_example_test_suite
plugins: anyio-3.7.0
collected 2 items

test_logging.py F.                                                                                                                                           [100%]

============================================================================ FAILURES =============================================================================
__________________________________________________________________________ test_logging2 __________________________________________________________________________

caplog = <_pytest.logging.LogCaptureFixture object at 0x000001396608B710>

    def test_logging2(caplog):
        logger = logging.getLogger(__name__)
        # caplog.set_level(logging.DEBUG)  # Debug level
        caplog.set_level(logging.ERROR)  # Error level
    
        logger.debug("This is a debug message.")
        logger.info("This is an info message.")
        logger.warning("This is a warning message.")
        logger.error("This is an error message.")
        logger.critical("This is a critical message.")

        # Printing to stdout and stderr
        print("This is a stdout message.")
        print("This is a stderr message.", file=sys.stderr)
>       assert False
E       assert False

test_logging.py:23: AssertionError
---------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------- 
This is a stdout message.
---------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------- 
This is a stderr message.
------------------------------------------------------------------------ Captured log call ------------------------------------------------------------------------ 
ERROR    test_logging:test_logging.py:17 This is an error message.
CRITICAL test_logging:test_logging.py:18 This is a critical message.
===================================================================== short test summary info ===================================================================== 
FAILED test_logging.py::test_logging2 - assert False
=================================================================== 1 failed, 1 passed in 0.29s =================================================================== 
PS C:\src\test\inc_dec_example_repo\inc_dec_example_test_suite> python -m pytest test_logging.py
======================================================================= test session starts =======================================================================
platform win32 -- Python 3.11.6, pytest-7.4.2, pluggy-1.3.0
rootdir: C:\src\test\inc_dec_example_repo\inc_dec_example_test_suite
plugins: anyio-3.7.0
collected 2 items                                                                                                                                                   

test_logging.py F.                                                                                                                                           [100%]

============================================================================ FAILURES ============================================================================= 
__________________________________________________________________________ test_logging2 __________________________________________________________________________ 

caplog = <_pytest.logging.LogCaptureFixture object at 0x000001A408DFB710>

    def test_logging2(caplog):
        logger = logging.getLogger(__name__)
        # caplog.set_level(logging.DEBUG)  # Debug level
        caplog.set_level(logging.ERROR)  # Error level

        logger.debug("This is a debug message.")
        logger.info("This is an info message.")
        logger.warning("This is a warning message.")
        logger.error("This is an error message.")
        logger.critical("This is a critical message.")

        # Printing to stdout and stderr
        print("This is a stdout message.")
        print("This is a stderr message.", file=sys.stderr)
>       assert False
E       assert False

test_logging.py:23: AssertionError
---------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------- 
This is a stdout message.
---------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------- 
This is a stderr message.
------------------------------------------------------------------------ Captured log call ------------------------------------------------------------------------ 
ERROR    test_logging:test_logging.py:17 This is an error message.
CRITICAL test_logging:test_logging.py:18 This is a critical message.
===================================================================== short test summary info ===================================================================== 
FAILED test_logging.py::test_logging2 - assert False
=================================================================== 1 failed, 1 passed in 0.24s =================================================================== 
PS C:\src\test\inc_dec_example_repo\inc_dec_example_test_suite>

test explorer:

CLIENT: Server listening on port 56911...
Received JSON data in run script
============================= test session starts =============================
platform win32 -- Python 3.11.6, pytest-7.4.2, pluggy-1.3.0
rootdir: c:\src\test\inc_dec_example_repo\inc_dec_example_test_suite
plugins: anyio-3.7.0
collected 2 items

test_logging.py::test_logging2 
-------------------------------- live log call --------------------------------
ERROR    test_logging:test_logging.py:17 This is an error message.
CRITICAL test_logging:test_logging.py:18 This is a critical message.
FAILED                                                                   [ 50%]
test_logging.py::TestLogger::test_logging 
-------------------------------- live log call --------------------------------
DEBUG    test_logging:test_logging.py:33 This is a debug message in a unittest.
INFO     test_logging:test_logging.py:34 This is an info message in a unittest.
WARNING  test_logging:test_logging.py:35 This is a warning message in a unittest.
ERROR    test_logging:test_logging.py:36 This is an error message in a unittest.
CRITICAL test_logging:test_logging.py:37 This is a critical message in a unittest.
PASSED                                                                   [100%]

================================== FAILURES ===================================
________________________________ test_logging2 ________________________________

caplog = <_pytest.logging.LogCaptureFixture object at 0x000002B122A66690>

    def test_logging2(caplog):
        logger = logging.getLogger(__name__)
        # caplog.set_level(logging.DEBUG)  # Debug level
        caplog.set_level(logging.ERROR)  # Error level
    
        logger.debug("This is a debug message.")
        logger.info("This is an info message.")
        logger.warning("This is a warning message.")
        logger.error("This is an error message.")
        logger.critical("This is a critical message.")
    
        # Printing to stdout and stderr
        print("This is a stdout message.")
        print("This is a stderr message.", file=sys.stderr)
>       assert False
E       assert False

test_logging.py:23: AssertionError
---------------------------- Captured stdout call -----------------------------
This is a stdout message.
---------------------------- Captured stderr call -----------------------------
This is a stderr message.
------------------------------ Captured log call ------------------------------
ERROR    test_logging:test_logging.py:17 This is an error message.
CRITICAL test_logging:test_logging.py:18 This is a critical message.
=========================== short test summary info ===========================
FAILED test_logging.py::test_logging2 - assert False
========================= 1 failed, 1 passed in 0.18s =========================
Finished running tests!

[rzhao271]

I have test_logging2 with logging.ERROR and assert True at the end. I have a workplace setting python.testing.pytestArgs set to ["--color=yes", "-o", "log_cli=true"].
When I run the test test_logging2 from the Test Explorer, I see the ERROR and CRITICAL lines being printed.
When I run the test test_logging2 from the Integrated Terminal, I don't see the ERROR and CRITICAL lines being printed.

[eleanorjboyd]

@rzhao271 did you run it from the Integrated Terminal with the "-o", "log_cli=true" args included? Like pytest test_logging.py -o log_cli=true?

[amunger]

I had different arguments, forgot that I had "-o", "log_cli=true" set

[eleanorjboyd]

aha that makes sense! Should have specified this in my instructions. Will confirm with @rzhao271 before closing the issue.

[rzhao271]

I read the instructions and didn't think I had to carry over the arguments to the integrated terminal. Testing now

[rzhao271]

Now the output matches, and that makes a lot of sense that the integrated terminal would also need the args.

[eleanorjboyd]

Thank you! Appreciate you taking those extra steps!