ci: Add simplified CI for Windows

[matthewfeickert]

Description

Resolves #2358

Add simplified CI for Windows. The workflow doesn't run on push or pull request events as it is slow but instead runs on a nightly schedule job and on demand with workflow dispatch. The workflow runs the normal test suite with the exception that it skips tests/test_scripts.py as these are currently broken on Windows. Python 3.8 is not added to the test matrix as there is no Python 3.8 version of jaxlib.

As this effectively adds testing support for Windows, additionally add operating system dependent PyPI trove classifiers.

Checklist Before Requesting Reviewer

• Tests are passing
• "WIP" removed from the title of the pull request
• Selected an Assignee for the PR to be responsible for the log summary

Before Merging

For the PR Assignees:

• Summarize commit messages into a comprehensive review of the PR

* Add simplfied CI workflow for Windows.
   - Runs on nighly schedule and with workflow dispatch.
* Add operating system PyPI trove classifier as support is added for Windows.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (6203a8f) 98.28% compared to head (c5d06cc) 98.28%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2363   +/-   ##
=======================================
  Coverage   98.28%   98.28%           
=======================================
  Files          69       69           
  Lines        4539     4539           
  Branches      803      803           
=======================================
  Hits         4461     4461           
  Misses         45       45           
  Partials       33       33           

Flag	Coverage Δ
contrib	97.86% <ø> (ø)
doctest	60.71% <ø> (ø)
unittests-3.10	96.29% <ø> (ø)
unittests-3.11	96.29% <ø> (ø)
unittests-3.8	96.32% <ø> (ø)
unittests-3.9	96.34% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[kratsg]

The workflow runs the normal test suite with the exception that it skips tests/test_scripts.py as these are currently broken on Windows.

what's broken on windows here? It could just be a missing path.

[matthewfeickert]

what's broken on windows here? It could just be a missing path.

There we multiple errors but here's an example of one FileNotFoundError

================================== FAILURES ===================================
___________________ test_import_prepHistFactory[inprocess] ____________________

E   FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_import_prepHistFactory_in0\\parsed_output.json'
All traceback entries are hidden. Pass `--full-trace` to see hidden and internal frames.

During handling of the above exception, another exception occurred:

tmpdir = local('C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_import_prepHistFactory_in0')
script_runner = <ScriptRunner inprocess>

    def test_import_prepHistFactory(tmpdir, script_runner):
        temp = tmpdir.join("parsed_output.json")
        command = f'pyhf xml2json validation/xmlimport_input/config/example.xml --basedir validation/xmlimport_input/ --output-file {temp.strpath:s} --hide-progress'
        ret = script_runner.run(shlex.split(command))
        assert ret.success
        assert ret.stdout == ''
        assert ret.stderr == ''
    
>       parsed_xml = json.loads(temp.read())

command    = 'pyhf xml2json validation/xmlimport_input/config/example.xml --basedir validation/xmlimport_input/ --output-file C:\\U...ata\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_import_prepHistFactory_in0\\parsed_output.json --hide-progress'
ret        = <pytest_console_scripts.RunResult object at 0x000002E66E873C40>
script_runner = <ScriptRunner inprocess>
temp       = local('C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_import_prepHistFactory_in0\\parsed_output.json')
tmpdir     = local('C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_import_prepHistFactory_in0')

tests\test_scripts.py:68: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\_py\path.py:377: in read
    with self.open(mode) as f:
        mode       = 'r'
        self       = local('C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_import_prepHistFactory_in0\\parsed_output.json')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = local('C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_import_prepHistFactory_in0\\parsed_output.json')
mode = 'r', ensure = False, encoding = None

    def open(self, mode="r", ensure=False, encoding=None):
        """Return an opened file with the given mode.
    
        If ensure is True, create parent directories if needed.
        """
        if ensure:
            self.dirpath().ensure(dir=1)
        if encoding:
            return error.checked_call(io.open, self.strpath, mode, encoding=encoding)
>       return error.checked_call(open, self.strpath, mode)
E       py.error.ENOENT: [No such file or directory]: open('C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_import_prepHistFactory_in0\\parsed_output.json', 'r')

encoding   = None
ensure     = False
mode       = 'r'
self       = local('C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_import_prepHistFactory_in0\\parsed_output.json')

C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\_py\path.py:[76](https://github.com/scikit-hep/pyhf/actions/runs/6635684924/job/18027015790#step:6:77)1: ENOENT
---------------------------- Captured stdout call -----------------------------
# Running console script: ['pyhf', 'xml2json', 'validation/xmlimport_input/config/example.xml', '--basedir', 'validation/xmlimport_input/', '--output-file', 'C:UsersrunneradminAppDataLocalTemppytest-of-runneradminpytest-0test_import_prepHistFactory_in0parsed_output.json', '--hide-progress']
# Script return code: 0
# Script stdout:

# Script stderr:

[henryiii]

tmp_path is a modern replacement for tmpdir, by the way. Anyway, the problem is the string:

command = f'pyhf xml2json validation/xmlimport_input/config/example.xml --basedir validation/xmlimport_input/ --output-file {temp.strpath:s} --hide-progress'
ret = script_runner.run(shlex.split(command))

Paths have backslashes on windows, which worries me a bit there, and https://docs.python.org/3/library/shlex.html#shlex.split says that posix=True is the default, but you aren't running on a posix shell on Windows. Why are you putting a string in then splitting it up? And if you use shlex.quote in the runner to put it back together, that's 100% UNIX only.

[matthewfeickert]

To make things compatible with Windows this is going to require some work, so I'm going to merge this in and then we can fix up the test suite in a follow up PR.

tmp_path is a modern replacement for tmpdir, by the way.

Thanks and noted.

From the pytest docs (https://docs.pytest.org/en/7.1.x/how-to/tmp_path.html#the-tmpdir-and-tmpdir-factory-fixtures):

The tmpdir and tmpdir_factory fixtures are similar to tmp_path and tmp_path_factory, but use/return legacy py.path.local objects rather than standard pathlib.Path objects.

Note

These days, it is preferred to use tmp_path and tmp_path_factory.

In order to help modernize old code bases, one can run pytest with the legacypath plugin disabled:

pytest -p no:legacypath

This will trigger errors on tests using the legacy paths. It can also be permanently set as part of the addopts parameter in the config file.

We'll fix this up in another PR. 👍

Anyway, the problem is the string:

command = f'pyhf xml2json validation/xmlimport_input/config/example.xml --basedir validation/xmlimport_input/ --output-file {temp.strpath:s} --hide-progress'
ret = script_runner.run(shlex.split(command))

Paths have backslashes on windows, which worries me a bit there, and https://docs.python.org/3/library/shlex.html#shlex.split says that posix=True is the default, but you aren't running on a posix shell on Windows. Why are you putting a string in then splitting it up? And if you use shlex.quote in the runner to put it back together, that's 100% UNIX only.

Looking at an example:

pyhf/tests/test_scripts.py

Lines 60 to 70 in 6203a8f

	def test_import_prepHistFactory(tmpdir, script_runner):
	temp = tmpdir.join("parsed_output.json")
	command = f'pyhf xml2json validation/xmlimport_input/config/example.xml --basedir validation/xmlimport_input/ --output-file {temp.strpath:s} --hide-progress'
	ret = script_runner.run(shlex.split(command))
	assert ret.success
	assert ret.stdout == ''
	assert ret.stderr == ''
	parsed_xml = json.loads(temp.read())
	spec = {'channels': parsed_xml['channels']}
	pyhf.schema.validate(spec, 'model.json')

command is a single string for convenience of readability for us as pytest-console-scripts expects a list of command fragrments that would be space seperated, similar to nox. This is not nice to look at once the commands get more than a few segements long.

Internally pytest-console-scripts only uses shlex.split, and we could do something similar to what they do with

import os
shlex.split(command, posix=os.name == 'posix')