pandas v2

[zmoon]

Some tests failing with pandas v2, noticed first in #110

=========================== short test summary info ============================
FAILED tests/test_airnow.py::test_add_data_hourly - assert 0 >= 1
 +  where 0 = RangeIndex(start=0, stop=0, step=1).size
 +    where RangeIndex(start=0, stop=0, step=1) = Empty DataFrame\nColumns: [time, siteid, site, utcoffset, time_local, latitude, longitude, cmsa_name, msa_code, msa_name, state_name, epa_region]\nIndex: [].index
FAILED tests/test_airnow.py::test_add_data_daily - assert 0 >= 1
 +  where 0 = RangeIndex(start=0, stop=0, step=1).size
 +    where RangeIndex(start=0, stop=0, step=1) = Empty DataFrame\nColumns: [time, siteid, site, latitude, longitude, cmsa_name, msa_code, msa_name, state_name, epa_region]\nIndex: [].index
FAILED tests/test_aeronet.py::test_serial_freq - TypeError: Could not convert lev15lev15lev15lev15lev15lev15lev15lev15lev15lev15lev15lev15lev15lev15lev15lev15lev15lev15lev15 to numeric
FAILED tests/test_ish.py::test_ish_one_site[False] - TypeError: Could not convert FM-15FM-15FM-15 to numeric
FAILED tests/test_ish.py::test_ish_one_site[True] - TypeError: Could not convert FM-15FM-15FM-15 to numeric
FAILED tests/test_ish_lite.py::test_ish_lite_resample - ValueError: cannot insert siteid, already exists
FAILED tests/test_ish.py::test_ish_resample - TypeError: Could not convert FM-15FM-15FM-15FM-15FM-15FM-15FM-15FM-15FM-15 to numeric
============ 7 failed, 54 passed, 2 skipped, 57 warnings in 39.29s =============

[zmoon]

For these tests, can mark and test with both pandas v1 and v2 to make sure changes don't break pandas v1 compat.

Would be nice to be able to check that results are the same too, easiest way would be to store the v1 results

[zmoon]

Scheduled CI failed today due to this.

[zmoon]

Looks like OpenAQ also needs some work to support pandas v2. @jordanschnell reported:

... @?T@?T@?T@?T@?T@?T@?T@?T@?T@?T@?T@?T@?T@?T@?T@?T@?T@?T@?T@?T@?T@?T@?T@?T@?T@?T@?T@?T@?T@?T@?T@?T@?T@?~U?

AssertionError

Which seems to come from this line:

 assert (
      zzz[~zzz.parameter.isin(["pm25", "pm4", "pm10", "bc"])].unit.dropna() == "ppm"
   ).all()

and needing to update the deprecated json_normalize call.

[zmoon]

Note
Note that the conda-forge recipe doesn't currently restrict the pandas version. Nor does the install_requires. So you need to specify when creating your environment or downgrade if necessary.

[blychs]

Hi @zmoon, is anything missing here? I think that the faster we can fix this, the better: it would allow us to lift the restrictions in matplotlib versions as well!

[zmoon]

Hi @blychs thanks for checking. The issue is that we have a significant amount of untested code, which may or may not require updates. But we can certainly start with the tested code and modify the CI to test both pandas v1 and v2. I think most (if not all) of the obs readers used in MM that would be affected have one test or more.

[zmoon]

Remember migrating from {error,warn}_bad_lines to on_bad_lines is part of this. AirNow in #146 has an example of how to support both v1 and v2.