conda-forge package does not support Python version >3.6, leads to package conflict resolution

[emiliom]

Copying in question from @DanCodigaMWRA posed at DOI-USGS/dataretrieval-python#8, as it pertains to ulmo specifically:

Looking for general advice. Or even just pointers to resources that would help me gather info to make a decision.

I used ulmo successfully a year or two ago to retrieve USGS river flow data. But now it doesn't seem very functional any more. (For one thing, I can't get it to install within conda, due to conflicts with other standard packages apparently.)

So it looks like I need to update my code using something other than ulmo.

@DanCodigaMWRA can you be more specific about the problems you're having with ulmo? Can you post the errors you're encountering?

I and others continue to use ulmo pretty sucessfully for both USGS NWIS and other data readers. The package is available on conda-forge. But there are some current constraints in version compatibility that should be removed. For example, it's not clear whether it's possible at this time to install it with Python 3.8 or 3.9. This is something I intend to address in the next release (https://github.com/ulmo-dev/ulmo/milestone/5).

Thanks!

[ghost]

Thanks for the reply. I am glad to learn that you and others are using ulmo and intending to support it into the future. It would be great if I could just use my code that relies on ulmo-- I found ulmo useful and it was great that I could get it via conda-forge.

The trouble I'm having now is that when I try to include ulmo, when creating my conda environment, my "conda env create" command leads to conficts. My environment includes many many packages such as numpy, pandas, xarray etc. I have not done extensive troubleshooting yet because it is so slow when conflicts occur. But some of the diagnostic messages led me to think ulmo was responsible. So I did a test, took out ulmo, and there were no more conflicts.

This is all with python 3.7 by the way.

I'm not sure if this is enough information to be all that helpful to you. If helpful I could at some point go back and keep track of the details of the conflicts that the conda package resolver is giving me-- or perhaps better try some simpler combinations of packages (instead of my "kitchen sink" environment) and try to home in on exactly what the conflicts are.

[mroberge]

I was curious so tried installing Ulmo several different ways:

• Using conda install ulmo in a Python 3.8 environment failed. See error message below.
• Using pip install ulmo in a Python 3.8 environment worked.
• Using conda install ulmo in a Python 3.6 environment worked. See below.

Based on reading the error message below, my guess is that the conda-stock recipe has some sort of statement limiting ulmo installations to only Python 2.7, 3.4, 3.5, and 3.6. My guess is that this should be an easy fix. (I don't know anything about making recipes, however!) Note that Ulmo works fine in my Python 3.8 environment, although I didn't try to run the tests. I just downloaded some NWIS data.

BTW, @DanCodigaMWRA , it is easy to install ulmo using pip. Pip is already included with Python, and you just type pip install ulmo and everything should work fine.

Here is the Python 3.8 error message:

(py38dev) C:\Users\Marty>conda install ulmo
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment:
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
-failed

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

Specifications:

• ulmo -> python[version='2.7.|3.5.|3.6.|3.4.']

Your python: python=3.8

If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.

To test out my ability to install Ulmo in a Python 3.6 environment, I did the following:

conda create --name py36ulmo Python=3.6 activate py36ulmo conda install ulmo

This worked fine so I tried this:

python
>>>import ulmo
>>>result = ulmo.usgs.nwis.get_site_data('01584050', 'dv', period='P1D')
>>>result

[ghost]

Yes, @mroberge I have used pip before. (It's not a requirement that I stay with conda only and not use pip, just simply a strong preference. Helps me to more easily update my virtual environment.)

I just installed ulmo by pip, which of course worked fine. I ran my old code, which uses ulmo, and it works fine. So I am all set for the short-term.

Hopefully whatever change is needed so that I can get ulmo in the future using conda-forge without any conda conflicts is easy for you folks to figure out and implement, so that next time I update my conda env I can just include ulmo there instead of using pip.

Thank you both for your tips.

[emiliom]

@DanCodigaMWRA thanks for the follow ups. They are very helpful. I definitely want to make sure the conda-forge package works well and doesn't impose artificial limitations and conflicts. I too create "kitchen sink" conda environments, as do many people!

I've started to track where this Python 3.6 limitations is happening. @mroberge, thanks so much for posting your detective work!! I'm very glad the Python version issue doesn't impact installations via pip. I may have found the source of the problem. I'll report back as I make progress.

[mroberge]

Excellent! My pleasure!
I hope I can contribute more in the future.

[emiliom]

Potential Python pinning to look into, here (as opposed to on the conda-forge feedstock, https://github.com/conda-forge/ulmo-feedstock/blob/master/.ci_support/linux_64_.yaml#L12):

• https://github.com/ulmo-dev/ulmo/blob/master/.travis.yml#L11
• https://github.com/ulmo-dev/ulmo/blob/master/tox.ini#L2
• https://github.com/ulmo-dev/ulmo/blob/master/setup.py#L94

Another avenue to investigate is whether suds-jurko is imposing a downstream limitation. See #189

[emiliom]

An update on the issue of Python 3.6 locking and dependency incompatibilities. It looks like I've got this under control now. FYI, it had to do with two interrelated issues: the conda-forge package being built with dependencies that were intended for testing & development only, not for actual use; and pinning of a couple of those dependencies to very old versions. I've already addressed both issues.

We're making steady progress towards a new release, including a new conda package. It should be out by early next week, if not earlier.

[emiliom]

ulmo is now building for Python 3.6 - 3.9. The new GitHub Action runs the tests (successfully) for those Python versions. The next conda package release will include this support. That's definitely happening this week (finally!). I'm closing this issue, but will reopen if something comes up while building the conda package.