halotools.test_installation() fails

[MatteoSerafino]

Dear all,
I installed halotools as explained in the guide, step by step.
However, when the installation test always fails.
Here the exit on the terminal:

ImportError while loading conftest '/home/matteo.serafino/virtualenv/venv/lib/python3.5/site-packages/halotools/conftest.py'.
venv/lib/python3.5/site-packages/halotools/conftest.py:5: in
from astropy.tests.pytest_plugins import *
E ImportError: No module named 'astropy.tests.pytest_plugins'
4

It seems related to some changes on the astropy package.
How may I solve it?

[aphearin]

Not sure offhand, @MatteoSerafino - it looks like could be an incompatibility between halotools and different versions of astropy. How did you try to install the code? With pip, conda, or building from source? Did you use a virtual environment? Could you review what you did to install?

To help identify the problem, could you include information about your python distribution and/or conda environment, and also version information for both halotools and astropy?

@bsipocz or @eteq - do you know whether the existence of pytest_plugins varies across astropy versions?

[MatteoSerafino]

Sure.
I'm working on a virtual environment since I'm on a server where I do not have sudo permission.
I'm working with Python 3.5.2 and I installed the code with pip as reported in your guide.
The version of the code the pip installed is the 0.6, that I suppose is the most released recent one.
The problem seems to be related to the issue #660, but I do not know how to solve it.
Astropy version 3.2.1.

[MatteoSerafino]

Sorry the one I suppose it's the related issue is the following: astropy/ccdproc#660

[aphearin]

@MatteoSerafino - would it work for you to instead install the code from source with a later version of python? For example, the following conda environment should resolve the issue:

conda create -n ht3 python=3 numpy scipy matplotlib cython h5py astropy ipython jupyter

After creating the environment, you should be able to install the latest version of halotools as follows:

$ conda activate ht3
$ cd path/to/halotools
$ python setup.py install

[MatteoSerafino]

Dear Aphearin,
I exactly did now what you suggested to me. So uninstalled everything and then I proceeded as you said:
conda create -n ht3 python=3 numpy scipy matplotlib cython h5py astropy ipython jupyter

and then

$ conda activate ht3
$ cd path/to/halotools
$ python setup.py install
Indeed in this way, I got the version 0.7.dev5027. However, I still face the same problem.

[aphearin]

Hmm, dang, sorry that didn't fix the problem. When you activate the ht3 environment, could you report on which version of python and astropy is installed?

[aphearin]

I should have also asked: is it just the testing suite that is the problem? Or have you noticed other functionality impacted? If the lack of existence of astropy.tests.pytest_plugins is the only bug, then in principle the only problem is that you can't locally run the test suite, but other than that the code may be perfectly good to go for your science application. I would still want to fix this, of course, but that doesn't need to be your problem in case you just to get on with your research.

[MatteoSerafino]

Hi,
the version of python is the Python 3.5.2. Astropy 3.2.1.
Indeed I tried:

from halotools.empirical_models import NFWPhaseSpace
nfw = NFWPhaseSpace()
data = nfw.mc_generate_nfw_phase_space_points(Ngals = 100, mass = 1e13, conc = 10)

and indeed it is working. So it seems only the test is not working.
However when I run the first line I obtain a warning, but I suppose it is not a problem.

WARNING: AstropyDeprecationWarning: astropy.extern.six will be removed in 4.0, use the six module directly if it is still needed [astropy.extern.six]

[aphearin]

Great, ok, that's good to hear. Since the problem has to do with the existence of astropy.tests.pytest_plugins across versions, and since this module is not used by any halotools source code, only the test suite, then I think you should be good to go with your science.

Based on the discussion in the Astropy issue you pointed to, it looks like if you modify the conda environment I previously sent you to use python 3.6.x, then this may be also resolve the problem with the test suite:

conda create -n ht3 python=3.6 numpy scipy matplotlib cython h5py astropy ipython jupyter

Although I think this should only be necessary if you want to run the test suite, and shouldn't have any impact on the source code behavior.

Re:

WARNING: AstropyDeprecationWarning: astropy.extern.six will be removed in 4.0, use the six module directly if it is still needed [astropy.extern.six]

Yes, the current version of the code raises a bunch of warnings about future compatibility. Fixing this is one of the last obstacles remaining before the v0.7 release. For now they should be harmless and can be ignored.

I'll leave this issue open until I either resolve the installation problem, or at least add a note to the documentation.

[MatteoSerafino]

Ok, thank you.
I really appreciated. I also have one more question about the function >>>nfw.mc_generate_nfw_radial_positions

All the results are obtained by imposing the value of Δref(z)=200.
Is there a way to change this value without changing internally the code?

[aphearin]

The NFWProfile class accepts a mdef argument that lets you change mass definition.

>>> nfw = NFWProfile(mdef='2500m') 
>>> nfw = NFWProfile(mdef='500c') 

Alternatively, you can just drill down into the module subdirectories and import the standalone function, which has a halo_radius argument that can simply be any positive float or ndarray of positive floats.

from halotools.empirical_models.phase_space_models.analytic_models.satellites.nfw.kernels.mc_generate_nfw_radial_positions import mc_generate_nfw_radial_positions         

[MehnaazAsad]

Hi @aphearin! I am commenting here since this is an issue I have been facing too. I recently re-installed Anaconda and all my packages due to a bug I had been facing and I was able to install the stable version of halotools (v0.6). When testing it, I ran into the same issue pointed out here with pytest_plugins which I was able to fix by making the relevant changes mentioned at the bottom of https://docs.astropy.org/en/stable/whatsnew/3.0.html in conftest.py . It seems that in Astropy v3.0 they have moved some pytest plugins to their own packages.

However, the AstropyDeprecationWarning regarding the six module is not letting me run the following import I need for my script:

from halotools.empirical_models import SubhaloModelFactory

It was originally also not working for:

from halotools.sim_manager import CachedHaloCatalog

but I was able to fix it by manually going into the sim_manager directory and editing the .py files from import astropy.extern.six to import six.

Should I, for now, just change the files connected to the packages I need until this warning has been resolved in v0.7?

[bsipocz]

@MehnaazAsad - those two issues got fixed yesterday, so if you try to install the dev version from here, it should work. The alternative workaround is to downgrade your astropy install to <3.2

However it's interesting what you're saying about the deprecation. As it's a warning, it should not be a blocker to do any of the imports, they may be another issue somewhere else.

[MehnaazAsad]

@bsipocz Yes, I too found it strange that a warning was performing more like an error but making the same changes I made to get the other import to work fixed the issue. It still shows up as a warning when I try to import SubhaloModelFactory but now it behaves purely like a warning.

Despite fixing it, I installed the dev version. The halotools.test_installation() still produces errors but none of the ones mentioned above. Moreover, I am able to use the two packages I need so I will just ignore those errors for now.

[aphearin]

Thanks for the fixes @bsipocz! There are a lot of annoying warnings I'm working on fixing so that I can finish and release v0.7. It's been pretty tedious, so in the meantime thanks for the patience @MehnaazAsad.