Adding Own Environment Install Error

[CameronStockton]

I have followed all the steps to add my own packages to an environment set out on this page: https://github.com/LSSTDESC/desc-python/wiki/Add-Your-Own-Packages-to-the-desc-python-Environment

I just want to install the cesium package from here: http://cesium-ml.org but the error I am getting is that I need NumPy installed in my environment and so I've installed NumPy with pip install NumPy --user and that said it was successfully installed but then I get the same error when trying to install cesium, that I need NumPy. So I did Conda install NumPy --user and that also said it was installed successfully but yet I get the same error when installing cesium. The same error also occurs when trying to install scikit-learn.

[johannct]

hi, can you provide the exact sequence of command lines you executed? In case you tried with jupyter, can you please try first with a python/ipython session?

[CameronStockton]

Here are my commands, I also attached the errors that come up (requirements.txt has all the dependencies I need to install, I can provide that if you'd like as well):
`conda create --clone desc -p /global/homes/c/cstock/mydesc
conda activate /global/homes/c/cstock/mydesc
pip install -r requirements.txt --user

ERROR: Command errored out with exit status 1:
command: /global/common/software/lsst/common/miniconda/py3.7-4.7.12.1-v2/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-l7gdy7yd/cesium/setup.py'"'"'; file='"'"'/tmp/pip-install-l7gdy7yd/cesium/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-xt470mxx --python-tag cp37
cwd: /tmp/pip-install-l7gdy7yd/cesium/
Complete output (6 lines):
To install cesium from source, you will need numpy.
Install numpy with pip:
pip install numpy
Or using conda:
conda install numpy
or use your operating system package manager.

Building wheel for scikit-learn (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /global/common/software/lsst/common/miniconda/py3.7-4.7.12.1-v2/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-g499o_ch/scikit-learn/setup.py'"'"'; file='"'"'/tmp/pip-install-g499o_ch/scikit-learn/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-aj7pvk0s --python-tag cp37
cwd: /tmp/pip-install-g499o_ch/scikit-learn/
Complete output (19 lines):
Partial import of sklearn during the build process.
Traceback (most recent call last):
File "/tmp/pip-install-g499o_ch/scikit-learn/setup.py", line 168, in get_numpy_status
import numpy
ModuleNotFoundError: No module named 'numpy'
Traceback (most recent call last):
File "/tmp/pip-install-g499o_ch/scikit-learn/setup.py", line 148, in get_scipy_status
import scipy
ModuleNotFoundError: No module named 'scipy'
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-g499o_ch/scikit-learn/setup.py", line 269, in
setup_package()
File "/tmp/pip-install-g499o_ch/scikit-learn/setup.py", line 249, in setup_package
.format(numpy_req_str, instructions))
ImportError: Numerical Python (NumPy) is not installed.
scikit-learn requires NumPy >= 1.8.2.
Installation instructions are available on the scikit-learn website: http://scikit-learn.org/stable/install.html

pip install numpy --user

Installing collected packages: numpy
WARNING: The scripts f2py, f2py3 and f2py3.7 are installed in '/global/homes/c/cstock/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed numpy-1.19.5

pip install -r requirements.txt --user

(Same error as above)
conda install numpy
pip install -r requirements.txt --user

(Same error as above)
`

I'm not familiar with the NERSC environment, can you elaborate on how to try this with a python/ipython session?

[heather999]

Let's start from the beginning - we really wouldn't recommend doing pip install --user for anything as it is a bit error prone. We might need to clean that up by looking your $HOME/.local/lib/ area and removing those files, but let's try to proceed without worrying about that yet.
I would remove anything that was put into your $HOME area such as the /global/homes/c/cstock/mydesc directory and start over. after cleaning up, and starting a fresh bash shell or even logging into cori again to start a new session:

source /global/common/software/lsst/common/miniconda/setup_current_python.sh
conda create --clone desc -p /global/homes/c/cstock/mydesc
conda activate /global/homes/c/cstock/mydesc

If that is successful, then you can add additional packages - I'll edit this further, but post this now so you can get started.

At that point, you have an environment that already has numpy installed, as is scikit-learn, so you don't need to install them again. You can really just install cesium, which is available on conda-forge, by doing:
conda install -c conda-forge cesium
That will install cesium in your cloned environment that is in /global/homes/c/cstock/mydesc.
Then to make sure you continue to use your cloned environment going forward, let's say when you log back into Cori in the future, you would need to add an environment variable that points to your cloned environment. That way, every time you go to activate desc-python, it would automatically load your cloned environment, whether you do that at the Cori command line or in jupyter, making your installed packages like cesium available. So I would add to your .bashrc (assuming bash is your shell!):
export DESCUSERENV=/global/homes/c/cstock/mydesc
if you later decide you want to ignore or get rid of your /global/homes/c/cstock/mydesc you would remove that environment variable from your .basrhc and it will no longer be used when you do a source /global/common/software/lsst/common/miniconda/setup_current_python.sh and you will be using the original DESC desc-python.

[CameronStockton]

Where would I find my .bashrc? Also, if I wanted to use cesium, would the file that uses it need to be within the new folder that this environment created in my $HOME?

[heather999]

Look inside your $HOME, so do:
ls -a $HOME/.bashrc
To make sure bash is your shell, you can do: echo $SHELL and it should say something like: /bin/bash

I'm not sure about the 2nd question - are you referring to script or notebooks that use cesium? Those scripts or notebooks could be anywhere on Cori - they don't have to be located in or near your new environment.

[CameronStockton]

I ran into this problem earlier with Renee... I dont seem to have a .bashrc file. I confirmed that when I run echo $SHELL it returns bin/bash.

(/global/homes/c/cstock/mydesc_new) cstock@cori02:~> ls -a $HOME/.bashrc
ls: cannot access '/global/homes/c/cstock/.bashrc': No such file or directory

I only ask my second question because when I run the notebook that contains the cesium import it still does not run. I get "no module named 'cesium'" as my error, which has been the error the entire time.

[heather999]

I think you can go ahead and create $HOME/.bashrc and then just add this one line to it:

export DESCPYTHONPATH=/global/homes/c/cstock/mydesc
I apologize..this wasn't what I should have written.. it should have been
export DESCUSERENV=/global/homes/c/cstock/mydesc

basically, point it at the directory where you cloned desc-python.
To test that it is working - I would do a fresh login into Cori and then do:
source /global/common/software/lsst/common/miniconda/setup_current_python.sh
and part of the output from that should mention that it is pointing to your $HOME/mydesc directory.
Then I would restart all kernels in jupyter and try to access cesium again.

[CameronStockton]

I have done all these steps and it has said I have installed cesium correctly. I created a .bashrc file and added that line to it (it has since disappeared) and yet my file is still not able to import cesium correctly. I'm not getting any errors except when I run my notebook which tries to import cesium.

[cwwalter]

As a check: You will need to make sure you are using a Jupyter kernel for your custom environment.

[heather999]

Hi @CameronStockton Let's start by making sure things are working at the command line on Cori and then get things working on jupyter.
You mentioned that $HOME/.bashrc disappeared. From the Cori command line can you do:
cat $HOME/.bashrc and show the output here?
If that file does not exist, or if it doesn't contain the line:
export DESCUSERENV=/global/homes/c/cstock/mydesc
then please try doing this at the Cori command line:
echo "export DESCUSERENV=/global/homes/c/cstock/mydesc" > $HOME/.bashrc
Please note this is different than what I wrote last night (I just edited my comment from last night).. I wasn't thinking.. you want to create an environment variable named DESCUSERENV
After you do this, please open up a new Cori command line..and then do
source /global/common/software/lsst/common/miniconda/setup_current_python.sh
and can you show us here what that outputs?
From there you should be able to start up python from the Cori command line:
python
and then do an import cesium and that should work. If not, then we still have to fix something. If it's not working, I'd probably want to see the output from which python to make sure the right python installation is in use.

If after all that, it does work.. then I would open up a fresh jupyter.nersc.gov, restart any running kernels, make sure to connect to the desc-python kernel and see if you can do import cesium in a notebook.

[CameronStockton]

@heather999 Here is my output from your steps:

cstock@cori08:~> cat $HOME/.bashrc export DESCPYTHONPATH=/global/homes/c/cstock/mydesccstock@cori08:~> echo "export/.bashrcRENV=/global/homes/c/cstock/mydesc" > $HOME/
cstock@cori08:~> cat $HOME/.bashrc export DESCUSERENV=/global/homes/c/cstock/mydesc
cstock@cori08:~> source /global/common/software/lsst/common/miniconda/setup_current_python.sh Including user python path: /global/homes/c/cstock/mydesc Now using /global/common/software/lsst/common/miniconda/py3.7-4.7.12.1-v2/envs/desc/bin/python
(desc) cstock@cori08:~> python
Python 3.7.6 | packaged by conda-forge | (default, Jun 1 2020, 18:57:50)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import cesium
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'cesium'

I ran which python and here is my output:

/global/common/software/lsst/common/miniconda/py3.7-4.7.12.1-v2/envs/desc/bin/python

I also just tried to install cesium again from your command above and try to import cesium again and it did not work either.

[heather999]

It's difficult to read the above due the formatting. If you can put code in code blocks it will format correctly, see the <> button in the GitHub Issue editor. When you click that, it will insert two back-ticks and you can put code in between those characters.

After you updated your $HOME\.bashrc by doing echo export DESCUSERENV=/global/homes/c/cstock/mydesc" > $HOME/.bashrc
Did you logout and back into Cori? Please do that and then redo source /global/common/software/lsst/common/miniconda/setup_current_python.sh that output should look a little different. What you posted suggests that your current session on Cori has not been updated for the changes you made in $HOME\.bashrc

We might want to try scheduling 15 minutes on zoom to get this cleared up. I'm in US Eastern Time - so maybe you can suggest a couple times and we can see what works out.

[johannct]

@CameronStockton It would help if you could turn your working area readable.

When basic things fail as here, it is always good to go back to basic. Can you try :

source /global/common/software/lsst/common/miniconda/setup_current_python.sh
mkdir cesium_test
export PYTHONPATH=~/cesium_test/lib/python3.7/site-packages
python -c 'import cesium;print(cesium.file)'
pytest cesium_test/lib/python3.7/site-packages/cesium/tests/

and report results?

[CameronStockton]

Hi everyone, thanks so much for your help.

After restarting Cori I ran an import cesium in python in the terminal with no error! I just need help figuring out how to access this new kernel in JupyterLab. This should be the final step... I hope!

[heather999]

@CameronStockton That sounds like good news.. so to recap, it sounds like you have a $HOME/.bashrc with the contents:
export DESCUSERENV=/global/homes/c/cstock/mydesc
and when at the Cori command line, you can do:
source /global/common/software/lsst/common/miniconda/setup_current_python.sh
and if you start up python, you can successfully do import cesium

To use this in jupyter at NERSC, do you already have the DESC jupyter kernels set up? Meaning, you already have access to desc-python? Have you already followed the steps here to set up the DESC kernels? Specifically, if you haven't already, you need to run this script once:
source /global/common/software/lsst/common/miniconda/kernels/setup.sh
if you have done that already, there is no need to do it again.
Once that is done, you will have access to DESC jupyter kernels when you visit jupyter.nersc.gov and when you start up jupyter, you can choose desc-python as your kernel and then you should have access to both the DESC python environment, plus cesium.

[CameronStockton]

@heather999 Yes I have set up my kernels. Whenever I choose desc python as my kernel, it will show that it is connecting and then disconnect and leave me with No Kernel. Odd, because this does not happen for any of the other kernels.

[heather999]

@CameronStockton - can you try running this script that will dump some information into a log file that I can take a look at?
/global/common/software/lsst/common/miniconda/kernels/versions/jupyter-kernels-current/scripts/jupyter-debug.sh > /global/cfs/cdirs/lsst/groups/CO/heatherk/cstock-jupyter-debug-info.log
It sounds like there is some trouble setting up the kernel, the above script will print some of your environment variables and the tail of the jupyter log file which might reveal some clues.

[CameronStockton]

@heather999 Here is the output:

Currently Loaded Modulefiles:

1. modules/3.2.11.4
2. altd/2.0
3. darshan/3.1.7
4. gcc/8.3.0
5. cray-mpich/7.7.10
6. craype-haswell
7. craype-hugepages2M
8. /opt/cray/ari/modulefiles/udreg/2.3.2-7.0.1.1_3.41__g8175d3d.ari
9. /opt/cray/ari/modulefiles/ugni/6.0.14.0-7.0.1.1_7.43__ge78e5b0.ari
10. pmi/5.0.14
11. /opt/cray/ari/modulefiles/dmapp/7.1.1-7.0.1.1_4.56__g38cf134.ari
12. /opt/cray/ari/modulefiles/gni-headers/5.0.12.0-7.0.1.1_6.33__g3b1768f.ari
13. /opt/cray/ari/modulefiles/xpmem/2.2.20-7.0.1.1_4.16__g0475745.ari
14. /opt/cray/ari/modulefiles/job/2.2.4-7.0.1.1_3.43__g36b56f4.ari
15. /opt/cray/ari/modulefiles/dvs/2.12_2.2.164-7.0.1.1_13.3__g354a5276
16. /opt/cray/ari/modulefiles/alps/6.6.58-7.0.1.1_6.13__g437d88db.ari
17. /opt/cray/ari/modulefiles/rca/2.2.20-7.0.1.1_4.56__g8e3fb5b.ari
18. atp/2.1.3
19. PrgEnv-gnu/6.0.5
    cat: /global/homes/c/cstock/.condarc: No such file or directory
    cat: /global/homes/c/cstock/.bashrc.ext: No such file or directory
    cat: /global/homes/c/cstock/.profile.ext: No such file or directory
    cat: /global/homes/c/cstock/.bash_profile.ext: No such file or directory

[heather999]

Thanks! One more thing.. can you do:
cp ~/.jupyter.log /global/cfs/cdirs/lsst/groups/CO/heatherk/cstock-jupyter.log
I need to go further back in the jupyter log file than the script printed. There is probably some error in that log that will tell us what is going on.

[CameronStockton]

@heather999 Just ran it! There was no output but im guessing you can see stuff on your end?

[heather999]

I see the file @CameronStockton I just need the permissions set to be able to read it (forgot about that!) Can you do:
chmod g+rx /global/cfs/cdirs/lsst/groups/CO/heatherk/cstock-jupyter.log

[heather999]

Just a note here, we were able to find the problem in the ~/.jupyter.log file,

Now using /global/homes/c/cstock/mydesc/bin/python
/global/homes/c/cstock/mydesc/bin/python: No module named ipykernel

which pointed to the need to remove the local conda cloned environment and start over to clone the env and add packages. Once that was done, importing cesium worked as expected when setting DESCUSERENV and using the desc-python kernel in jupyter at NERSC.
Also realized that the notebook being run is from: https://github.com/LSSTDESC/plasticc-kit where it is suggested to do this python installation using their requirements.txt - but those directions really aren't meant for use desc-python, so I think that made things more complicated. I'm not sure if we should request some note be added to this 2018 repo to allow people who might want to play with this in the future to utilize a DESC python environment at NERSC or CC-IN2P3.

[herjy]

Hi I seem to have a similar issue. I scrupulously followed the steps to add my own package to the desc-stack..
It works for the most part but I can't seem to access the packages I install in my jupyter envs or when starting a new command line session. Here is what works: the environment is correctly setup and substitutes for the desc-stack-weekly:

remyj@cori19:~> python /global/common/software/lsst/common/miniconda/start-kernel-cli.py desc-stack-weekly
Starting an environment for desc-stack-weekly now...
This may take a while. Be patient...
Activated your DESCSTACKUSERENV: /global/homes/r/remyj/mydescstack
Wondering Why? DESCSTACKUSERENV is likely set in your /global/homes/r/remyj/.basrhc, /global/homes/r/remyj/.bashrc.ext, or similar config script

So far so good. I have managed to install my package (a few lines of code at the moment, nothing fancy) and when I conda list I can see my package (desclamp) in the list. However if I start python and try to import, it I get ModuleNotFoundError. I can install it again and then it works fine, but still no way to have a notebook with desc-stack-weekly as a kernel that knows of my package.

[heather999]

Hi @herjy
Can you provide any details about how someone else could try to install desclamp - I couldn't find it quickly in github or pypi.
I'm going to run through the instructions again in add my own package to the desc-stack and see what happens!

[herjy]

The package is here. I used pip install ./DESC-Lamp from outside and python setup.py develop from within the repo. Both work and the package ends up listed in the environment. I don't think it's package dependent. Thanks for your prompt response!

[herjy]

I have realised that I actually needed to restart my hub (not just the kernels or terminals) for some of the changes too take effect.
But now when I try to setup the desc-stak-weekly from within my environment, I get an error message I'm havinng trouble with:

(/global/homes/r/remyj/Mystack) bash-4.2$ python /global/common/software/lsst/common/miniconda/start-kernel-cli.py desc-stack-weekly
Starting an environment for desc-stack-weekly now...
This may take a while. Be patient...
Traceback (most recent call last):
  File "/global/common/software/lsst/common/miniconda/start-kernel-cli.py", line 62, in <module>
    subprocess.call(command)
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/lib/python3.8/subprocess.py", line 340, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/lib/python3.8/subprocess.py", line 1702, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'shifter' 

[heather999]

HI @herjy Thanks for the poke and sorry for the delay.

First to address the explicit error above about "No such file or directory: 'shifter' "
It is true that desc-stack-weekly currently is a shifter image and to start it up would require shifter be available. I'm guessing that when you do:
(/global/homes/r/remyj/Mystack) bash-4.2$ python /global/common/software/lsst/common/miniconda/start-kernel-cli.py desc-stack-weekly
you are not in a typical Cori login node? Is this a terminal from jupyter.nersc.gov or a command line where you have some other environment enabled? If you were to do which shifter, I suspect it would be undefined, but that shouldn't be the case. So I think this is a separate issue from the desclamp installation.

Now concerning the ongoing issue to just add desclamp to your cloned environment of desc-stack-weekly, since this is a develop install via python setup.py develop (actually I used python -m pip install -e ./DESC-Lamp) which is basically equivalent, I have to update PYTHONPATH and add PathToMyCloneEnv/lib/python3.8/site-packages/desclamp.egg-link and also the path to my DESC-Lamp repo and then desclamp is fully recognized. Attached is a screenshot from my jupyter notebook.

I can repeat that same experiment with a fully released python module on PyPI, like for example fxpmath, do pip install fxpmath, and it works just fine without any need to modify PYTHONPATH.

[herjy]

Hi, thank you for your answer!

I think the first point is actually related, because if I can't execute this line (/global/homes/r/remyj/Mystack) bash-4.2$ python /global/common/software/lsst/common/miniconda/start-kernel-cli.py desc-stack-weekly from within my environment, then I can't enable my desc-stack-weekly conda env, and indeed, when I am in my jupyter environment on the desc-stack-weekly sourced from my cloned environment, import lsst fails.

[heather999]

Hi @herjy
Can you start a fresh Cori login session and then do:
python /global/common/software/lsst/common/miniconda/start-kernel-cli.py desc-stack-weekly ?

I want to see whether this is failing in your default Cori environment. This was working for your originally, so I suspect you have set up some other conda environment or some other settings first and that's why your prompt looks like:
(/global/homes/r/remyj/Mystack) bash-4.2$

Or show the output from running env so I can see the environment?

Even more information could be seen if you do:
Log in to cori and run

/global/common/software/lsst/common/miniconda/kernels/versions/jupyter-kernels-current/scripts/jupyter-debug.sh > jupyter-debug-info.log

and post the output log to an area at NERSC I can read so I can review the output.

[herjy]

No it works on my default env for sure. Yes, it is inn my own environment that it fails. But according to the insstructions, that's what we're supposed to do once the env is activated.

[heather999]

I think there is a misunderstanding and I am rewording the instructions to hopefully make it clearer. Going back to the instructions here:
https://github.com/LSSTDESC/desc-python/wiki/Add-Your-Own-Packages-to-the-desc-stack-Environment

You started here and followed these directions on a freshly started Cori login node to set up your cloned environment. These steps are only required once, to get your cloned environment set up:
https://github.com/LSSTDESC/desc-python/wiki/Add-Your-Own-Packages-to-the-desc-stack-Environment#to-install-the-packages-in-your-own-conda-environment
and you have done the conda activate of your cloned environment and pip installed desclamp.
In that specific session of Cori - there is nothing else you need to do (except update the PYTHONPATH for desclamp as I mentioned previously), you are all set and are in your cloned environment.

If you want your cloned environment to be set up automatically the next time you are in a fresh Cori login or in jupyter.nersc.gov using the desc-stack-weekly jupyter kernel, then you should make sure to create a DESCSTACKUSERENV environment variable as described here:
https://github.com/LSSTDESC/desc-python/wiki/Add-Your-Own-Packages-to-the-desc-stack-Environment#set-up-descstackuserenv-env-variable

Then the next time you start up a fresh jupyter session and use desc-stack-weekly, you will be in your cloned environment.

Now, the next time you start a fresh Cori login session and want to use your cloned desc-stack-weekly environment, you would do:
python /global/common/software/lsst/common/miniconda/start-kernel-cli.py desc-stack-weekly
and that's it, you should then be in your cloned environment, since you have set up that DESCSTACKUSERENV environment variable. Due to the develop pip install of desclamp, you'd still need to update PYTHONPATH as mentioned before.

If you try to do python /global/common/software/lsst/common/miniconda/start-kernel-cli.py desc-stack-weekly again in that same Cori login session after you've already run it - you will see that error about shifter, because you are already in a shifter environment.
Does that make sense?

[heather999]

@herjy are you able to run your cloned env?

[herjy]

I'm trying to figure out why my cloned environment is not doing what it's supposed to: when activated, import lsst raises ModuleNotFoundError: No module named 'lsst'.
I cloned it from lsst-scipipe-0.5.0 as I can't find lsst-scipipe-0.6.0.

[heather999]

The version of lsst-scipipe for desc-stack-weekly is indeed 0.5.0, so that is fine... what is written in the wiki is just an example and does not necessarily refer to the current version of the LSST Science Pipelines. I just update dthe doc to mention that to hopefully make that clear.

Now I do realize that my instructions are not correct though... and I see what you mean about being unable to import lsst.. and didn't realize it until now.
We shouldn't be cloning desc-stack (like we do for desc-python) and I somehow mistakenly thought that would work. I need to fix these instructions in the wiki and I apologize for the time wasted..

Here's how to do this correctly:

heatherk@cori10:~> bash
heatherk@cori10:~> python /global/common/software/lsst/common/miniconda/start-kernel-cli.py desc-stack-weekly
Starting an environment for desc-stack-weekly now...
This may take a while. Be patient...
(lsst-scipipe-0.5.0) bash-4.2$ 

That's without any cloned environment, I don't have any DESCSTACKUSERENV defined. Instead.. we should just pip install into another area (I truly don't suggest pip install --user because that is difficult to keep track of). Instead, I would utilize --root and --prefix to point to some other area .. so for an editable install of desclamp, I do this:

(lsst-scipipe-0.5.0) bash-4.2$ git clone https://github.com/herjy/DESC-Lamp.git
Cloning into 'DESC-Lamp'...
remote: Enumerating objects: 74, done.
remote: Total 74 (delta 0), reused 0 (delta 0), pack-reused 74
Receiving objects: 100% (74/74), 9.51 MiB | 14.71 MiB/s, done.
Resolving deltas: 100% (28/28), done.
(lsst-scipipe-0.5.0) bash-4.2$ python -m pip install --root /global/common/software/lsst/users/heatherk/mydescstack-0.5.0 --prefix /global/common/software/lsst/users/heatherk/mydescstack-0.5.0 -e ./DESC-Lamp
Obtaining file:///global/common/software/lsst/users/heatherk/lamp3/DESC-Lamp
Installing collected packages: desclamp
  Running setup.py develop for desclamp
Successfully installed desclamp
(lsst-scipipe-0.5.0) bash-4.2$ export PYTHONPATH=/global/common/software/lsst/users/heatherk/mydescstack-0.5.0/lib/python3.8/site-packages:/global/common/software/lsst/users/heatherk/lamp3/DESC-Lamp:$PYTHONPATH
(lsst-scipipe-0.5.0) bash-4.2$ python
Python 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:22:27) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import lsst
>>> import desclamp
>>> 

As before - we still need to update the PYTHONPATH - and to use this in your jupyter environment, you'll still need to do that.. but there is no need to clone (and in fact as you've seen cloning in this case doesn't provide access to the LSST Sci Pipeline env) and no need of that $DESCSTACKUSERENV environment variable.

sorry again for this long confusing thread... I"m hopeful this will get you going.

[herjy]

Brilliant! It works now, thanks a lot for your efforts and I'm glad if it helps others who might encounter the issue in the future. As far as I'm concerned, the issue can be closed. Thanks again @heather999 .