Skip to content

Add Your Own Packages to the desc stack Environment

Heather Kelly edited this page Apr 2, 2024 · 7 revisions

If you are new to DESC and DESC python environments, please consult our related documentation for NERSC or IN2P3.

You may have additional packages or an updated version of a package that you want to install alongside desc-stack (or desc-stack-weekly or desc-stack-weekly-latest).

  • If this is a package of general interest, please consult within your Working Groups and then request the package be added to desc-python and/or desc-stack by opening an issue in desc-help.

To add package(s) locally to your desc-stack environment:

Create a directory to store the installation artifacts associated with your new packages, create the DESCSTACKUSERBASE environment variable and then pip install the packages you wish to add.

  •  cd $SCRATCH   # I just chose SCRATCH.. you could use another area
     mkdir <yourInstallArea>
     export DESCSTACKUSERBASE=$PWD/<yourInstallArea>   # This env var is recognized by the desc-stack-weekly setup, so --user installs will use this dir
     # For desc-python, use DESCPYTHONUSERAREA.  For desc-python-bleed, use DESCPYTHONBLEEDUSERBASE
     python /global/common/software/lsst/common/miniconda/start-kernel-cli.py desc-stack-weekly
     pip install --user --no-build-isolation <NewPackage>
    

To enable your installed packages the next time you set up desc-stack-weekly in Jupyter or on the command line, you will need to set the following in your $HOME/.bashrc:

export DESCSTACKUSERBASE=<fullPathtoYourInstallArea>

Deprecated Instructions Below

To install the package(s) in your own conda environment:

  • Create a full clone of the lsst-scipipe environment in your own area (this will take a few minutes). You will want to note the environment name after running start-kernel-cli.py and use it when running conda create --clone
    •  python /global/common/software/lsst/common/miniconda/start-kernel-cli.py desc-stack-weekly
       conda create --clone lsst-scipipe-0.6.0 -p <PathToYourDir>/mydescstack
      
      Please note that lsst-scipipe-0.6.0 is just an example, the version of the LSST Science Pipelines associated with desc-stack changes as we upgrade to more recent weekly builds.
    • Activate your new environment and add your desired packages
         conda activate <PathToYourDir>/mydescstack
         conda install <Your Package> OR pip install <Your Package> OR mamba install <Your Package>
      

Your current NERSC session now has your cloned conda environment set up and it is ready to use.

Set up DESCSTACKUSERENV env variable

  • To enable your cloned environment automatically the next time you set up the desc-stack-weekly env, including when using Jupyter at NERSC, you need to set up an environment variable in your NERSC $HOME/.bashrc or $HOME/.cshrc.
    If using bash, you would add this to your $HOME/.bashrc or $HOME/.bashrc.ext (create this file if it does not already exist):

export DESCSTACKUSERENV=<FullPathToYourCondaEnv>

Example:

export DESCSTACKUSERENV=/global/common/software/lsst/users/heatherk/mydescstack

With the DESCSTACKUSERENV environment variable set, the next time you start the desc-stack-weekly jupyter kernel, or in a new session on Cori, you will be using your own cloned environment, rather than the official desc-stack-weekly.

Full Example

Here is a full concrete example at NERSC

python /global/common/software/lsst/common/miniconda/start-kernel-cli.py desc-stack-weekly
conda create --clone desc -p /global/common/software/lsst/users/heatherk/mydescstack
conda activate /global/common/software/lsst/users/heatherk/mydescstack
conda install -c conda-forge astrocats

Now your package(s) are installed in your personal clone of the desc-stack-weekly conda environment. Your cloned conda environment is set up and ready to use in your current NERSC session.

  • In order to enable your cloned environment the next time you set up desc-stack-weekly or in jupyter.nersc.gov, you should set up a DESCSTACKUSERENV environment variable in your $HOME/.bashrc or $HOME/.bashrc.ext`, as described above.

  • If you later decide to ignore or discard your cloned desc conda environment, just unset or remove DESCSTACKUSERENV and the next time you enabledesc-stack-weekly you will be using the default desc conda environment.