Merge base-environment.yml into solver environments

Easier to explain and less fragile, at the expense of redundancy in individual environment.yml files. It's a shame they don't have an include mechanism. Refactored environment documentation into environments/README.rst.
usnistgov · Dec 12, 2023 · 1c1afa5 · 1c1afa5
1 parent 4c201df
commit 1c1afa5
Show file tree

Hide file tree

Showing 10 changed files with 162 additions and 112 deletions.
diff --git a/.azure/templates/install.yml b/.azure/templates/install.yml
@@ -49,12 +49,8 @@ steps:
   # Daily midnight build uses environment.yml files to catch regressions
   # due to updates in dependencies
   - bash: |
-      python -m pip install conda-merge
-      conda-merge environments/base-environment.yml \
-        environments/${{ parameters.solver }}-environment.yml \
-        > environments/merged-environment.yml
       conda env create --name myEnvironment \
-        --file environments/merged-environment.yml
+        --file environments/${{ parameters.solver }}-environment.yml
     displayName: Create Anaconda environment from environments
     condition: eq(variables['Build.CronSchedule.DisplayName'], 'Daily midnight build')
 

diff --git a/INSTALLATION.rst b/INSTALLATION.rst
@@ -66,127 +66,71 @@ Recommended Method
    These days, the most useful channel is conda-forge_, which provides
    everything necessary to install :term:`FiPy`.
 
-* `install Miniconda`_ on your computer
-* Install :term:`FiPy` :ref:`REQUIREDPACKAGES` with one of the following:
+Install conda_
+==============
 
-    * Conda environment files::
+`Install conda`_ or `install micromamba`_ on your computer.
 
-        $ conda env create --name <MYFIPYENV> \
-            --file environments/base-environment.yml
 
-      followed by::
+.. include:: CONDA_ENVIRONMENT.rst
 
-        $ conda env update --name <MYFIPYENV> \
-            --file environments/<SOLVER>-environment.yml
 
-      You can try to include multiple solver suites, but be aware that
-      different suites may have incompatible requirements, or may restrict
-      installation to obsolete versions of Python.  Given that :term:`FiPy`
-      can only use one solver suite during a run, installing more than one
-      solver in an environment isn't necessary.
+Install :term:`FiPy`
+====================
 
-      .. attention::
-
-         Successively updating an environment can be unpredictable, as
-         later packages may conflict with earlier ones.  Unfortunately,
-         ``conda env create`` `does not support multiple environment files
-         <https://github.com/conda/conda/issues/9294>`_.
-
-         Alternatively, combine :file:`environments/base-environment.yml`
-         with any :file:`environments/<SOLVER>-environment.yml` you wish to
-         use (`conda-merge <https://github.com/amitbeka/conda-merge>`_ may
-         prove useful).  Then execute::
-
-           $ conda env create --name <MYFIPYENV> --file <MYMERGEDENVIRONMENT>.yml
-
-    * `conda-lock <https://github.com/conda/conda-lock>`_ lockfiles::
-
-        $ conda-lock install --name <MYFIPYENV> \
-            environments/locks/conda-<SOLVER>-lock.yml
-
-      or, to be really explicit::
-
-        $ conda create --name <MYFIPYENV> \
-            --file environments/locks/conda-<SOLVER>-<PLATFORM>.lock
-
-      These lockfiles list the specific package versions used to test
-      :term:`FiPy` and should provide the most reproducible environment.
-
-    * Directly from conda-forge_, picking and choosing desired packages, e.g.::
-
-        $ conda create --name <MYFIPYENV> --channel conda-forge \
-            python=3 numpy scipy matplotlib-base future packaging mpich \
-            mpi4py petsc4py mayavi "gmsh <4.0|>=4.5.2"
-
-      or::
-
-        $ conda create --name <MYFIPYENV> --channel conda-forge \
-            python=2.7 numpy scipy matplotlib-base future packaging \
-            pysparse mayavi "traitsui<7.0.0" "gmsh<4.0"
-
-      .. attention::
-
-         Bit rot has started to set in for Python 2.7.  One consequence is that
-         :class:`~fipy.viewers.vtkViewer.VTKViewer`\s can raise errors
-         (probably other uses of :term:`Mayavi`, too). Hence, the constraint
-         of `"traitsui<7.0.0"`.
-
-  .. note::
-
-     Each of these commands create a self-contained conda_ environment and
-     then download and populate the environment with the prerequisites for
-     :term:`FiPy` from the conda-forge_ channel at https://anaconda.org.
-
-* Install :term:`FiPy` itself::
+::
 
     $ conda install --name <MYFIPYENV> --channel conda-forge fipy
 
-  .. note::
+.. note::
+
+   The `fipy conda-forge`_ package used to be "batteries included", but
+   we found this to be too fragile.  It now only includes the bare
+   minimum for :term:`FiPy` to function.
 
-     The `fipy conda-forge`_ package used to be "batteries included", but
-     we found this to be too fragile.  It now only includes the bare
-     minimum for :term:`FiPy` to function.
+Enable conda_ environment
+=========================
 
-* enable this new environment with::
+Enable your new environment with::
 
     $ conda activate <MYFIPYENV>
 
-  or
+or::
 
     $ source activate <MYFIPYENV>
 
-  .. note::
+or, on Windows_::
 
-     ``$ activate <MYFIPYENV>`` on Windows_
+    $ activate <MYFIPYENV>
 
-* move on to :ref:`USAGE`.
+You're now ready to move on to :ref:`USAGE`.
 
-  .. note::
+.. note::
 
-     conda_ can be
-     `quite <https://www.anaconda.com/blog/understanding-and-improving-condas-performance>`_
-     `slow <https://medium.com/@marius.v.niekerk/conda-metachannel-f962241c9437>`_
-     to resolve all dependencies when performing
-     an installation.  You may wish to consider using the alternative
-     mamba_ installation manager to speed things up.
+   conda_ can be
+   `quite <https://www.anaconda.com/blog/understanding-and-improving-condas-performance>`_
+   `slow <https://medium.com/@marius.v.niekerk/conda-metachannel-f962241c9437>`_
+   to resolve all dependencies when performing
+   an installation.  You may wish to consider using the alternative
+   mamba_ installation manager to speed things up.
 
-  .. note::
+.. note::
 
-     On Linux_ and `Mac OS X`_, you should have a pretty complete system
-     to run and visualize :term:`FiPy` simulations. On Windows_, there
-     are fewer packages available via conda_, particularly amongst the
-     sparse matrix :ref:`SOLVERS`, but the system still should be
-     functional. Significantly, you will need to download and install
-     :term:`Gmsh` manually when using Python 2.7.
+   On Linux_ and `Mac OS X`_, you should have a pretty complete system
+   to run and visualize :term:`FiPy` simulations. On Windows_, there
+   are fewer packages available via conda_, particularly amongst the
+   sparse matrix :ref:`SOLVERS`, but the system still should be
+   functional. Significantly, you will need to download and install
+   :term:`Gmsh` manually when using Python 2.7.
 
 .. attention::
 
    When installed via conda_ or :term:`pip`, :term:`FiPy` will not include
    its :ref:`examples <part:examples>`.  These can be obtained by
    `cloning the repository`_ or downloading a `compressed archive`_.
 
-.. _install Miniconda: https://conda.io/projects/conda/en/latest/user-guide/install
-.. _guyer: https://anaconda.org/guyer
+.. _install conda: https://conda.io/projects/conda/en/latest/user-guide/install/
+.. _install micromamba: https://mamba.readthedocs.io/en/latest/installation/micromamba-installation.html
 .. _conda-forge: https://conda-forge.github.io/
 .. _Mac OS X: http://www.apple.com/macosx/
 .. _Linux: http://www.linux.org/
@@ -406,8 +350,8 @@ of the system package manager and the system directories.  These utilities
 include conda_, Nix_, Stow_, Virtualenv_ and Buildout_, amongst others.
 Conda_ and Nix_ are only ones of these we have the resources to support.
 
-Our preferred development environment is set up with the initial steps of
-the :ref:`RECOMMENDED_METHOD`, without :term:`FiPy` itself, followed by::
+Our preferences is to :ref:`CREATE_CONDA_ENVIRONMENT` for development,
+followed by::
 
    $ source activate <MYFIPYENV>
    $ python -m pip install scikit-fmm

diff --git a/docs/source/ADMINISTRATA.rst b/docs/source/ADMINISTRATA.rst
@@ -129,7 +129,6 @@ The `conda-lock <https://github.com/conda/conda-lock>`_ lockfiles in
     $ for solver in petsc pysparse scipy trilinos
       do
         conda-lock lock \
-          --file environments/base-environment.yml \
           --file environments/${solver}-environment.yml \
           --lockfile environments/locks/conda-${solver}-lock.yml
         conda-lock render \

diff --git a/docs/source/CONDA_ENVIRONMENT.rst b/docs/source/CONDA_ENVIRONMENT.rst
@@ -0,0 +1 @@
+.. include:: ../../environments/README.rst
diff --git a/environments/README.rst b/environments/README.rst
@@ -0,0 +1,87 @@
+.. _CREATE_CONDA_ENVIRONMENT:
+
+Create a conda_ environment
+===========================
+
+Use one of the following methods to create a self-contained conda_
+environment and then download and populate the environment with the
+prerequisites for :term:`FiPy` from the conda-forge_ channel at
+https://anaconda.org.
+
+* Conda_ environment files
+
+  This option is the most upgradable in the future and probably the best
+  for development.
+
+  ::
+
+    $ conda env create --name <MYFIPYENV> \
+        --file environments/<SOLVER>-environment.yml
+
+  .. note::
+
+     You can try to include multiple solver suites using ``conda env
+     update``, but be aware that different suites may have incompatible
+     requirements, or may restrict installation to obsolete versions of
+     Python.  Given that :term:`FiPy` can only use one solver suite during
+     a run, installing more than one solver in an environment isn't
+     necessary.
+
+     .. attention::
+
+        Successively updating an environment can be unpredictable, as later
+        packages may conflict with earlier ones.  Unfortunately, ``conda
+        env create`` `does not support multiple environment files
+        <https://github.com/conda/conda/issues/9294>`_.
+
+        Alternatively, combine the different
+        :file:`environments/<SOLVER>-environment.yml` files you wish to
+        use, along with `environment.yml` files for any other packages you
+        are interested in (`conda-merge
+        <https://github.com/amitbeka/conda-merge>`_ may prove useful).
+        Then execute::
+
+          $ conda env create --name <MYFIPYENV> --file <MYMERGEDENVIRONMENT>.yml
+
+* conda-lock_ lockfiles
+
+  This option will pin all the packages, so is the most reproducible, but
+  not particularly upgradable.  For most, this is the safest way to
+  generate a FiPy environment that consistently works.
+
+  ::
+
+    $ conda-lock install --name <MYFIPYENV> \
+        environments/locks/conda-<SOLVER>-lock.yml
+
+  or, to be really explicit (and obviating the need for conda-lock_)::
+
+    $ conda create --name <MYFIPYENV> \
+        --file environments/locks/conda-<SOLVER>-<PLATFORM>.lock
+
+* Directly from conda-forge_, picking and choosing desired packages
+
+  This option is the most flexible, but has the highest risk of missing or
+  incompatible packages.
+
+  e.g.::
+
+    $ conda create --name <MYFIPYENV> --channel conda-forge \
+        python=3 numpy scipy matplotlib-base future packaging mpich \
+        mpi4py petsc4py mayavi "gmsh <4.0|>=4.5.2"
+
+  or::
+
+    $ conda create --name <MYFIPYENV> --channel conda-forge \
+        python=2.7 numpy scipy matplotlib-base future packaging \
+        pysparse mayavi "traitsui<7.0.0" "gmsh<4.0"
+
+  .. attention::
+
+     Bit rot has started to set in for Python 2.7.  One consequence is that
+     :class:`~fipy.viewers.vtkViewer.VTKViewer`\s can raise errors
+     (probably other uses of :term:`Mayavi`, too). Hence, the constraint
+     of `"traitsui<7.0.0"`.
+
+.. _conda-lock: https://github.com/conda/conda-lock
+
diff --git a/environments/base-environment.yml b/environments/base-environment.yml
diff --git a/environments/petsc-environment.yml b/environments/petsc-environment.yml
@@ -1,6 +1,15 @@
 channels:
   - conda-forge
 dependencies:
+  - python
+  - numpy
+  - scipy
+  - matplotlib-base
+  - future
+  - packaging
+  - mayavi
+  - gmsh <4.0|>=4.5.2
+  - pip
   - mpich
   - mpi4py
   # petsc 3.20.2_*_102 is broken on macOS,

diff --git a/environments/pysparse-environment.yml b/environments/pysparse-environment.yml
@@ -1,8 +1,17 @@
 channels:
   - conda-forge
 dependencies:
-  - pysparse
+  - python
+  - numpy
+  - scipy
+  - matplotlib-base
+  - future
+  - packaging
   - mayavi # [unix]
+  - traitsui <7.0.0
+  - gmsh <4.0|>=4.5.2
+  - pip
+  - pysparse
 platforms:
   - linux-64
   - osx-64
diff --git a/environments/scipy-environment.yml b/environments/scipy-environment.yml
@@ -3,4 +3,12 @@
 channels:
   - conda-forge
 dependencies:
+  - python
+  - numpy
   - scipy
+  - matplotlib-base
+  - future
+  - packaging
+  - mayavi
+  - gmsh <4.0|>=4.5.2
+  - pip
diff --git a/environments/trilinos-environment.yml b/environments/trilinos-environment.yml
@@ -1,6 +1,15 @@
 channels:
   - conda-forge
 dependencies:
+  - python
+  - numpy
+  - scipy
+  - matplotlib-base
+  - future
+  - packaging
+  - mayavi
+  - gmsh <4.0|>=4.5.2
+  - pip
   - mpich
   - mpi4py
   - pytrilinos