Skip to content

Commit

Permalink
Adding packages metadata and documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
jordancaraballo committed Aug 21, 2023
1 parent def5181 commit 39e73f2
Show file tree
Hide file tree
Showing 6 changed files with 78 additions and 32 deletions.
9 changes: 9 additions & 0 deletions .readthedocs.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
version: 2

build:
os: "ubuntu-20.04"
tools:
python: "3.8"

sphinx:
fail_on_warning: true
21 changes: 13 additions & 8 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,11 @@ directory so projects, examples, and notebooks can be run.
Note: PIP installations do not include CUDA libraries for GPU support. Make sure NVIDIA libraries
are installed locally in the system if not using conda/mamba.

```bash
.. code-block:: bash
module load singularity # if a module needs to be loaded
singularity build --sandbox pytorch-caney-container docker://nasanccs/pytorch-caney:latest
```


Why Caney?
---------------
Expand Down Expand Up @@ -66,24 +67,28 @@ Pre-training with Masked Image Modeling
-----------------------------------------

To pre-train the swinv2 base model with masked image modeling pre-training, run:
```bash

.. code-block:: bash
torchrun --nproc_per_node <NGPUS> pytorch-caney/pytorch_caney/pipelines/pretraining/mim.py --cfg <config-file> --dataset <dataset-name> --data-paths <path-to-data-subfolder-1> --batch-size <batch-size> --output <output-dir> --enable-amp
```

For example to run on a compute node with 4 GPUs and a batch size of 128 on the MODIS SatVision pre-training dataset with a base swinv2 model, run:

```bash
.. code-block:: bash
singularity shell --nv -B <mounts> /path/to/container/pytorch-caney-container
Singularity> export PYTHONPATH=$PWD:$PWD/pytorch-caney
Singularity> torchrun --nproc_per_node 4 pytorch-caney/pytorch_caney/pipelines/pretraining/mim.py --cfg pytorch-caney/examples/satvision/mim_pretrain_swinv2_satvision_base_192_window12_800ep.yaml --dataset MODIS --data-paths /explore/nobackup/projects/ilab/data/satvision/pretraining/training_* --batch-size 128 --output . --enable-amp
```


This example script runs the exact configuration used to make the SatVision-base model pre-training with MiM and the MODIS pre-training dataset.
```bash

.. code-block:: bash
singularity shell --nv -B <mounts> /path/to/container/pytorch-caney-container
Singularity> cd pytorch-caney/examples/satvision
Singularity> ./run_satvision_pretrain.sh
```


Fine-tuning Satvision-base
-----------------------------
Expand Down
24 changes: 0 additions & 24 deletions docs/conf.py
Original file line number Diff line number Diff line change
@@ -1,27 +1,14 @@
# Configuration file for the Sphinx documentation builder.
#
# For the full list of built-in configuration values, see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
import os
import sys

sys.path.insert(0, os.path.abspath('..'))

import pytorch_caney

# package_path = os.path.abspath('..')
# os.environ['PYTHONPATH'] = ':'.join((package_path, os.environ.get('PYTHONPATH', '')))

# -- Project information -----------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information

project = 'pytorch-caney'
copyright = '2023, Jordan A. Caraballo-Vega'
author = 'Jordan A. Caraballo-Vega'

# -- General configuration ---------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration

extensions = [
'sphinx.ext.autodoc',
'sphinx_autodoc_typehints',
Expand All @@ -43,15 +30,6 @@
templates_path = ['_templates']
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']

# -- Options for HTML output -------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output

#source_suffix ={
# '.rst': 'restructuredtext',
# '.txt': 'markdown',
# '.md': 'markdown',
# '.ipynb': 'myst-nb'
#}
master_doc = "index"

version = release = pytorch_caney.__version__
Expand All @@ -63,8 +41,6 @@
html_theme = 'sphinx_rtd_theme'
html_logo = 'static/DSG_LOGO_REDESIGN.png'

# html_static_path = ['_static/']

myst_enable_extensions = [
"amsmath",
"colon_fence",
Expand Down
6 changes: 6 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
[build-system]
# Minimum requirements for the build system to execute.
requires = ["setuptools", "wheel"]

[tool.black]
target_version = ['py39']
1 change: 1 addition & 0 deletions pytorch_caney/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
__version__ = "0.1.0"
49 changes: 49 additions & 0 deletions setup.cfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
[metadata]
name = pytorch-caney
version = attr: pytorch_caney.__version__
description = Methods for pytorch deep learning applications
long_description = file: README.md
long_description_content_type = text/markdown
keywords = pytorch-caney, deep-learning, machine-learning
url = https://github.com/nasa-nccs-hpda/pytorch-caney
author = jordancaraballo
author_email = [email protected]
license = MIT
license_file = LICENSE.md
classifiers =
Development Status :: 4 - Beta
Intended Audience :: Developers
Intended Audience :: Science/Research
Topic :: Software Development :: Libraries :: Python Modules
License :: OSI Approved :: MIT License
Programming Language :: Python :: 3 :: Only
project_urls =
Documentation = https://github.com/nasa-nccs-hpda/pytorch-caney
Source = https://github.com/nasa-nccs-hpda/pytorch-caney
Issues = https://github.com/nasa-nccs-hpda/pytorch-caney/issues

[options]
packages = find:
zip_safe = True
include_package_data = True
platforms = any
python_requires = >= 3.7
install_requires =
omegaconf
numpy
pandas
tqdm
xarray
rioxarray
numba

[options.extras_require]
test =
pytest
coverage[toml]
black
docs =
pdoc==8.0.1
all =
%(docs)s
%(test)s

0 comments on commit 39e73f2

Please sign in to comment.