Merge branch 'master' into #93

.gitignore

@@ -39,3 +39,7 @@ nosetests.xml
 *.aux
 *.log
 
+# Sphinx build directories
+doc/_build
+doc/html
+doc/latex

CHANGELOG.md

@@ -1,2 +1,4 @@
-2/10/15 Add a setup.py installer (issue #57)
+3/17/16: Update documentation to Sphinx standard and add documentation build files (issue #17)
+2/17/16: Changes to correlation function plots & documentation (issue #77) 
+2/10/15: Add a setup.py installer (issue #57)
 8/26/14: Change from relying on compiled C-code corr2 to Python package TreeCorr (issue #33)

README.md

@@ -6,7 +6,8 @@ Stile: the Systematics Tests In LEnsing pipeline
 -------------------------------------
 #### Installation instructions ####
 
-Stile is a pure python package, no compilation needed.  We do not currently have a utility that will install it in your system path for you, but as long as this directory is in your $PYTHONPATH, you should be able to import and run.
+Stile is a pure python package, no compilation needed.  You can install it using:
+> python setup.py install
 
 #### Dependencies ####
 To run Stile, you must have:
@@ -22,6 +23,11 @@ We also recommend:
 
 More dependencies may be added in the future.  
 
+-------------------------------------
+#### Documentation ####
+
+The documentation is available online at http://stile.readthedocs.io/.  You can also build it using Sphinx in the `doc/` directory.
+
 -------------------------------------
 
 #### Current functionality ####
@@ -30,6 +36,9 @@ Right now, Stile can:
 
  - Generate an average shear signal around a set of points (if you've installed corr2), given two catalogs that you specify in a particular way, and plot the results (if you have matplotlib).
  - Perform a number of basic statistics measurements and print the results.
+ - Make whisker plots.
+ - Make scatter plots with trendlines.
+ - Generate histograms.
  - Perform any of the above tests for data with different binning schemes applied, using a simple method of specifying the bins.
  - Interface with sufficiently recent versions of the HSC pipeline.
 
@@ -39,11 +48,8 @@ Right now, Stile can:
 
 Over the upcoming months, we plan to add:
 
- - A number of other systematics tests, including whisker plots, scatter plots/trend-fitting, histograms, and a broader array of correlation function tests.
  - A more flexible way of specifying your data set than hard-coding the file names in your script.
  - Automatic drivers to run as many tests as your data set allows.
  - A larger suite of example code.
- - More robust documentation of both the code and the tests.
- - Installation scripts.
  - Tests on images, and the utilities to make those tests easier.
 

devel/checklist.txt

@@ -0,0 +1,7 @@
+Before making a PR:
+ - Run all the tests in the tests/ directory and make sure they run
+ - Run all the example scripts in the examples/ directory and make sure they run
+ - Update any documentation and run `make clean html` to regenerate the docs and check that they look okay
+ - Check for line length and whitespace errors (eg with pep8ify)
+ - Add an update to the CHANGELOG.md in the main directory
+ - Check that you've merged the most recent version of master

doc/Makefile

@@ -0,0 +1,192 @@
+# Makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line.
+SPHINXOPTS    =
+SPHINXBUILD   = sphinx-build
+PAPER         =
+BUILDDIR      = _build
+
+# User-friendly check for sphinx-build
+ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1)
+	$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don\'t have Sphinx installed, grab it from http://sphinx-doc.org/)
+endif
+
+# Internal variables.
+PAPEROPT_a4     = -D latex_paper_size=a4
+PAPEROPT_letter = -D latex_paper_size=letter
+ALLSPHINXOPTS   = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
+# the i18n builder cannot share the environment and doctrees with the others
+I18NSPHINXOPTS  = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
+
+.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest coverage gettext
+
+help:
+	@echo "Please use \`make <target>' where <target> is one of"
+	@echo "  html       to make standalone HTML files"
+	@echo "  dirhtml    to make HTML files named index.html in directories"
+	@echo "  singlehtml to make a single large HTML file"
+	@echo "  pickle     to make pickle files"
+	@echo "  json       to make JSON files"
+	@echo "  htmlhelp   to make HTML files and a HTML help project"
+	@echo "  qthelp     to make HTML files and a qthelp project"
+	@echo "  applehelp  to make an Apple Help Book"
+	@echo "  devhelp    to make HTML files and a Devhelp project"
+	@echo "  epub       to make an epub"
+	@echo "  latex      to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
+	@echo "  latexpdf   to make LaTeX files and run them through pdflatex"
+	@echo "  latexpdfja to make LaTeX files and run them through platex/dvipdfmx"
+	@echo "  text       to make text files"
+	@echo "  man        to make manual pages"
+	@echo "  texinfo    to make Texinfo files"
+	@echo "  info       to make Texinfo files and run them through makeinfo"
+	@echo "  gettext    to make PO message catalogs"
+	@echo "  changes    to make an overview of all changed/added/deprecated items"
+	@echo "  xml        to make Docutils-native XML files"
+	@echo "  pseudoxml  to make pseudoxml-XML files for display purposes"
+	@echo "  linkcheck  to check all external links for integrity"
+	@echo "  doctest    to run all doctests embedded in the documentation (if enabled)"
+	@echo "  coverage   to run coverage check of the documentation (if enabled)"
+
+clean:
+	rm -rf $(BUILDDIR)/*
+
+html:
+	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
+	@echo
+	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
+
+dirhtml:
+	$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
+	@echo
+	@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
+
+singlehtml:
+	$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
+	@echo
+	@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
+
+pickle:
+	$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
+	@echo
+	@echo "Build finished; now you can process the pickle files."
+
+json:
+	$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
+	@echo
+	@echo "Build finished; now you can process the JSON files."
+
+htmlhelp:
+	$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
+	@echo
+	@echo "Build finished; now you can run HTML Help Workshop with the" \
+	      ".hhp project file in $(BUILDDIR)/htmlhelp."
+
+qthelp:
+	$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
+	@echo
+	@echo "Build finished; now you can run "qcollectiongenerator" with the" \
+	      ".qhcp project file in $(BUILDDIR)/qthelp, like this:"
+	@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/Stile.qhcp"
+	@echo "To view the help file:"
+	@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/Stile.qhc"
+
+applehelp:
+	$(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp
+	@echo
+	@echo "Build finished. The help book is in $(BUILDDIR)/applehelp."
+	@echo "N.B. You won't be able to view it unless you put it in" \
+	      "~/Library/Documentation/Help or install it in your application" \
+	      "bundle."
+
+devhelp:
+	$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
+	@echo
+	@echo "Build finished."
+	@echo "To view the help file:"
+	@echo "# mkdir -p $$HOME/.local/share/devhelp/Stile"
+	@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/Stile"
+	@echo "# devhelp"
+
+epub:
+	$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
+	@echo
+	@echo "Build finished. The epub file is in $(BUILDDIR)/epub."
+
+latex:
+	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
+	@echo
+	@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
+	@echo "Run \`make' in that directory to run these through (pdf)latex" \
+	      "(use \`make latexpdf' here to do that automatically)."
+
+latexpdf:
+	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
+	@echo "Running LaTeX files through pdflatex..."
+	$(MAKE) -C $(BUILDDIR)/latex all-pdf
+	@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
+
+latexpdfja:
+	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
+	@echo "Running LaTeX files through platex and dvipdfmx..."
+	$(MAKE) -C $(BUILDDIR)/latex all-pdf-ja
+	@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
+
+text:
+	$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
+	@echo
+	@echo "Build finished. The text files are in $(BUILDDIR)/text."
+
+man:
+	$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
+	@echo
+	@echo "Build finished. The manual pages are in $(BUILDDIR)/man."
+
+texinfo:
+	$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
+	@echo
+	@echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
+	@echo "Run \`make' in that directory to run these through makeinfo" \
+	      "(use \`make info' here to do that automatically)."
+
+info:
+	$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
+	@echo "Running Texinfo files through makeinfo..."
+	make -C $(BUILDDIR)/texinfo info
+	@echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."
+
+gettext:
+	$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
+	@echo
+	@echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."
+
+changes:
+	$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
+	@echo
+	@echo "The overview file is in $(BUILDDIR)/changes."
+
+linkcheck:
+	$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
+	@echo
+	@echo "Link check complete; look for any errors in the above output " \
+	      "or in $(BUILDDIR)/linkcheck/output.txt."
+
+doctest:
+	$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
+	@echo "Testing of doctests in the sources finished, look at the " \
+	      "results in $(BUILDDIR)/doctest/output.txt."
+
+coverage:
+	$(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage
+	@echo "Testing of coverage in the sources finished, look at the " \
+	      "results in $(BUILDDIR)/coverage/python.txt."
+
+xml:
+	$(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml
+	@echo
+	@echo "Build finished. The XML files are in $(BUILDDIR)/xml."
+
+pseudoxml:
+	$(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml
+	@echo
+	@echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml."

doc/binning.rst

@@ -0,0 +1,6 @@
+===============
+Binning methods
+===============
+
+.. automodule:: stile.binning
+   :members:

doc/binning_basics.rst

@@ -0,0 +1,115 @@
+Using the binning module
+========================
+
+Some systematics tests will benefit from being performed on subsamples of the data binned in some 
+way. For example, PSF models may be more accurate in regions of high stellar density, so producing
+PSF characterization tests as a function of galactic latitude might be useful.  Shape measurement
+may be more likely to fail for low signal-to-noise objects or objects which are small relative to
+the PSF, so checking shape measurement accuracy in signal-to-noise or size bins may reveal problems
+that cannot be seen when most of the signal comes from larger or more well-resolved objects.  Some
+tests may also benefit from selection cuts, which you can also think of as a single broad bin with
+a defined lower edge and an infinite upper edge.
+
+To make these tests easier, Stile contains some simple functions to bin your data.  Two of
+them--:class:`stile.BinStep <stile.binning.BinStep>` and 
+:class:`stile.BinList <stile.binning.BinList>`--have simple, predefined ways of acting on your
+data set. The third, :class:`stile.BinFunction <stile.binning.BinFunction>`, uses user-defined
+functions to split your data.
+
+Basic interface
+---------------
+
+To start binning your data, you create a :class:`Bin*` object that will contain binning
+definitions.  For instance, if you wanted to bin the ``ra`` column in 10 bins:
+
+>>> bin_object = stile.BinStep(field='ra', n_bins=10, low=0, high=360)
+
+To use it, call the object; it returns a list of objects which you can apply to your data to
+produce properly binned subsets.
+
+>>> for single_bin in bin_object():
+>>>     binned_data = single_bin(data)
+
+``binned_data`` is a subset of ``data`` with the same format.
+
+The ``single_bin`` above is actually another class called a :class:`stile.binning.SingleBin`.  It
+knows its boundaries and it also contains a string you can use in program outputs.
+
+>>> single_bin = bin_object()[0]
+>>> print single_bin.low
+0.0
+>>> print single_bin.hi
+36.0
+>>> print single_bin.short_name
+'0'
+
+Types of binning schemes
+------------------------
+
+The :class:`stile.BinList <stile.binning.BinList>` is the simplest class.  To create it, call it
+with a list of bin edges and a field name (see the :doc:`data` documentation for more information
+on field names).
+
+>>> bin_list_object = stile.BinList(field='g1', 
+      bin_list=[-10, -1, -0.5, -0.3, -0.1, -0.05, 0, 0.05, 0.1, 0.3, 0.5, 1, 10])
+
+:class:`stile.BinStep <stile.binning.BinStep>` is also fairly simple.  It generates bins that are
+equally spaced in linear or log space based on the provided arguments.  It is created using at
+least three of the arguments ``low`` (the low edge of the lowest bin), ``high`` (the high edge of
+the highest bin), ``n_bins`` (the number of bins to create), and ``step`` (the step size for the
+bin).  All four arguments may be passed, but will be checked for consistency if so.
+
+>>> bin_step_object = stile.BinStep(field='g2', low=-2, high=2, step=0.1)
+>>> bin_step_object = stile.BinStep(field='g2', low=-2, high=2, n_bins=40)
+
+will create identical binning schemes.
+
+Finally, :class:`stile.BinFunction <stile.binning.BinFunction>` is available for more complex
+binning schemes, especially those that rely on more than one field of data.  To use it, you will
+need a function that either 1) accepts an entire data array (with fields defined as described
+in :doc:`data`) and returns a vector of integers corresponding to the bin number for each row in
+the data array, or 2) accepts an entire data array plus an integer bin number and returns a Boolean
+mask.  You will also need to specify the maximum expected number of bins, either as an argument
+passed to the constructor or as an attribute of the function.  Then, you define the bin object as
+
+>>> bin_function_object = stile.BinFunction(func, n_bins=n_bins)
+
+if the function returns a vector of bin indices, or
+
+>>> bin_function_object = stile.BinFunction(func, n_bins=n_bins, returns_bools=True)
+
+if it returns Boolean masks.  This object can be called like any other :class:`Bin*` object to
+create a list of callable objects, and it will work with :func:`stile.ExpandBinList
+<stile.binning.ExpandBinList>` as well.  However, the child objects it creates when you call it
+don't have ``.low`` or ``.high`` attributes, so any automatic processing or looping that assumes
+these attributes exist (such as for naming files) will fail.
+
+Combining binning schemes
+-------------------------
+
+Maybe you have two binning schemes you'd like to use at once: a binning in magnitude and a binning
+in galaxy weight ``'w'``.  There is a function, :func:`stile.ExpandBinList
+<stile.binning.ExpandBinList>`, to automatically loop through all the possible pairs of those
+binning schemes.
+
+.. note::
+  The interface for :func:`ExpandBinList` may be changing in the near future--see `Stile issue 82
+  <https://github.com/msimet/Stile/issues/82>`_.
+
+:func:`stile.ExpandBinList <stile.binning.ExpandBinList>` returns a `list of lists`.  The inner
+lists are all possible pairs (tuples) of the binning schemes passed to the function.  So, for
+example, given the magnitude binning object ``magnitude_bin_object`` and the galaxy weight binning
+object ``weight_bin_object``, the data would be binned like this:
+
+>>> for bin_set in stile.ExpandBinList(magnitude_bin_object, weight_bin_object):
+>>>     binned_data = data
+>>>     for bin in bin_set:
+>>>         binned_data = bin(binned_data)
+
+:func:`stile.ExpandBinList <stile.binning.ExpandBinList>` can accept any number of bin objects as
+arguments (including none).  In the lists it returns, the first object passed as an argument
+changes most slowly, followed by the second, etc (so the first item in the list it returns will be
+``[magnitude_bin_object_0, weight_bin_object_0]``, the second will be
+``[magnitude_bin_object_0, weight_bin_object_1]``, etc).
+
+

doc/codedoc.rst

@@ -0,0 +1,15 @@
+==================
+Code documentation
+==================
+
+This section contains the docstrings for the main classes and functions that users will see.
+
+.. toctree::
+   :maxdepth: 2
+
+   binning
+   file_io
+   stile_utils
+   sys_tests
+   treecorr_utils
+   hsc