Merge pull request #100 from NREL/development

Automated testing and deployment along with bug fixes

.gitattributes

@@ -1 +1,5 @@
 rdtools/_version.py export-subst
+
+*.ipynb	diff=jupyternotebook
+
+*.ipynb	merge=jupyternotebook

.gitignore

@@ -7,10 +7,20 @@
 # ignore byte compiled files
 *.py[co]
 
+# ignore coveralls yaml, coverge dir
+.coveralls.yml
+.coverage
+
+# ignore test cache
+.pytest_cache
+
 .DS_Store
 
 docs/.ipynb_checkpoints/degradation_example-checkpoint.ipynb
 *.ipynb_checkpoints*
 
-# ignore egg-info
-rdtools.egg-info*
+# ignore setup and egg-info
+.eggs/
+build/
+dist/
+rdtools.egg-info*

.travis.yml

@@ -0,0 +1,32 @@
+sudo: false
+
+language: python
+
+python:
+  - "2.7"
+  - "3.6"
+
+# Test two environments:
+# 1) dependencies with pinned versions from requirements.txt
+# 2) 'pip install --upgrade --upgrade-strategy=eager .' to install upgraded
+#    dependencies from PyPi using version ranges defined within setup.py
+env:
+  - REQ_ENV='-r requirements.txt .'
+  - REQ_ENV='--upgrade --upgrade-strategy=eager .'
+
+install:
+  - pip install $REQ_ENV
+
+script:
+  - pytest
+
+deploy:
+  provider: pypi
+  user: RdTools
+  password:
+    secure: GjzVQdzwnXUx+nJV5R7FdZRiI+6amQAklzS2rj0/i23B2QCh0qnnbIjcatodiglBhrH/i/ccNT6HVvYt21IDxux50VuGCSgvvX1I+QfBtDCV0Wt61j8Ms9C7Xbbdt/jXeI4KXQYDq118GJgWPQDfZPDo0okttwrRYNuYhTPYdJ7qmnn3pKOKwrDcjRNiO8dWv8tcY8bfzp7WEEhWhUqHbWGf3EprbVALuS7DF5BzoxKz6np869dTj/5j7BdCF19vscQnP1p8rM7H7J+0m6SNH9slbTE1Z/wfY+97906+zvHOU3rAC2q6NBKRArSIcWy/RsYQcdmuXpZPw7fk04whMZ3V7zojmtkeggTp7AbKlf7qmwKLkDGfNofztQNojNg171eLFSz3GLlYzraw8wLa1Zq8APjq/gG9TkqQkRZ0n73MuYTtZz2ARQZiuDWPJ8sjyoonp04FXRqTK+zKCaFvSaATEXMj4N9GY1yQawrNCOPodR42sz82aCIp/Yu5bvrP1YuMAHkYttzD0HScMRN/ebHethSdRc2S3x2yyX6r7B76lfcx1tS6vcbSVc6B8gg59cZk+82AlpbF7LxJnmenThfKp3cdiYAUI7oYo8+Bfp6ZI44UH8ODY8d4Lhx00o55ojAb7tiwJSWLio3NSC5LGagCwfJmiQPze1luPiKIqBM=
+  on:
+    tags: true
+    branch: master
+  distributions: "sdist bdist_wheel"
+  skip_cleanup: true

README.md

@@ -1,8 +1,11 @@
 # About RdTools
 
+Master branch: [![Build Status](https://travis-ci.org/NREL/rdtools.svg?branch=master)](https://travis-ci.org/NREL/rdtools)  
+Development branch: [![Build Status](https://travis-ci.org/NREL/rdtools.svg?branch=development)](https://travis-ci.org/NREL/rdtools)
+
 RdTools is a set of Python tools for analysis of photovoltaic data.
 In particular, PV production data is evaluated over several years
-to obtain rates of performance degradation over time. Rdtools can
+to obtain rates of performance degradation over time. RdTools can
 handle both high frequency (hourly or better) or low frequency (daily, weekly, etc.)
 datasets. Best results are obtained with higher frequency data.
 
@@ -28,7 +31,7 @@ degradation. The width of the distribution provides information about the uncert
 estimate via a bootstrap calculation. The [example notebook](./docs/degradation_example.ipynb) uses the output of `degradation.degradation_year_on_year()`
 to visualize the calculation.
 
-<img src="./screenshots/Clearsky_result.png" width="600" height="456" alt="RdTools Result"/>
+<img src="./screenshots/Clearsky_result_updated.png" width="600" height="456" alt="RdTools Result"/>
 
 
 Two workflows are available for system performance ratio calculation, and illustrated in an example notebook. 
@@ -50,14 +53,14 @@ Alternatively it can be installed manually using the command line:
 2. Navigate to the repository: `cd rdtools`
 3. Install via pip: `pip install .`
 
-On some systems installation with `pip` can fail due to problems installing requireiments. If this occurs, the requirements specified in `setup.py` may need to be seperately installed (for example by using `conda`) before installing `rdtools`.
+On some systems installation with `pip` can fail due to problems installing requirements. If this occurs, the requirements specified in `setup.py` may need to be separately installed (for example by using `conda`) before installing `rdtools`.
 
 RdTools currently runs in both Python 2.7 and 3.6.
 
-## Usage
+## Usage and examples
 
 
-Full workflow examples are found in the notebooks in [rdtools/docs](./docs/degradation_example.ipynb).
+Full workflow examples are found in the notebooks in [rdtools/docs](./docs/degradation_example.ipynb). The examples are designed to work with python 3.6. For a consistent experience, we recommend installing the packages and versions documented in `docs/notebook_requirements.txt`. This can be achieved in your environment by first installing RdTools as described above, then running `pip install -r docs/notebook_requirements.txt` from the base directory.
 
 The following functions are used for degradation analysis:
 
@@ -126,12 +129,6 @@ Other useful references which may also be consulted for degradation rate methodo
   - D. Jordan, S. Kurtz, PV Degradation Rates – an Analytical Review, Progress in Photovoltaics: Research and Application, 2013, 21(1), 12 - 29.
   - E. Hasselbrink, M. Anderson, Z. Defreitas, M. Mikofski, Y.-C.Shen, S. Caldwell, A. Terao, D. Kavulak, Z. Campeau, D. DeGraaff, “Validation of the PVLife model using 3 million module-years of live site data”, 39th IEEE Photovoltaic Specialists Conference, Tampa, FL, USA, 2013, p. 7 – 13, DOI: 10.1109/PVSC.2013.6744087.
 
-## Unit tests
-
-To run tests from the main directory:
-```
-$ tests/run_tests
-```
 ## Further Instructions and Updates
 
 Check out the [wiki](https://github.com/NREL/rdtools/wiki) for additional usage documentation, and for information on development goals and framework.

docs/notebook_requirements.txt

@@ -0,0 +1,50 @@
+appnope==0.1.0
+backcall==0.1.0
+bleach==2.1.3
+cycler==0.10.0
+decorator==4.3.0
+entrypoints==0.2.3
+html5lib==1.0.1
+ipykernel==4.8.2
+ipython==5.8.0
+ipython-genutils==0.2.0
+ipywidgets==7.3.0
+jedi==0.12.1
+Jinja2==2.10
+jsonschema==2.6.0
+jupyter==1.0.0
+jupyter-client==5.2.3
+jupyter-console==5.2.0
+jupyter-core==4.4.0
+kiwisolver==1.0.1
+MarkupSafe==1.0
+matplotlib==2.2.2
+mistune==0.8.3
+nbconvert==5.3.1
+nbformat==4.4.0
+notebook==5.6.0
+numpy==1.15.2
+pandas==0.23.4
+pandocfilters==1.4.2
+parso==0.3.1
+patsy==0.5.0
+pexpect==4.6.0
+pickleshare==0.7.4
+prometheus-client==0.3.0
+prompt-toolkit==1.0.15
+ptyprocess==0.6.0
+pvlib==0.5.2
+Pygments==2.2.0
+pyparsing==2.2.0
+pyzmq==17.1.0
+qtconsole==4.3.1
+Send2Trash==1.5.0
+simplegeneric==0.8.1
+tables==3.4.4
+terminado==0.8.1
+testpath==0.3.1
+tornado==5.1
+traitlets==4.3.2
+wcwidth==0.1.7
+webencodings==0.5.1
+widgetsnbextension==3.3.0

rdtools/aggregation.py

@@ -1,5 +1,6 @@
-import pandas as pd
-
+'''
+Aggregation Helper Functions
+'''
 
 def aggregation_insol(normalized_energy, insolation, frequency='D'):
     '''

rdtools/clearsky_temperature.py

@@ -70,9 +70,9 @@ def get_clearsky_tamb(times, latitude, longitude, window_size=40, gauss_std=20):
         day = _get_pixel_value(a, lon_index, lat_index, k, radius)
         night = _get_pixel_value(b, lon_index, lat_index, k, radius)
 
-        if day == float("NaN"):
+        if pd.isnull(day):
             day = a[:, lat_index, k]
-        if night == float("NaN"):
+        if pd.isnull(night):
             night = a[:, lat_index, k]
 
         ave_day.append(day)
@@ -112,13 +112,13 @@ def _get_pixel_value(data, i, j, k, radius):
                 continue
 
             value = data[x, y, k]
-            if value == float("NaN"):
+            if pd.isnull(value):
                 continue
 
             list.append(value)
 
     if len(list) == 0:
-        return float("NaN")
+        return np.nan
 
     return pd.Series(list).median()
 

rdtools/degradation.py

@@ -190,8 +190,9 @@ def degradation_year_on_year(normalized_energy, recenter=True, exceedance_prob=9
     tuple of (degradation_rate, confidence_interval, calc_info)
         degradation_rate:  float
             rate of relative performance change in %/yr
-        confidence_interval:  float
-            one-sigma confidence interval of degradation rate estimate
+        confidence_interval:  numpy ndarray
+            confidence interval (size specified by confidence_level) of degradation
+            rate estimate
         calc_info:  dict
             ('YoY_values') pandas series of right-labeled year on year slopes
             ('renormalizing_factor') float of value used to recenter data

rdtools/normalization.py

@@ -69,6 +69,7 @@ def normalize_with_pvwatts(energy, pvwatts_kws):
     ----------
     energy: Pandas Series (numeric)
         Energy time series to be normalized in watt hours.
+        Must be a right-labeled regular time series.
     pvwatts_kws: dictionary
         Dictionary of parameters used in the pvwatts_dc_power function.
 
@@ -99,10 +100,7 @@ def normalize_with_pvwatts(energy, pvwatts_kws):
             Insolation associated with each normalized point
     '''
 
-    if energy.index.freq is None:
-        freq = pd.infer_freq(energy.index)
-    else:
-        freq = energy.index.freq
+    freq = check_series_frequency(energy, 'energy')
 
     dc_power = pvwatts_dc_power(**pvwatts_kws)
     irrad = pvwatts_kws['poa_global']
@@ -213,6 +211,7 @@ def normalize_with_sapm(energy, sapm_kws):
     ----------
     energy: Pandas Series (numeric)
         Energy time series to be normalized  in watt hours.
+        Must be a right-labeled regular time series.
     sapm_kws: dictionary
         Dictionary of parameters required for sapm_dc_power function.
 
@@ -234,10 +233,7 @@ def normalize_with_sapm(energy, sapm_kws):
             Insolation associated with each normalized point
     '''
 
-    if energy.index.freq is None:
-        freq = pd.infer_freq(energy.index)
-    else:
-        freq = energy.index.freq
+    freq = check_series_frequency(energy, 'energy')
 
     dc_power, irrad = sapm_dc_power(**sapm_kws)
 
@@ -271,11 +267,23 @@ def normalize_with_sapm(energy, sapm_kws):
 
 
 def delta_index(series):
-    # Takes a panda series as input and returns (time step sizes, average time step size)
-    # Length of each interval calculated by using 'int64' to convert to nanoseconds
+    '''
+    Takes a panda series with a DatetimeIndex as input and
+    returns (time step sizes, average time step size) in hours
+    '''
 
-    deltas = (series.index - series.index.shift(-1)).astype('int64') / (10.0**9 * 3600.0)
-    return deltas, np.mean(deltas)
+    if series.index.freq is None:
+        # If there is no frequency information, explicily calculate interval sizes
+        # Length of each interval calculated by using 'int64' to convert to nanoseconds
+        hours = pd.Series(series.index.astype('int64') / (10.0**9 * 3600.0))
+        hours.index = series.index
+        deltas = hours.diff()
+    else:
+        # If there is frequency information, pandas shift can be used to gain a meaningful
+        # interful for the first element of the timeseries
+        # Length of each interval calculated by using 'int64' to convert to nanoseconds
+        deltas = (series.index - series.index.shift(-1)).astype('int64') / (10.0**9 * 3600.0)
+    return deltas, np.mean(deltas.dropna())
 
 
 def irradiance_rescale(irrad, modeled_irrad, max_iterations=100, method=None):
@@ -358,3 +366,18 @@ def _rmse(fact):
 
     else:
         raise ValueError('Invalid method')
+
+
+def check_series_frequency(series, series_description):
+    '''Returns the inferred frequency of a pandas series, raises ValueError
+    using series_description if it can't. series_description should be a string'''
+
+    if series.index.freq is None:
+        freq = pd.infer_freq(series.index)
+        if freq is None:
+            error_string = ('Could not infer frequency of ' + series_description +
+                            ', which must be a regular time series')
+            raise ValueError(error_string)
+    else:
+        freq = series.index.freq
+    return freq

tests/clearsky_temperature_test.py → rdtools/test/clearsky_temperature_test.py

@@ -36,6 +36,9 @@ def test_hour_offset(self):
         self.assertTrue(east_hottest_hour > 12)
         self.assertTrue(west_hottest_hour > east_hottest_hour)
 
+# TODO:
+# Test irradiance_rescale
+
 
 if __name__ == '__main__':
     unittest.main()