Added PV Fleets QA pipeline examples

[kperrynrel]

• Closes Power, irradiance, and temperature examples from PV Fleets pipeline? #201
• Non-API functions clearly documented with docstrings or comments as necessary.
• Adds description and name entries in the appropriate "what's new" file
  in docs/whatsnew
  for all changes. Includes link to the GitHub Issue with :issue:`num`
  or this Pull Request with :pull:`num`. Includes contributor name
  and/or GitHub username (link with :ghuser:`user`).
• Pull request is nearly complete and ready for detailed review.
• Maintainer: Appropriate GitHub Labels and Milestone are assigned to the Pull Request and linked Issue.

This PR includes QA examples of the PV fleets pipeline, adapted from the following repo: https://github.com/kperrynrel/fleets_qa_examples

[cwhanse]

@kperrynrel I think you'll need a README.rst file in order for the section of the Examples to be created. I would call this group of examples PVFleets QA pipeline.

[kandersolar]

The test failures can be ignored here; they'll be fixed in #203.

[kperrynrel]

@kandersolar these are ready for review. Do we have a limit on example data size? The files are bigger for these examples and I could host them on the Duramat datahub if need be. Let me know!

[cwhanse]

@kperrynrel many of the style-related comments in the irradiance example also apply to the power example.

These are lengthy scripts, thanks for preparing them.

Are there steps that you think could be packaged into new pvanalytics functions? Something to consider for future work.

[cwhanse]

This sequential filtering makes me uncomfortable. Although the plot shows that time_series maintains its original length, the statements make me think that points are being dropped. I may be reading these lines as arrays and these are OK because pandas handles the indexing.

[kperrynrel]

The asfreq() call at the end of this returns the data to its original frequency, with NaN's for the data that's been filtered out. To make this less sequential, I have updated this to a single call:

# Filter the time series, taking out all of the issues
issue_mask = ((~stale_data_mask) & (~negative_mask) &
          (~erroneous_mask) & (~zscore_outlier_mask))
time_series = time_series[issue_mask]
time_series = time_series.asfreq(data_freq)

[cwhanse]

That's much easier to read, thanks

[cwhanse]

Same comment as in the irradiance example: do these filters change the length of the Series?

[kperrynrel]

Did the same here as above to make the mask a single call. asfreq() returns the time series to original data frequency with NaN's:

# Filter the time series, taking out all of the issues
issue_mask = ((~stale_data_mask) & (temperature_limit_mask) &
              (~zscore_outlier_mask))
time_series = time_series[issue_mask]
time_series = time_series.asfreq(data_freq)

[kandersolar]

Do we have a limit on example data size? The files are bigger for these examples and I could host them on the Duramat datahub if need be.

Looks like adding these files will increase the size of the data directory from ~2 MB to ~80 MB. It would be nice to keep both the repository and the PyPI distribution files lightweight if we can, and that big of an increase for data files that many/most installations won't ever touch does seem a bit wasteful. Keeping the datasets on the Data Hub instead of this repository comes with the downside of having to make network requests during the docs build, which isn't desirable either.

I explored a few alternative formats for system_50_ac_power_2_full_DST.csv. Of course storing as a binary format like parquet helps, as does cutting off unnecessary digits of precision in the power data, and using an encoding for the timestamp index that exploits the consistent 1-minute interval. Results:

• CSV (current form): 50.8 MB
• Zipped CSV: 7.9 MB
• parquet: 14.8 MB
• parquet, float32: 14.2 MB
• parquet, float32, delta-encoded index: 4.6 MB
• parquet, float32, delta-encoded index, negative values clipped to zero: 3.2 MB

For a dataset with ~1.4 million values (plus as many timestamps), I doubt there's much more opportunity for easy reduction via encoding tweaks. For reference, here is the code to produce the 3.2 MB file:

df = pd.read_csv("system_50_ac_power_2_full_DST.csv", index_col=0, parse_dates=True)
df.astype(np.float32).clip(lower=0).reset_index().to_parquet("system_50_encoded.parquet",
                                                             use_dictionary=False,
                                                             column_encoding={"measured_on": "DELTA_BINARY_PACKED"})

The next question is: do we need the full data file? It's currently ~2.5 years. Cutting it down to one year gets the size down to 1.2 MB. 6 months is only 600 kB. Would shorter data periods still achieve the goals of these examples? Or resample the file to 5-minute averages?

[kperrynrel]

@kandersolar I upsampled all the data so it's 15 minute averaged, and then converted to parquet format. We are now getting an error on parquet support:

• Missing optional dependency 'pyarrow'. pyarrow is required for parquet support. Use pip or conda to install pyarrow.
• Missing optional dependency 'fastparquet'. fastparquet is required for parquet support. Use pip or conda to install fastparquet

You ok with me adding either fastpaquet or pyarrow as a dependency? Also, do you have a preference on one?

[kandersolar]

You ok with me adding either fastpaquet or pyarrow as a dependency? Also, do you have a preference on one?

Yep! I think we can get away with keeping it as a docs requirement rather than an installation requirement. I'd say mild preference for pyarrow for its maturity.

[kperrynrel]

@kandersolar @cwhanse let me know if there's anything else I can update here before the 2.0 release! I updated some of the matplotlib rendering yesterday and am feeling ok with how it is looking. Thanks!

[cwhanse]

This plot isn't rendering

[cwhanse]

That plot still isn't rendering, idk why not. @kperrynrel

[kperrynrel]

Looks like it's rendering now, not sure what was going on with it:

[cwhanse]

Thanks @kperrynrel

[kandersolar]

Thanks @kperrynrel! One minor comment, otherwise good to merge

[kandersolar]

This "less than 10% of daily mean" rule is used in the power analysis code, but not here. Should this text say something about the daily minimum being above 50 W/m2?

[kperrynrel]

Good catch at @kandersolar, updated docstring to the following in commit 0cadb6e:

3. "Abnormal" data periods, which are defined as less than the daily minimum of 50 OR any data greater than 1300

Looks like the docs rendering in the checks for some reason isn't updating, even though the underlying code has been updated. Not sure what's going on with that

[kandersolar]

Should it be "greater than" instead of "less than"?

I often have to manually refresh the RTD page to see updates after a rebuild. Something to do with browser caching. I'm guessing that's what you're experiencing too.

[kperrynrel]

oops, updated in 5a01c3a