Anomalies in the passiv data

[peterdudfield]

Plot of max valye per day

suggestions:

• capacity could defined 99% of data, remove anaything above
• All could remove 5stds away from the mean

Note any removal of data will cause gaps in the data

[jacobbieker]

What about taking the mean value of the points around the anomaly and replacing it with that? Would at least keep the gaps filled

[peterdudfield]

What about taking the mean value of the points around the anomaly and replacing it with that? Would at least keep the gaps filled

yea, or just linear interpolate. Depends how big the anomly is. If its > 12 hours, then probably just remove the data. If its one point, i agree, can just be filled in. But data pipes would be able to handle the gap in data.

Whenever I get a chance, Ill try to point out the advantages to ocf datapipes

[peterdudfield]

@simlmx reference here

[jacobbieker]

Yeah, currently for training if there is a gap, it'll mean the valid time period selection will say there is an issue, but yeah, I agree if its for awhile, then just remove it all.

[simlmx]

Making sure to use absolute loss |y - pred| instead of squared loss (y-pred)**2 when training my model fixed all the issues with outliers. When using absolute loss, removing the problematic values didn't help in a significant way.

[dfulu]

This might be partially solved now. Using a percentile for the estimated capacity was added in #225. the mentioned filtering isn't included yet