Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/MIDAS transformer #1820

Merged
merged 73 commits into from
Nov 18, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
e55bc8a
MIDASTransformer now outputs a low sample variant of the high sample …
Beerstabr Feb 27, 2023
1622a34
MIDASTransformer now outputs a low sample variant of the high sample …
Beerstabr Feb 27, 2023
268bdc2
extracted '_create_midas_df' from 'ts_transform'
Beerstabr Feb 27, 2023
dde5b20
extracted '_create_midas_df' from 'ts_transform'
Beerstabr Feb 27, 2023
1f8843f
Added some comments to helper functions
Beerstabr Feb 27, 2023
99fa5d2
changed some variable names
Beerstabr Feb 27, 2023
e4403e1
changed some variable names
Beerstabr Feb 27, 2023
a61c4d6
added warning if target frequency and input frequency don't match up …
Beerstabr Mar 8, 2023
24daea5
add _transform_iterator like the one in window_transformer
Beerstabr Mar 8, 2023
a7b86a2
docstring finished, including example
Beerstabr Mar 10, 2023
27a5999
added comments and description to 'ts_transform()'
Beerstabr Mar 10, 2023
800b98b
more robust up and downsampling in order to get for example 3 months …
Beerstabr Mar 17, 2023
e64cef2
tests are coming along, but still giving some errors
Beerstabr Mar 17, 2023
98cf5a7
tests work
Beerstabr Mar 17, 2023
998f4b6
small comment change
Beerstabr Mar 17, 2023
84ba2bf
Merge branch 'unit8co:master' into feature/add_midas_transformer
Beerstabr Mar 18, 2023
9f1d713
Merge branch 'unit8co:master' into feature/add_midas_transformer
Beerstabr Mar 24, 2023
01fc781
Merge branch 'unit8co:master' into feature/add_midas_transformer
Beerstabr Mar 31, 2023
d9918c9
adapted to 'params['fixed']['variable_name']' way of dealing with args
Beerstabr Mar 31, 2023
383ceb5
Merge branch 'master' into feature/add_midas_transformer
madtoinou Jun 7, 2023
9dd8522
feat: attemp to make midas invertible
madtoinou Jun 8, 2023
cdd1192
feat: multivariate ts are supported by the inverse_transform
madtoinou Jun 8, 2023
1bc278a
fix: updated changelog
madtoinou Jun 8, 2023
5ed1121
Merge branch 'master' into feat/add_midas_transformer
dennisbader Jun 10, 2023
0e8c7bc
feat: make the transformer fittable to improve inversability
madtoinou Jun 13, 2023
c6829e8
Merge branch 'feat/add_midas_transformer' of https://github.com/unit8…
madtoinou Jun 24, 2023
619b559
fix: when using anchored low freq, try to adjust the time index when …
madtoinou Jun 26, 2023
9f899e5
Merge branch 'master' into feat/add_midas_transformer
madtoinou Jun 29, 2023
548dc3b
Merge branch 'master' into feat/add_midas_transformer
madtoinou Jun 30, 2023
78b29d2
Merge branch 'master' into feat/add_midas_transformer
madtoinou Jul 4, 2023
053eb67
Merge branch 'master' into feat/add_midas_transformer
madtoinou Jul 4, 2023
5342e76
Merge branch 'master' into feat/add_midas_transformer
madtoinou Jul 5, 2023
7775efe
Merge branch 'master' into feat/add_midas_transformer
dennisbader Jul 10, 2023
12cf652
Merge branch 'master' into feat/add_midas_transformer
madtoinou Jul 13, 2023
dbfd3f4
fix: addressed reviewer comments
madtoinou Jul 14, 2023
f1b6f6c
fix: bug in TimeSeries constructor when the length of the pd.Series c…
madtoinou Jul 14, 2023
a2e95a2
fix: bug in static_covariates, component-specific static covariates w…
madtoinou Jul 14, 2023
020fa88
fix: new argument in pandas 2.0
madtoinou Jul 17, 2023
2beced0
fix: improved finite row detection
madtoinou Jul 17, 2023
1290d9f
fix: revert changes to timeseries.py
madtoinou Jul 17, 2023
6b055d6
fix: tests properly account for static covariates representation depe…
madtoinou Jul 17, 2023
c8ce8bd
test: added test for strip=True
madtoinou Jul 17, 2023
8a65800
Merge branch 'master' into feat/add_midas_transformer
dennisbader Jul 18, 2023
493e732
feat: adding how argument to TimeSeries.strip(), updated tests
madtoinou Jul 25, 2023
7bf95b1
fix: MIDAS properly leverage the TimeSeries.strip method
madtoinou Jul 25, 2023
afef4f8
fix: MIDAS properly leverage the TimeSeries.strip method
madtoinou Jul 25, 2023
cfdc197
Merge branch 'master' into feat/add_midas_transformer
madtoinou Jul 25, 2023
e5f4984
fix: fixed bug when series was sliced prior to inverse transform
madtoinou Jul 27, 2023
795ba01
Merge branch 'master' into feat/add_midas_transformer
madtoinou Jul 27, 2023
7bb0715
Merge branch 'master' into feat/add_midas_transformer
dennisbader Jul 31, 2023
8ed12d4
Merge branch 'master' into feat/add_midas_transformer
madtoinou Aug 2, 2023
bdbd787
Merge branch 'master' into feat/add_midas_transformer
madtoinou Aug 3, 2023
13affc6
Merge branch 'master' into feat/add_midas_transformer
madtoinou Nov 1, 2023
1a216c2
fix: for anchored low freq, the first value of the row correspond to …
madtoinou Nov 2, 2023
b5ac093
fix: splitting aligned/shifted unit-tests
madtoinou Nov 2, 2023
30d8b9a
Merge branch 'master' into feat/add_midas_transformer
madtoinou Nov 2, 2023
1d5b9be
Merge branch 'master' into feat/add_midas_transformer
madtoinou Nov 6, 2023
6c0f4a3
Merge branch 'master' into feat/add_midas_transformer
madtoinou Nov 6, 2023
761e1a3
Merge branch 'master' into feat/add_midas_transformer
madtoinou Nov 7, 2023
b84eb6d
Merge branch 'master' into feat/add_midas_transformer
dennisbader Nov 10, 2023
7a1e9c7
work with moving windows for midas transform
dennisbader Nov 11, 2023
5f0c2a7
Merge branch 'master' into feat/add_midas_transformer
dennisbader Nov 11, 2023
5655d45
Merge branch 'master' into feat/add_midas_transformer
dennisbader Nov 11, 2023
9cb6772
fix all tests
dennisbader Nov 11, 2023
d906e9d
update midas transform
dennisbader Nov 16, 2023
7329fd7
Merge branch 'master' into feat/add_midas_transformer
dennisbader Nov 16, 2023
2707638
fix transformation
dennisbader Nov 16, 2023
a697316
refactor transform
dennisbader Nov 17, 2023
ff0825d
update midas feature separator
dennisbader Nov 17, 2023
8326e60
remove old create midas df
dennisbader Nov 17, 2023
68cfa01
probabilistic support
dennisbader Nov 18, 2023
bb4a330
update changelog
dennisbader Nov 18, 2023
103b777
Merge branch 'master' into feat/add_midas_transformer
dennisbader Nov 18, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,11 @@ but cannot always guarantee backwards compatibility. Changes that may **break co
- `XGBModel` now leverages XGBoost's native Quantile Regression support that was released in version 2.0.0 for improved probabilistic forecasts. [#2051](https://github.com/unit8co/darts/pull/2051) by [Dennis Bader](https://github.com/dennisbader).
- Other improvements:
- Added support for time index time zone conversion with parameter `tz` before generating/computing holidays and datetime attributes. Support was added to all Time Axis Encoders, standalone encoders and forecasting models' `add_encoders`, time series generation utils functions `holidays_timeseries()` and `datetime_attribute_timeseries()`, and `TimeSeries` methods `add_datetime_attribute()` and `add_holidays()`. [#2054](https://github.com/unit8co/darts/pull/2054) by [Dennis Bader](https://github.com/dennisbader).
- Added new data transformer: `MIDAS`, which uses mixed-data sampling to convert `TimeSeries` from high frequency to low frequency (and back). [#1820](https://github.com/unit8co/darts/pull/1820) by [Boyd Biersteker](https://github.com/Beerstabr), [Antoine Madrona](https://github.com/madtoinou) and [Dennis Bader](https://github.com/dennisbader).
- Added optional keyword arguments dict `kwargs` to `ExponentialSmoothing` that will be passed to the constructor of the underlying `statsmodels.tsa.holtwinters.ExponentialSmoothing` model. [#2059](https://github.com/unit8co/darts/pull/2059) by [Antoine Madrona](https://github.com/madtoinou).
- Added new dataset `ElectricityConsumptionZurichDataset`: The dataset contains the electricity consumption of households in Zurich, Switzerland from 2015-2022 on different grid levels. We also added weather measurements for Zurich which can be used as covariates for modelling. [#2039](https://github.com/unit8co/darts/pull/2039) by [Antoine Madrona](https://github.com/madtoinou) and [Dennis Bader](https://github.com/dennisbader).
- Added new arguments `fit_kwargs` and `predict_kwargs` to `historical_forecasts()`, `backtest()` and `gridsearch()` that will be passed to the model's `fit()` and / or `predict` methods. E.g., you can now set a batch size, static validation series, ... depending on the model support. [#2050](https://github.com/unit8co/darts/pull/2050) by [Antoine Madrona](https://github.com/madtoinou)
- For transparency, we issue a (removable) warning when performing auto-regressive forecasts with past covariates (with `n >= output_chunk_length`) to inform users that future values of past covariates will be accessed. [#2049](https://github.com/unit8co/darts/pull/2049) by [Antoine Madrona](https://github.com/madtoinou)

**Fixed**
- Fixed a bug when calling optimized `historical_forecasts()` for a `RegressionModel` trained with unequal component-specific lags. [#2040](https://github.com/unit8co/darts/pull/2040) by [Antoine Madrona](https://github.com/madtoinou).
Expand Down
1 change: 1 addition & 0 deletions darts/dataprocessing/transformers/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
from .fittable_data_transformer import FittableDataTransformer
from .invertible_data_transformer import InvertibleDataTransformer
from .mappers import InvertibleMapper, Mapper
from .midas import MIDAS
from .missing_values_filler import MissingValuesFiller
from .reconciliation import (
BottomUpReconciliator,
Expand Down
6 changes: 3 additions & 3 deletions darts/dataprocessing/transformers/boxcox.py
Original file line number Diff line number Diff line change
Expand Up @@ -67,10 +67,10 @@ def __init__(
Specifies which method to use to find an optimal value for the lmbda parameter.
Either 'mle' or 'pearsonr'. Ignored if `lmbda` is not `None`.
global_fit
Optionally, whether all of the `TimeSeries` passed to the `fit()` method should be used to fit
Optionally, whether all `TimeSeries` passed to the `fit()` method should be used to fit
a *single* set of parameters, or if a different set of parameters should be independently fitted
to each provided `TimeSeries`. If `True`, then a `Sequence[TimeSeries]` is passed to `ts_fit`
and a single set of parameters is fitted using all of the provided `TimeSeries`. If `False`, then
and a single set of parameters is fitted using all provided `TimeSeries`. If `False`, then
each `TimeSeries` is individually passed to `ts_fit`, and a different set of fitted parameters
if yielded for each of these fitting operations. See `FittableDataTransformer` documentation for
further details.
Expand Down Expand Up @@ -136,7 +136,7 @@ def ts_fit(
params: Mapping[str, Any],
*args,
**kwargs
) -> Union[Sequence[float], pd.core.series.Series]:
) -> Union[Sequence[float], pd.Series]:
lmbda, method = params["fixed"]["_lmbda"], params["fixed"]["_optim_method"]
# If `global_fit` is `True`, then `series` will be ` Sequence[TimeSeries]`;
# otherwise, `series` is a single `TimeSeries`:
Expand Down
10 changes: 5 additions & 5 deletions darts/dataprocessing/transformers/fittable_data_transformer.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,10 +64,10 @@ def __init__(
be passed as a keyword argument, but won't automatically be applied to the input timeseries.
See `apply_component_mask` method of `BaseDataTransformer` for further details.
global_fit
Optionally, whether all of the `TimeSeries` passed to the `fit()` method should be used to fit
Optionally, whether all `TimeSeries` passed to the `fit()` method should be used to fit
a *single* set of parameters, or if a different set of parameters should be independently fitted
to each provided `TimeSeries`. If `True`, then a `Sequence[TimeSeries]` is passed to `ts_fit`
and a single set of parameters is fitted using all of the provided `TimeSeries`. If `False`, then
and a single set of parameters is fitted using all provided `TimeSeries`. If `False`, then
each `TimeSeries` is individually passed to `ts_fit`, and a different set of fitted parameters
if yielded for each of these fitting operations. See `ts_fit` for further details.

Expand All @@ -83,9 +83,9 @@ def __init__(
of fitted parameters.

Note that if an invertible *and* fittable data transformer is to be globally fitted, the data transformer
class should first inherit from `FittableDataTransformer` and then from `InveritibleDataTransformer`. In
other words, `MyTransformer(FittableDataTransformer, InveritibleDataTransformer)` is correct, but
`MyTransformer(InveritibleDataTransformer, FittableDataTransformer)` is **not**. If this is not implemented
class should first inherit from `FittableDataTransformer` and then from `InvertibleDataTransformer`. In
other words, `MyTransformer(FittableDataTransformer, InvertibleDataTransformer)` is correct, but
`MyTransformer(InvertibleDataTransformer, FittableDataTransformer)` is **not**. If this is not implemented
correctly, then the `global_fit` parameter will not be correctly passed to `FittableDataTransformer`'s
constructor.

Expand Down
Loading
Loading