Feat/time net model #2538

gdevos010 · 2024-09-23T03:34:01Z

Checklist before merging this PR:

Mentioned all issues that this PR fixes or addresses.
Summarized the updates of this PR under Summary.
Added an entry under Unreleased in the Changelog.

TODO

update doc strings
add tests
add proper attribution
add futures covariate support (this will probably need to be seperate MR)

Fixes #2537.

Summary

Adds the TimesNet mode based on this code
https://github.com/thuml/Time-Series-Library/blob/main/models/TimesNet.py

Other Information

codecov · 2024-09-25T19:39:55Z

Codecov Report

Attention: Patch coverage is 95.19651% with 11 lines in your changes missing coverage. Please review.

Project coverage is 94.11%. Comparing base (d909589) to head (c985d54).

Files with missing lines	Patch %	Lines
darts/models/forecasting/times_net_model.py	93.20%	7 Missing ⚠️
darts/models/components/embed.py	97.56%	3 Missing ⚠️
darts/models/__init__.py	66.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2538      +/-   ##
==========================================
- Coverage   94.14%   94.11%   -0.04%     
==========================================
  Files         139      141       +2     
  Lines       14884    15113     +229     
==========================================
+ Hits        14013    14223     +210     
- Misses        871      890      +19

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

gdevos010 · 2024-09-29T04:47:36Z

@dennisbader This model is ready for review.

madtoinou

Thanks a lot for this great PR @gdevos010 and sorry for the delay, we have been quite busy with others responsibilities.

Some comments to reduce the code redundancy in the implementation of this quite complex model.

Also, could you please also include an example notebook to compare it to models such as TiDEModel and/or TSMixerModel on a toy example, just to make sure that the default configuration yield acceptable results?

madtoinou · 2024-11-12T17:19:50Z

darts/models/components/embed.py

+from darts.utils.torch import MonteCarloDropout
+
+
+class PositionalEmbedding(nn.Module):


the PositionalEncoding class already implements this logic in darts/models/forecasting/transformer_model.py, let's maybe move it here and keep the old name

madtoinou · 2024-11-12T17:22:36Z

darts/models/components/embed.py

+        w = torch.zeros(c_in, d_model).float()
+        w.require_grad = False
+
+        position = torch.arange(0, c_in).float().unsqueeze(1)
+        div_term = (
+            torch.arange(0, d_model, 2).float() * -(math.log(10000.0) / d_model)
+        ).exp()
+
+        w[:, 0::2] = torch.sin(position * div_term)
+        w[:, 1::2] = torch.cos(position * div_term)


this code snippet can also be found in PositionalEncoding, let's abstract it to reduce redundancy

madtoinou · 2024-11-12T17:24:02Z

darts/models/components/embed.py

+        return self.dropout(x)
+
+
+class DataEmbedding_inverted(nn.Module):


Suggested change

class DataEmbedding_inverted(nn.Module):

class DataEmbeddingInverted(nn.Module):

madtoinou · 2024-11-12T17:27:58Z

darts/models/components/embed.py

+        return self.dropout(x)
+
+
+class DataEmbedding_wo_pos(nn.Module):


The logic is so similar to DataEmbedding and DataEmbedding_inverted that they should probably be combined into a single class and the logic difference should be implemented in the forward() method instead (by adding a parameter/attribute type that could take the values "normal", "inverted" or "wopos".

madtoinou · 2024-11-12T17:29:26Z

darts/models/forecasting/times_net_model.py

+from darts.models.forecasting.torch_forecasting_model import PastCovariatesTorchModel
+
+
+class Inception_Block_V1(nn.Module):


Just of the sake of homogeneity.

Suggested change

class Inception_Block_V1(nn.Module):

class InceptionBlock(nn.Module):

madtoinou · 2024-11-12T17:32:15Z

darts/models/forecasting/transformer_model.py

@@ -480,7 +480,7 @@ def encode_year(idx):
                }
            ..
        random_state
-            Control the randomness of the weights initialization. Check this
+            Control the randomness of the weight's initialization. Check this


Let's revert this change, since there are several weights, the apostrophe does not seem appropriate.

Or we could eventually rephrase it into "Control the randomness in the initialization of the weights" if you have the impression that the original sentence was not clear.

madtoinou · 2024-11-12T17:35:41Z

darts/models/forecasting/times_net_model.py

Let's rename the file "timesnet_model.py" so that it's homogeneous with the others

madtoinou · 2024-11-12T17:36:16Z

darts/tests/models/forecasting/test_times_net_model.py

Let's rename the file "test_timesnet_model.py" to keep the number of underscores to the minimum

madtoinou · 2024-11-12T17:44:26Z

CHANGELOG.md

@@ -619,13 +620,13 @@ Patch release
 - Added support for past and future covariates to `residuals()` function. [#1223](https://github.com/unit8co/darts/pull/1223) by [Eliane Maalouf](https://github.com/eliane-maalouf).
 - Added support for retraining model(s) every `n` iteration and on custom conditions in `historical_forecasts` method of `ForecastingModel`s. [#1139](https://github.com/unit8co/darts/pull/1139) by [Francesco Bruzzesi](https://github.com/fbruzzesi).
 - Added support for beta-NLL in `GaussianLikelihood`s, as proposed in [this paper](https://arxiv.org/abs/2203.09168). [#1162](https://github.com/unit8co/darts/pull/1162) by [Julien Herzen](https://github.com/hrzn).
- New LayerNorm alternatives, RMSNorm and LayerNormNoBias [#1113](https://github.com/unit8co/darts/pull/1113) by [Greg DeVos](https://github.com/gdevos010).
+- New LayerNorm alternatives, RMSNorm and LayerNormNoBias [#1113](https://github.com/unit8co/darts/issues/1113) by [Greg DeVosNouri](https://github.com/gdevos010).


Incorrect link

gdevos010 added 2 commits September 22, 2024 17:20

timeNetModel

3cf2e54

change name

9bbd1c9

gdevos010 requested review from dennisbader and madtoinou as code owners September 23, 2024 03:34

gdevos010 marked this pull request as draft September 23, 2024 03:45

add TODO

3edce3a

gdevos010 force-pushed the feat/time_net_model branch from d60d5e4 to 3edce3a Compare September 23, 2024 05:27

gdevos010 and others added 3 commits September 22, 2024 22:28

Merge remote-tracking branch 'unit8co/master' into feat/time_net_model

34b55b9

Merge branch 'unit8co:master' into feat/time_net_model

d2f6d68

tests

5d57e23

gdevos010 marked this pull request as ready for review September 25, 2024 18:35

Greg DeVosNouri and others added 5 commits September 26, 2024 14:34

test_embed

c4c2d6f

lint

409c56f

FFT_for_period

63b1358

FFT_for_period

d9a0299

Merge branch 'master' into feat/time_net_model

3e5b913

madtoinou requested changes Nov 12, 2024

View reviewed changes

Merge branch 'master' into feat/time_net_model

c985d54

madtoinou reviewed Nov 12, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/time net model #2538

Feat/time net model #2538

gdevos010 commented Sep 23, 2024 •

edited

Loading

codecov bot commented Sep 25, 2024 •

edited

Loading

gdevos010 commented Sep 29, 2024 •

edited

Loading

madtoinou left a comment

madtoinou Nov 12, 2024

madtoinou Nov 12, 2024

madtoinou Nov 12, 2024

madtoinou Nov 12, 2024

madtoinou Nov 12, 2024

madtoinou Nov 12, 2024

madtoinou Nov 12, 2024

madtoinou Nov 12, 2024

madtoinou Nov 12, 2024

		from darts.utils.torch import MonteCarloDropout


		class PositionalEmbedding(nn.Module):

		return self.dropout(x)


		class DataEmbedding_inverted(nn.Module):

	class DataEmbedding_inverted(nn.Module):
	class DataEmbeddingInverted(nn.Module):

		return self.dropout(x)


		class DataEmbedding_wo_pos(nn.Module):

		from darts.models.forecasting.torch_forecasting_model import PastCovariatesTorchModel


		class Inception_Block_V1(nn.Module):

	class Inception_Block_V1(nn.Module):
	class InceptionBlock(nn.Module):

Feat/time net model #2538

Are you sure you want to change the base?

Feat/time net model #2538

Conversation

gdevos010 commented Sep 23, 2024 • edited Loading

Summary

Other Information

codecov bot commented Sep 25, 2024 • edited Loading

Codecov Report

gdevos010 commented Sep 29, 2024 • edited Loading

madtoinou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gdevos010 commented Sep 23, 2024 •

edited

Loading

codecov bot commented Sep 25, 2024 •

edited

Loading

gdevos010 commented Sep 29, 2024 •

edited

Loading