How to evaluate models on my own datasets (hyper-params)

[Caelyn-gao]

Hi, thanks for the great work. It is very useful to help us understand the area of time series forecasting.

If I want to use your pipeline to evaluate models on my own datasets, may I ask how to do it? The important thing is the hyper-params. I noticed that per model, you have given the hyper-params in the scripts you offered. I guess these hyper-params are selected according to hyper-parameter searches or settings in the original papers. If I want to use my own datasets, in the TFB pipeline, is there any module that I can get the best hyper-parameters when training on my own dataset?

Thanks a lot in advance! Have a good day!

[qiu69]

Hello, to evaluate models on my own datasets, you can follow the steps below：
1、Process your dataset into three columns in TFB format. You can refer to the dataset in TFB.
2、Put the processed dataset under: dataset/forecasting folder.
3、Run for example：Please replace "--data-name-list" with your own dataset name；
python ./scripts/run_benchmark.py --config-path "rolling_forecast_config.json" --data-name-list "ILI.csv" --strategy-args '{"horizon":24}' --model-name "time_series_library.DLinear" --model-hyper-params '{"batch_size": 16, "d_ff": 512, "d_model": 256, "lr": 0.01, "horizon": 24, "seq_len": 104}' --adapter "transformer_adapter" --gpus 0 --num-workers 1 --timeout 60000 --save-path "ILI/DLinear".

Parameter search requires trying multiple parameter combinations on your own, and TFB does not provide code for parameter search.

I hope it can help you. If you have any questions, you can send me an email，[email protected].

[Caelyn-gao]

Thanks for the response. It is great!

[qiu69]

We provide tutorial about how to evaluate your own time series, please see https://github.com/decisionintelligence/TFB/blob/master/docs/tutorials/steps_to_evaluate_your_own_time_series.md