ENH: Switch to version 1.0 of config file format, fix #685 #345 #748 #750

NickleDave · 2024-05-05T03:22:38Z

This pull request does the following:

switches to using tomlkit to read/write toml, as in DEV/ENH: Switch to using tomlkit #685
make a new version of the config file format (version 1.0) that better leverages TOML, as in change config file format to leverage toml tables #345
as part of better leveraging TOML format, adds a dataset sub-table to top-level eval/learncurve/predict/train tables, as in ENH: Add dataset sub-table to config file, remove other dataset/transform param keys #748

…names

…/validators.py

…is not a top-level table and will be an attribute of prep instead

…onfig_from_toml_dict to look in specific section

…classmethods from_config_dict

….py module

…set_config parameters, removing parameters val/train_transform_params + val/train_dataset_params and dataset_path

…set_config parameters, removing parameters transform_params + dataset_params and dataset_path

…onfig parameters, removing parameters transform_params + dataset_params and dataset_path

…meters, removing parameters transform_params + dataset_params and dataset_path

…dule.eval, remove dataset_path/transform_params/datset_params arguments

…n.py, instead of directly getting window_size from the params dict

…d dataset_config parameters, removing parameters transform_params + dataset_params and dataset_path

…config parameters, removing parameters transform_params + dataset_params and dataset_path

…earning_curve.learncurve, remove dataset_path/transform_params/datset_params arguments

…ataset_config parameters, removing parameters transform_params + dataset_params and dataset_path

…t_config parameters, removing parameters transform_params + dataset_params and dataset_path

… parameters, removing parameters transform_params + dataset_params and dataset_path

…el_name in src/vak/learncurve/learncurve.py

…el_name in src/vak/learncurve/frame_classification.py

…ameters, removing parameters transform_params + dataset_params and dataset_path

…sification.py

…led correctly

…est cases we parametrize with in tests/test_config/

…asets

…set_config parameters, removing parameters transform_params + dataset_params and dataset_path

…els/test_base.py

… modules

NickleDave added 30 commits May 1, 2024 19:23

WIP: Add src/vak/config/dataset.py

476b9da

Add module-level docstring + type annotations in src/vak/config/parse.py

a625bb7

WIP: Fix how cli.prep adds dataset path to toml config file

272f67d

Change table names in src/vak/config/valid.toml

fa35def

Rename section -> table in config/parse.py

988c150

In cli/prep change 'section' -> 'table' and lowercase table names

72e11e8

In config/config.py, change 'section' -> 'table' and lowercase table …

7181f71

…names

Change '[PREP]' -> '[vak.prep]' in config/prep.py

9d4467c

WIP: Change table names in config files in tests/data_for_tests/configs

b5a6413

Make tomlkit a dependency in pyproject.toml, drop toml

27f0596

Change config/parse.py to use tomlkit

62fe4f4

Update example configs in doc/toml/

3253a4e

Add link to example config files in docs, in error messages in config…

fb70733

…/validators.py

Remove 'spect_params' from REQUIRED_OPTIONS in config/parse.py, this …

b653138

…is not a top-level table and will be an attribute of prep instead

Rename 'config_toml' -> 'config_dict' in config/parse.py

83eddf0

Fix function _validate_tables_arg_convert_list in config/parse.py

1160a6b

Fix error message formatting in src/vak/config/validators.py

9796624

Add ModelConfig class to config/model.py, add type annotations, fix c…

60f7b02

…onfig_from_toml_dict to look in specific section

Fixup fixing config_from_toml_dict to look in specific section

910c5df

Rewrite config/eval.py with 'modern' attrs

32995bf

Fixup rewrite config/eval with 'modern attrs

f775749

Rewrite config/learncurve.py with 'modern' attrs

060abaa

Rewrite config/predict.py with 'modern' attrs

71e89bc

Rewrite config/prep.py with 'modern' attrs

f6cf945

Rewrite config/train.py with 'modern' attrs

9798707

Rename Dataset -> DatasetConfig in config/dataset.py

9c0d301

Add are_table_options_valid to config/validators.py, will be used by …

0de12da

…classmethods from_config_dict

WIP: Add from_config_dict classmethod to EvalConfig

2c272bc

WIP: Add tests/test_config/test_dataset.py

53447f5

Make fixes to ModelConfig class, fix circular imports in config/model…

a6cb674

….py module

NickleDave added 25 commits May 4, 2024 20:14

Rewrite src/vak/train/parametric_umap.py to use model_config and data…

bebaa3f

…set_config parameters, removing parameters val/train_transform_params + val/train_dataset_params and dataset_path

Rewrite vak/eval/frame_classification.py to use model_config and data…

8f6ef37

…set_config parameters, removing parameters transform_params + dataset_params and dataset_path

Rewrite vak/eval/parametric_umap.py to use model_config and dataset_c…

9768b5d

…onfig parameters, removing parameters transform_params + dataset_params and dataset_path

Rewrite vak/eval/eval_.py to use model_config and dataset_config para…

519f7d4

…meters, removing parameters transform_params + dataset_params and dataset_path

Rewrite cli.eval to pass model_config and dataset_config into eval_mo…

e68ad49

…dule.eval, remove dataset_path/transform_params/datset_params arguments

Unpack dataset_config[params] with ** inside trak/frame_classificatio…

dca625f

…n.py, instead of directly getting window_size from the params dict

Rewrite vak/learncurve/frame_classification.py to use model_config an…

43d1a99

…d dataset_config parameters, removing parameters transform_params + dataset_params and dataset_path

Rewrite vak/learncurve/learncurve.py to use model_config and dataset_…

af971a0

…config parameters, removing parameters transform_params + dataset_params and dataset_path

Rewrite cli.learncurve to pass model_config and dataset_config into l…

2257144

…earning_curve.learncurve, remove dataset_path/transform_params/datset_params arguments

Rewrite vak/predict/frame_classification.py to use model_config and d…

cd60eab

…ataset_config parameters, removing parameters transform_params + dataset_params and dataset_path

Rewrite vak/predict/parametric_umap.py to use model_config and datase…

571a9c6

…t_config parameters, removing parameters transform_params + dataset_params and dataset_path

Rewrite vak/predict/predict.py to use model_config and dataset_config…

aba3353

… parameters, removing parameters transform_params + dataset_params and dataset_path

Fix dataset_path -> dataset_config[path] and add missing variable mod…

3e237b4

…el_name in src/vak/learncurve/learncurve.py

Fix dataset_path -> dataset_config[path] and add missing variable mod…

9eb7c0c

…el_name in src/vak/learncurve/frame_classification.py

Rewrite vak/cli/predict.py to use model_config and dataset_config par…

32d9604

…ameters, removing parameters transform_params + dataset_params and dataset_path

Remove non-existent dataset_params variable in vak/predict/frame_clas…

87e1a79

…sification.py

Fix unit tests for DatasetConfig to test 'params' attribute gets hand…

bfb4a9b

…led correctly

Remove train/val_dataset_params and train/val_transform_params from t…

d70886a

…est cases we parametrize with in tests/test_config/

Use DatasetConfig.params attribute where we need to in tests/test_dat…

401f1a3

…asets

Fix method name ModelConfig.to_dict -> asdict in tests/

0ae33a8

In tests for eval/learncurve/predict/train, use model_config and data…

e033b25

…set_config parameters, removing parameters transform_params + dataset_params and dataset_path

Fix use of default transform and dataset.params attribute in test_mod…

9d7ec3e

…els/test_base.py

Fix config snippets in docs

33f578b

Apply linting to src/

84c316b

Raise 'from e' with errors in eval/predict/train/frame_classification…

bc0b487

… modules

NickleDave merged commit df876e3 into main May 5, 2024
0 of 4 checks passed

This was referenced May 5, 2024

change config file format to leverage toml tables #345

Closed

ENH: Add dataset sub-table to config file, remove other dataset/transform param keys #748

Closed

NickleDave added a commit that referenced this pull request May 5, 2024

DOC: Update CHANGELOG after merging #750 [skip ci]

1e6abe6

NickleDave deleted the update-config-add-dataset-table branch August 24, 2024 12:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Switch to version 1.0 of config file format, fix #685 #345 #748 #750

ENH: Switch to version 1.0 of config file format, fix #685 #345 #748 #750

NickleDave commented May 5, 2024

ENH: Switch to version 1.0 of config file format, fix #685 #345 #748 #750

ENH: Switch to version 1.0 of config file format, fix #685 #345 #748 #750

Conversation

NickleDave commented May 5, 2024