Issue: Implement YAML Configuration for Dynamic Model and Ensemble Paths (Directories and Scripts)

[Polichinel]

Issue: Implement YAML Configuration for Dynamic Model and Ensemble Paths (Directories and Scripts)

Description
Implement four YAML configuration files to dynamically set up directories and scripts within the ModelPath and Ensemble classes.

The point is to have files that determine which model-specific dirs can be accessed as attributes by the ModelPath and EnsemblePath respectively. This ensures we don't have to change the script of the class if we decide to change something in the models-specific dirs.

This approach will also ensure that there is a clear separation between models and ensembles and between directories and scripts, streamlining configuration management and making the code more adaptable.

Note that the root level dirs are still expected to be had coded in the two path classes. This issue only pertains to the model-specific dirs and configs.

Details and Requirements

1. YAML Files:
   Add the following YAML configuration files to common_configs:

   • model_dirs.yaml: Contains directory paths specific to each model.
   • model_scripts.yaml: Contains script paths specific to each model.
   • ensemble_dirs.yaml: Contains directory paths for ensemble models.
   • ensemble_scripts.yaml: Contains script paths for ensemble models.

2. Dynamic Attribute Creation:

   • Use _initialize_directories() and _initialize_scripts() to parse these files and set attributes dynamically.
   • Class attributes generated from the YAMLs should use snake_case formatting.

Examples of something like this can be found here: https://github.com/prio-data/views_pipeline/tree/simon_stabs_at_path_stuff

3. Logging:
   • Use Python’s logger for basic error handling, ensuring any parsing issues with YAML files are captured and logged.
   • Include logs in _initialize_directories() and _initialize_scripts() for easy troubleshooting.

Example YAML Structure:

# model_dirs.yaml
model_name_1:
   output_dir: "/path/to/output"
   cache_dir: "/path/to/cache"
model_name_2:
   ...

Tasks

• Create model_dirs.yaml and model_scripts.yaml for model-specific paths in common_configs.
• Create ensemble_dirs.yaml and ensemble_scripts.yaml for ensemble-specific paths in common_configs.
• Implement dynamic attribute parsing in _initialize_directories() for directory paths.
• Implement dynamic attribute parsing in _initialize_scripts() for script paths.
• Add logging in _initialize_directories() and _initialize_scripts() for any YAML parsing errors.

Labels
enhancement, yaml, configuration, logging, error-handling