-
Notifications
You must be signed in to change notification settings - Fork 0
File based Parameter Values
Home > Common Topics > File-based Parameter Values
This topic describes how parameter values can be be represented in files in OpenM++. File-based parameter values are used by several OpenM++ components including build, run, UI, scripting, import, export.
- Modgen-specific: CsvToDat utility: Command-line utility to convert CSV parameters to DAT format
In several contexts, the value(s) of a parameter can be in a file whose name is the name of the parameter followed by a suffix which indicates content and format.
For example, if a file named AgeBaselinePreg1.csv
is present in the folder `RiskPaths/parameters/Default', it will be used by the OpenM++ compiler to provide values for that parameter when publishing the Default scenario when the model is built.
The suffix of the file indicates how the contents of the file are interpreted, as follows:
File suffix | Meaning |
---|---|
.csv | Comma-separated values (CSV) with header and dimension values, one value per line |
.tsv | Tab-separated values (TSV) with header and dimension values, one value per line |
.id.csv | CSV with header and dimension values as 0-based ID's |
.id.tsv | TSV with header and dimension values as 0-based ID's |
.value.csv | CSV with values only (no header or dimension values) |
.value.tsv | TSV with values with (no header or dimension values) |
The header line contains the column names of indices as specified in the parameter declaration in model code. The header line can optionally start with an initial column named sub_id
. If so, the first value in each line is the 0-based index of the replicate (aka sub, subsample, member).
You can use CSV or TSV files to supply input parameter values for your model. For example, if the RiskPaths model has the SeparationDurationBaseline
parameter:
partition DISSOLUTION_DURATION //EN Duration since union dissolution
{
2, 6, 10, 15
};
..........
// Separation Duration Baseline of 2nd Formation
double SeparationDurationBaseline[DISSOLUTION_DURATION] = {
0.1995702, 0.1353028, 0.1099149, 0.0261186, 0.0456905
};
then you can supply parameter values from RiskPaths/parameters/RiskPths.dat
or from RiskPaths/parameters/SeparationDurationBaseline.csv
:
dim0, param_value
"(-∞,2)", 0.1995702
"[2,6)", 0.1353028
"[6,10)", 0.1099149
"[10,15)", 0.0261186
"[15,∞)", 0.0456905
If parameter values are coming from CSV or TSV file then you can use Markdown file(s) to provide parameter value notes.
For example, RiskPaths/parameters/SeparationDurationBaseline.EN.md
:
This is a parameter values for Separation Duration Baseline of 2nd Formation.
It is a test sample model and data may not be accurate.
and RiskPaths/parameters/SeparationDurationBaseline.FR.md
:
**Translation below created by Google, please provide a proper French translation before publishing the model**
Il s'agit d'une valeur de paramètre pour la ligne de base de la durée de séparation de la 2e formation.
Il s'agit d'un modèle d'échantillon de test et les données peuvent ne pas être exactes.
The following rules are applied to parameter CSV or TSV files:
- file name must be the same as parameter name, which is
SeparationDurationBaseline
in example above - it can be parameter.csv with comma-separated values or parameter.tsv with tab-separated values
- if dimension item name or parameter value contains comma then it must be "quoted"
- header line of the file must contain dimension names and "param_value", for example:
dim0,otherDim,dim3,param_value
- (optional) if parameter has multiple sub-values then header line must start with "sub_id" (see example below)
- if the scenario directory contains a .dat file with values for ParameterName and also a file named ParameterName.csv the CSV values will override the .dat values.
The following formats of CSV or TSV files are supported:
- CSV or TSV file with dimension(s) and parameter value(s)
- CSV or TSV file with dimension(s) and multiple parameter sub-values
- CSV or TSV files with IDs as dimension(s) items
- CSV or TSV file with parameter values only, without dimensions (omc only)
Subsequent sub-topics describe each in turn, with examples.
Dimension items must be supplied as item "names", similar to partition DISSOLUTION_DURATION
above. Or take a look into another example of two-dimensional parameter:
classification SEX //EN Sex
{
FEMALE,
MALE
};
range YEAR //EN Year
{
2019,
2021
};
............
double GenderByYearRatio[SEX][YEAR] = {
0.1, (2)0.2, 0.5, 0.6, 0.7
};
GenderByYearRatio.csv file:
dim0, dim1, param_value
FEMALE, 2019, 0.1
FEMALE, 2020, 0.2
FEMALE, 2021, 0.2
MALE, 2019, 0.5
MALE, 2020, 0.6
MALE, 2021, 0.7
- Line order in the file is not important and openM++ will sort it automatically.
- If there some line(s) are missing then parameter value will be a default for that parameter type, for example: 0.0.
It can be used for uncertainty probabilistic analysis, file example:
sub_id, dim0, dim1, param_value
0, FEMALE, 2019, 0.1
0, FEMALE, 2020, 0.2
0, FEMALE, 2021, 0.2
0, MALE, 2019, 0.5
0, MALE, 2020, 0.6
0, MALE, 2021, 0.7
1, FEMALE, 2019, 1.1
1, FEMALE, 2020, 1.2
1, FEMALE, 2021, 1.2
1, MALE, 2019, 1.5
1, MALE, 2020, 1.6
1, MALE, 2021, 1.7
...........
- Header line must start with "sub_id" in order to indicate presnse of sub values in parameter.csv file.
- Sub value id's can be any integer values, for example: -1, 0, 4, 8, 12, 42. It must be integer but does not need to be positive or sequential.
- The first sub value id in a CSV file is considered to be the "default" sub value for that parameter.
You can use dimension item ID instead of item names, for example SeparationDurationBaseline.id.csv:
dim0, param_value
0, 0.1995702
1, 0.1353028
2, 0.1099149
3, 0.0261186
4, 0.0456905
GenderByYearRatio.id.csv:
dim0, dim1, param_value
0, 0, 0.1
0, 1, 0.2
0, 2, 0.2
1, 0, 0.5
1, 1, 0.6
1, 2, 0.7
Please notice file naming convention: ParameterName.id.csv
or ParameterName.id.tsv
This format is only supported for parameters read at build time by the OpenM++ compiler (omc).
In this format the file contains only parameter value(s) without dimensions, for example SeparationDurationBaseline.value.csv
:
0.1995702, 0.1353028, 0.1099149, 0.0261186, 0.0456905
GenderByYearRatio.value.csv
:
0.1, 0.2, 0.2,
0.5,
0.6, 0.7
- Please note the file naming convention:
ParameterName.value.csv
orParameterName.value.tsv
- Values in the file must be ordered according to the dimension order in the parameter declaration, ie the same order as in a .dat file.
- Any number of values may be given on each line.
- Each pair of values must be separated by a comma, including the last value on a line and the first value on the subsequent line.
- Windows: Quick Start for Model Users
- Windows: Quick Start for Model Developers
- Linux: Quick Start for Model Users
- Linux: Quick Start for Model Developers
- MacOS: Quick Start for Model Users
- MacOS: Quick Start for Model Developers
- Model Run: How to Run the Model
- MIT License, Copyright and Contribution
- Model Code: Programming a model
- Windows: Create and Debug Models
- Linux: Create and Debug Models
- MacOS: Create and Debug Models
- MacOS: Create and Debug Models using Xcode
- Modgen: Convert case-based model to openM++
- Modgen: Convert time-based model to openM++
- Modgen: Convert Modgen models and usage of C++ in openM++ code
- Model Localization: Translation of model messages
- How To: Set Model Parameters and Get Results
- Model Run: How model finds input parameters
- Model Output Expressions
- Model Run Options and ini-file
- OpenM++ Compiler (omc) Run Options
- OpenM++ ini-file format
- UI: How to start user interface
- UI: openM++ user interface
- UI: Create new or edit scenario
- UI: Upload input scenario or parameters
- UI: Run the Model
- UI: Use ini-files or CSV parameter files
- UI: Compare model run results
- UI: Aggregate and Compare Microdata
- UI: Filter run results by value
- UI: Disk space usage and cleanup
- UI Localization: Translation of openM++
- Authored Model Documentation
- Built-in Attributes
- Censor Event Time
- Create Import Set
- Derived Tables
- Entity Attributes in C++
- Entity Function Hooks
- Entity Member Packing
- Entity Tables
- Enumerations
- Events
- Event Trace
- External Names
- Generated Model Documentation
- Groups
- Illustrative Model
Align1
- Lifecycle Attributes
- Local Random Streams
- Memory Use
- Microdata Output
- Model Code
- Model Documentation
- Model Languages
- Model Localization
- Model Metrics Report
- Model Resource Use
- Model Symbols
- Parameter and Table Display and Content
- Population Size and Scaling
- Screened Tables
- Symbol Labels and Notes
- Tables
- Test Models
- Time-like and Event-like Attributes
- Use Modules
- Weighted Tabulation
- File-based Parameter Values
- Oms: openM++ web-service
- Oms: openM++ web-service API
- Oms: How to prepare model input parameters
- Oms: Cloud and model runs queue
- Use R to save output table into CSV file
- Use R to save output table into Excel
- Run model from R: simple loop in cloud
- Run RiskPaths model from R: advanced run in cloud
- Run RiskPaths model in cloud from local PC
- Run model from R and save results in CSV file
- Run model from R: simple loop over model parameter
- Run RiskPaths model from R: advanced parameters scaling
- Run model from Python: simple loop over model parameter
- Run RiskPaths model from Python: advanced parameters scaling
- Windows: Use Docker to get latest version of OpenM++
- Linux: Use Docker to get latest version of OpenM++
- RedHat 8: Use Docker to get latest version of OpenM++
- Quick Start for OpenM++ Developers
- Setup Development Environment
- 2018, June: OpenM++ HPC cluster: Test Lab
- Development Notes: Defines, UTF-8, Databases, etc.
- 2012, December: OpenM++ Design
- 2012, December: OpenM++ Model Architecture, December 2012
- 2012, December: Roadmap, Phase 1
- 2013, May: Prototype version
- 2013, September: Alpha version
- 2014, March: Project Status, Phase 1 completed
- 2016, December: Task List
- 2017, January: Design Notes. Subsample As Parameter problem. Completed
GET Model Metadata
- GET model list
- GET model list including text (description and notes)
- GET model definition metadata
- GET model metadata including text (description and notes)
- GET model metadata including text in all languages
GET Model Extras
GET Model Run results metadata
- GET list of model runs
- GET list of model runs including text (description and notes)
- GET status of model run
- GET status of model run list
- GET status of first model run
- GET status of last model run
- GET status of last completed model run
- GET model run metadata and status
- GET model run including text (description and notes)
- GET model run including text in all languages
GET Model Workset metadata: set of input parameters
- GET list of model worksets
- GET list of model worksets including text (description and notes)
- GET workset status
- GET model default workset status
- GET workset including text (description and notes)
- GET workset including text in all languages
Read Parameters, Output Tables or Microdata values
- Read parameter values from workset
- Read parameter values from workset (enum id's)
- Read parameter values from model run
- Read parameter values from model run (enum id's)
- Read output table values from model run
- Read output table values from model run (enum id's)
- Read output table calculated values from model run
- Read output table calculated values from model run (enum id's)
- Read output table values and compare model runs
- Read output table values and compare model runs (enun id's)
- Read microdata values from model run
- Read microdata values from model run (enum id's)
- Read aggregated microdata from model run
- Read aggregated microdata from model run (enum id's)
- Read microdata run comparison
- Read microdata run comparison (enum id's)
GET Parameters, Output Tables or Microdata values
- GET parameter values from workset
- GET parameter values from model run
- GET output table expression(s) from model run
- GET output table calculated expression(s) from model run
- GET output table values and compare model runs
- GET output table accumulator(s) from model run
- GET output table all accumulators from model run
- GET microdata values from model run
- GET aggregated microdata from model run
- GET microdata run comparison
GET Parameters, Output Tables or Microdata as CSV
- GET csv parameter values from workset
- GET csv parameter values from workset (enum id's)
- GET csv parameter values from model run
- GET csv parameter values from model run (enum id's)
- GET csv output table expressions from model run
- GET csv output table expressions from model run (enum id's)
- GET csv output table accumulators from model run
- GET csv output table accumulators from model run (enum id's)
- GET csv output table all accumulators from model run
- GET csv output table all accumulators from model run (enum id's)
- GET csv calculated table expressions from model run
- GET csv calculated table expressions from model run (enum id's)
- GET csv model runs comparison table expressions
- GET csv model runs comparison table expressions (enum id's)
- GET csv microdata values from model run
- GET csv microdata values from model run (enum id's)
- GET csv aggregated microdata from model run
- GET csv aggregated microdata from model run (enum id's)
- GET csv microdata run comparison
- GET csv microdata run comparison (enum id's)
GET Modeling Task metadata and task run history
- GET list of modeling tasks
- GET list of modeling tasks including text (description and notes)
- GET modeling task input worksets
- GET modeling task run history
- GET status of modeling task run
- GET status of modeling task run list
- GET status of modeling task first run
- GET status of modeling task last run
- GET status of modeling task last completed run
- GET modeling task including text (description and notes)
- GET modeling task text in all languages
Update Model Profile: set of key-value options
- PATCH create or replace profile
- DELETE profile
- POST create or replace profile option
- DELETE profile option
Update Model Workset: set of input parameters
- POST update workset read-only status
- PUT create new workset
- PUT create or replace workset
- PATCH create or merge workset
- DELETE workset
- POST delete multiple worksets
- DELETE parameter from workset
- PATCH update workset parameter values
- PATCH update workset parameter values (enum id's)
- PATCH update workset parameter(s) value notes
- PUT copy parameter from model run into workset
- PATCH merge parameter from model run into workset
- PUT copy parameter from workset to another
- PATCH merge parameter from workset to another
Update Model Runs
- PATCH update model run text (description and notes)
- DELETE model run
- POST delete model runs
- PATCH update run parameter(s) value notes
Update Modeling Tasks
Run Models: run models and monitor progress
Download model, model run results or input parameters
- GET download log file
- GET model download log files
- GET all download log files
- GET download files tree
- POST initiate entire model download
- POST initiate model run download
- POST initiate model workset download
- DELETE download files
- DELETE all download files
Upload model runs or worksets (input scenarios)
- GET upload log file
- GET all upload log files for the model
- GET all upload log files
- GET upload files tree
- POST initiate model run upload
- POST initiate workset upload
- DELETE upload files
- DELETE all upload files
Download and upload user files
- GET user files tree
- POST upload to user files
- PUT create user files folder
- DELETE file or folder from user files
- DELETE all user files
User: manage user settings
Model run jobs and service state
- GET service configuration
- GET job service state
- GET disk usage state
- POST refresh disk space usage info
- GET state of active model run job
- GET state of model run job from queue
- GET state of model run job from history
- PUT model run job into other queue position
- DELETE state of model run job from history
Administrative: manage web-service state
- POST a request to refresh models catalog
- POST a request to close models catalog
- POST a request to close model database
- POST a request to open database file
- POST a request to cleanup database file
- GET the list of database cleanup log(s)
- GET database cleanup log file(s)
- POST a request to pause model run queue
- POST a request to pause all model runs queue
- PUT a request to shutdown web-service