Tests for all combinations of settings

[svdhoog]

Test data sets

See: circles model.

Config.yaml settings

• Types of analysis (4)
  agent, multiple_run, multiple_batch, multiple_set

• Types of plots (4)
  timeseries, boxplot, histogram, scatterplot

• Types of statistical summary (9)
  full, mean, median, quantile, custom_quantile, maximum, minimum, upper_quartile, lower_quartile

• Nature of Plots (2) (option is set per analysis, ie per variable or entry in the config.yaml file)
  one, many
  one: a single plot with all curves or plots is produced in 1 file
  many: multiple plots are produced in multiple files, with 1 file per curve or plot

Total combinations: 4x4x9x2 = 288

Some in-build exceptions that have been explicitly excluded:

• agent_analysis + one_output : "Quantile not possible for agent level analysis!"
  The reason is that an agent only has scalar values, so there is no range for quantiles.

• agent_analysis + many_output : "- Warning: too many plots will be produced !!! "
  This is a precautionary warning.

• agent_analysis + custom_quantiles : "Quantile not possible for agent level analysis"
  The reason is that an agent only has scalar values, so there is no range for quantiles.

• scatter_plot + custom_quantiles: "Delay not supported for Quantiles! Adjust parameter and retry!"
  This combinations has not yet been implemented.

General notes

• summary==full + many_output : Does not make sense? Since summary==full creates the full ensemble of data, so there is only one data set to plot. Hence, this will always result in 1 plot in 1 file. So here many_output has no effect, and always yield the same result as one_output.

[svdhoog]

Test Circle model

The test data allows to test these settings:

• Types of analysis (4)
  agent, multiple_run, multiple_batch, multiple_set

• Types of plots (4)
  timeseries, boxplot, histogram, scatterplot

• Types of statistical summary (3)
  full, mean,custom_quantile

[the remaining 6 options are: median, custom_quantile, maximum, minimum, upper_quartile, lower_quartile]

• Nature of Plots (2)
  one, many

Total combinations: 4x4x3x2 = 96

[svdhoog]

Scatterplot mean

one plot

summary!=full

plot 61

Shows a summary across agents, for each run, each set (hence: 8 circles)

Notes:

• The first circle (set=1, run=1) is centred on (a,b)=(3.5,1) because we have 6 agents, with a=1...6, b=1. The means are E[a]=3.5 and E[b]=1, resp.
• The 2nd circle (set=1, run=2) is centred on (a,b)=(3.5,2) because b=run_no.
• All concentric circles that are centred on (a,b)=(3.5,1) belong to run 1 and increase with the set number.
• All concentric circles that are centred on (a,b)=(3.5,2) belong to run 2 and increase with the set number.

plot 62

Shows a summary across agents & runs, for each set (hence: 4 circles)

Notes:

• The circle for set 1 is centred on (a,b)=(3.5,1.5) because we are now averaging across runs, and the two blue circles in plot60 were centred on b=1 and b=2, resp.
• The circles for sets 2-4are again increasing in radius with the set number.

plot 63

shows summary across agents & runs & sets (hence: 1 circle)

Notes:

• The single circle is the average across the 4 circles in plot62, which were centred on a=3.5 with b=1...4. Hence the average is centred on a=3.5,b=2 with an average radius of r=2.

many plot

Plot 64

Shows agent-level analysis with many plots. This produces 1 file per (set,run,agent) combination (48 files for the 48 circle plots in plot 60).

Plot 65

Shows multi-run analysis with many plots. This produces 1 file per (set,run) combination (8 files for the 8 circle plots in plot 61).

Plot 66

Shows multi-batch analysis with many plots. This produces 1 file per set (4 files for the 4 circle plot in plot 62).

Plot 67

Shows multi-set analysis with many plots. This produces 1 file (1 files for the 1 circle plot in plot 63).

Scatterplot full

one plot

summary=full

Plot 70

Summary=full is not possible with agent-level analysis.

Plot 71

Plot 72

Plot 73

many plot

summary=full

Plot 74

Summary=full is not possible with agent-level analysis.

Plot 75

Plot 76

Plot 77