fix(evaluation): correct stress-des 5.2 random sampling

evaluation/reporting.qmd

@@ -16,8 +16,8 @@ This page evaluates the extent to which the journal article meets the criteria f
 Of the **24** items in the checklist:
 
 * **14** were met fully (✅)
-* **6** were partially met (🟡)
-* **3** were not met (❌)
+* **5** were partially met (🟡)
+* **4** were not met (❌)
 * **1** was not applicable (N/A)
 
 | Item | Recommendation | Met by study? | Evidence |
@@ -47,7 +47,7 @@ Of the **24** items in the checklist:
 | 4.3 Estimation approach | State the method used to account for the stochasticity: For example, two common methods are multiple replications or batch means. Where multiple replications have been used, state the number of replications and for batch means, indicate the batch length and whether the batch means procedure is standard, spaced or overlapping. For both procedures provide a justification for the methods used and the number of replications/size of batches. | 🟡 Partially | Number of replications stated but not justified.<br><br>`Methods: Statistics and Software`: "Each scenario... was simulated 30 times"<br><br>@huang_optimizing_2019 |
 | **Implementation** |
 | 5.1 Software or programming language | State the operating system and version and build number.<br>State the name, version and build number of commercial or open source DES software that the model is implemented in.<br>State the name and version of general-purpose programming languages used (e.g. Python 3.5).<br>Where frameworks and libraries have been used provide all details including version numbers. | 🟡 Partially | Some details provided - `Methods: Statistics and Software`: "The DES model was built with Simmer (version 4.1.0), a DES package for R. The interactive web application was built with R-Shiny"<br><br>@huang_optimizing_2019 |
-| 5.2 Random sampling | State the algorithm used to generate random samples in the software/programming language used e.g. Mersenne Twister.<br>If common random numbers are used, state how seeds (or random number streams) are distributed among sampling processes. | 🟡 Partially | Sampling described for arrivals but not for length of time with resources. Doesn't mention whether seeds are used.<br><br>`Methods: Model Properties: Patients`: "Patients are generated by a Poissone process with an inter-arrival time as specified in Table 1."<br>![Table 1. DES model inputs](../original_study/tab1.jpg)<br>@huang_optimizing_2019 |
+| 5.2 Random sampling | State the algorithm used to generate random samples in the software/programming language used e.g. Mersenne Twister.<br>If common random numbers are used, state how seeds (or random number streams) are distributed among sampling processes. | ❌ Not met | Doesn't mention algorithm or whether seeds or streams are used (know from code that it does not though). |
 | 5.3 Model execution | State the event processing mechanism used e.g. three phase, event, activity, process interaction.<br>*Note that in some commercial software the event processing mechanism may not be published. In these cases authors should adhere to item 5.1 software recommendations.*<br>State all priority rules included if entities/activities compete for resources.<br>If the model is parallel, distributed and/or use grid or cloud computing, etc., state and preferably reference the technology used.  For parallel and distributed simulations the time management algorithms used.  If the HLA is used then state the version of the standard, which run-time infrastructure (and version), and any supporting documents (FOMs, etc.) | 🟡 Partially | Does not state event processing mechanism. Does describe priority rules - `Methods: Model Properties: Queueing` - e.g. "n our model, emergency IR and stroke patients have higher priority than elective patients for resources. Specifically, angioINRs are capable of both INR and IR procedures, although all patient types... |
 | 5.4 System specification | State the model run time and specification of hardware used.  This is particularly important for large scale models that require substantial computing power.  For parallel, distributed and/or use grid or cloud computing, etc. state the details of all systems used in the implementation (processors, network, etc.) | ❌ Not met | - | 
 | **Code access** |

evaluation/reproduction_report.qmd

@@ -120,7 +120,7 @@ eval_dict = {
   'STARS (optional)': [2, 0, 3, 0],
   'Badges (criteria)': [3, 0, 9, 0],
   'Badges (badges)': [0, 0, 12, 0],
-  'STRESS-DES': [14, 6, 3, 1],
+  'STRESS-DES': [14, 5, 4, 1],
   'ISPOR-SDM': [7, 2, 7, 2]
 }
 eval_wide = pd.DataFrame(eval_dict, index=col).T

logbook/posts/2024_11_04/index.qmd

@@ -0,0 +1,19 @@
+---
+title: "Day 18"
+author: "Amy Heather"
+date: "2024-11-04"
+categories: [evaluation]
+bibliography: ../../../quarto_site/references.bib
+---
+
+## Untimed: Amendment to evaluation
+
+I noticed a mistake in my evaluation of the repository against STRESS-DES.
+
+**Criteria:** 5.2 Random sampling
+
+**Original decision:** 🟡 Partially met - no seeds but describes sampling of arrivals
+
+**New decision:** ❌ Not met - as sampling of arrivals wasn't relevant to this criteria (I had misunderstood)
+
+I amended `reporting.qmd`, `reproduction_report.qmd` and [stars_wp1_summary](https://github.com/pythonhealthdatascience/stars_wp1_summary) accordingly.