-
Notifications
You must be signed in to change notification settings - Fork 5
Scoring & Benchmarking
The scoring is done by simple addition. Each response meeting the criteria set for a specific Title (I know this language is awkward but it corresponds to the code) will add 1 to the result. It allows specific questions to be included in multiple Titles. It also allows multiple responses for a specific question to be included in a single Title.
To implement scoring you must provide a scoring json file which specifies the Title, Category and Measures to be used. Where the Category does not correspond to an item_type within the csv file containing the questions, you must also include a column called scoring_category which holds the Category.
Consider this extract from the English_WG_scoring.json.
{
"title" : "First Gestures",
"category" : "gestures_first",
"measure" : "sometimes;often",
"order" : 1
},
{
"title" : "Games Gestures",
"category" : "gestures_games",
"measure" : "yes",
"order" : 2
},
{
"title" : "Early Gestures",
"category" : "gestures_first;gestures_games",
"measure" : "sometimes;often;yes",
"order" : 3
},
{
"title" : "Object Gestures",
"category" : "gestures_objects",
"measure" : "yes",
"order" : 4
},
{
"title" : "Adult Gestures",
"category" : "gestures_adults",
"measure" : "yes",
"order" : 5
},
{
"title" : "Parent Gestures",
"category" : "gestures_parents",
"measure" : "yes",
"order" : 6
},
{
"title" : "Later Gestures",
"category" : "gestures_parents;gestures_adults;gestures_objects",
"measure" : "yes",
"order" : 7
},
{
"title" : "Total Gestures",
"category" : "gestures_parents;gestures_adults;gestures_objects;gestures_first;gestures_games",
"measure" : "sometimes;often;yes",
"order" : 8
}
Within each set of curly brackets ( {} ) is the definition to calculate a single score. Each set contains the 3 fields title, category, measure and order.
order specifies the order they will appear in the data analysis.
title provides the name that will be used for the score - it must be unique for the file.
category is the item_type or scoring_category from the csv file that is to be used. If you want to use more than 1, separate them with a semi-colon (;). An example of using multiple categorys is the last set in the example which is called Total Gestures and uses 5 different categories.
measure is the responses that are acceptable for the score to be incremented. With the example of Total Gestures, any response of sometimes, often or yes against any of the categories will result in the score being incremented by 1.
A new, optional, item has been added to the json file. kind specifies the type of summary reporting to be provided. It defaults to count which sums the scores as described above. It can also be list, which will produce a list of responses (this is useful for where the respondent has type in sentences).
Perhaps an exaggeration of title, but complex scoring allows different responses to get a different score (rather than simply 1. Should complex scoring be required, we add "scores" to sections where the complex score is to be used and for each measure we specify it's score in a "key", "value" list. For example, if we wanted "understands" to score 1 and "produces" to score 2 in "Total Understood", then the dictionary for that section would look like:
{
"title" : "Total Understood",
"category" : "word",
"measure" : "understands;produces",
"order" : 1,
"scores" : [
{
"key" : "understands",
"value" : 1
},
{
"key" : "produces",
"value" : 2
}
]
},
If the "scores" section is not included, 1 point is applied for every occurrance of items in the "measure" list.
Benchmarking works through csv files held in cdi_forms>form_data>benchmarking. The csv files held here are specified within instruments.json using the key benchmark followed by the file location, for example: "benchmark" : "cdi_forms/form_data/benchmarking/English_WG_benchmark.csv"
The file structure is as follows: The header row contains: title,percentile,age,raw_score,raw_score_boy,raw_score_girl Each subsequent row contains the values for the particular benchmark, so if you have benchmarks every 5% you will have 20 rows for a single measure. The title must match exactly the scoring title within the scoring json file for this instrument.
If there are NO unisex scores available for a specific measure, raw_score should be set to 9999. If there is only a single measurement for each age category, for example you are showing the average score achieved at this age rather than the percentile, then the percentile value is set to 999.
Examples of the above 2 can be found in English_WG (Phrases) and English_WS (Combining) respectively.
There is a field in cdi_forms.instrument_forms called scoring_category. This is used to specify the scoring category. This value is included in the csv data files held in cdi_form_csv. If a scoring_category column is not included, then the item_type is used instead.
A new table has been created in researcher_ui.instrumentscore. This table has a foreign key link to research_ui.instrument and includes title, category and measure fields. The title is the string that will show in the downloaded data file, the category is a semi-colon delimited list of categories (as specified in cdi_forms.instrument_forms.scoring_category and measure is a semi-colon delimited list of answers that will be used to increment the score.
researcher_ui.instrumentscore is populated by running manage.py populate_scoring. This program reads json files found in cdi_forms.scoring directory. The json file contains the title, category and measure for each score to be created. In order to associate the correct json file with the correct instrument, the field scoring_json in instruments.json specifies the path to the scoring mechanism to use for each instrument.