Calculation of confidence intervals (CIs)

[janezlapajne]

Hello!

I am a PhD student and I have found a great interest in the paper: Sounds of COVID-19: exploring the realistic performance of audio-based digital testing. I have decided to contact you regarding the methodology used in the article, specificaly about the calculation of confidence intervals (CIs) for various metrics.

In the paper, it is stated that a total of 800 samples were used for training and 200 samples for testing (with balanced covid participants). If I understand correctly, the model was only trained once. Then the bootstrapping resampling was used to evaluate the model on the testing samples. Furthermore, the CIs were calculated based on bootstrapped samples extracted from only test data. I discern the stated from the paper and from source code (function that calculates CIs):

covid19-sounds-npjDM/COVID19_prediction/COVID_model/model_util.py

Line 379 in 4ae96ef

def get_CI(data, AUC, Sen, Spe):

I am interested to know, how did you justify the CI calculation, since normally, the bootstrapping is performed by following this pseudocode:

statistics = []
for i in bootstraps:
	train, test = select_sample_with_replacement(data, size)
	model = train_model(train)
	stat = evaluate_model(test)
	statistics.append(stat) 

In other words, the model is usually refitted for each bootstrapped example and only then tested on the remaining data. However, in this article, the bootstrapping is only made on the already produced classification results. I hope you can elaborate on that, since I would like to know the details. Thank you!

I hope I did not miss anything crucial, otherwise, I am giving my sincerest apologies ahead of time.