Report confidence for detections other than the best-confidence match?

[conormcmahon]

In the online version of the BirdNET tool (https://birdnet.cornell.edu/api/) it reports confidence for the best 10 species matches, instead of just the highest confidence species like BirdNET-Analyzer reports. Is there any way to get BirdNET-Analyzer to do something similar? Often in cases where a few species have similar vocalizations I might be interested in the second or third best match, as well as the best match.
Thanks!

[Mattk70]

You’re mistaken Conor. All results with a confidence that reach your chosen threshold are reported by the tool, not just the top result.

[conormcmahon]

Oh, interesting - thanks Matt. I don't know why I thought this. This changes the way I'll think about the tool quite a bit - definitely this is a much better behavior than what I thought it did.

Is this documented on the ReadME anywhere? If it's not I think it might be worth stating explicitly. I've spoken to other users of the package who had the same misapprehension as me.

[conormcmahon]

I'm surprised I don't see more results with lower-confidence secondary classifications for 'correct' species when the classifier mistakenly IDs something. For example, I have a dataset I'm working with right now from Santa Barbara, CA. In one set of recordings I currently have 183,000 detections of Eastern Towhee and 42,460 of Spotted Towhee, using a minimum confidence threshold of 0.1. Eastern Towhee has never been reported in the county (maybe never in the state?) and Spotted is very common, so I think all of the misidentified EATO are actually SPTO.

But if I look at individual 3-second snippets in which an EATO was "detected," only 3.7% had a SPTO detected (mean confidence of 0.32 when it was detected).

About 78% of EATO false positives had some other species also detected but there's no clear culprit to explain the really high false positive rate - no one species was co-detected with more than 14% of EATO detections. I just manually checked a handful of the highest confidence EATO detections and they all sound fine for SPTO but none had a co-occurring SPTO 'second guess' match.

[abfleishman]

This is a common issue and not as simplistic as you and I want it to be! While your two towhees are closely related, their calls are not super similar ( to my eye/ear). And there is a gradient in vocalizations east to west in both species.

One thing to remember is that the confidence scores that are returned are not probabilities of presence in the clip. See recent papers that have suggestions for how to over come this. I think you likely may need to reduce the threshold for SPTO in this case and ignore the EATO detections. Another thing that we have done is first confirm that the EATO detections are indeed all misclassified SPTO and then just treat them as such!

[conormcmahon]

Thanks Abram. Yeah, on first look at my data and results it feels like this might be a case where I'd benefit from eventually training a custom classifier for my sites to get over that regional variation. I grew up in Texas and the SPTO there do sound pretty different from the ones here in California. And if I had a locally trained classifier I could just omit EATO to begin with.

I don't think I can fix this by reducing the confidence threshold for SPTO because even though I've set it to 0.1 when running BirdNET, I already end up rejecting most of the detections with much higher confidence than that in post processing. But I might end up doing something like your other suggestion if I can verify that most of the EATO mis-IDs are some other species. So far (but have only listened to a handful) they've all been SPTO.