Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validate 3D metrics #261

Open
wants to merge 16 commits into
base: master
Choose a base branch
from
Open

Validate 3D metrics #261

wants to merge 16 commits into from

Conversation

Anirudh-Scale
Copy link

Added 3D IOU, BEV 2D IOU, and recall, precision metrics. Tested end-to-end with celery in models repo (see associated PR there).

@@ -45,7 +45,7 @@ click = ">=7.1.2,<9.0" # NOTE: COLAB has 7.1.2 and has problems updating
rich = "^10.15.2"
shellingham = "^1.4.0"
scikit-learn = ">=0.24.0"
Shapely = ">=1.7.1"
Shapely = "^1.8.1"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gatli didn't we run into problems with Shapely at some point?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, we can't use it. It doesn't have wide enough binary support.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So what should we do about this? @Anirudh-Scale how much do we rely in the shapely dependency? Any chance we can get around this? If not, @gatli how can we still make this work?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pfmark I can implement a function to calculate the area intersection without Shapely, it's just going to be significantly slower. If that's a trade-off we're ok with, I can do that.

Copy link
Contributor

@phil-scale phil-scale Mar 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's already a function for polygon area intersection in the directory under geometry.py. I'd suggest using that since it looks like you're using shapely only for polygon intersection area.

Edit: it looks like the native polygon was used originally before changing to shapely? The native polygon intersection function is quite slow since it's written in Python, but there's a speed up for convex polygons

Copy link
Contributor

@pfmark pfmark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

few smaller comments. In addition, do we have a way of easily testing the correctness of the metrics in a test unit test? I think that would be great!

nucleus/metrics/cuboid_metrics.py Outdated Show resolved Hide resolved
nucleus/metrics/cuboid_metrics.py Outdated Show resolved Hide resolved
nucleus/metrics/cuboid_utils.py Outdated Show resolved Hide resolved
nucleus/metrics/cuboid_utils.py Outdated Show resolved Hide resolved

def __init__(
self,
enforce_label_match: bool = False,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This argument (and maybe confidence_threshold?) should be required in the constructor, and if we want to set defaults then set them in the child classes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants