[DE-4880] Update annotation related endpoints for multiple ground truth sets #444
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
iz18
force-pushed
the
iz/nucleusTaskBasedAnnotations
branch
from
8818b62 to
421282f
Compare
daveguo-scale
approved these changes
Dec 9, 2024
…m parent classes but not needing the task id field
iz18
force-pushed
the
iz/nucleusTaskBasedAnnotations
branch
from
f4ccf81 to
9c918fa
Compare
|
Removed the black formatting in circle ci because it doesn't behave the same way as local black formatter on the same version. Weird issue and maybe should be looked into? Will put a ticket into backlog. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With this feature, we will have multiple sets of ground truth if a dataset item is labeled across multiple tasks. To not suddenly break everything on the ML side, we want to have the option to return the legacy results, which is only the set of annotations that correspond to the most recent task. This is also changed to be consistent with the model results shown on Nucleus itself where it will only pull the most recent task. The task id is also now returned with all the annotations.
Moving forward, ML should keep track of which set of ground truth they trained their model with, and they can pull back that set of ground truth by setting
only_most_recent_taskto False. I think a new endpoint where they can request a set of ground truth based on a specifictask_idwould also be helpful in the future.Accompanies this pr