refactor: parallelizate fl annotation processing #182
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…nd DeepenToT4Converter Signed-off-by: Shunsuke Miura <[email protected]> # Conflicts: # perception_dataset/deepen/deepen_to_t4_converter.py # perception_dataset/fastlabel_to_t4/fastlabel_2d_to_t4_converter.py # perception_dataset/t4_dataset/annotation_files_generator.py
for more information, see https://pre-commit.ci
Signed-off-by: Shunsuke Miura <[email protected]>
for more information, see https://pre-commit.ci
ktro2828
added a commit
that referenced
this pull request
Jan 2, 2025
…#172) * feat: add T4 updater from FastLabel annotations Signed-off-by: ktro2828 <[email protected]> * fix: update conversion Signed-off-by: ktro2828 <[email protected]> * test: update tests Signed-off-by: ktro2828 <[email protected]> * feat: update FastLabel merger (#178) * update configs Signed-off-by: Shunsuke Miura <[email protected]> * add label converter to FastLabel2dToT4Converter Signed-off-by: Shunsuke Miura <[email protected]> * fix bug in from_json of surface_ann Signed-off-by: Shunsuke Miura <[email protected]> * add automatic_annotation key to sample_annotation/object_ann/surface_ann Signed-off-by: Shunsuke Miura <[email protected]> * accelerate default object_mask generation Signed-off-by: Shunsuke Miura <[email protected]> * feat: improve T4 updater functionality and directory handling Signed-off-by: Shunsuke Miura <[email protected]> * feat: support polygon-to-bbox conversion and improve polygon-to-RLE conversion * update object config Signed-off-by: Shunsuke Miura <[email protected]> * feat: update annotation loader * feat: only convert to bbox for objects labels Signed-off-by: Shunsuke Miura <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: handling automatic_annotation entry Signed-off-by: Shunsuke Miura <[email protected]> * fix: bug fix in rle_from_points Signed-off-by: Shunsuke Miura <[email protected]> * fix: add default value for automatic_annotation Signed-off-by: Shunsuke Miura <[email protected]> --------- Signed-off-by: Shunsuke Miura <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * feat: enhance surface category handling in AnnotationFilesGenerator and DeepenToT4Converter (#180) * feat: enhance surface category handling in AnnotationFilesGenerator and DeepenToT4Converter Signed-off-by: Shunsuke Miura <[email protected]> * delete unnecessary import Signed-off-by: Shunsuke Miura <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Shunsuke Miura <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * refactor: parallelizate fl annotation processing (#182) * feat: enhance surface category handling in AnnotationFilesGenerator and DeepenToT4Converter Signed-off-by: Shunsuke Miura <[email protected]> # Conflicts: # perception_dataset/deepen/deepen_to_t4_converter.py # perception_dataset/fastlabel_to_t4/fastlabel_2d_to_t4_converter.py # perception_dataset/t4_dataset/annotation_files_generator.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * refactor: parallelizate fl annotation processing Signed-off-by: Shunsuke Miura <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Shunsuke Miura <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * remove duplicate key in object.yaml Signed-off-by: Shunsuke Miura <[email protected]> * feat(fastalbel_to_t4dataset): fix keyframe handling (#187) * merge _load_annotation_jsons function to one Signed-off-by: Shunsuke Miura <[email protected]> * add default value for automatic_annotation Signed-off-by: Shunsuke Miura <[email protected]> * change to non-keyframe for no-annotation frame Signed-off-by: Shunsuke Miura <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove unused import Signed-off-by: Shunsuke Miura <[email protected]> --------- Signed-off-by: Shunsuke Miura <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * change import path Signed-off-by: Shunsuke Miura <[email protected]> * Revert "refactor: parallelizate fl annotation processing (#182)" This reverts commit 3872c99. * refactor: FastLabel annotation processing for parallel execution Signed-off-by: Shunsuke Miura <[email protected]> * supress no-3D annotation warning Signed-off-by: Shunsuke Miura <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: ktro2828 <[email protected]> Signed-off-by: Shunsuke Miura <[email protected]> Co-authored-by: Shunsuke Miura <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Shunsuke Miura <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces parallel processing to the
fastlabel_2d_to_t4_converter.pyto improve the performance of annotation processing. The most important changes include importing necessary modules for parallel processing, refactoring the annotation formatting function, and adding new methods to handle parallel processing.Parallel processing improvements:
perception_dataset/fastlabel_to_t4/fastlabel_2d_to_t4_converter.py: ImportedProcessPoolExecutorandas_completedfromconcurrent.futuresto enable parallel processing.perception_dataset/fastlabel_to_t4/fastlabel_2d_to_t4_converter.py: Refactored_format_fastlabel_annotationto process annotations for each file in parallel using the new_process_annotations_for_filemethod.perception_dataset/fastlabel_to_t4/fastlabel_2d_to_t4_converter.py: Addedprocess_annotationmethod to handle individual annotation processing tasks.perception_dataset/fastlabel_to_t4/fastlabel_2d_to_t4_converter.py: Added_process_annotations_for_filemethod to manage parallel processing of annotations usingProcessPoolExecutor.