Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Tracker] Fix tests added to xfail_set due to PyTorch version update #3796

Open
85 tasks
vivekkhandelwal1 opened this issue Oct 15, 2024 · 7 comments
Open
85 tasks

[Tracker] Fix tests added to xfail_set due to PyTorch version update #3796

vivekkhandelwal1 opened this issue Oct 15, 2024 · 7 comments
Assignees
@vivekkhandelwal1

Comments

@vivekkhandelwal1
Copy link
Collaborator

vivekkhandelwal1 commented Oct 15, 2024 •
edited
Loading

This issue is for tracking the failing tests due to the roll PyTorch update done here: #3727. Some of the tests are common for the FX_IMPORTER_XFAIL_SET and FX_IMPORTER_STABLEHLO_XFAIL_SET.

FX_IMPORTER_XFAIL_SET

  • "AdaptiveAvgPool1dGeneralDynamic_basic",
  • "AdaptiveAvgPool1dStaticEvenMultiple_basic",
  • "AdaptiveAvgPool1dStaticLargerOutput_basic",
  • "AdaptiveAvgPool2dDynamicNoBatch_basic",
  • "AdaptiveAvgPool2dDynamic_basic",
  • "AdaptiveMaxPool1dDynamicNoBatch_basic",
  • "AdaptiveMaxPool1dDynamic_basic",
  • "AdaptiveMaxPool1dStatic_basic",
  • "CrossEntropyLossModule_basic",
  • "CrossEntropyLossNoReductionModule_basic",
  • "ElementwiseExpm1IntModule_basic",
  • "ElementwiseExpm1Module_basic",
  • "IndexPutImpl1DFloatAccumulateModule_basic",
  • "IndexPutImpl1DFloatNonAccumulateModule_basic",
  • "IndexPutImpl1DIntAccumulateModule_basic",
  • "IndexPutImpl1DIntNonAccumulateModule_basic",
  • "IndexPutImpl2DFloatNonAccumulateModule_basic",
  • "IndexPutImpl2DImplicitModule_basic",
  • "IndexPutImpl2DIndexModule_basic",
  • "IndexPutImpl2DNoneIndexStaticModule_basic",
  • "IndexPutImpl3DFloatNonAccumulateModule_basic",
  • "IndexPutImplIndexWithNoneModule_basic",
  • "InterpolateDynamicModule_sizes_nearest",
  • "IouOfModule_basic",
  • "MeshgridIndexingIJ_basic",
  • "MeshgridIndexingXY_basic",
  • "Meshgrid_basic",
  • "OneHotModule_basic",

FX_IMPORTER_CRASHING_SET

  • "AdaptiveAvgPool1dGeneralDynamicNoBatches_basic",

FX_IMPORTER_STABLEHLO_XFAIL_SET

  • "AdaptiveAvgPool1dGeneralDynamic_basic",
  • "AdaptiveAvgPool1dGeneralDynamicNoBatches_basic",
  • "AdaptiveAvgPool1dStaticEvenMultiple_basic",
  • "AdaptiveAvgPool1dStaticLargerOutput_basic",
  • "AdaptiveAvgPool2dDynamicNoBatch_basic",
  • "AdaptiveAvgPool2dDynamic_basic",
  • "AddIntModule_basic",
  • "AtenIntTensorByteDtypeModule_basic",
  • "AtenIntTensorCharDtypeModule_basic",
  • "AtenItemIntOpModule_basic",
  • "CrossEntropyLossModule_basic",
  • "CrossEntropyLossNoReductionModule_basic",
  • "EinsumStaticContractRhsModule_basic",
  • "EinsumStaticFourDimensionModule_basic",
  • "EinsumStaticModule_basic",
  • "EinsumStaticWithEllipsisSlicingAndBroadcastModule_basic",
  • "EinsumStaticWithEllipsisSlicingModule_basic",
  • "ElementwiseExpm1IntModule_basic",
  • "ElementwiseExpm1Module_basic",
  • "InterpolateDynamicModule_sizes_nearest",
  • "IouOfModule_basic",
  • "IscloseStaticModuleTrue_basic",
  • "IscloseStaticModule_basic",
  • "MeshgridIndexingIJ_basic",
  • "MeshgridIndexingXY_basic",
  • "Meshgrid_basic",
  • "MulIntModule_basic",
  • "OneHotModule_basic",
  • "ReduceFrobeniusNormComplexModule_basic",
  • "ScalarImplicitIntModule_basic",
  • "ScaledDotProductAttentionBoolMaskModule_basic",
  • "ScaledDotProductAttentionDifferentCausalModule_basic",
  • "ScaledDotProductAttentionDifferentDynamicCausalModule_basic",
  • "ScaledDotProductAttentionDifferentModule_basic",
  • "ScaledDotProductAttentionMaskModule_basic",
  • "ScaledDotProductAttentionSameCausalModule_basic",
  • "ScaledDotProductAttentionSameDynamicModule_basic",
  • "ScaledDotProductAttentionSameModule_basic",
  • "SubIntModule_basic",
  • "TensorToIntZeroRank_basic",
  • "UpSampleNearest2dDynamicFactor_basic",
  • "UpSampleNearest2dDynamicSize_basic",
  • "UpSampleNearest2dStaticFactor_basic",
  • "UpSampleNearest2dStaticSize_basic",
  • "UpSampleNearest2d_basic",

FX_IMPORTER_TOSA_XFAIL_SET

  • "AdaptiveAvgPool1dNonUnitOutputSizeDynamicModule_basic"
  • "AdaptiveAvgPool1dNonUnitOutputSizeStaticModule_basic"
  • "AdaptiveAvgPool1dUnitOutputSizeDynamicModule_basic"
  • "AdaptiveAvgPool1dUnitOutputSizeStaticModule_basic"
  • "ElementwiseRreluTrainModule_basic"
  • "ElementwiseRreluTrainStaticModule_basic"
  • "MaxPool1dEmptyStrideStaticModule_basic"
  • "MaxPool1dStaticCeilModeTrueModule_basic"
  • "MaxPool1dStaticModule_basic"
  • "ReduceFrobeniusNormKeepDimModule_basic"
  • "RepeatInterleaveSelfIntModule_basic"
@vivekkhandelwal1 vivekkhandelwal1 mentioned this issue Oct 15, 2024
Merged
@vivekkhandelwal1 vivekkhandelwal1 self-assigned this Oct 15, 2024
vivekkhandelwal1 added a commit that referenced this issue Oct 18, 2024
@vivekkhandelwal1
build: manually update PyTorch version (#3727)
9c70676 
Set PyTorch and TorchVision version to nightly release 2024-10-15.

Tracker issue for the failing tests added to xfail_set in this PR.
Issue: #3796
This commit disables the failing sparse tensor tests since they are not 
maintained on day-to-day basis and blocks the roll PyTorch update for now.

Signed-Off By: Vivek Khandelwal <[email protected]>
@zjgarvey zjgarvey mentioned this issue Oct 23, 2024
Open
25 tasks
@justin-ngo-arm
Copy link
Contributor

justin-ngo-arm commented Nov 1, 2024 •
edited
Loading

Hi @vivekkhandelwal1, @sjarus and I are also tracking fx_importer_tosa regressions from this update. Here are some additional tests that failed on fx_importer_tosa that haven't been included in your list. I also have the full list if you want:

  • "AdaptiveAvgPool1dNonUnitOutputSizeDynamicModule_basic"
  • "AdaptiveAvgPool1dNonUnitOutputSizeStaticModule_basic"
  • "AdaptiveAvgPool1dUnitOutputSizeDynamicModule_basic"
  • "AdaptiveAvgPool1dUnitOutputSizeStaticModule_basic"
  • "ElementwiseRreluTrainModule_basic"
  • "ElementwiseRreluTrainStaticModule_basic"
  • "MaxPool1dEmptyStrideStaticModule_basic"
  • "MaxPool1dStaticCeilModeTrueModule_basic"
  • "MaxPool1dStaticModule_basic"
  • "ReduceFrobeniusNormKeepDimModule_basic"
  • "RepeatInterleaveSelfIntModule_basic"

@vivekkhandelwal1
Copy link
Collaborator Author

Hi @vivekkhandelwal1, @sjarus and I are also tracking fx_importer_tosa regressions from this update. Here are some additional tests that failed on fx_importer_tosa that haven't been included in your list. I also have the full list if you want:

Hi @justin-ngo-arm, thanks for the update. Actually, the CI does not run the fx_importer_tosa tests hence I missed them. I have updated the above list.

@sjarus
Copy link
Collaborator

sjarus commented Nov 6, 2024

@vivekkhandelwal1 which targets does CI run now ? We'd previously discussed potentially swapping one or more existing TOSA targets for fx_importer_tosa.

@vivekkhandelwal1
Copy link
Collaborator Author

@vivekkhandelwal1 which targets does CI run now ? We'd previously discussed potentially swapping one or more existing TOSA targets for fx_importer_tosa.

Hi @sjarus, we run the following script https://github.com/llvm/torch-mlir/blob/main/build_tools/ci/test_posix.sh to run the tests in CI. The configs for which tests are run are: onnx, fx_importer, fx_importer_stablehlo.

@sjarus
Copy link
Collaborator

sjarus commented Nov 12, 2024

@vivekkhandelwal1 which targets does CI run now ? We'd previously discussed potentially swapping one or more existing TOSA targets for fx_importer_tosa.

Hi @sjarus, we run the following script https://github.com/llvm/torch-mlir/blob/main/build_tools/ci/test_posix.sh to run the tests in CI. The configs for which tests are run are: onnx, fx_importer, fx_importer_stablehlo.

Thanks! What would be the logistical impact of adding fx_importer_tosa to that list ? There are multiple contributors to the TorchToTosa path who would benefit from this.

@vivekkhandelwal1
Copy link
Collaborator Author

@vivekkhandelwal1 which targets does CI run now ? We'd previously discussed potentially swapping one or more existing TOSA targets for fx_importer_tosa.

Hi @sjarus, we run the following script https://github.com/llvm/torch-mlir/blob/main/build_tools/ci/test_posix.sh to run the tests in CI. The configs for which tests are run are: onnx, fx_importer, fx_importer_stablehlo.

Thanks! What would be the logistical impact of adding fx_importer_tosa to that list ? There are multiple contributors to the TorchToTosa path who would benefit from this.

I don't exactly remember why did we remove this but AFAICR the reason might have been irregular maintenance of this code path. If you think we should have this in CI again then I think @stellaraccident is the best person to take a call on this.

@stellaraccident
Copy link
Collaborator

If it is maintained and of reasonable duration, I think we can handle that. Certainly open to give it a try and see how it goes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vivekkhandelwal1 vivekkhandelwal1

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@stellaraccident @sjarus @vivekkhandelwal1 @justin-ngo-arm