Add Grouped Convolution and GEMM documentation #1719
+350
−16
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@bartekxk What about using BF16, FP16 and FP32 instead of bhalf_t half_t and float ? |
bartekxk
requested review from
junliume,
illsilin,
carlushuang,
qianfengz,
aosewski,
poyenc,
geyyer,
andriy-ca and
a team
as code owners
aosewski
reviewed
Dec 18, 2024
Comment on lines
+7
to
+12
| * **BLayout** - B layout (RowMajor/ColumnMajor). | ||
| * **CLayout** - C layout (RowMajor/ColumnMajor). | ||
| * **ADataType** - A data type. | ||
| * **BDataType** - B data type. | ||
| * **CDataType** - C data type. |
Comment on lines
+13
to
+15
| * **BElementwiseOperation** - fused operation on tensor B. | ||
| * **CElementwiseOperation** - fused operation on tensor C. |
There was a problem hiding this comment.
Let's start from capital letter.
Comment on lines
+13
to
+14
| * **BElementwiseOperation** - fused operation on tensor B. |
There was a problem hiding this comment.
Important to note this elementwise operation is done before multiplication of those two tensors - that is on their (A/B tensor) elements.
| * **AElementwiseOperation** - fused operation on tensor A. | ||
| * **BElementwiseOperation** - fused operation on tensor B. | ||
| * **CElementwiseOperation** - fused operation on tensor C. |
There was a problem hiding this comment.
Operation is done on the output (result of two matrix multiplication) matrix elements.
| * **CElementwiseOperation** - fused operation on tensor C. | ||
|
|
||
| For matrices with large K dimension `DeviceGemmSplitK` implementation is available. This implementation allows user to split K dimension between work groups. This implementation requires AtomicAdd operation on global memory (output buffer must be set to zeroes). |
There was a problem hiding this comment.
Suggested change
| For matrices with large K dimension `DeviceGemmSplitK` implementation is available. This implementation allows user to split K dimension between work groups. This implementation requires AtomicAdd operation on global memory (output buffer must be set to zeroes). | |
| For matrices with large K dimension `DeviceGemmSplitK` implementation is available. This implementation allows user to split K dimension between work groups. This implementation uses `AtomicAdd` operation on global memory, thus need to zero-out output buffer for correct results. |
|
|
||
| For fused operations with additional tensor there are `DeviceGemmMultipleABD` or `DeviceGemmMultipleD` operation which require following parameters: | ||
| * **DsLayout** - layouts for additional tensors for fused operations. |
There was a problem hiding this comment.
Please add note that all DsLayout have to be the same as output matrix C layout.
| * **DeviceGemmDl** - Device operation with DL instructions. | ||
| * **DeviceGemmDpp** - Device operation with DL instructions with DPP instructions during data load. | ||
| * **DeviceGemmWmma_CShuffle** - Device operation with WMMA instructions with CShuffle optimization for more optimized data store. |
There was a problem hiding this comment.
Suggested change
| * **DeviceGemmWmma_CShuffle** - Device operation with WMMA instructions with CShuffle optimization for more optimized data store. | |
| * **DeviceGemmWmma_CShuffle** - Device operation with WMMA instructions and CShuffle optimization for more optimized data store. |
| * **DeviceGemm_Xdl_CShuffleV2** - Device operation with XDL instructions with CShuffle optimization for more optimized data store. GEMM pipeline has been optimized compared to **DeviceGemm_Xdl_CShuffle**. | ||
| * **DeviceGemmXdlSkipBLds** - Device operation with XDL instructions. Load to shared memory has been skiped for B matrix. | ||
| * **DeviceGemm_Xdl_WaveletModel_CShuffle** - Device operation with XDL instructions with CShuffle optimization for more optimized data store. |
There was a problem hiding this comment.
Please not is uses producer+consumer scheme cooperation between waves in workgroup.
| * **DeviceGemmXdl** - Device operation with XDL instructions. | ||
|
|
||
| Table of supported cases by instance factory with XDL instruction for Row/Row/Row, Row/Column/Row, Column/Row/Row or Column/Column/Row: |
There was a problem hiding this comment.
Are you positive that we support all those layouts for all data types?
Comment on lines
+89
to
+99
| * **BLayout** - B layout (RowMajor/ColumnMajor). | ||
| * **CLayout** - C layout (RowMajor/ColumnMajor). | ||
| * **ADataType** - A data type. | ||
| * **BDataType** - B data type. | ||
| * **CDataType** - C data type. | ||
| * **AElementwiseOperation** - fused operation on tensor A. | ||
| * **BElementwiseOperation** - fused operation on tensor B. | ||
| * **CElementwiseOperation** - fused operation on tensor C. | ||
|
|
||
| This implementation allows user to split K dimension between work groups. This implementation requires AtomicAdd operation on global memory (output buffer must be set to zeroes if splitK parameter is larger than one). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.