Skip to content

Commit

Permalink
[RISCV] Lower (vector_interleave X, undef) to (vzext_vl X). (#87283)
Browse files Browse the repository at this point in the history
If the odd vector is undef or poison, the widening add and multiply trick
doesn't work unless we freeze the odd vector.

Unfortunately, freezing doesn't work when the operand is provably
undef/poison. MIR doesn't have a representation for freeze so it
just becomes a COPY from IMPLICIT_DEF which freely propagates undef
to each operand independently.

To work around this, check for undef explicitly and lower to a VZEXT_VL
of the even vector. This produces better code than we'd get from a
freeze anyway.

I've left a FIXME for adding a freeze. I'll do that as a separate patch
as it affects other tests and doesn't help with the new test.
  • Loading branch information
topperc committed Apr 2, 2024
1 parent 8c1dc5d commit a9af66a
Show file tree
Hide file tree
Showing 2 changed files with 16 additions and 14 deletions.
11 changes: 10 additions & 1 deletion llvm/lib/Target/RISCV/RISCVISelLowering.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -4624,7 +4624,13 @@ static SDValue getWideningInterleave(SDValue EvenV, SDValue OddV,
SDValue Passthru = DAG.getUNDEF(WideContainerVT);

SDValue Interleaved;
if (Subtarget.hasStdExtZvbb()) {
if (OddV.isUndef()) {
// If OddV is undef, this is a zero extend.
// FIXME: Not only does this optimize the code, it fixes some correctness
// issues because MIR does not have freeze.
Interleaved =
DAG.getNode(RISCVISD::VZEXT_VL, DL, WideContainerVT, EvenV, Mask, VL);
} else if (Subtarget.hasStdExtZvbb()) {
// Interleaved = (OddV << VecVT.getScalarSizeInBits()) + EvenV.
SDValue OffsetVec =
DAG.getSplatVector(VecContainerVT, DL,
Expand All @@ -4635,6 +4641,9 @@ static SDValue getWideningInterleave(SDValue EvenV, SDValue OddV,
Interleaved = DAG.getNode(RISCVISD::VWADDU_W_VL, DL, WideContainerVT,
Interleaved, EvenV, Passthru, Mask, VL);
} else {
// FIXME: We should freeze the odd vector here. We already handled the case
// of provably undef/poison above.

// Widen EvenV and OddV with 0s and add one copy of OddV to EvenV with
// vwaddu.vv
Interleaved = DAG.getNode(RISCVISD::VWADDU_VL, DL, WideContainerVT, EvenV,
Expand Down
19 changes: 6 additions & 13 deletions llvm/test/CodeGen/RISCV/rvv/vector-interleave.ll
Original file line number Diff line number Diff line change
Expand Up @@ -656,26 +656,19 @@ define <vscale x 16 x double> @vector_interleave_nxv16f64_nxv8f64(<vscale x 8 x
ret <vscale x 16 x double> %res
}

; FIXME: The last operand to the vwaddu.vv and vwmaccu.vx are both undef. They
; need to be the same register with the same contents. Otherwise, the even
; elements will not contain just the values from %a.
define <vscale x 8 x i32> @vector_interleave_nxv8i32_nxv4i32_poison(<vscale x 4 x i32> %a) {
; CHECK-LABEL: vector_interleave_nxv8i32_nxv4i32_poison:
; CHECK: # %bb.0:
; CHECK-NEXT: vsetvli a0, zero, e32, m2, ta, ma
; CHECK-NEXT: vwaddu.vv v12, v8, v10
; CHECK-NEXT: li a0, -1
; CHECK-NEXT: vwmaccu.vx v12, a0, v8
; CHECK-NEXT: vmv4r.v v8, v12
; CHECK-NEXT: vsetvli a0, zero, e64, m4, ta, ma
; CHECK-NEXT: vzext.vf2 v12, v8
; CHECK-NEXT: vmv.v.v v8, v12
; CHECK-NEXT: ret
;
; ZVBB-LABEL: vector_interleave_nxv8i32_nxv4i32_poison:
; ZVBB: # %bb.0:
; ZVBB-NEXT: li a0, 32
; ZVBB-NEXT: vsetvli a1, zero, e32, m2, ta, ma
; ZVBB-NEXT: vwsll.vx v12, v10, a0
; ZVBB-NEXT: vwaddu.wv v12, v12, v8
; ZVBB-NEXT: vmv4r.v v8, v12
; ZVBB-NEXT: vsetvli a0, zero, e64, m4, ta, ma
; ZVBB-NEXT: vzext.vf2 v12, v8
; ZVBB-NEXT: vmv.v.v v8, v12
; ZVBB-NEXT: ret
%res = call <vscale x 8 x i32> @llvm.experimental.vector.interleave2.nxv8i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> poison)
ret <vscale x 8 x i32> %res
Expand Down

0 comments on commit a9af66a

Please sign in to comment.