[tracking] Pass RISC-V compliance test suite

[imphil]

We want to execute and pass the RISC-V compliance tests on Ibex.

Setup tasks

• Create target to run the compliance tests in Verilator: Run RISC-V Compliance tests through Verilator #99

Known issues:

• Standards-compliant trap/interrupts: Implement trap handling according to latest RISC-V spec #70
• Implement the MSCRATCH CSR: Implement the mscratch CSR #97

[davideschiavone]

We added MSCRATCH and ported RISCY to 1.10 spec.
You will find the TB to run the compliance tests here: https://github.com/pulp-platform/riscv/tree/master/tb/core

Best
Davide

[imphil]

We now pass all tests, except for four tests which require a writable mtvec. Requiring this is a bug in the compliance test suite, as a read-only mtvec is explicitly allowed by the standard.

[imphil]

Follow-up discussion to fix this and other issues with riscv-compliance is happening at #214.

[imphil]

We are now using the unmodified upstream test suite and have writable mtvec. But we still fail on these five tests:

Check      I-EBREAK-011,4d0
< 00000003
< 11111111
< 00000000
< 00000000
 ... FAIL
Check       I-ECALL-011,4d0
< 0000000b
< 11111111
< 00000000
< 00000000
 ... FAIL
Check     I-FENCE.I-011,4d0
< 00000030
< 00000012
< 00000042
< 001101b3
 ... FAIL
Check I-MISALIGN_JMP-011,36d0
< 00000002
< 00000000
< 11111111
< 22222222
< 33333333
< 44444444
< 00000002
> ffffffff
> ffffffff
> ffffffff
 ... FAIL
Check I-MISALIGN_LDST-012,10c2,10
< 91a1b1c1
< 91a1b1c1
< 91a1b1c1
< 00000002
< 00000006
< 00000003
< 00000006
---
> 00000099
> 00009900
> 00990000
> ff000000
> ffffffff
> ffffffff
> ffffffff
> ffffffff
> ffffffff
38c38
< 99999999
---
> 99000099
40,44c40,44
< 99999999
< 00000001
< 00000006
< 00000003
< 00000006
---
> 00999999
> ffffff00
> ffffffff
> ffffffff
> ffffffff
 ... FAIL

More investigation is needed to see what's actually going on with these tests.

[imphil]

@eroom1966 did you already look into some of these issues?

[towoe]

Had a quick look at the failures.
I-EBREAK-01.stdout shows 105: Illegal instruction (hart 0) at PC 0x00000108: 0x10529073
which is the same for I-ECALL-01.stdout:
10529073 csrw stvec,t0
Seems right as the CSR stvec (0x105) is not defined.

I-FENCE.I is tracked in #391

I-MISALIGN_JMP-01.stdout: 101: Illegal instruction (hart 0) at PC 0x0000016e: 0x00010413
the disassembly:

00a0006f                j       16e <begin_testcode+0x2a>
00000113                li      sp,0
00002097                auipc   ra,0x2
ea008093                addi    ra,ra,-352 # 200c <test_A2_res>

I-MISALIGN_LDST-01 shows no obvious error in the stdout.

[imphil]

Seems right as the CSR stvec (0x105) is not defined.

Yes, but the test shouldn't rely on a CSR which isn't required. Looks like these tests will need to be rewritten in some way.

The two MISALIGN tests look strange indeed, and I didn't spend much time looking into what's exactly going on there. Could be an Ibex thing (though we have reasonably good DV on misaligned instructions these days), or something in the test.

[towoe]

Seems right as the CSR stvec (0x105) is not defined.

Yes, but the test shouldn't rely on a CSR which isn't required. Looks like these tests will need to be rewritten in some way.

The test actually checks for a routine stvec_handler to be defined and only writes to stvec in this case. The problem was that mtvec had an entry which was not 256-byte aligned. This resulted in a instruction fetch to an address behind the check for the stvec_handler and before the write to stvec.

[towoe]

I-FENCE.I is moved in riscv-non-isa/riscv-arch-test@df18fa8 from rv32i to rv32Zifencei.
Expected failure updated in #458

[tomeroberts]

This might already have been debugged and recorded somewhere, but recording here for completeness.
The MISALIGN-JMP test fails as it assumes the core will enforce 32-bit alignment on instruction fetches, i.e. it assumes no C extension.
Ibex always has the C extension "enabled" and therefore it fetches and executes a 16bit aligned instruction (which decodes as an illegal instruction in the test).

Having said that, it doesn't look like Ibex ever generates unaligned fetch exceptions (even if the address isn't 16bit aligned)

[tomeroberts]

It looks like the MISALIGN-LDST fails due to the compliance suite expecting traps on unaligned ldst whereas Ibex has hardware support.

[vogelpi]

Related to #428 and some recent changes that were done upstream to address the mtvec issues, I wanted to quickly check the riscv-compliance suite. Using v.2.1 from upstream (Dec 2020), I see many failures.

In CI, we are running an older version 844c666 (Feb 2020). Are there any plans to update the version?

[imphil]

I'm not sure how much work we want to put into the "old" riscv-compliance framework. @towoe might know more.

[towoe]

I think we want to keep using a fixed old version for the compliance check we do right now and once version 3 is ready we can remove those tests in favor of the check which are then hopefully based on #1251.
See #1233 for a description of the upstream changes.

[vogelpi]

Ah, I see. This makes sense. Thanks @towoe for the clarification.

[zeeshanrafique23]

Hmm, I use this version of riscv-compliance, and with this getting 4 instruction failures instead of 5.

git clone https://github.com/riscv/riscv-compliance --branch 1.0

Check              I-EBREAK-011,4d0
< 00000003
< 11111111
< 00000000
< 00000000
 ... FAIL
Check               I-ECALL-011,4d0
< 0000000b
< 11111111
< 00000000
< 00000000
 ... FAIL
Check        I-MISALIGN_JMP-011,36d0
< 00000002
< 00000000
< 11111111
< 22222222
< 33333333
< 44444444
< 00000002
< 00000000
< 55555555
< 00000002
< 00000000
< 66666666
< 00000002
< 00000000
< 77777777
< 00000002
< 00000000
< 88888888
< 00000002
< 00000000
< 99999999
< 00000002
< 00000000
< aaaaaaaa
< 00000002
< 00000000
< bbbbbbbb
< 00000002
< 00000000
< cccccccc
< 00000002
< 00000000
< dddddddd
< 00000002
< 00000000
< eeeeeeee
 ... FAIL
Check       I-MISALIGN_LDST-012,10c2,10
< 91a1b1c1
< 91a1b1c1
< 91a1b1c1
< 00000001
< 00000004
< 00000002
< 00000004
< 00000003
< 00000004
---
> 0291a1b1
> f20291a1
> e2f20291
> ffffffff
> ffffffff
> ffffffff
> ffffffff
> ffffffff
> ffffffff
12,13c12
< fffff202
< ffffd2e2
---
> ffffe2f2
14a14
> 000000d2
16,17c16
< 0000f202
< 0000d2e2
---
> 0000e2f2
19,26c18,26
< 00000001
< 00000004
< 00000003
< 00000004
< 00000001
< 00000004
< 00000003
< 00000004
---
> 000000d2
> ffffffff
> ffffffff
> ffffffff
> ffffffff
> ffffffff
> ffffffff
> ffffffff
> ffffffff
28,36c28,36
< 99999999
< 99999999
< 99999999
< 00000001
< 00000006
< 00000002
< 00000006
< 00000003
< 00000006
---
> 00000099
> 00009900
> 00990000
> ff000000
> ffffffff
> ffffffff
> ffffffff
> ffffffff
> ffffffff
38c38
< 99999999
---
> 99000099
40,44c40,44
< 99999999
< 00000001
< 00000006
< 00000003
< 00000006
---
> 00999999
> ffffff00
> ffffffff
> ffffffff
> ffffffff
 ... FAIL

--------------------------------
FAIL: 4/48 RISCV_TARGET=ibex RISCV_DEVICE=rv32imc RISCV_ISA=rv32i

Fence is passing.

Compare to reference files ... 

Check             I-FENCE.I-01 ... OK
--------------------------------
OK: 1/1 RISCV_TARGET=ibex RISCV_DEVICE=rv32imc RISCV_ISA=rv32Zifencei