running miri on rustc's test suite (run-pass)

[oli-obk]

output of

time MIRI_SYSROOT=~/.xargo/HOST MIRI_RUSTC_TEST=../rust/src/test/run-pass cargo run --release --manifest-path="rustc_tests/Cargo.toml" 2> cargo_test_output.txt > /dev/null

1617 success, 558 no mir, 205 crate not found, 158 failed
1630 success, 560 no mir, 205 crate not found, 143 failed
1632 success, 561 no mir, 205 crate not found, 140 failed
1632 success, 562 no mir, 205 crate not found, 139 failed
1633 success, 563 no mir, 205 crate not found, 137 failed
1638 success, 554 no mir, 205 crate not found, 141 failed
1864 success, 211 no mir, 205 crate not found, 258 failed
1867 success, 202 no mir, 205 crate not found, 264 failed
1868 success, 202 no mir, 205 crate not found, 94 failed, 43 C fn, 9 ABI, 102 unsupported, 5 intrinsic
1878 success, 201 no mir, 205 crate not found, 58 failed, 42 C fn, 9 ABI, 121 unsupported, 5 intrinsic
1883 success, 201 no mir, 205 crate not found, 55 failed, 42 C fn, 9 ABI, 121 unsupported, 6 intrinsic
1885 success, 201 no mir, 205 crate not found, 53 failed, 42 C fn, 9 ABI, 121 unsupported, 6 intrinsic
1886 success, 201 no mir, 205 crate not found, 52 failed, 42 C fn, 9 ABI, 121 unsupported, 6 intrinsic
1902 success, 202 no mir, 206 crate not found, 60 failed, 44 C fn, 0 ABI, 122 unsupported, 6 intrinsic
1901 success, 202 no mir, 206 crate not found, 54 failed, 44 C fn, 9 ABI, 122 unsupported, 6 intrinsic
1904 success, 202 no mir, 206 crate not found, 49 failed, 44 C fn, 9 ABI, 122 unsupported, 6 intrinsic
1905 success, 202 no mir, 206 crate not found, 52 failed, 44 C fn, 9 ABI, 122 unsupported, 2 intrinsic
1907 success, 202 no mir, 206 crate not found, 52 failed, 44 C fn, 9 ABI, 122 unsupported, 0 intrinsic
1913 success, 202 no mir, 206 crate not found, 50 failed, 44 C fn, 9 ABI, 122 unsupported, 0 intrinsic
1933 success, 201 no mir, 209 crate not found, 49 failed, 38 C fn, 0 ABI, 122 unsupported, 6 intrinsic
1985 success, 206 no mir, 219 crate not found, 62 failed, 40 C fn, 0 ABI, 122 unsupported, 6 intrinsic
2016 success, 208 no mir, 225 crate not found, 57 failed, 34 C fn, 0 ABI, 123 unsupported, 10 intrinsic
2022 success, 208 no mir, 225 crate not found, 51 failed, 34 C fn, 0 ABI, 123 unsupported, 10 intrinsic
2025 success, 208 no mir, 225 crate not found, 48 failed, 34 C fn, 0 ABI, 123 unsupported, 10 intrinsic
2179 success, 2 no mir, 235 crate not found, 68 failed, 145 C fn, 0 ABI, 15 unsupported, 6 intrinsic
2179 success, 3 no mir, 235 crate not found, 73 failed, 42 C fn, 0 ABI, 112 unsupported, 6 intrinsic
2188 success, 14 no mir, 236 crate not found, 79 failed, 44 C fn, 0 ABI, 113 unsupported, 6 intrinsic
2457 success, 5 no mir, 252 crate not found, 100 failed, 44 C fn, 0 ABI, 118 unsupported, 14 intrinsic

Tracking the state in gist from now on, so we can see diffs

https://gist.github.com/oli-obk/5a0832eef3124ad9088748fc9e759318

[solson]

I have implementations for atomic_load, atomic_store, and atomic_xadd_relaxed which don't really do anything atomic, but it shouldn't matter when we don't have actual parallelism. A few others like the relaxed/non-relaxed variants and volatile_{load,store} should be just as easy.

I'll submit them soon and see how the test results change.

[oli-obk]

I implemented the missing non atomic* intrinsics. Rustc seriously needs more tests, every intrinsic should at least have a run-pass test...

[oli-obk]

use of fn(usize) -> Foo as fn(usize) -> u32
use of fn(&isize) -> Option<&isize> as fn(&isize) -> &isize

These are used when working with ffi... Should we support these cases?
I mean, any &isize can be transmuted to Option<&isize> and will be perfectly fine. Afaik this is guaranteed.

[oli-obk]

to pass more tests we need to implement panic and print, some minor bugs are left, but these make up most of our failed tests.

[eddyb]

@oli-obk Any chance you could gist the entire test output? Assuming it contains both names and messages, it would let people like me to idly rummage through some of the maybe-UB tests.

[oli-obk]

cargo_test_output.txt

see in first post for frequently updating file

[oli-obk]

https://github.com/rust-lang/rust/blob/master/src/test/run-pass/mir_raw_fat_ptr.rs#L129 compares pointers into different allocs ( same in raw_fat_ptr.rs and mir_raw_fat_ptr.rs)

[solson]

I believe we could impose an ordering between unrelated allocations, but it wouldn't really be based on anything (it would just have to be deterministic).

I've seen a C interpreter that essentially has an arbitrary deterministic ordering, if I recall correctly.

[eddyb]

@solson Not in CTFE mode, though, right?

[solson]

@eddyb No, even in CTFE mode, if we can guarantee determinism.

[oli-obk]

That's gonna expose some internals and differ depending on the mir optimizations that are run before evaluation.

[eddyb]

@solson If that determinism depends on the order things were evaluated in, that's not deterministic enough. It should be impossible for constant evaluation to observe any past interactions.
So maybe it'd make sense for "stack memory"? Not worth risking it though.

[solson]

It doesn't really expose anything. It's fine if it differs, so long as it runs the same given the same inputs (sources, flags, etc).

[solson]

@eddyb Yeah, it wouldn't work if you re-use the same uncleared Memory for different runs in some manner.

[solson]

I agree it sounds too risky for CTFE now.

[eddyb]

so long as it runs the same given the same inputs

Nope, that's not enough, the "inputs" are the MIR to evaluate and its generic parameters, not the whole compilation. I've described how getting this wrong even one bit can impact coherence and thus soundness.
Presumably there'd be some global Memory to cache results but its state should not be observable.

[oli-obk]

I updated the top post with a list of the missing MIRs and a list of all panics happening inside miri. I also attached an updated raw output file.

[solson]

@oli-obk Can you re-run and update the OP? I was having some odd problems running it, but I can investigate stuff based on your results for now.

[oli-obk]

sure

[oli-obk]

I think I broke it in 8b8c743

fixed it, report and PR coming in around an hour.

[oli-obk]

updated log

[solson]

There are a number of "invalid enum discriminant value read" results in your log that I can't reproduce. When I run those tests, they succeed.

[oli-obk]

our rustc versions and/or test suite may differ due to different checkouts. My checkout is from February 2nd. My rustc is rustc 1.16.0-nightly (24055d0f2 2017-01-31)

[solson]

rustc 1.17.0-nightly (ea7a6486a 2017-02-04) with the latest checkout here. Maybe your packed struct PR a few days ago fixed things?

[oli-obk]

I'll investigate

[oli-obk]

ok, so updating rustc and the checkout fixed these bugs, so it must be something rustc changed that fixed it and not us, because I used the same miri code base. Regenerating log...

[oli-obk]

updated log.

[oli-obk]

every NO MIR FOR ...panic... is a test failure.

     17 NO MIR FOR `std::panicking::rust_panic_with_hook`
      7 NO MIR FOR `std::rt::begin_panic_fmt`
      7 NO MIR FOR `std::panicking::panicking`

some of these are useless... like a test testing --test compileflag, which we simply ignore and try to run the main function, which simply panics.

rerunning after 1 > -1 is true again, since that fixes a few of those.

[gnzlbg]

Is there a plan for supporting inline assembly?

[solson]

@gnzlbg No.

[gnzlbg]

So there will never be a way to work around that?

[gnzlbg]

If rust code could detect that is being run at compile-time, and branch on it, it could provide a different implementation of some functionality without features that miri will probably never understand (like inline assembly, SIMD intrinsics, llvm intrinsics, etc).

[solson]

@gnzlbg Yeah, I refer to that idea as #[cfg(const)]. I guess bring it up again once we have RFCs for the more advanced const features Miri will enable.

[gnzlbg]

Cool! Something like #[cfg(const)] would be a really pragmatic hack!

[solson]

@oli-obk It'd be nice to split the "error" count in the OP into true errors and "unsupported", so things like lack of threading don't inflate the error count. I'd also prefer to have them in separate lists if possible.

[eddyb]

• #[cfg(const)] is syntactical whereas this needs to decide based on what executes it

[solson]

Good point, it wouldn't actually use cfg.

[oli-obk]

fun fact: over the last month miri has gotten faster by a factor of two. At least the rustc test suite is finished twice as fast (number of tests went up! not down). This might either be due to improvements in rustc or due to improvements in miri.

[oli-obk]

I'm running this now with full std mir enabled. We seem to have some previously unknown issues. But at least "no mir" errors have gone down to 2.

[eddyb]

@oli-obk If you look at your 2 "MIR not found" errors, I'm pretty sure the :::: is from extern {...}, so these are most likely FFI.

[RalfJung]

Is there any way to run miri on the tests (#[test]) embedded in e.g. libstd? Most of the functional tests are actually such embedded tests, not run-pass tests.

[oli-obk]

hmm... I think I implemented this in cargo miri. But I'm not sure how to run that inside the rustc build system.

[RalfJung]

Also see https://github.com/RalfJung/miri-test-libstd/issues/1