Stream: t-compiler/wg-rls-2.0

Topic: testing type inference


Florian Diebold (Apr 14 2019 at 12:17, on Zulip):

I've been thinking how to write more focused type checking tests (where just one or a few specific expressions are being tested). One thing I'd like to do is validate them against rustc, i.e. make sure that rustc infers the same type for that expression. I'm currently considering two ways:
1. a macro that defines a test: The end result would maybe look somewhat like this:

ty_test!(method_resolution_trait_before_autoderef {
    trait Trait { fn foo(self) -> u128; }
    struct S;
    impl S { fn foo(&self) -> i8 { 0 } }
    impl Trait for S { fn foo(self) -> u128 { 0 } }
    assert_type!(S.foo(), u128)
});

and it would on the one hand include that code in the actual test (with assert_type expanding to something that fails to build if the type doesn't match), and on the other hand use stringify to generate a test that runs RA's type inference on the code and checks that the asserted type is correct (with a different definition of assert_type).
2. use compiletest and test via diagnostics: We'd need to first add diagnostics for type checking errors (maybe toggleable somehow), and then we'd implement enough of compiletest to compare RA's diagnostics to the ones defined in the test file, so something like

trait Trait { fn foo(self) -> u128; }
struct S;
impl S { fn foo(&self) -> i8 { 0 } }
impl Trait for S { fn foo(self) -> u128 { 0 } }
fn test() -> i8 { S.foo() } // ~ERROR mismatched types

The nice thing about this would be that it could be used for other diagnostics as well, but the error annotations in the compiletest files would have to be written to match both rustc's and RA's diagnostics, which would probably be complicated. Also it'd make the tests somewhat dependent on rustc's error messages.

What do you think?

matklad (Apr 14 2019 at 13:23, on Zulip):

Good questions!

Having focused unit-test would be really helpful. Long run, I imagine having our own well-factored test-suite will be better than direct comparison with rustc (especially if we are aiming at merging with rustc typchecker: it would be a shame if sharing type-checking code makes all the tests void :D ). I also feel like we shouldn't rely on compiletest, at lest directly: there's going to be a huge number of tests, and any communication with compiler is bound to make them slower. So, I'd game for something like insta, but where the gold values to compare against could be generated by using rustc, once.

If we are going to have astronomical number of tests here, than maybe going full-data would be beneficial? Basically, using the same setup we use for parsing --- a .data folder with cases and gold results.

matklad (Apr 14 2019 at 13:29, on Zulip):

Do we actually get a big long-term value out of validation with rustc?

I imagine the value here is "we can confidently replace typechecker in the compiler", but I imagine we could write comparison test when actually switching the typecher (and even than, I'd imagine we'd just run both typechecker simultaneously for some time to check the results on real code).

For "making sure that type-checker works mostly correct", I feel like just writing test by hand will be faster.

What would help is a minimal corpus of rust code that exercises type-checker in an interesting way. I think reusing IntelliJ test-suite would be a smart move here, it should be pretty well factored

Florian Diebold (Apr 14 2019 at 13:32, on Zulip):

My goal isn't to directly compare to rustc, but rather to make sure that the 'gold' value is the same as in rustc (or in other words, to be able to run the same test against rustc's type checker)

matklad (Apr 14 2019 at 13:32, on Zulip):

Hm, actually, I think there could be a huge value in the compiler-independent rust-typecheker test suite!

Florian Diebold (Apr 14 2019 at 13:33, on Zulip):

so e.g. in the first example, the test defines what type is expected, but it's written in a way that it wouldn't even compile if the expected type is not the same one as rustc would infer

Florian Diebold (Apr 14 2019 at 13:34, on Zulip):

as a way to 'prove' that the expectation is actually correct ;) (since e.g. I didn't know beforehand in which order autoderefs happen vs. looking at extension methods)

matklad (Apr 14 2019 at 13:35, on Zulip):

Yeah, I see how this could be useful in various corner cases. OTOH, for, for example, autoderef one can write a test-case manually and see. That is, such cases seems pretty rare to me

matklad (Apr 14 2019 at 13:38, on Zulip):

coming back to the idea of the compiler-independent test-suite: let's say that a test is a rust-file (possibly with special mark-up, like <|> thing), and the result of the test is a human-readable representation of the type.

We than have a "driver" which feeds this into rust-analyzer, and a driver which feeds it into rustc.

By not depending on precise diagnostics, this should actually be feasible to implement

matklad (Apr 14 2019 at 13:38, on Zulip):

I guess we can invent something for non-denotable types like closures?

matklad (Apr 14 2019 at 13:39, on Zulip):

The benefit I see in this setup is that the test-suite is 100% text files, so it could be reused by other projects with minimal effort

Florian Diebold (Apr 14 2019 at 13:40, on Zulip):

I imagine it will be possible to always write tests without having to denote non-denotable types

Florian Diebold (Apr 14 2019 at 13:41, on Zulip):

(another reason is btw that I'm always annoyed that I'm writing rust code inside a string inside rust code, without any editor support ;) )

Florian Diebold (Apr 14 2019 at 13:42, on Zulip):

the tests could use an attribute like #[expected_type(i128)], we'd just have to turn that into tests compileable with rustc as well

matklad (Apr 14 2019 at 13:43, on Zulip):

cool! Yeah, a "special" mark-up could be a magic macro call!

And it could expand to let _: $ty = $expr; in rustc

matklad (Apr 14 2019 at 13:43, on Zulip):

LIke, expect_ty!(2 + 90, i32)

Florian Diebold (Apr 14 2019 at 13:44, on Zulip):

with which we're almost at my option 1 ;)

Florian Diebold (Apr 14 2019 at 13:45, on Zulip):

except with separate files

matklad (Apr 14 2019 at 13:47, on Zulip):

Yeah.... I feel like option 1 wouldn't work for all of the cases. Specifically, I think we might want to specify lang-items inside the tests

matklad (Apr 14 2019 at 13:48, on Zulip):

There's also a tradeoff:

inline test data, like in macro or in a string literal, is easier to work with

out-of-line data in the directory is easier to reuse in some yet-unknown projects

Florian Diebold (Apr 14 2019 at 13:55, on Zulip):

yeah

Florian Diebold (Apr 14 2019 at 13:56, on Zulip):

I'll write the test in the straight-forward way for now anyway, we need a bit better macro support before we can do this :)

matklad (Apr 14 2019 at 14:10, on Zulip):

FWIW, long-term, specifying tests in strings should give use more editor support than specifying them in macros

matklad (Apr 14 2019 at 14:10, on Zulip):

pasted image

matklad (Apr 14 2019 at 14:11, on Zulip):

though I am not sure how this whole "Language Injection" workflow should work in the LSP setting...

Last update: Nov 12 2019 at 16:30UTC