Stream: t-compiler/help

Topic: MIR transformations


Claire Nord (May 22 2020 at 11:58, on Zulip):

Hi all,

I'm doing research on static analysis and optimization in Rust for my Master's thesis and I'm interested in how MIR optimization / transformation works.

In particular for my project I want to perform static analysis across the entire crate, then perform some transformations. I've read some of the existing transform passes and compiler queries but I don't see an example of this. I've written my own transform passes as well, but I'm stuck on how to get some sort of entire-crate analysis result.

I've read most of sections 2 and 3 of the Rustc dev book but haven't figured out how to solve my problem.

I was wondering if someone has time to answer some questions -- it would be great to discuss this at a high level.

Thanks!

bjorn3 (May 22 2020 at 12:16, on Zulip):

but I'm stuck on how to get some sort of entire-crate analysis result.

That is a hard problem to solve using the current architecture. I can see several problems:

  1. How do you collect all functions? Some functions like shims only pop into existence during the collector, which requests already optimized MIR. While other functions are never used at all.
  2. How do you handle incremental compilation? If you analyse all functions and then make the optimized MIR depend on that analysis any change to any function would likely cause all codegened object files to be codegened again.
  3. There is currently only one optimization that supports non-local transformations: inlining. Unfortunately it has several problems. One of which is that it doesn't handle recursive functions very well. It works by requesting the already optimized mir for the callee. Doing so would cause cycles, so there is a hack to prevent inlining back edges by only inlining functions that come before the current function in the source code.

If you solve the first problem you may be able to add a new query to request the analysis and make that query depend on tcx.mir_validated. Just make sure to depend on the query for all tcx.optimized_mir calls, as any tcx.optimized_mir call before the query has analysed anything will make tcx.mir_validated for that specific function impossible.

Also may I ask what kind of analysis you want to perform?

bjorn3 (May 22 2020 at 12:17, on Zulip):

@Claire Nord ^

Claire Nord (Jun 01 2020 at 15:24, on Zulip):

@bjorn3 Thank you for your response!

  1. How do you collect all functions?

What is an example of a shim? This might not be a problem for our initial MVP. We only care about developer-defined functions in source code.

  1. How do you handle incremental compilation?

We're not really concerned about compile time, so I think it's okay to just re-compile with any change to any function.

  1. inlining

Yeah, it was helpful to read through the inlining code.

Just make sure to depend on the query for all tcx.optimized_mir calls, as any tcx.optimized_mir call before the query has analysed anything will make tcx.mir_validated for that specific function impossible.

When you say "depend on the query for all tcx.optimized_mir calls, do you mean that fn optimized_mir\(...\) source code should contain a call to my query, or that I should call my query before each time tcx.optimized_mir is called in the compiler code? Right now I'm running into a cycle by depending on tcx.mir_validated for every def_id in tcx.mir_keys(LOCAL_CRATE).

For more context, my project involves def-use analysis for every object of a certain type in a crate. I want to check every time an object of that type is allocated and used across functions, closures, etc. Once I have that global def-use information, I use that for an optimization step.

Thank you for your responses so far. Can you further explain what you had in mind for query dependencies? I'd love to get your perspective on this either in chat or voice call, if that's easier.

bjorn3 (Jun 01 2020 at 15:59, on Zulip):

@Claire Nord

What is an example of a shim? This might not be a problem for our initial MVP. We only care about developer-defined functions in source code.

A shim is a function generated by rustc to implement certain things. For example the CloneShim is used for the compiler implementation of Clone for [T; n].

See https://github.com/rust-lang/rust/blob/d3cba254e464303a6495942f3a831c2bbd7f1768/src/librustc_mir/shim.rs#L33-L112 for the full list of shims.

When you say "depend on the query for all tcx.optimized_mir calls, do you mean that fn optimized_mir\(...\) source code should contain a call to my query, or that I should call my query before each time tcx.optimized_mir is called in the compiler code? Right now I'm running into a cycle by depending on tcx.mir_validated for every def_id in tcx.mir_keys(LOCAL_CRATE).

You have to run it from fn optimized_mir(...) before calling body.steal(): https://github.com/rust-lang/rust/blob/d3cba254e464303a6495942f3a831c2bbd7f1768/src/librustc_mir/transform/mod.rs#L401

Claire Nord (Jun 01 2020 at 16:27, on Zulip):

You have to run it from fn optimized_mir(...) before calling body.steal()

Ah yes, I’m doing that. I’m running into a query cycle, though. :slight_frown:

My query basically does for def_id in tcx.mir_keys(crate_num) { tcx.subquery(def_id); }, where subquery calls tcx.mir_validated and performs def-use analysis.

The compiler detects a cycle where my first query calls itself. I’m working on getting a more verbose trace of the cycle now, but do you have an idea of why this would happen? @bjorn3

bjorn3 (Jun 01 2020 at 16:29, on Zulip):

@Claire Nord What is the query stack during the cycle error? (Run rustc with RUST_BACKTRACE=1 to show it)

Claire Nord (Jun 02 2020 at 16:55, on Zulip):
error[E0391]: cycle detected when [ performing my query ]
     |
note: ...which requires processing `unix::linux_like::linux::CPU_EQUAL`...
    --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.66/src/macros.rs:192:17
     |
192  | /                 pub unsafe extern fn $i($($arg: $argty),*
193  | |                 ) -> $ret {
194  | |                     $($body);*
195  | |                 }
     | |_________________^
     |
    ::: /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.66/src/unix/linux_like/linux/mod.rs:2307:1
     |
2307 | / f! {
2308 | |     pub fn NLA_ALIGN(len: ::c_int) -> ::c_int {
2309 | |         return ((len) + NLA_ALIGNTO - 1) & !(NLA_ALIGNTO - 1)
2310 | |     }
...    |
2406 | |     }
2407 | | }
     | |_- in this macro invocation
note: ...which requires getting validated MIR...
    --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.66/src/macros.rs:192:17
     |
192  | /                 pub unsafe extern fn $i($($arg: $argty),*
193  | |                 ) -> $ret {
194  | |                     $($body);*
195  | |                 }
     | |_________________^
     |
    ::: /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.66/src/unix/linux_like/linux/mod.rs:2307:1
     |
2307 | / f! {
2308 | |     pub fn NLA_ALIGN(len: ::c_int) -> ::c_int {
2309 | |         return ((len) + NLA_ALIGNTO - 1) & !(NLA_ALIGNTO - 1)
2310 | |     }
...    |
2406 | |     }
2407 | | }
     | |_- in this macro invocation
note: ...which requires getting pre-const evaluation MIR...
    --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.66/src/macros.rs:192:17
     |
192  | /                 pub unsafe extern fn $i($($arg: $argty),*
193  | |                 ) -> $ret {
194  | |                     $($body);*
195  | |                 }
     | |_________________^
     |
    ::: /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.66/src/unix/linux_like/linux/mod.rs:2307:1
     |
2307 | / f! {
2308 | |     pub fn NLA_ALIGN(len: ::c_int) -> ::c_int {
2309 | |         return ((len) + NLA_ALIGNTO - 1) & !(NLA_ALIGNTO - 1)
2310 | |     }
...    |
2406 | |     }
2407 | | }
     | |_- in this macro invocation
note: ...which requires processing `unix::linux_like::linux::CPU_EQUAL`...
    --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.66/src/macros.rs:192:17
     |
192  | /                 pub unsafe extern fn $i($($arg: $argty),*
193  | |                 ) -> $ret {
194  | |                     $($body);*
195  | |                 }
     | |_________________^
     |
    ::: /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.66/src/unix/linux_like/linux/mod.rs:2307:1
     |
2307 | / f! {
2308 | |     pub fn NLA_ALIGN(len: ::c_int) -> ::c_int {
2309 | |         return ((len) + NLA_ALIGNTO - 1) & !(NLA_ALIGNTO - 1)
2310 | |     }
...    |
2406 | |     }
2407 | | }
     | |_- in this macro invocation
note: ...which requires processing `unix::linux_like::linux::CPU_EQUAL`...
    --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.66/src/macros.rs:192:17
     |
192  | /                 pub unsafe extern fn $i($($arg: $argty),*
193  | |                 ) -> $ret {
194  | |                     $($body);*
195  | |                 }
     | |_________________^
     |
    ::: /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.66/src/unix/linux_like/linux/mod.rs:2307:1
     |
2307 | / f! {
2308 | |     pub fn NLA_ALIGN(len: ::c_int) -> ::c_int {
2309 | |         return ((len) + NLA_ALIGNTO - 1) & !(NLA_ALIGNTO - 1)
2310 | |     }
...    |
2406 | |     }
2407 | | }
     | |_- in this macro invocation
note: ...which requires processing `unix::linux_like::linux::CPU_EQUAL`...
    --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.66/src/macros.rs:192:17
     |
192  | /                 pub unsafe extern fn $i($($arg: $argty),*
193  | |                 ) -> $ret {
194  | |                     $($body);*
195  | |                 }
     | |_________________^
     |
    ::: /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.66/src/unix/linux_like/linux/mod.rs:2307:1
     |
2307 | / f! {
2308 | |     pub fn NLA_ALIGN(len: ::c_int) -> ::c_int {
2309 | |         return ((len) + NLA_ALIGNTO - 1) & !(NLA_ALIGNTO - 1)
2310 | |     }
...    |
2406 | |     }
2407 | | }
     | |_- in this macro invocation
note: ...which requires const-evaluating + checking `unix::linux_like::linux::cpu_set_t::bits::{{constant}}#0`...
    --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.66/src/unix/linux_like/linux/mod.rs:148:21
     |
148  |         bits: [u64; 16],
     |                     ^^
note: ...which requires const-evaluating + checking `unix::linux_like::linux::cpu_set_t::bits::{{constant}}#0`...
    --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.66/src/unix/linux_like/linux/mod.rs:148:21
     |
148  |         bits: [u64; 16],
     |                     ^^
note: ...which requires const-evaluating `unix::linux_like::linux::cpu_set_t::bits::{{constant}}#0`...
    --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.66/src/unix/linux_like/linux/mod.rs:148:21
     |
148  |         bits: [u64; 16],
     |                     ^^
note: ...which requires getting optimized MIR...
    --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.66/src/unix/linux_like/linux/mod.rs:148:21
     |
148  |         bits: [u64; 16],
     |                     ^^
     = note: ...which again requires [ performing my query ], completing the cycle
note: cycle used when getting optimized MIR
    --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.66/src/unix/linux_like/mod.rs:35:33
     |
35   |         pub sa_data: [::c_char; 14],
     |                                 ^^

error: aborting due to previous error

For more information about this error, try `rustc --explain E0391`.
error: could not compile `libc`.

@bjorn3 my query is [ performing my query ], and the other descriptions should be intuitive (I modified them because I couldn't figure out how to set tcx.sess.opts.debugging_opts.verbose -- it's not cargo -vv, is it?).

lcnr (Jun 02 2020 at 16:56, on Zulip):

The query names are printed when using RUST_BACKTRACE=1 afaik

Claire Nord (Jun 02 2020 at 16:59, on Zulip):

@lcnr hm, the command I ran was RUSTC_LOG=rustc_mir::transform=debug RUST_BACKTRACE=1 cargo +stage1 build -vv |& tee out.log

lcnr (Jun 02 2020 at 17:01, on Zulip):

What is the result of RUSTC_LOG=rustc_mir::transform=debug RUST_BACKTRACE=1 cargo +stage1 build -vv --rustc-args -Ztreat-err-as-bug |& tee out.log

bjorn3 (Jun 02 2020 at 17:02, on Zulip):

I think the problem here is that const evaluated code is also optimized. This however can run during typechecking, which is required to generate MIR for other functions, thus causing a cycle.

Claire Nord (Jun 02 2020 at 17:03, on Zulip):

@lcnr

$ RUSTC_LOG=rustc_mir::transform=debug RUST_BACKTRACE=1 cargo +stage1 build -vv --rustc-args -Ztreat-err-as-bug  |& tee out.log
error: Found argument '--rustc-args' which wasn't expected, or isn't valid in this context

USAGE:
    cargo build --verbose

For more information try --help
lcnr (Jun 02 2020 at 17:06, on Zulip):

RUSTC_LOG=rustc_mir::transform=debug RUST_BACKTRACE=1 cargo +stage1 rustc -- -Ztreat-err-as-bug |& tee out.log

lcnr (Jun 02 2020 at 17:07, on Zulip):

mixed up cargo and x.py arguments

Claire Nord (Jun 02 2020 at 17:22, on Zulip):

@lcnr

$ RUSTC_LOG=rustc_mir::transform=debug RUST_BACKTRACE=1 cargo +stage1 rustc -- -Ztreat-err-as-bug --bin txcell |& tee out.log
error: extra arguments to `rustc` can only be passed to one target, consider filtering
the package by passing, e.g., `--lib` or `--bin NAME` to specify a single target
$ RUSTC_LOG=rustc_mir::transform=debug RUST_BACKTRACE=1 cargo +stage1 rustc --lib libc -- -Ztreat-err-as-bug |& tee out.log
... same output as before, e.g.
note: ...which requires processing `x86_64::<impl at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs:188:1: 195:2>::new128`...

As of 2 days ago this should be okay because of 8894bd220b82de486f2a8aecec6753c1b416b1f2, though? I'm on an older version of rustc.

Claire Nord (Jun 02 2020 at 17:24, on Zulip):

bjorn3 said:

I think the problem here is that const evaluated code is also optimized. This however can run during typechecking, which is required to generate MIR for other functions, thus causing a cycle.

Ah, I see; thank you so much for pointing this out. Do you have advice on how to prevent this cycle?

bjorn3 (Jun 02 2020 at 17:26, on Zulip):

Not really, maybe @oli?

oli (Jun 02 2020 at 17:28, on Zulip):

I don't think there's any way you can reasonably call a complex query on all items from another query without running into cycles

oli (Jun 02 2020 at 17:30, on Zulip):

why do you need to run it on all items? Can't you just call it on the item at hand and thus get it cached?

oli (Jun 02 2020 at 17:30, on Zulip):

and call it on all items from somwhere else?

oli (Jun 02 2020 at 17:31, on Zulip):

sorry I haven't read the entire thread here, doing that now

oli (Jun 02 2020 at 17:32, on Zulip):

ah

oli (Jun 02 2020 at 17:32, on Zulip):

you want to use the result of a whole program analysis to optimize specific functions

oli (Jun 02 2020 at 17:32, on Zulip):

uuuuh XD

oli (Jun 02 2020 at 17:32, on Zulip):

I don't think the query system is up for that

oli (Jun 02 2020 at 17:33, on Zulip):

think about it this way: you created a dependency for the optimization of a single function on all other functions in the crate

oli (Jun 02 2020 at 17:33, on Zulip):

but the query system expects only functions to have a dependency on other functions if they (transitively) use it

oli (Jun 02 2020 at 17:34, on Zulip):

so one of these other functions gets type checked. As already noted above, type checking involves array lengths and thus constants

oli (Jun 02 2020 at 17:34, on Zulip):

constants can do a lot of stuff, including: function calls

oli (Jun 02 2020 at 17:34, on Zulip):

so if that constant wants to call your function... kaboom

Claire Nord (Jun 02 2020 at 17:34, on Zulip):

Darn, I see. Yeah, that makes sense...

oli (Jun 02 2020 at 17:35, on Zulip):

now... let's go back to the drawing board and figure out whether you can have your cake and eat it, too

oli (Jun 02 2020 at 17:37, on Zulip):

the query system expects there to be no dependency edge between functions that have nothing to do with each other. so a query processing fn foo() {} and cannot by definition ever call a query that processes fn bar() {}. One super notable exception is type Foo = impl Trait; fn foo() -> Foo {...} fn bar() -> foo {...}.

oli (Jun 02 2020 at 17:37, on Zulip):

I don't remember the exact algorithm there, but it was taken a lot of care that we never create a dependency edge that would get us into trouble

oli (Jun 02 2020 at 17:37, on Zulip):

now... for the actual thing you want to analyze, will fn foo() {}'s MIR ever influence the optimization of fn bar() {}?

oli (Jun 02 2020 at 17:38, on Zulip):

or can you judge whether you need a dependency edge by looking at e.g. the signature of fn foo() while optimizing fn bar()?

oli (Jun 02 2020 at 17:39, on Zulip):

note: I'm on and off rn, no promises I'll respond until tomorrow

Claire Nord (Jun 02 2020 at 17:53, on Zulip):

Yes, fn foo() {}'s MIR can influence the optimization of fn bar() {}.

Claire Nord (Jun 02 2020 at 17:54, on Zulip):

Oh wait, let me think about whether the signature is sufficient with some assumptions...

oli (Jun 03 2020 at 07:05, on Zulip):

if it is, you can make the decision to fetch the more advanced data of the function by first fetching the signature

oli (Jun 03 2020 at 07:06, on Zulip):

if everything fails, you could also do all this work post-"mir-optimized" by adding another query "mir-codegen" which processes the optimized MIR for codegen and thus has no query edges from earlier steps to it

Claire Nord (Jun 03 2020 at 15:35, on Zulip):

Oh wow, the latter seems promising. I'd still be able to transform the MIR or LLVM IR in this query, right? Can you point me to any existing post-tcx.optimized_mir transformations?

How does this break the cycle? Is it that everything has already been generated, so my query cannot create additional work that requires my query?

Claire Nord (Jun 03 2020 at 15:43, on Zulip):

oli said:

if it is, you can make the decision to fetch the more advanced data of the function by first fetching the signature

I thought about it more and concluded that the function signature alone is not sufficient.

oli (Jun 03 2020 at 16:32, on Zulip):

There aren't any post-optimized_mir transformations. You'd have to create all relevant changes for this and in a way that it won't impact the regular compiler negatively if you want to upstream those changes. You could create a query that returns a Cow<'tcx, mir::Body<'tcx>> and ensure that all sites that use it in codegen call that query instead of optimized_mir. Then, when you build your own compiler driver, you can override that query with one that clones all the MIRs that you want to modify, modifies them and returns the modified MIR. The regular compiler would just return a Cow::Borrowed so the additional query would be very cheap (especially since only codegen uses it). If you want to do such a change, I suggest you write an MCP (major change proposal) for the compiler team.

oli (Jun 03 2020 at 16:32, on Zulip):

There may be other/better ways, but I can't think of any off the top of my head

lcnr (Jun 03 2020 at 16:34, on Zulip):

Only somewhat related, but how big are the perf costs of not using Steal for mir_built?

oli (Jun 03 2020 at 16:35, on Zulip):

probably very high

oli (Jun 03 2020 at 16:35, on Zulip):

ok

oli (Jun 03 2020 at 16:35, on Zulip):

hehe

oli (Jun 03 2020 at 16:35, on Zulip):

that's an idea

oli (Jun 03 2020 at 16:35, on Zulip):

you can not modify anything

oli (Jun 03 2020 at 16:35, on Zulip):

but make your driver replace optimized_mir and make it clone mir_validated instead of stealing it

oli (Jun 03 2020 at 16:35, on Zulip):

(and then invoke the original optimized_mir)

bjorn3 (Jun 03 2020 at 16:35, on Zulip):

You also have to take into account that that would increase max-rss due to more mir::Body instances being stored.

oli (Jun 03 2020 at 16:36, on Zulip):

I'm assuming all of this is just for research, the compiler will never be able to support such a scheme as far as I can tell

oli (Jun 03 2020 at 16:38, on Zulip):

hmm... at which point, you may just want to change your custom driver to run all optimized_mir queries, then replace the optimized_mir query with your own, which injects your optimization, and only then run codegen

bjorn3 (Jun 03 2020 at 16:39, on Zulip):

Then the queries won't run again. They are cached.

oli (Jun 03 2020 at 16:39, on Zulip):

oh right

oli (Jun 03 2020 at 16:39, on Zulip):

yea idk, this optimization goes so much against the query system...

bjorn3 (Jun 03 2020 at 16:39, on Zulip):

Also I don't think there is any public api to change the Providers struct after construction of TyCtxt.

Claire Nord (Jun 03 2020 at 23:14, on Zulip):

create all relevant changes for this and in a way that it won't impact the regular compiler negatively if you want to upstream those changes

If you want to do such a change, I suggest you write an MCP (major change proposal) for the compiler team.

the compiler will never be able to support such a scheme as far as I can tell

Yeah, this is just for research, so I'm not planning on upstreaming.

Claire Nord (Jun 04 2020 at 16:50, on Zulip):

oli said:

but make your driver replace optimized_mir and make it clone mir_validated instead of stealing it

I'd like to do whatever's easiest to get working, so this approach seems most promising :sweat_smile:

To clarify, do you mean removing Steal entirely from mir_built, mir_const, and mir_validated and using clone() instead? Would the return value for each of these queries then be &'tcx mir::Body<'tcx> rather than &'tcx Steal<mir::Body<'tcx>>?

oli (Jun 04 2020 at 16:53, on Zulip):

no, just keep the steal there

oli (Jun 04 2020 at 16:54, on Zulip):

modify as little as possible

oli (Jun 04 2020 at 16:54, on Zulip):

just remove the .steal() call in optimized_mir

oli (Jun 04 2020 at 16:54, on Zulip):

and replace it with .clone() (may need some derefs)

Claire Nord (Jun 04 2020 at 16:59, on Zulip):

Hm, but mir_validated returns &'tcx Steal<mir::Body<'tcx>>, so wouldn't the .clone()d value still be wrapped in the Steal? run_optimization_passes requires &mut Body without the Steal.

bjorn3 (Jun 04 2020 at 16:59, on Zulip):

(*mir).clone()?

Claire Nord (Jun 04 2020 at 17:02, on Zulip):

The specific line is here.

Hm, @bjorn3 it says "type rustc::ty::steal::Steal<rustc::mir::Body<'_>> cannot be dereferenced".

bjorn3 (Jun 04 2020 at 17:03, on Zulip):

(**mir.borrow()).clone()

Claire Nord (Jun 04 2020 at 17:07, on Zulip):

Trying this out now. (**body.borrow()).clone(); says "type rustc::mir::Body<'_> cannot be dereferenced", so I'm trying (*body.borrow()).clone();, which I think works.

Just to clarify, how does this resolve the cycle?

Claire Nord (Jun 04 2020 at 17:07, on Zulip):

Ahaha, yes.

Claire Nord (Jun 04 2020 at 17:23, on Zulip):

Just rebuilt the compiler and still ran into the cycle error -- cloning alone won't solve this, right?

Claire Nord (Jun 04 2020 at 17:23, on Zulip):

Was there a separate recommendation that I missed?

Claire Nord (Jun 05 2020 at 01:15, on Zulip):

@oli could you clarify how you meant cloning instead of stealing would prevent the cycle?

oli (Jun 05 2020 at 06:26, on Zulip):

You can now access mir_validated from anywhere, without worrying that it's result will disappear. So every single mir_optimized call cann now invoke all other mir_validated. This by itself will not break the cycle you are seeing, but since the only cycle you are seeing is through constants, you can now do an eager check before calling mir_validated on all items, and only do that if the current optimized_mir query's DefId is not a constant

oli (Jun 05 2020 at 06:28, on Zulip):

hmm... this may still invoke const fn cyclically, because the constants can call const fn and we may currently be in a const fn. So I guess you need to exclude const fn, from your optimizations, too

Claire Nord (Jun 09 2020 at 15:36, on Zulip):

Ah I see -- using tcx.is_const_fn to check that now

Claire Nord (Jun 09 2020 at 16:13, on Zulip):

oli said:

hmm... this may still invoke const fn cyclically, because the constants can call const fn and we may currently be in a const fn. So I guess you need to exclude const fn, from your optimizations, too

Hm, I see. Is this what's going on in the following cycle? Now I only call tcx.mir_validated(def_id) if !tcx.is_const_fn(def_id), but still encounter this cycle.

Thank you so much for all your help so far! It's really helpful to talk to someone who understands the existing query graph.

error[E0391]: cycle detected when [ performing my query ]
    |
note: ...which requires processing `x86_64::<impl at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs:188:1: 195:2>::new128`...
   --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs:189:5
    |
189 |     pub fn new128(xs: [vec128_storage; 4]) -> Self {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...which requires getting validated MIR...
   --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs:189:5
    |
189 |     pub fn new128(xs: [vec128_storage; 4]) -> Self {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...which requires getting pre-const evaluation MIR...
   --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs:189:5
    |
189 |     pub fn new128(xs: [vec128_storage; 4]) -> Self {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...which requires processing `x86_64::<impl at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs:188:1: 195:2>::new128`...
   --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs:189:5
    |
189 |     pub fn new128(xs: [vec128_storage; 4]) -> Self {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...which requires processing `x86_64::<impl at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs:188:1: 195:2>::new128`...
   --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs:189:5
    |
189 |     pub fn new128(xs: [vec128_storage; 4]) -> Self {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...which requires processing `x86_64::<impl at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs:188:1: 195:2>::new128`...
   --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs:189:5
    |
189 |     pub fn new128(xs: [vec128_storage; 4]) -> Self {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...which requires const-evaluating + checking `x86_64::<impl at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs:188:1: 195:2>::new128::{{constant}}#0`...
   --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs:189:40
    |
189 |     pub fn new128(xs: [vec128_storage; 4]) -> Self {
    |                                        ^
note: ...which requires const-evaluating + checking `x86_64::<impl at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs:188:1: 195:2>::new128::{{constant}}#0`...
   --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs:189:40
    |
189 |     pub fn new128(xs: [vec128_storage; 4]) -> Self {
    |                                        ^
note: ...which requires const-evaluating `x86_64::<impl at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs:188:1: 195:2>::new128::{{constant}}#0`...
   --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs:189:40
    |
189 |     pub fn new128(xs: [vec128_storage; 4]) -> Self {
    |                                        ^
note: ...which requires getting optimized MIR...
   --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs:189:40
    |
189 |     pub fn new128(xs: [vec128_storage; 4]) -> Self {
    |                                        ^
    = note: ...which again requires [ performing my query ], completing the cycle
note: cycle used when getting optimized MIR
   --> /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs:111:18
    |
111 |     u32x4: [u32; 4],
    |                  ^
Claire Nord (Jun 09 2020 at 16:29, on Zulip):

Also is this what you mean by const fn? I think it's fine to exclude them from my analysis, then.

Claire Nord (Jun 09 2020 at 16:41, on Zulip):

Oh wait, if tcx.is_const_fn checks for const fn, how do you check if a DefId is a constant otherwise? I think I misunderstood your first message.

Claire Nord (Jun 09 2020 at 16:44, on Zulip):

I've seen Operand::Constant and ty::Const, but not sure what you mean by a DefId being a constant.

Claire Nord (Jun 09 2020 at 17:02, on Zulip):

Oh, I see now that the cycle starts with DefId(0:1545 ~ ppv_lite86[f354]::x86_64[0]::vec128_storage[0]::u32x4[0]::{{constant}}[0]), which has {{constant}} in it...

Claire Nord (Jun 09 2020 at 17:04, on Zulip):

And I see that the {{constant}} comes from AnonConst in DefPathData

bjorn3 (Jun 09 2020 at 17:07, on Zulip):

You could match on tcx.def_kind(def_id) and then check for DefKind::Const | DefKind::AssocConst, like here:

https://doc.rust-lang.org/nightly/nightly-rustc/src/rustc_mir/const_eval/eval_queries.rs.html#337

Claire Nord (Jun 09 2020 at 17:15, on Zulip):

Ah great, thank you :)

Claire Nord (Jun 09 2020 at 17:44, on Zulip):

Oh, tcx.def_kind of DefId(0:1545 ~ ppv_lite86[f354]::x86_64[0]::vec128_storage[0]::u32x4[0]::{{constant}}[0]) is actually None.

Claire Nord (Jun 09 2020 at 17:45, on Zulip):

I'll play around with eager checks using tcx.def_kind to decide when to actually run my query.

Claire Nord (Jun 09 2020 at 18:44, on Zulip):

Eager checking with tcx.def_kind prevents the cycle :grinning_face_with_smiling_eyes:

Claire Nord (Jun 17 2020 at 17:02, on Zulip):

My research project is working now :grinning_face_with_smiling_eyes: thank you so much @.oli @.bjorn3 @.lcnr for working with me so patiently!

Last update: Jan 22 2021 at 13:15UTC