Stream: t-compiler/wg-incr-comp

Topic: "local" queries/tasks

view this post on Zulip Anton Golov (Aug 23 2021 at 13:34):


Right now, some passes over the HIR (e.g. rustc_passes::dead) operate on an entire crate. Based on a suggestion from @cjgillot , I've been looking at how to split this up to work on individual definitions instead.

In particular, we'd like to implement a way to create "queries" that would not need to be declared in query/, but would still be persisted for use in incremental computation. The interface would likely be something like tcx.make_local_task(def_id, |id| { ... }). (Perhaps the name/id of the pass would be included as well, if we can't automatically use the call site or something.)

The difficulty is in creating the dependency graph node for this task. To identify the node, we'd need to include both which make_local_task call it was created by (i.e. which pass this is) and the def_id that it was called with. I could create such a node by adding another value to DepKind, with something like (PassId, DefPath) as the key type. However, if I understand correctly it would be hashed as a single value, and I'm not sure I could recover that information: I can see how to lookup DepNode -> DefPathHash -> DefId, but I don't see a Hash -> DepNode lookup.

Would it make sense to add that and implement something like the following?

  1. Pass DefId to make_local_task(possibly together with some PassId)
  2. Convert the DefId to a DefPathId
  3. Hash (PassId, DefIdPath)
  4. If the node with this hash exists in the old graph and all its dependencies are green, no action needed.(*)
  5. Otherwise, recompute.

I'm not sure how invasive such a change would be and how complicated it would be to key the nodes by their hash.

(*) Actually, since this is for lint passes and such, we'd also like to store whether there were any lints, and force a re-run if there were.

view this post on Zulip cjgillot (Aug 23 2021 at 18:18):

You can create a custom DepKind in rustc_middle::dep_graph::dep_node. Just after CompileCodegenUnit, you can add a line: [anon] Custom((PassId, LocalDefId)) with an associated make_custom_dep_node(TyCtxt<'_>, PassId, LocalDefId) -> DepNode like make_compile_codegen_unit does. The anon flag means the query system cannot call any code with just the DepNode, which is what we want.

view this post on Zulip Anton Golov (Aug 23 2021 at 19:43):

I may be mistaken, but my impression is that this wouldn't work for the incremental behaviour we want, though? rustc_middle::dep_graph::dep_node says that anonymous queries are always recomputed. My impression is that in order to be able to see the results from the previous computation, there currently needs to be some way of recovering the key from its hash, which is done for DefPath via a hashtable, but it wouldn't be feasible to decode a (PassId, LocalDefId) hash in the same way (I think)

view this post on Zulip cjgillot (Aug 23 2021 at 20:54):

A key difference is that you know the key: it is passed as an argument to make_local_task. From there, you can compute its hash, and compare it to known hashes. In that sense, the dep-graph works as a hash table that you can query from a key's hash.

view this post on Zulip Anton Golov [they/them] (Aug 23 2021 at 20:59):

Ah, okay; then I guess there is a hash -> DepNode map and I just missed it. Thanks, I'll take another look :)

Last updated: Oct 21 2021 at 21:02 UTC