Stream: t-compiler

Topic: design meeting 2019-10-11


nikomatsakis (Oct 11 2019 at 13:23, on Zulip):

Dear @T-compiler/meeting,

Today we will be having a design meeting. The topic was originally sketched as "some Zoxc PR". We've since narrowed that down to discuss #62038, which is a refactoring to how dep-graph loading occurs. @Zoxc wrote up a comment giving a summary of the ideas. Note that this PR itself is an incremental step towards #60035, which aims to make dep-graph loading/saving more continuous.

I'd also like to discuss briefly how we should document these changes. We currently have some rustc-chapters on incremental compilation (e.g., this chapter goes into detail). I would like to move us to a world where major refactorings like #60035 (but not limited to this one -- I think e.g. my recent PR and work on lazy norm fits the bill) come along with a rustc-guide chapter that documents the new state of the world. Maybe we discuss some how that might work and -- in the case of THIS PR -- who might do that documentation work (I don't necessarily think it has to be @Zoxc, though they're also an obvious candidate). (In my ideal world, drafts of that chapter would be available before the PR, but at minimum I think such a chapter should be in place to help with reviewing.)

nikomatsakis (Oct 11 2019 at 14:01, on Zulip):

Hey @T-compiler/meeting! Meeting starting in a few minutes. As you arrive, please click on the :wave: emoji so we have some idea who is here today.

Announcements

Zoxc (Oct 11 2019 at 14:01, on Zulip):

The A Tale Of Two DepGraphs: The Old And The New part you linked could be updated, but it seems like the rest of the docs don't go enough into details for this PR to matter.

mw (Oct 11 2019 at 14:02, on Zulip):

I think updating that section would suffice, yes

nikomatsakis (Oct 11 2019 at 14:02, on Zulip):

(In general, I definitely think there's a bootstrapping problem that the old docs don't exist, which makes writing the first round of new docs harder.)

nikomatsakis (Oct 11 2019 at 14:03, on Zulip):

Announcement: @davidtwco wrote an awesome blog post on the recent diagnostic work they've been working on.

nikomatsakis (Oct 11 2019 at 14:05, on Zulip):

Here is a hackmd document that we can use to document and guide this conversation -- at the end, it will hopefully also serve as the minutes.

nikomatsakis (Oct 11 2019 at 14:05, on Zulip):

I meant to do this before but I got caught up in other crap -- but maybe use a few minutes to jot down questions or thoughts, if you've had a chance to peruse the links above

Santiago Pastorino (Oct 11 2019 at 14:07, on Zulip):

Announcement: davidtwco wrote an awesome blog post on the recent diagnostic work they've been working on.

wonder how much of that may end in rustc-guide :)

nikomatsakis (Oct 11 2019 at 14:09, on Zulip):

OK, let's get started?

centril (Oct 11 2019 at 14:09, on Zulip):

was just gonna say :P

nikomatsakis (Oct 11 2019 at 14:09, on Zulip):

There are some notes happening in the hackmd too :)

nikomatsakis (Oct 11 2019 at 14:10, on Zulip):

(I advise you to tile your windows if possible :)

nikomatsakis (Oct 11 2019 at 14:10, on Zulip):

So, maybe we can start by talking a bit about how the PR works?

nikomatsakis (Oct 11 2019 at 14:11, on Zulip):

So as @Zoxc wrote, we currently load he entire previous dep-graph into memory

nikomatsakis (Oct 11 2019 at 14:11, on Zulip):

I have a few questions to refresh my memory

nikomatsakis (Oct 11 2019 at 14:11, on Zulip):

About existing system

nikomatsakis (Oct 11 2019 at 14:12, on Zulip):

Basically, when we start a query, I think we go check against this old graph to see if we can re-use the results, right?

nikomatsakis (Oct 11 2019 at 14:12, on Zulip):

If so, we copy into the new graph that we are building up for the current session?

mw (Oct 11 2019 at 14:12, on Zulip):

yes

nikomatsakis (Oct 11 2019 at 14:13, on Zulip):

We have also this color scheme -- green nodes being ones whose results were unchanged from last time etc -- but that is only relevant to the new graph, right?

Zoxc (Oct 11 2019 at 14:13, on Zulip):

That's only for the old graph

nikomatsakis (Oct 11 2019 at 14:14, on Zulip):

Hmm, I guess that surprises me, but likely i'm confused :)

nikomatsakis (Oct 11 2019 at 14:14, on Zulip):

What I think happens when we start a query is something like this:

nikomatsakis (Oct 11 2019 at 14:15, on Zulip):
mw (Oct 11 2019 at 14:15, on Zulip):

yes, when you ask if node X is green or red, and node X did not exist in the old graph, then you don't have to look any further

nikomatsakis (Oct 11 2019 at 14:15, on Zulip):
Zoxc (Oct 11 2019 at 14:15, on Zulip):

The green color means that the result of a query is the same in the previous graph as the current one. If the query doesn't exist in the previous graph, it has no color.

nikomatsakis (Oct 11 2019 at 14:15, on Zulip):
nikomatsakis (Oct 11 2019 at 14:16, on Zulip):

Right, ok, it sounds like we're roughly saying the same thing. So the color is a property of the new graph. Nobody cares what color it had in the old graph.

nikomatsakis (Oct 11 2019 at 14:16, on Zulip):

(Sound right?)

mw (Oct 11 2019 at 14:16, on Zulip):

@nikomatsakis almost: we don;t add grey nodes. rather we don't do anything with the current node yet

nikomatsakis (Oct 11 2019 at 14:16, on Zulip):

OK.

nikomatsakis (Oct 11 2019 at 14:17, on Zulip):

/me wonders about parallelism

nikomatsakis (Oct 11 2019 at 14:17, on Zulip):

ah I guess that's handled separately

mw (Oct 11 2019 at 14:17, on Zulip):

marking is racy, but deterministic currently

nikomatsakis (Oct 11 2019 at 14:17, on Zulip):

i.e., the query mechanism intercepts parallel attempts to compute same value earlier

mw (Oct 11 2019 at 14:17, on Zulip):

I think

Zoxc (Oct 11 2019 at 14:17, on Zulip):

Colors are comparisons between the new and old graph and don't get stored on disk

nikomatsakis (Oct 11 2019 at 14:17, on Zulip):

ok, let's leave parallelism out of it for a second

nikomatsakis (Oct 11 2019 at 14:18, on Zulip):

I have another question:

This means that the ids in the previous dep graph and current dep graph are distinct.

nikomatsakis (Oct 11 2019 at 14:18, on Zulip):

These ids you are referring to, @Zoxc, are they the DepGraphNodeIndex values?

nikomatsakis (Oct 11 2019 at 14:19, on Zulip):

To give those following along some context:

mw (Oct 11 2019 at 14:20, on Zulip):

These ids you are referring to, @Zoxc, are they the DepGraphNodeIndex values?

yes. DepGraphNodeIndex -> DepNodeIndex

nikomatsakis (Oct 11 2019 at 14:20, on Zulip):

er, ok

nikomatsakis (Oct 11 2019 at 14:20, on Zulip):

OK, so in the new PR, let's maybe walk through the editing process a bit?

nikomatsakis (Oct 11 2019 at 14:21, on Zulip):

Basically, the change is to load up the old graph and use it as the starting point, editing it in place, rather than copying things out from it into the new graph (right?)

Zoxc (Oct 11 2019 at 14:21, on Zulip):

Yeah. That's the key idea.

nikomatsakis (Oct 11 2019 at 14:22, on Zulip):

Do we ever delete nodes?

nikomatsakis (Oct 11 2019 at 14:22, on Zulip):

Like, if we go to evaluate some query, and we find it had a correpsonding node, but that node has a red input

nikomatsakis (Oct 11 2019 at 14:22, on Zulip):

We will re-execute the query, producing a result -- this result may have distinct edges from before, I guess

nikomatsakis (Oct 11 2019 at 14:22, on Zulip):

But we keep the same DepNode index, we just adjust its set of edges in place?

Zoxc (Oct 11 2019 at 14:23, on Zulip):

No, we only overwrite nodes with new dependencies.

Zoxc (Oct 11 2019 at 14:23, on Zulip):

My PR will delete nodes at the end of the compiler, but that's the only way nodes are removed.

nikomatsakis (Oct 11 2019 at 14:24, on Zulip):

The main scenario (at present) that I think something could need to be deleted is:

Zoxc (Oct 11 2019 at 14:24, on Zulip):

As in queries which did not get executed are removed from the dep graph. In the current state of the compiler, these wouldn't get copied to the new dep graph from the old.

nikomatsakis (Oct 11 2019 at 14:25, on Zulip):

OK, so we do a final sweep over the graph

nikomatsakis (Oct 11 2019 at 14:25, on Zulip):

Checking which queries were used this time

nikomatsakis (Oct 11 2019 at 14:25, on Zulip):

and deleting those that were not?

nikomatsakis (Oct 11 2019 at 14:25, on Zulip):

what about so-called "anonymous nodes"?

mw (Oct 11 2019 at 14:26, on Zulip):

everything that got used is either red or green, right? (including anon nodes)

mw (Oct 11 2019 at 14:26, on Zulip):

the rest is "unknown"

mw (Oct 11 2019 at 14:26, on Zulip):

and we should get rid of all excess nodes by deleting all the unknown ones

mw (Oct 11 2019 at 14:27, on Zulip):

(which is true only under the assumption that we did a full compilation that touched everything)

nikomatsakis (Oct 11 2019 at 14:27, on Zulip):

ok, so we now have a use for the "grey" color :P

mw (Oct 11 2019 at 14:27, on Zulip):

yes, in the PR, the current graph starts out as a grey copy of the old graph

nikomatsakis (Oct 11 2019 at 14:27, on Zulip):

however, I take it that when we delete nodes, we don't have any mechanism to "re-use" the old indices

nikomatsakis (Oct 11 2019 at 14:28, on Zulip):

this is presumably the "index leak" that @Zoxc was referring to?

Zoxc (Oct 11 2019 at 14:28, on Zulip):

The PR stores the deletes ids to disk to be reused.

nikomatsakis (Oct 11 2019 at 14:28, on Zulip):

OK. So the main problem is that the "high water mark" never goes down?

mw (Oct 11 2019 at 14:28, on Zulip):

so is there still a drift?

nikomatsakis (Oct 11 2019 at 14:28, on Zulip):

plus fragmentation

nikomatsakis (Oct 11 2019 at 14:28, on Zulip):

i.e., maybe they are scattered through the "index space"

Zoxc (Oct 11 2019 at 14:29, on Zulip):

The problem is when you have a crate with a lot of ids, which then could have less ids in the future.

nikomatsakis (Oct 11 2019 at 14:29, on Zulip):

yeah, that's what I was referring to by "high water mark"

Zoxc (Oct 11 2019 at 14:29, on Zulip):

Usually crates grow larger though, but there could be like an edge case with recursive types, etc. that crates a lot of ids due to an error.

mw (Oct 11 2019 at 14:30, on Zulip):

are the DepNodeIndex values visible outside the graph?

nikomatsakis (Oct 11 2019 at 14:30, on Zulip):

Presently we dont' store the dep-graph on error

nikomatsakis (Oct 11 2019 at 14:30, on Zulip):

(But obviously we'd like to change that eventually)

mw (Oct 11 2019 at 14:30, on Zulip):

i.e. are we free to re-assign them during serialization if fragmentation gets too high?

nikomatsakis (Oct 11 2019 at 14:31, on Zulip):

it seems like if we are doing a "remove old node" sweep

nikomatsakis (Oct 11 2019 at 14:31, on Zulip):

it's not hard to imagine a "compression step" as well

nikomatsakis (Oct 11 2019 at 14:31, on Zulip):

not saying it has to block this PR, but there seems to be room to fix that in the future if we had a problem

nikomatsakis (Oct 11 2019 at 14:31, on Zulip):

(is there any fundamental reason that is hard?)

nikomatsakis (Oct 11 2019 at 14:31, on Zulip):

i.e. are we free to re-assign them during serialization if fragmentation gets too high?

basically this, yes

mw (Oct 11 2019 at 14:31, on Zulip):

I'm trying to remember if the query result cache uses these indices as keys or something

nikomatsakis (Oct 11 2019 at 14:32, on Zulip):

are the DepNodeIndex values visible outside the graph?

@Zoxc, do you recall :point_up:

Zoxc (Oct 11 2019 at 14:32, on Zulip):

The query result cache does indeed use them as keys.

nikomatsakis (Oct 11 2019 at 14:32, on Zulip):

ok I confess I don't really remember anything about query result cache

Zoxc (Oct 11 2019 at 14:32, on Zulip):

So if we want to garbage collect them, it must be done in sync with the query result cache.

nikomatsakis (Oct 11 2019 at 14:33, on Zulip):

presumably this is stored on the disk ?

mw (Oct 11 2019 at 14:33, on Zulip):

well, right now the cache is entirely re-written anyway :)

nikomatsakis (Oct 11 2019 at 14:33, on Zulip):

So if we want to garbage collect them, it must be done in sync with the query result cache.

yeah, I guess the question is "is that hard?"

mw (Oct 11 2019 at 14:33, on Zulip):

compression would have to record the mapping

Zoxc (Oct 11 2019 at 14:34, on Zulip):

I think the query result cache will require garbage collection anyway though, for it to be incremental (I have a branch with that), since it uses interned recursive stuff like types.

mw (Oct 11 2019 at 14:34, on Zulip):

and cache serialization would map on the fly during serialization?

nikomatsakis (Oct 11 2019 at 14:35, on Zulip):

well, right now the cache is entirely re-written anyway :)

sorry I misunderstood this -- you mean "the compiler re-writes the cache after every exection"

nikomatsakis (Oct 11 2019 at 14:35, on Zulip):

I thought you meant "it has been fully refactored" or something

Zoxc (Oct 11 2019 at 14:35, on Zulip):

We'd probably want to garbage collect all the incremental state either before loading it or after compilation. Doing it on-the-fly is going to make the query code messier.

nikomatsakis (Oct 11 2019 at 14:35, on Zulip):

yes please :)

mw (Oct 11 2019 at 14:36, on Zulip):

with on-the-fly I mean during serialization

mw (Oct 11 2019 at 14:36, on Zulip):

instead of going into the query tables and rewriting there before serialization

nikomatsakis (Oct 11 2019 at 14:37, on Zulip):

(ps, for those following the hackmd, I'm finding lots of weird bugs owing to it being tiled to half a screen, hence the weird formatting :) I need my separate monitor! at a coffee house right now though...)

Zoxc (Oct 11 2019 at 14:37, on Zulip):

There's no separate serialization step with https://github.com/rust-lang/rust/pull/60035. We serialize while executing.

nikomatsakis (Oct 11 2019 at 14:37, on Zulip):

I think the real question regarding GC is whether we think it's necessary right now

nikomatsakis (Oct 11 2019 at 14:37, on Zulip):

and separately whether we see a path forward

nikomatsakis (Oct 11 2019 at 14:37, on Zulip):

I guess I mean like "should it block landing this PR", is question 1, and maybe also "how soon would we want to fix it"

nikomatsakis (Oct 11 2019 at 14:38, on Zulip):

I .. feel like it will probably be a problem at some point

nikomatsakis (Oct 11 2019 at 14:38, on Zulip):

but I also don't know that it needs to block current PR

nikomatsakis (Oct 11 2019 at 14:38, on Zulip):

I'd also like to do a time check - we are 40 minutes in

nikomatsakis (Oct 11 2019 at 14:38, on Zulip):

For me this has been very helpful, I understand much better what's happening, and I hope that for those who haven't been as actively participating it's been at least eductional :)

Zoxc (Oct 11 2019 at 14:38, on Zulip):

I don't think we need it now. Also having tons of ids likely won't affect performance much.

nikomatsakis (Oct 11 2019 at 14:38, on Zulip):

But I want to give a chance to turn to another topic

Zoxc (Oct 11 2019 at 14:39, on Zulip):

We might as well garbage collect ids if we are garbage collecting the query result cache in the future though.

qmx (Oct 11 2019 at 14:39, on Zulip):

it has been highly educational indeed

mw (Oct 11 2019 at 14:39, on Zulip):

we should also determine if want to do this at all

nikomatsakis (Oct 11 2019 at 14:39, on Zulip):

yes, that could be a good topic :)

nikomatsakis (Oct 11 2019 at 14:39, on Zulip):

Which seems like it should be decided as part of broader picture

mw (Oct 11 2019 at 14:39, on Zulip):

I'm not saying we shouldn't but there has been no discussion about whether #60035 is where we want to go

nikomatsakis (Oct 11 2019 at 14:40, on Zulip):

Can we summarize in a few words what #60035 is aiming to do? (Maybe @Zoxc?)

Zoxc (Oct 11 2019 at 14:41, on Zulip):

As in, do we want incremental data structures for the dep graph and query result cache or are we fine with a discrete step writing the incremental state taking up all the time in the clean incremental benchmarks? =P

nikomatsakis (Oct 11 2019 at 14:42, on Zulip):

Let me rephase that sarcasm into something more productive: can we lay out the motivations :)

mw (Oct 11 2019 at 14:42, on Zulip):

As in, what is the right approach to make these more incremental

nikomatsakis (Oct 11 2019 at 14:43, on Zulip):

I need to skim #60035 more closely

nikomatsakis (Oct 11 2019 at 14:43, on Zulip):

the PR description says

This PR changes how we save the dep graph. Instead of storing the nodes in memory until the end of the compilation they are streamed to a file (in the background with parallel_compiler). This should hopefully reduce the memory usage with incremental compilation somewhat.

nikomatsakis (Oct 11 2019 at 14:43, on Zulip):

But that doesn't feel complete

nikomatsakis (Oct 11 2019 at 14:44, on Zulip):

Or is it? I was assuming that dep-graph loading was also be affected

Zoxc (Oct 11 2019 at 14:44, on Zulip):

I'll quote myself =P

This PR (https://github.com/rust-lang/rust/pull/62038) is intended as an incremental step towards #60035 which goes further and removes the current dep graph from memory and instead stores it in an incremental key-value store on disk. There we apply changes to the on disk dep graph as we execute and we only need write the changes to the dep graph instead of writing the entire dep graph at the end.

Zoxc (Oct 11 2019 at 14:44, on Zulip):

The dep graph loading isn't really affected.

nikomatsakis (Oct 11 2019 at 14:44, on Zulip):

So in that model we would store a "diff" as we execute?

nikomatsakis (Oct 11 2019 at 14:45, on Zulip):

I don't think I had fully processed that part of the text before :)

Zoxc (Oct 11 2019 at 14:45, on Zulip):

Kind of. We still have a old dep graph in memory, but a new dep graph on disk which we change as we execute.

Zoxc (Oct 11 2019 at 14:46, on Zulip):

So unlike https://github.com/rust-lang/rust/pull/62038 it doesn't need to copy the dep graph when loading.

Zoxc (Oct 11 2019 at 14:46, on Zulip):

We have a copy on disk we can modify.

nikomatsakis (Oct 11 2019 at 14:46, on Zulip):

I'll make an observation: we probably don't have enough time to really talk out the new PR in a lot of technical depth.

nikomatsakis (Oct 11 2019 at 14:46, on Zulip):

Maybe we should do a 2nd meeting on that, as a follow-up?

nikomatsakis (Oct 11 2019 at 14:47, on Zulip):

Do we think that we can land the current PR either way? Off hand, it seems like it may just be a useful step period.

mw (Oct 11 2019 at 14:48, on Zulip):

/me looks at https://perf.rust-lang.org/compare.html?start=4a365a29d64bec75d107214319a129ba68fc12a3&end=7830caefb62c9c8d3fb7a742c66c64c89bf3aafe&stat=wall-time again

nikomatsakis (Oct 11 2019 at 14:48, on Zulip):

I guess the other question is whether people have other ideas for tackling the problems @Zoxc is trying to tackle, which reminds me that I think it'd be good to define those a bit more closely (you mentioned timing in incremental benchmarks, @Zoxc, can you elaborate a bit more?)

mw (Oct 11 2019 at 14:49, on Zulip):

The documentation in the rustc-guide should also be updated if the PR lands

qmx (Oct 11 2019 at 14:50, on Zulip):

one thing that got me wondering: do we have rough figures on the footprint of the depgraph, both in memory and in disk?

nikomatsakis (Oct 11 2019 at 14:50, on Zulip):
mw (Oct 11 2019 at 14:50, on Zulip):

also the wins in the PR might go away on a fully loaded system, right? because we are offloading work to the background

Zoxc (Oct 11 2019 at 14:50, on Zulip):

I don't want to waste time writing the entire dep graph and query result cache to disk each execution. In the clean incremental benchmark pretty much nothing changes so it's extremely wasteful.

Zoxc (Oct 11 2019 at 14:51, on Zulip):

@mw The instruction count regresses only slightly though, so at worst it will be pretty close.

nikomatsakis (Oct 11 2019 at 14:52, on Zulip):

one thing that got me wondering: do we have rough figures on the footprint of the depgraph, both in memory and in disk?

I was wondering the same thing, I don't know the answer, but I know that I have had trouble with huge amounts of incremental data from time to time. I don't know that this PR will affect that, I guess it could a little but I assume it's mostly from serialized reuslts and not the graph itself.

Zoxc (Oct 11 2019 at 14:52, on Zulip):

We pretty much move work from the query system to the background thread.

nikomatsakis (Oct 11 2019 at 14:52, on Zulip):

I don't want to waste time writing the entire dep graph and query result cache to disk each execution. In the clean incremental benchmark pretty much nothing changes so it's extremely wasteful.

This does feel like a very solid motivation :)

nikomatsakis (Oct 11 2019 at 14:53, on Zulip):

I remember @mw you and I talking a bit about things like "mmap'ing" incremental results (or at least making it so that loading and saving is very cheap)

nikomatsakis (Oct 11 2019 at 14:53, on Zulip):

It feels like this PR by @Zoxc also moves in that direction, since mutating the graph is more in line with that? (I guess it's somewhat separable)

Zoxc (Oct 11 2019 at 14:53, on Zulip):

Memory stats from https://github.com/rust-lang/rust/pull/60035: https://perf.rust-lang.org/compare.html?start=eeedd3a6e15d43d0cd3e860f36be737cb2c941ca&end=cfe977fc791e1a9305d2b79e47b448dfa50abb4a&stat=max-rss

nikomatsakis (Oct 11 2019 at 14:54, on Zulip):

This is because we don't have to retain the old graph in memory?

Zoxc (Oct 11 2019 at 14:54, on Zulip):

The new dep graph is only on disk. The old is still in memory.

nikomatsakis (Oct 11 2019 at 14:54, on Zulip):

In #60035, @Zoxc, do we always dump out a full new graph each time?

mw (Oct 11 2019 at 14:54, on Zulip):

I remember mw you and I talking a bit about things like "mmap'ing" incremental results (or at least making it so that loading and saving is very cheap)

Yes, a while ago. That would be independent of making the graph more incremental

nikomatsakis (Oct 11 2019 at 14:54, on Zulip):

The new dep graph is only on disk. The old is still in memory.

yeah, sorry, I meant to say the whole graph

Zoxc (Oct 11 2019 at 14:54, on Zulip):

Which suggest that most of the compiler memory usage is just the dep graph in some cases...

nikomatsakis (Oct 11 2019 at 14:55, on Zulip):

In #60035, Zoxc, do we always dump out a full new graph each time?

in other words, how I am understanding this PR is something like this:

nikomatsakis (Oct 11 2019 at 14:55, on Zulip):

this is based on basically zero reading of the PR itself :)

nikomatsakis (Oct 11 2019 at 14:56, on Zulip):

so feel free to correct me

Zoxc (Oct 11 2019 at 14:57, on Zulip):

We only 'dump' out nodes that changed or a new relative to the old dep graph, but it otherwise sounds right.

nikomatsakis (Oct 11 2019 at 14:57, on Zulip):

Do we delete nodes from the old graph?

Zoxc (Oct 11 2019 at 14:58, on Zulip):

No. We delete the entire old graph from the disk (and all incremental state) from the previous session.

Zoxc (Oct 11 2019 at 14:59, on Zulip):

The old dep graph is read-only in memory and doesn't change.

nikomatsakis (Oct 11 2019 at 14:59, on Zulip):

So if something doesn't change, and we delete it, but don't write a new copy, what happens to it?

Zoxc (Oct 11 2019 at 15:00, on Zulip):

We make a copy of the entire incremental state in the start of the compiler, and we only mutate that copy of the dep graph

nikomatsakis (Oct 11 2019 at 15:01, on Zulip):

OK I don't understand but it doesn't matter

nikomatsakis (Oct 11 2019 at 15:01, on Zulip):

I started taking some notes on "conclusions" in the hackmd

nikomatsakis (Oct 11 2019 at 15:01, on Zulip):

pasting current status here

nikomatsakis (Oct 11 2019 at 15:01, on Zulip):
nikomatsakis (Oct 11 2019 at 15:01, on Zulip):

In particular, the "conclusion" there is kind of a "what do you all think of this"?

nikomatsakis (Oct 11 2019 at 15:02, on Zulip):

Ah we did not discuss documentation

mw (Oct 11 2019 at 15:02, on Zulip):

I think we should make the graph and the result cache more incremental

nikomatsakis (Oct 11 2019 at 15:02, on Zulip):

But I think I've made clear what I Think we sohuld do there and we can discuss it somewhat separately

mw (Oct 11 2019 at 15:02, on Zulip):

I would like a more detailed description of what the final state looks like

nikomatsakis (Oct 11 2019 at 15:02, on Zulip):

To be clear, I am not proposing to land #62038 before we discuss #60035

mw (Oct 11 2019 at 15:02, on Zulip):

i.e. the incremental file format for the dep graph

nikomatsakis (Oct 11 2019 at 15:03, on Zulip):

But I thought it would be useful to do some review of it while my understanding is fresh :)

mw (Oct 11 2019 at 15:03, on Zulip):

regarding #62038, it does regress memory usage quite a bit in some cases: https://perf.rust-lang.org/compare.html?start=4a365a29d64bec75d107214319a129ba68fc12a3&end=7830caefb62c9c8d3fb7a742c66c64c89bf3aafe&stat=max-rss

nikomatsakis (Oct 11 2019 at 15:03, on Zulip):

That's useful data

nikomatsakis (Oct 11 2019 at 15:06, on Zulip):

So @Zoxc how do you feel about the plan of

I would expect to hold a follow-up meeting in ~3 weeks -- next week is busy, week after that is planning meeting. But that seems ok.

nikomatsakis (Oct 11 2019 at 15:06, on Zulip):

And I guess I want to know if there are alternative designs in mind :)

nikomatsakis (Oct 11 2019 at 15:06, on Zulip):

Or anyone has strong objections to this change

nikomatsakis (Oct 11 2019 at 15:06, on Zulip):

(Feel free to message me privately if you prefer :)

Zoxc (Oct 11 2019 at 15:07, on Zulip):

It seems fine given that https://github.com/rust-lang/rust/pull/62038 has drawbacks fixed by https://github.com/rust-lang/rust/pull/60035.

nikomatsakis (Oct 11 2019 at 15:07, on Zulip):

Yes, I was going to mention that

nikomatsakis (Oct 11 2019 at 15:07, on Zulip):

it'd be best to land them in close succession

nikomatsakis (Oct 11 2019 at 15:08, on Zulip):

OK, I gotta go. I've occupied this cafe seat for too long.

Zoxc (Oct 11 2019 at 15:08, on Zulip):

I'm not sure the drawback are that severe, but we should decide that we want to do something like 60035.

mw (Oct 11 2019 at 15:08, on Zulip):

for the next discussion we might want to think about whether we also want to tackle saving the dep-graph if compilation errors or if that is off the table for the time being

mw (Oct 11 2019 at 15:08, on Zulip):

I do think we want something like #60035

nikomatsakis (Oct 11 2019 at 15:08, on Zulip):

Yes, I want to spend a bit of time trying to prepare for next discussion, @mw and @Zoxc

mw (Oct 11 2019 at 15:09, on Zulip):

but I'd like to have a clearer picture and some evaluation of it

nikomatsakis (Oct 11 2019 at 15:09, on Zulip):

let's create a hackmd talk and I'll reach out to you all to schedule some time to take notes in it

nikomatsakis (Oct 11 2019 at 15:09, on Zulip):

ok, seems like we're on same page, I feel the same -- this work seems to be in the right direction; I'd like to understand it better, but overall I'm feeling positive

nikomatsakis (Oct 11 2019 at 15:10, on Zulip):

I also want to give a shout out to @Zoxc for pursuing so much of this work. It's very cool stuff. =)

mw (Oct 11 2019 at 15:13, on Zulip):

alright, thanks everyone! I think it is very useful to discussion bigger changes like the ones proposed here!

mw (Oct 11 2019 at 15:13, on Zulip):

/me needs to run

Last update: Nov 16 2019 at 01:05UTC