Stream: t-compiler

Topic: design meeting 2019-11-01


pnkfelix (Nov 01 2019 at 12:50, on Zulip):

just a reminder to @T-compiler/meeting : we will be having our first design meeting for this four week cycle in 1 hour and 10 minutes.

pnkfelix (Nov 01 2019 at 12:51, on Zulip):

this week's topic is "incremental dep-graph storage"; see rust-lang/compiler-team#199

pnkfelix (Nov 01 2019 at 12:53, on Zulip):

and also, there has been pre-meeting discussion over the last week over in a parallel zulip topic. (My hope is that we will start today's meeting by summarizing any points that arose there or elsehwere, rather than presuming that everyone's already read through that thread or others.)

pnkfelix (Nov 01 2019 at 13:02, on Zulip):

notes from that parallel zulip topic were accumulated in this hackmd document

nikomatsakis (Nov 01 2019 at 13:40, on Zulip):

and also, there has been pre-meeting discussion over the last week over in a parallel zulip topic. (My hope is that we will start today's meeting by summarizing any points that arose there or elsehwere, rather than presuming that everyone's already read through that thread or others.)

(the hope is indeed that the hackmd suffices :)

nikomatsakis (Nov 01 2019 at 13:41, on Zulip):

One thing I would add is that you may want to refresh your memory on the design meeting from 2019-10-11, which discussed #62038 -- I know I do.

pnkfelix (Nov 01 2019 at 13:41, on Zulip):

zulip stream for aforementioned design meeting

nikomatsakis (Nov 01 2019 at 14:01, on Zulip):

Shall we start the meeting?

nikomatsakis (Nov 01 2019 at 14:01, on Zulip):

Hey @T-compiler/meeting -- design meeting starting now! Add a :wave: emoji to show you're here :)

nikomatsakis (Nov 01 2019 at 14:01, on Zulip):

Announcements

nikomatsakis (Nov 01 2019 at 14:02, on Zulip):
nikomatsakis (Nov 01 2019 at 14:03, on Zulip):

thanks to @Matthew Jasper, @centril, and @simulacrum for pushing that forward (did I forget anyone?)

nikomatsakis (Nov 01 2019 at 14:03, on Zulip):

also cc @eddyb since apparently they don't get notifications?? (they're in the user group, for sure...)

nikomatsakis (Nov 01 2019 at 14:03, on Zulip):

oh wait, they waved :)

nikomatsakis (Nov 01 2019 at 14:05, on Zulip):

OK, let's get started I guess :)

nikomatsakis (Nov 01 2019 at 14:05, on Zulip):

I guess we should start by setting the context a bit?

nikomatsakis (Nov 01 2019 at 14:06, on Zulip):

Last time on 2019-10-11 we discussed @Zoxc's PR that modified how we handle the dep-graph. Whereas today we read in the old dep-graph and then copy bits of it to create a new one, that PR had us modify the graph in place and then write out the final result.

nikomatsakis (Nov 01 2019 at 14:07, on Zulip):

That was a mid-way step towards an end-goal in which we load the previous dep-graph into memory but then never modify it all -- instead we write those changes out to disk in a streaming fashion, so that we only have to keep one copy in memory at a time

nikomatsakis (Nov 01 2019 at 14:07, on Zulip):

Is that accurate, @Zoxc ?

nikomatsakis (Nov 01 2019 at 14:07, on Zulip):

(Good way for me to check myself...)

Zoxc (Nov 01 2019 at 14:07, on Zulip):

Seem accurate enough

nikomatsakis (Nov 01 2019 at 14:08, on Zulip):

(One quick question -- did we discuss "GC" when talking about the previous PR?)

Zoxc (Nov 01 2019 at 14:08, on Zulip):

No.

nikomatsakis (Nov 01 2019 at 14:08, on Zulip):

So, we created this hackmd document with various notes and questiosn -- I think probably the best way to start is to walk through the proposed design and try to answer some of the questions within

Zoxc (Nov 01 2019 at 14:09, on Zulip):

And the GC in this PR is for the key-value store used to store the graph. There's no GC on the graph itself.

nikomatsakis (Nov 01 2019 at 14:09, on Zulip):

ok, let's start with a basic question

nikomatsakis (Nov 01 2019 at 14:09, on Zulip):

What does the on-disk format look like?

nikomatsakis (Nov 01 2019 at 14:09, on Zulip):

In particular, how do we store the graph today, and how does it change in this PR?

nikomatsakis (Nov 01 2019 at 14:10, on Zulip):

(Also, i'm going to be updating the design doc live with links to key comments and things, feel free to add things if you like, or use :point_up: to draw attention)

mw (Nov 01 2019 at 14:11, on Zulip):

this is how the graph is stored today: https://github.com/rust-lang/rust/blob/master/src/librustc/dep_graph/serialized.rs#L13-L26

nikomatsakis (Nov 01 2019 at 14:11, on Zulip):

Yeah, so there's one big file that we encode automatically

Zoxc (Nov 01 2019 at 14:11, on Zulip):

The on disk format is basically just a list of Action here: https://github.com/rust-lang/rust/blob/9f268ad2f5009213d5f33fea02d359536cc89cec/src/librustc/dep_graph/serialized.rs#L340

It's basically just a list of the changes, and when rustc runs, it appends more to the dep graph file.

nikomatsakis (Nov 01 2019 at 14:12, on Zulip):

so when rustc starts, we read in all the actions to start

nikomatsakis (Nov 01 2019 at 14:12, on Zulip):

that's "loading" the dep-graph?

nikomatsakis (Nov 01 2019 at 14:12, on Zulip):

and presumably if a node is overwritten, we'll create it twice

Zoxc (Nov 01 2019 at 14:12, on Zulip):

Yeah, we apply all the actions to an empty dep graph basically.

nikomatsakis (Nov 01 2019 at 14:13, on Zulip):

i.e., if we have a node in the dep-graph like A -> [B, C] and it gets overwritten with A -> [D, E], we'll first create A -> [B, C] and then trash that and create A -> [D, E] to replace it?

Zoxc (Nov 01 2019 at 14:13, on Zulip):

Yeah.

Zoxc (Nov 01 2019 at 14:14, on Zulip):

The GC also does a similar thing where it notes down what was overwritten.

nikomatsakis (Nov 01 2019 at 14:14, on Zulip):

(OK, and this in turn explains why you changed the in-memory format to be a Box<[Index]>, but we'll get to that)

nikomatsakis (Nov 01 2019 at 14:14, on Zulip):

One question I have, maybe dumb -- are we able to do this loading in parallel?

Zoxc (Nov 01 2019 at 14:14, on Zulip):

It happens in the background thread, so yes/no?

nikomatsakis (Nov 01 2019 at 14:15, on Zulip):

do we start the background thread and then ensure that it has completed by the time we enter the "incremental" phase?

mw (Nov 01 2019 at 14:15, on Zulip):

that does sound a bit wasteful. do we allocate those slices individually?

Zoxc (Nov 01 2019 at 14:16, on Zulip):

They're allocated individually.

nikomatsakis (Nov 01 2019 at 14:16, on Zulip):

I guess maybe we should discuss the in-memory format just a bit :) topics kind of bleed in together. Just for context (and to check my understanding), currently we store all edges in one big vector, and how each node we store a pair of indices into that vector

nikomatsakis (Nov 01 2019 at 14:17, on Zulip):

so

edge_list_indices: IndexVec<SerializedDepNodeIndex, (u32, u32)>,
edge_list_data: Vec<SerializedDepNodeIndex>,
nikomatsakis (Nov 01 2019 at 14:17, on Zulip):

but this is hard to "update"

nikomatsakis (Nov 01 2019 at 14:17, on Zulip):

because to remove a node from the middle you'd have to shift over the elements etc etc

nikomatsakis (Nov 01 2019 at 14:17, on Zulip):

so instead in the PR we are using

nikomatsakis (Nov 01 2019 at 14:17, on Zulip):
edges: IndexVec<DepNodeIndex, Option<Box<[DepNodeIndex]>>>,
nagisa (Nov 01 2019 at 14:18, on Zulip):

Are these SerializedDepNodeIndexes large enough to warrant removing them individually rather than all at once at some well specified point in process?

nikomatsakis (Nov 01 2019 at 14:19, on Zulip):

in other words

nagisa (Nov 01 2019 at 14:19, on Zulip):

i.e. an actual mark and sweep GC kind of thing at the end.

nikomatsakis (Nov 01 2019 at 14:19, on Zulip):

how bad is it if we waste some slots in the edge vector?

nikomatsakis (Nov 01 2019 at 14:20, on Zulip):

I think this is an interesting question, but it in the grand scheme of things it seems like a relatively minor point -- one best answered by measuring?

mw (Nov 01 2019 at 14:20, on Zulip):

currently the SerializedDepGraph (which is the in-memory version we are using for the previous dep-graph) is just 4 flat arrays, which is pretty efficient

nikomatsakis (Nov 01 2019 at 14:20, on Zulip):

it seems clear that we could optimize this in various ways, or maybe it's good enough as is?

Zoxc (Nov 01 2019 at 14:20, on Zulip):

Depends on how often we GC.

Zoxc (Nov 01 2019 at 14:20, on Zulip):

I was hoping to move it to arena allocation

pnkfelix (Nov 01 2019 at 14:21, on Zulip):

I'm not clear on how we can say "good enough as is" (without data), given that one primary goal here is to reduce peak mem usage?

pnkfelix (Nov 01 2019 at 14:22, on Zulip):

((but maybe "need data" was implicit in niko's comment, esp since they had just said "best answered by measuring" a few lines above...))

Zoxc (Nov 01 2019 at 14:22, on Zulip):

That's more of a secondary goal, and it is already massively reduced.

nikomatsakis (Nov 01 2019 at 14:22, on Zulip):

I was hoping to move it to arena allocation

can you say a bit more what you mean by that?

nikomatsakis (Nov 01 2019 at 14:22, on Zulip):

do you mean allocating &'tcx [Index] instead of box<[Index>]?

Zoxc (Nov 01 2019 at 14:22, on Zulip):

Yes

Zoxc (Nov 01 2019 at 14:23, on Zulip):

The arenas would have to be moved earlier first

nikomatsakis (Nov 01 2019 at 14:23, on Zulip):

in that case

nikomatsakis (Nov 01 2019 at 14:23, on Zulip):

well, why is that better than the vector?

nikomatsakis (Nov 01 2019 at 14:23, on Zulip):

i.e., it is still wasting the memory for the old slics

Zoxc (Nov 01 2019 at 14:23, on Zulip):

No allocation slowing down loading, and no memory fragmentation.

mw (Nov 01 2019 at 14:24, on Zulip):

can the data be read backwards so that overwritten entries can be ignored altogether?

Zoxc (Nov 01 2019 at 14:24, on Zulip):

We don't waste any memory now. The old boxes are freed during loading

nikomatsakis (Nov 01 2019 at 14:24, on Zulip):

There is allocation (of the arena) and there is fragmentation (within the arena) -- but ok, I definitely think this is a "measure" question. I'm willing to believe arena allocation is more efficienct since you're combining in a larger pool.

Zoxc (Nov 01 2019 at 14:24, on Zulip):

@mw Probably?

Zoxc (Nov 01 2019 at 14:25, on Zulip):

Can you decode LEB128 backwards? =P

nikomatsakis (Nov 01 2019 at 14:25, on Zulip):

ok, time check, it's 10:25

pnkfelix (Nov 01 2019 at 14:25, on Zulip):

That's more of a secondary goal, and it is already massively reduced.

This is actually important

pnkfelix (Nov 01 2019 at 14:26, on Zulip):

The mem usage data quoted in the hackmd is based on perf runs from May

mw (Nov 01 2019 at 14:27, on Zulip):

Can you decode LEB128 backwards? =P

We don't have to encode everything in LEB128.

nikomatsakis (Nov 01 2019 at 14:27, on Zulip):

yep, it'd be good to get more up-to-date data

nikomatsakis (Nov 01 2019 at 14:27, on Zulip):

ok, time check, it's 10:25

I was going to suggest that maybe we want to turn to Garbage Collection --

mw (Nov 01 2019 at 14:27, on Zulip):

if memory reduction is a secondary goal, what is the primary goal then?

nikomatsakis (Nov 01 2019 at 14:27, on Zulip):

(it seems like we've identified some interesting thoughts regarding loading, but there were some big questions around GC and I'd like to make sure we talk about it)

nikomatsakis (Nov 01 2019 at 14:28, on Zulip):

if memory reduction is a secondary goal, what is the primary goal then?

but this is also imp't, maybe I misunderstood what @Zoxc was saying

eddyb (Nov 01 2019 at 14:28, on Zulip):

as a data point, I think we got a speedup in proc_macro by not using LEB128 for the RPC stuff

Zoxc (Nov 01 2019 at 14:28, on Zulip):

LEB128 is just for saving disk space

Zoxc (Nov 01 2019 at 14:28, on Zulip):

The goal of the PR is performance, not having to store 2 dep graph in memory is a happy side effect =P

nikomatsakis (Nov 01 2019 at 14:29, on Zulip):

That's more of a secondary goal, and it is already massively reduced.

I thought @Zoxc was saying here that the question of memory usage around edges was of secondary imporance

nikomatsakis (Nov 01 2019 at 14:29, on Zulip):

ok, I guess I misunderstood :)

nikomatsakis (Nov 01 2019 at 14:29, on Zulip):

it seems though that we don't actually get many perf wins

nikomatsakis (Nov 01 2019 at 14:29, on Zulip):

but perhaps I am misreading the perf results?

nikomatsakis (Nov 01 2019 at 14:30, on Zulip):

I was debating if I should have kicked off the meeting with a "goals" discussion --

nikomatsakis (Nov 01 2019 at 14:30, on Zulip):

but let's do it now :)

mw (Nov 01 2019 at 14:30, on Zulip):

persisting the dep-graph already happens "in the background" for cases where LLVM is doing something

mw (Nov 01 2019 at 14:30, on Zulip):

at least if I remember correctly and the situation hasn't changed

pnkfelix (Nov 01 2019 at 14:30, on Zulip):

three advantages listed in hackmd doc here

nikomatsakis (Nov 01 2019 at 14:30, on Zulip):

yeah

pnkfelix (Nov 01 2019 at 14:31, on Zulip):
  1. Lower Peak Memory Usage
  2. Avoids blocking on writing out dep graph at end of compilation
  3. Avoids work of re-writing unchanged portions of dep-graph
nikomatsakis (Nov 01 2019 at 14:31, on Zulip):

it seems like the biggest wins come from memory usage (e.g., reduction of peak usage by 30% in some cases), but we do see some performance wins too (I saw measurements that suggested max wins of about 5%?)

nikomatsakis (Nov 01 2019 at 14:32, on Zulip):

but obviously conceptually it is better to avoid having to rewrite the whole graph

pnkfelix (Nov 01 2019 at 14:32, on Zulip):

@Zoxc do you agree all of these remain boons acheived in this PR (we can side-step question of primary vs secondary for moment...)

pnkfelix (Nov 01 2019 at 14:33, on Zulip):

I'd love to have data on 3

nikomatsakis (Nov 01 2019 at 14:33, on Zulip):

don't we?

nikomatsakis (Nov 01 2019 at 14:33, on Zulip):

we have runs where literally nothing changes

mw (Nov 01 2019 at 14:34, on Zulip):

(perf.rlo doesn't seem to work for me, right :/)

mw (Nov 01 2019 at 14:34, on Zulip):

https://perf.rust-lang.org/detailed-query.html?commit=c553e8e8812c19809e70523064989e66c5cfd3f1&benchmark=inflate-debug&run_name=clean%20incremental

mw (Nov 01 2019 at 14:34, on Zulip):

if you search for dep_graph you can see how much time we spend on it

mw (Nov 01 2019 at 14:34, on Zulip):

i.e. on loading and persisting it

pnkfelix (Nov 01 2019 at 14:34, on Zulip):

we have runs where literally nothing changes

yeah but I meant in the not-quite-so trivial cases

pnkfelix (Nov 01 2019 at 14:35, on Zulip):

e.g. when you change just comments

pnkfelix (Nov 01 2019 at 14:35, on Zulip):

(vs touching a file)

nikomatsakis (Nov 01 2019 at 14:35, on Zulip):

ok, i'm not sure why that number is interesting, but it seems gatherable :)

mw (Nov 01 2019 at 14:35, on Zulip):

the link above should be the worst case for the current impl

mw (Nov 01 2019 at 14:36, on Zulip):

no changes, nothing for LLVM to do

pnkfelix (Nov 01 2019 at 14:37, on Zulip):

@mw you mean "run clean incremental" in general, right? Not just for inflate-debug in particular ?

mw (Nov 01 2019 at 14:37, on Zulip):

"clean incremental" in general

nikomatsakis (Nov 01 2019 at 14:38, on Zulip):

it seems like the biggest wins come from memory usage (e.g., reduction of peak usage by 30% in some cases), but we do see some performance wins too (I saw measurements that suggested max wins of about 5%?)

@Zoxc do you agree with this? did I miss some measurements?

nikomatsakis (Nov 01 2019 at 14:38, on Zulip):

(also, other thoughts around goals?)

mw (Nov 01 2019 at 14:39, on Zulip):

(5% seems like a realistic upper bound at the moment)

Zoxc (Nov 01 2019 at 14:39, on Zulip):

@pnkfelix Yeah

Zoxc (Nov 01 2019 at 14:41, on Zulip):

@nikomatsakis Seems to be about 4-9%, but there's probably some stuff that can be made faster

nikomatsakis (Nov 01 2019 at 14:42, on Zulip):

(what is an example that gets 9%?)

nikomatsakis (Nov 01 2019 at 14:42, on Zulip):

doesn't matter, I can look later

nikomatsakis (Nov 01 2019 at 14:42, on Zulip):

shall we discuss GC?

pnkfelix (Nov 01 2019 at 14:42, on Zulip):

I figure I'll just point out unused-warnings-check is a benchmark

Zoxc (Nov 01 2019 at 14:42, on Zulip):

regex

nikomatsakis (Nov 01 2019 at 14:42, on Zulip):

(we're at 42 minutes)

pnkfelix (Nov 01 2019 at 14:42, on Zulip):

unused-warnings-check is benchmark where clean incremental currently costs more than clean

pnkfelix (Nov 01 2019 at 14:43, on Zulip):

(and I believe, based on analysis I did some weeks ago, that is solely to due cost of unnecessarily re-writing the dep graph. This work would fix that.)

nikomatsakis (Nov 01 2019 at 14:43, on Zulip):

cool

nikomatsakis (Nov 01 2019 at 14:44, on Zulip):

may 7 perf results

nikomatsakis (Nov 01 2019 at 14:44, on Zulip):

pasted image

nikomatsakis (Nov 01 2019 at 14:44, on Zulip):

that is CPU clock

pnkfelix (Nov 01 2019 at 14:44, on Zulip):

hmm that does not support my claim

nikomatsakis (Nov 01 2019 at 14:45, on Zulip):

keep in mind the default is instructions, which doesn't show parallelism wins -- but still interesting!

nikomatsakis (Nov 01 2019 at 14:45, on Zulip):

I'd expect in this case to see wins from instructions?

Zoxc (Nov 01 2019 at 14:45, on Zulip):

GC reads the dep graph file again, notes down the byte positions of stuff getting overwritten and deleted, and then writes the rest back as raw bytes. So it skips LEB128 encoding, but still needs LEB128 decoding when loading the file.

nikomatsakis (Nov 01 2019 at 14:45, on Zulip):

still, I'd really like to spend some time on GC :hint hint:

nikomatsakis (Nov 01 2019 at 14:45, on Zulip):

so can we talk about that and come back to the measurements?

nikomatsakis (Nov 01 2019 at 14:46, on Zulip):

@Zoxc you mentioned early on that that the current PR doesn't do any GC on the dep graph?

mw (Nov 01 2019 at 14:46, on Zulip):

so the GC as implemented in the PR just removes "actions" from the stream that are later overwritten?

nikomatsakis (Nov 01 2019 at 14:46, on Zulip):

For context, the need for GC can arise for two reasons (copying from hackmd):

Zoxc (Nov 01 2019 at 14:47, on Zulip):

Yes. It just removes stuff that isn't in the graph anymore. It removes the redundant actions from the file. It isn't an operation on the graph itself.

mw (Nov 01 2019 at 14:47, on Zulip):

it also does not seem to compact the used DepNodeIndex range, right?

Zoxc (Nov 01 2019 at 14:47, on Zulip):

It's just an internal detail of the key-value store we encode the dep graph onto

Zoxc (Nov 01 2019 at 14:48, on Zulip):

It does nothing to dep node indices.

nikomatsakis (Nov 01 2019 at 14:48, on Zulip):

I'm wondering where it fits in the "pipeline" --

mw (Nov 01 2019 at 14:48, on Zulip):

so the in-memory representation of the graph would keep growing

nikomatsakis (Nov 01 2019 at 14:48, on Zulip):

when we start, we read the actions in

nikomatsakis (Nov 01 2019 at 14:48, on Zulip):

and build a graph

nikomatsakis (Nov 01 2019 at 14:48, on Zulip):

then we append new actions to the file

nikomatsakis (Nov 01 2019 at 14:49, on Zulip):

but sometimes we run the GC and .. overwrite the file so as to prune old actions?

nikomatsakis (Nov 01 2019 at 14:49, on Zulip):

when do we run this GC relative to other things?

nikomatsakis (Nov 01 2019 at 14:49, on Zulip):

so the in-memory representation of the graph would keep growing

presumably the next compilation will load a "pruned" action list, right?

Zoxc (Nov 01 2019 at 14:49, on Zulip):

@mw Only if the indices get fragmented, they are reused as in the previous PR

nikomatsakis (Nov 01 2019 at 14:49, on Zulip):

ah, yeah, it can keep growing from fragementation

Zoxc (Nov 01 2019 at 14:50, on Zulip):

We load the dep graph first. If we notice a lot of garbage we run the GC on dep graph file. All this happens in the background thread and anything incremental in the compiler is blocked on it

nikomatsakis (Nov 01 2019 at 14:51, on Zulip):

OK, so the full pipeline is something like:

Zoxc (Nov 01 2019 at 14:52, on Zulip):

Yeah. This means that right after a GC, rustc will crate more garbage =P

nikomatsakis (Nov 01 2019 at 14:52, on Zulip):

how do we measure the "amount" of garbage?

nikomatsakis (Nov 01 2019 at 14:52, on Zulip):

is that "overwritten" actions?

Zoxc (Nov 01 2019 at 14:53, on Zulip):

Yeah. We record the bytes of overwritten and non-overwritten actions and use a ratio to trigger GC

nikomatsakis (Nov 01 2019 at 14:53, on Zulip):

ok. so I don't quite see where "unreachable" nodes get collected

mw (Nov 01 2019 at 14:54, on Zulip):

(I'd be interested in how an alternative approach (that also does DepNodeIndex compaction and graph node GC) does: Still rewrite the whole dep-graph, but "stream" it to disk in the background. That should be doable with minimal changes and might have most of the benefits.)

nikomatsakis (Nov 01 2019 at 14:54, on Zulip):

ah hmm

nikomatsakis (Nov 01 2019 at 14:54, on Zulip):

so when nodes are first loaded from the disk, they are "grey" or something, right? and we give them a color if they are used

Zoxc (Nov 01 2019 at 14:54, on Zulip):

@nikomatsakis Those don't relate to the key value store and I'm not sure what you mean by that either

nikomatsakis (Nov 01 2019 at 14:54, on Zulip):

so presumably at the end of compilation, we can look at those colors to see nodes that were not relevant

pnkfelix (Nov 01 2019 at 14:55, on Zulip):

It just removes stuff that isn't in the graph anymore. It removes the redundant actions from the file. It isn't an operation on the graph itself.

maybe part of the point @Zoxc 's been saying above

Zoxc (Nov 01 2019 at 14:55, on Zulip):

@nikomatsakis Those nodes are deleted from the dep graph (in the previous PR).

nikomatsakis (Nov 01 2019 at 14:55, on Zulip):

in other words, that's an action?

nikomatsakis (Nov 01 2019 at 14:56, on Zulip):

that gets added at the end?

nikomatsakis (Nov 01 2019 at 14:56, on Zulip):

i.e., there is some final step that says "if this node never got a color, delete it"?

nikomatsakis (Nov 01 2019 at 14:56, on Zulip):

(iterating over all previous nodes)

Zoxc (Nov 01 2019 at 14:56, on Zulip):

There is yes.

nikomatsakis (Nov 01 2019 at 14:56, on Zulip):

OK, that makes sense. Neat.

nikomatsakis (Nov 01 2019 at 14:57, on Zulip):

(I'd be interested in how an alternative approach (that also does DepNodeIndex compaction and graph node GC) does: Still rewrite the whole dep-graph, but "stream" it to disk in the background. That should be doable with minimal changes and might have most of the benefits.)

maybe we want .. hmm, ok, 3 minuts :)

nikomatsakis (Nov 01 2019 at 14:57, on Zulip):

I was going to say @mw that maybe you could say a bit more what you have in mind :)

Zoxc (Nov 01 2019 at 14:58, on Zulip):

@mw I think we can do something like that if we spot dep node indices fragmentation, but I don't know if that is a problem is practice though.

nikomatsakis (Nov 01 2019 at 14:58, on Zulip):

but maybe we should try to wrap-up and leave that for follow-up?

Zoxc (Nov 01 2019 at 14:58, on Zulip):

But that is a lot of complexity for something we that don't know is a problem

mw (Nov 01 2019 at 14:59, on Zulip):

yeah, I only would consider it if it really is a small diff to the current implementation

mw (Nov 01 2019 at 14:59, on Zulip):

if it would turn out to be complex, I'd rather do something the is append-only

mw (Nov 01 2019 at 15:00, on Zulip):

so, the is an invalidation action in the stream that takes care of dep-graph GC?

nikomatsakis (Nov 01 2019 at 15:00, on Zulip):

OK, well, it's the end of the meeting -- I found this pretty helpful for understanding the PR, at least.

nikomatsakis (Nov 01 2019 at 15:00, on Zulip):

I've been trying to take minutes as we went and update the document

nikomatsakis (Nov 01 2019 at 15:01, on Zulip):

I'm not 100% sure how we go from here to a final decision

pnkfelix (Nov 01 2019 at 15:01, on Zulip):

There was something totally orthogonal to the topic that I wanted to mention

nikomatsakis (Nov 01 2019 at 15:01, on Zulip):

(But I think first step would be to take a moment to digest comments and review code and compare against them)

nikomatsakis (Nov 01 2019 at 15:01, on Zulip):

There was something totally orthogonal to the topic that I wanted to mention

go for it..

pnkfelix (Nov 01 2019 at 15:01, on Zulip):

I have beta-nom'ed this PR: #66018

pnkfelix (Nov 01 2019 at 15:02, on Zulip):

I think we discussed this in the compiler team meeting yesterday

pnkfelix (Nov 01 2019 at 15:02, on Zulip):

its a revert of the semantic effect of PR #64324

pnkfelix (Nov 01 2019 at 15:02, on Zulip):

(I believe)

mw (Nov 01 2019 at 15:02, on Zulip):

I'm not 100% sure how we go from here to a final decision

I'd like to know more about the fragmentation problem before making a decision. I.e. a risk assessment and possible solutions

pnkfelix (Nov 01 2019 at 15:03, on Zulip):

or really,I should have phrased the beta-nom this way:

pnkfelix (Nov 01 2019 at 15:03, on Zulip):

beta-nom: "Revert PR 64324: dylibs export generics again (for now)" #66018

pnkfelix (Nov 01 2019 at 15:03, on Zulip):

(I would personally wait for it to get r+'ed and land in Nightly before doing backport)

nikomatsakis (Nov 01 2019 at 15:04, on Zulip):

I'd like to know more about the fragmentation problem before making a decision. I.e. a risk assessment and possible solutions

it seems like a good thing would be to accumulate the things we'd like to investigate further

nikomatsakis (Nov 01 2019 at 15:05, on Zulip):

there is a "open question" at the end of the hackmd

nikomatsakis (Nov 01 2019 at 15:05, on Zulip):

I'll add it there

pnkfelix (Nov 01 2019 at 15:06, on Zulip):

I'll add my Q about whether this resolves the unused-warnings perf oddity

nikomatsakis (Nov 01 2019 at 15:07, on Zulip):

heh oh I realize now you meant that you actually edit the doc

nikomatsakis (Nov 01 2019 at 15:07, on Zulip):

go for it:)

nikomatsakis (Nov 01 2019 at 15:07, on Zulip):

though I sort of did so

pnkfelix (Nov 01 2019 at 15:07, on Zulip):

(though I wouldn't block landing this on finding the answer to that)

nikomatsakis (Nov 01 2019 at 15:07, on Zulip):

I'd also like @mw to understand a bit better what the alternative design you mention looks like, and if it's really possible in a small diff

pnkfelix (Nov 01 2019 at 15:08, on Zulip):

though I sort of did so

does anyone mind if I change the links to point to zulip-archive.rlo ?

nikomatsakis (Nov 01 2019 at 15:08, on Zulip):

I sort of do

nikomatsakis (Nov 01 2019 at 15:08, on Zulip):

those are significantly more annoying to read

nikomatsakis (Nov 01 2019 at 15:08, on Zulip):

I guess I don't really though

pnkfelix (Nov 01 2019 at 15:08, on Zulip):

yeah but you have to login otherwise

nikomatsakis (Nov 01 2019 at 15:08, on Zulip):

/me doesn't care

pnkfelix (Nov 01 2019 at 15:08, on Zulip):

so its less open

pnkfelix (Nov 01 2019 at 15:08, on Zulip):

heh

nikomatsakis (Nov 01 2019 at 15:08, on Zulip):

I won't stop you :)

pnkfelix (Nov 01 2019 at 15:09, on Zulip):

don't the zulip-archive posts link over here anyway?

pnkfelix (Nov 01 2019 at 15:09, on Zulip):

so its a "lossless" transformation?

nikomatsakis (Nov 01 2019 at 15:09, on Zulip):

I guess that would satisfy my use case :)

pnkfelix (Nov 01 2019 at 15:09, on Zulip):

I'll verify that claim

pnkfelix (Nov 01 2019 at 15:09, on Zulip):

(clearly we should just "fix" zulip archive to be more readable...)

pnkfelix (Nov 01 2019 at 15:10, on Zulip):

((the presentation there does lose some stuff, e.g. the emojis))

eddyb (Nov 01 2019 at 15:10, on Zulip):

I think they do link back to Zulip, so it's strictly better to use the archive for anything public

nikomatsakis (Nov 01 2019 at 15:11, on Zulip):

side note, I wonder if we can extend zulip somehow with a feature to make it easy to copy links to the archive

nikomatsakis (Nov 01 2019 at 15:11, on Zulip):

although I guess applying a regex to the files in the github repo is probably easy enough

nikomatsakis (Nov 01 2019 at 15:13, on Zulip):

OK, thanks everyone, and especially @Zoxc :heart:

nikomatsakis (Nov 01 2019 at 15:15, on Zulip):

@pnkfelix when you've finished updating URLs, do you want to upload the doc to compiler-team repo?

pnkfelix (Nov 01 2019 at 15:16, on Zulip):

okay sure

mw (Nov 01 2019 at 15:21, on Zulip):

I'd also like mw to understand a bit better what the alternative design you mention looks like, and if it's really possible in a small diff

I've added an outline to the hackmd doc

pnkfelix (Nov 01 2019 at 15:26, on Zulip):

pnkfelix when you've finished updating URLs, do you want to upload the doc to compiler-team repo?

okay posted as rust-lang/compiler-team#214

pnkfelix (Nov 01 2019 at 15:28, on Zulip):

(oh, and, I'm taking the above discussion of PR #66018 as a approval for beta-backport

Last update: Nov 22 2019 at 05:15UTC