hi @T-compiler/meeting , we're starting today's steering meeting now

We have an agenda, or at least some notes, prepared for us thanks to @simulacrum : https://docs.google.com/document/d/1DzQtKpudHqCiYPuRrFPMkhMCaQ884uG_8dh3Vufv9Vc/edit

I don't remember: Did we establish that we would have this meeting over Zulip?

(vs zoom?)

I think Zulip is fine

Okay, lets go with zulip then

so yeah, lets all maybe take five minutes right now to read the doc

and then we can talk about it

You all should be able to edit / leave comments btw

@simulacrum seems like this "A cold-cache (fresh checkout) should produce a [...]" was an unfinished sentence

ha! Nice fix!

yeah not sure what I was thinking

probably some other metric

I think its worth clarifying something up front: You would prefer the improvements here to be "general" in this sense that they would apply to other projects of rustc's size/structure, and/or would apply to a snapshot of rustc taken today, right?

I think sort of

I would focus 100% on rustc itself

when I first read this, I was thinking it might be more narrow, as in a solution of restructuring the rustc repo (e.g. fine-grain librarification) could address at least some of these issues

e.g., if it means we have some hacks that we "know" are true in rustc, I would be fine with that

but restructuring the rustc repo doesn't fix the problem for other projects

but I would expect the majority of the work to yield steps, at least, that others could follow

Oh I see

e.g. restructuring is something others can do, and we can say that's recommended

So restructuring can be part of the plan, okay.

that helps me understand, thanks.

I suspect that there could be wins from making some things around queries non-generic

and rustc_middle would profit from better CGU partitioning

(hey all, sorry, I'm in a meeting that's running over)

Yep, both of those seem like specific steps that we can also use to document some general steps

Okay, put a :thumbs_up: on this comment if you are finshed reading, and put a :thumbs_down: if you need more time. (and I'll assume non-responses are :thumbs_down: )

We should make an effort to document things that we change that aren't just making rustc itself faster in njn's perf book.

e.g. changing how we structure our crate graph, etc.

njn's book is focused on object code performance, not build times, right?

(or is it both?)

it has a chapter on compile times

https://nnethercote.github.io/perf-book/compile-times.html

okay so I think we're only waiting for @Léo Lanteri Thauvin and @nikomatsakis to finish reading at this point

sorry don't block on me

I will catch up

I can't focus right now

Lets go ahead and get started on discussion

if the PGO work I've been discussing with @simulacrum and others bares fruit, the bootstrap compiler would also become faster (~15%)

wait, I had thought PGO wasn't showing so much of a win in many cases?

what happened?

wall-time improvements look pretty promising: https://perf.rust-lang.org/compare.html?start=pgo-2020-10-30-none-20&end=pgo-2020-10-30-both-20&stat=wall-time

(am I misremembering, about PGO not showing a benefit in the past?)

well anyway that is promising

pure instruction counts look less good because PGO often improves caching and branch prediction

maybe I'm thinking of PGO not meeting expectations on Firefox or something

anyway

pnkfelix said:

maybe I'm thinking of PGO not meeting expectations on Firefox or something

Yes, I think that improved after they adapted their training sets

Okay. @simulacrum , am I right that the main actions proposed by the doc are the two bullets below the "We should aim for: ..." ?

Kind of -- not sure I'd call them actions, more goals

namely: 1. turn on incr-comp for contributors, and 2. make it easier/possible to build just individual components (like rustdoc, test suite, std lib)

but yes, in some sense

I guess I see them as strategies (or tactics? Still need to make that distinction clear in my head) for acheiving the end goal of better build times.

yes that's accurate

What are blockers for incremental? One big one is that we still don't have a great story around fixing bugs there

i.e. we count ourselves lucky when we get a bug report that we can actually reproduce

that isn't an objection per se to turning it on by default

yeah -- I think digging in here might be useful -- I've not heard a lot from incr WG though I know y'all have been meeting a bunch lately

the main thing I have personally experienced is that incremental kinda feels lackluster a lot of the time

but we don't want to be in a situation where we are repeatedly telling new people "oh, that ICE? yeah, you need to clean your build dir first"

I've usually turned incremental on when building rustc and don't run into many bugs

as in, turning incremental on doesn't actually yield the "feel" of an incremental environment ?

but I'm not building rustc that often these days

I've pretty much only used -i when building rustc for the last year or two.

simulacrum said:

the main thing I have personally experienced is that incremental kinda feels lackluster a lot of the time

@simulacrum what did "lackluster" here mean?

Yes, it doesn't feel like a big win

okay

Maybe I just have good hardware -- I've heard differently

but I never felt a marked difference with it on

no I think the wg-incr-comp agrees that it doesn't seem to yield the benefits you would expect

and the clean build is slower by our benchmarks, so I'm always kinda "meh" on turning it on

yeah, I think it performs worse on rustc than on "regular" crates

for me the felt difference is a lot bigger when I work on crates that don't require rustc_middle to be rebuilt

that might be my problem -- I usually end up touching middle or even parser in some of my work :)

I've definitely found it to be very effective sometimes -- but also sometimes completely ineffective

but, I guess, my concrete question for incr WG is: can we pull off these subminute recompiles?

like, is that a feasible goal?

one of the reasons I ask is because it's a key part -- imo -- of delivering e.g. documentation building quickly after edits to docs to users

I know I'm always super frustrated to type up some comments on a rustc_middle function for example

so if I recall correctly, one of the issues there is that the spans all change when you change comments, right?

I know there's been some work in that space

to make span changes not cause the rest of the stuff to have to be rebuilt

or at least ideas thrown around

but I don't remember what's actually happened at this point

@Wesley Wiser do you remember?

@cjgillot is currently working on this (#72015 and it's sub prs)

or at least has previously

the big issue here is that only changing some of the spans caused some ~5% perf regressions or so without any incremental benefits iirc

so they pretty much had to change all in one go which was both enormous and still had a small perf impact i think

The concrete proposal being made here, I guess, is that we try to focus compiler team implementation/design bandwidth on this goal of incremental performance / rustc build times. I'm not quite sure what that means -- maybe e.g. we'd focus in on prioritizing incremental bugs (both ICE and perf)

its a good idea

one thing I do know from the recent wg-incr-comp meetings

is that all the members are swamped

@pnkfelix I know there's tracking issues for that but I wasn't aware there was progress being made on them.

with things not related to incr-comp

I am very in favor of this as a focus

so progress on the tasks there themselves is also lackluster

is it other compiler tasks, or just other things?

basically "can we meaningfully prioritize"

Beyond what's in the doc, it seems to pay multiple dividends (e.g., there is a "positive feedback" where making rustc compile faster will yield more people to help make it compile faster)

But yeah the tactics is the big question, how do we organize to be successful at this goal?

simulacrum said:

is it other compiler tasks, or just other things?

At least for myself, it's a bit of both. But I'm starting to find more time for incr comp things.

at least the metrics are relatively clear -- but actually it'd be good to define specific workloads I think

editing a comment in rustc_middle seems very plausible

I guess my experience is that perf wins don't come until you can profile and stare at the actual code path

simulacrum said:

is it other compiler tasks, or just other things?

Likewise to Wesley Wiser , but for me, it's mostly life getting in the way but occasionally other PRs.

but ok I see we're talking more about distractions...

I feel like having a dedicated working group is certainly a start

I've not been following along but how much of a "roadmap" does the working group have?

(no judgement any which way)

mmm, I don't think we have a roadmap

more a set of tasks that we're all working on

and checking in with eachother about

what sort of tasks?

and triaging the archive of issues

(like, specific issues?)

/me pulls up notes from last meeting

specific features

nikomatsakis said:

what sort of tasks?

I suppose this is the time to post this since that's what I'm working on:

Don't want to derail the discussion too much - not sure if the discussion is intended to be focused on "should this be the goal" or "how can we achieve this goal", but as an intermediate goal:

We could consider turning on Split DWARF for rustc by default too - might help cut down on link times. I've been intending to do this once Split DWARF lands and gets into the bootstrap compiler.

(like, @Santiago Pastorino has been working on PR #76896 for a while, whch avoids making local copies fo inline fns in debug mode)

pnkfelix said:

/me pulls up notes from last meeting

Those notes, for those who are interested: https://hackmd.io/WCuUWLCAQ2WTjUQyZeSCgg

pnkfelix said:

(like, Santiago Pastorino has been working on PR #76896 for a while, whch avoids making local copies fo inline fns in debug mode)

well I'm not really working on that, just did the PR and failed on CI and didn't have time to repro it

:)

Yep, I think split dwarf and other measures that we may not be able to stabilize but can definitely turn on for local compiles is absolutely in scope

You were the lucky one with the first item on those notes

But the point is

I think its safe to say we don't have a roadmap

OK. I guess I'm thinking that there are two basic discussions we could have

• figuring out if there are ways to make more time for people to focus
• looking at what the roadmap / organization of working group should be

the former seems kind of hard to do, unless it's a matter of rustc-related tasks that we can "reapportion"

it's necessarily going to be kind of personal :)

maybe s/working group/team, in some sense

(though I think that it'd be interesting to try and do some 1:1 conversations around this, I've wanted us to get better at mentoring people in these respects, or e.g. in working with employers to get paid time...)

e.g. I think there's a big question of "Can compiler team as a whole roadmap productively?"

mm, well, I debated the choice of words though

(I don't know)

we're at the 45 min mark

and

my hunch is that it's better to talk about wg but there is a good question of managing the set of wgs and relative priorities

Changes from #77307 also looked quite promising in terms of bootstrap times.

WhileI think there's useful stuff to talk about w.r.t. incr-comp

there is also the other bullet that @simulacrum put on here

Contributors aiming to edit specific components (rustdoc, test suite, standard library) can build just those components, at the cost of some network usage.

This is sort of "just" rustbuild

/me contemplates a rebranding frm "incremental wg" to "project fast bootstrap"

but it kinda gets into our guarantees around ABI etc, I've long wanted to sit down with someone and work out exactly what we need for reliable linkage and such

(I would like that very much)

to the point, I'd like us to have a way to signal when an ABI-breaking change occurs and track that

but it is tricky

it's not something the compiler can do alone, changes to stdlib etc apply too

yep

but such changes are still rare, right?

I don't know

e.g., adding a field to some stdlib struct?

I would like to also consider e.g. matklad's musings on whether std and rustc need such a tight coupling

probably occurs in every release right now, though not every nightly

I would think the problem is tracking them. But your point is that maybe they are also happening too often?

but yeah, also: I wouldn't be surprised if its every release.

but I also would think that from nightly-to-nightly its often fine

simulacrum said:

e.g. I think there's a big question of "Can compiler team as a whole roadmap productively?"

so to this point, one thing we could be talking about is whether we can recruit more people (and broader mindset) to "incremental wg / project fast bootstrap", I guess

right, yes, that would seem potentially beneficial

specifically more compiler team contributors, probably, though I think it's good to think about new folks

e.g. I could imagine that we say that work on non-incr/bootstrap speed is going to be deprioritized in review queues, though that may be stronger than we want

a mix would be good

I would very much like it if we were able to fulfill the vision of having the compiler team manage its set of active projects (with perhaps the expectation that compiler team contributors get involved in some active project etc)

simulacrum said:

e.g. I could imagine that we say that work on non-incr/bootstrap speed is going to be deprioritized in review queues, though that may be stronger than we want

this is interesting. I wouldn't want to deprioritize for a whole year

but for a month?

could be very interesting

I don't think that this should be the only project (e.g., I intend to keep pushing on RFC 2229, chalk integration), but I do think it is an important one

I'm mostly thinking that if we dropped a bunch of other stuff for a month, how much could we improve the cycle time for compiler dev

in a month

simulacrum said:

e.g. I could imagine that we say that work on non-incr/bootstrap speed is going to be deprioritized in review queues, though that may be stronger than we want

I would .. hmm. I wouldn't put it like this, but I do think that thinking about review speed and tying to active projects etc is a good idea.

@pnkfelix I do think that sprints are a great way to build momentum

though people mean many different things when they say sprints

one thing we've done in polonius wg from time to time that I found super helpful was

scheduling a week

right -- in some sense, e.g., if we can expect say a 25% improvement from a month's time of "all hands on deck" then we absolutely should do that in jan/feb, right?

where we all cleared our calendars

and put in like 4 hours each morning

that sounds nice

I suspect that a full month may be too tall of an ask, but that we might be able to manage a push for 1 week (which generates follow-up items that are then prioritized somewhat less so over the next 3 weeks)

yeah I think that makes sense too

if we synchronzied it right

so that its a week of hacking

I think it'd be great to try to figure out those week sprints perhaps at our planning meetings?

followed by one or more weeks of review/polish

so lets try to work out what the concrete outcomes here are

I think that's a really great idea and I would definitely be willing to block off time for that. A whole month would probably be difficult but I could certainly do a week.

I want to first double-check about the linkage stuff for just building individual components

@simulacrum it sounds like there's more research that would need to be done here, right?

sort of -- I think probably a 1:1 session to talk through some things might even be enough

okay. I'd like to have it in scope for 2021.

mostly again needs a block out of time :)

but I'd also like to figure out when that discussion should happen, roughly

e.g. do we table it until later, like March or April

I think I cannot commit to more until January

so that we can focus on the stuff for first bullet until then

I expect this to happen in parallel, roughly

In my head, this whole conversation is about January

incremental is sort of "in component"

mmm

I don't know if there's a benefit to doing them in parallel?

hm not sure I follow

If we want to see a payoff for the time invested within next year, it would be better to do it earlier in the year. So I would advocate for January or possibly February instead of March or April.

Well, if we had infinite dev resources

Yeah, I think earlier is better than later

then doing them in parallel would make perfect sense

pnkfelix said:

In my head, this whole conversation is about January

definitely not December :)

but we don't

but I agree that they don't need to happen in parallel

and given the resources we do have

Wesley Wiser said:

I think that's a really great idea and I would definitely be willing to block off time for that. A whole month would probably be difficult but I could certainly do a week.

Ps, this is why in polonius we do a week, much easier for folks who are not full time :)

I think it will be better for groups to focus on one bullet or the other

I could see e.g. focusing on the separate components in January and incr in Feb

interesting

(Idea:

We could make this a regular compiler team thing?

you think we'll get more payoff from doing separate components first?

I think so yes

I'm contemplating like "first week of every month is hack week, where the team as a whole picks a project"

@nikomatsakis you mean sprints?

the other 3 weeks are your own ;)

that's interesting

that's probably too ambitious but maybe a mildly scaled back version

e.g., we have some such weeks

separate components already cuts build times strongly for a good portion of people (e.g., a rustdoc dev only changing rustdoc)

this is all intermixed with our schedules for design meetings and what not

and incremental sort of only matters for compiler devs (obviously important)

e.g. if you change a crate that has a <1 minute rebuild time from clean, then incremental isn't going to be a huge win I imagine

I don't know

7 seconds vs 14 is big to me

whereas if we can get a compiler rebuild from 5 minutes to 1 minute, that seems big

anyway

we're over time

and we haven't actually made a concrete plan

I like Niko's idea of sprint weeks, maybe scheduled at planning meetings

but it sounds like @simulacrum is optimistic that we could see real progress in January on the indivdual component builds

if we focus on it.

right?

yes

okay.

simulacrum said:

I like Niko's idea of sprint weeks, maybe scheduled at planning meetings

yeah i'm into this idea, I think monthly is too much, but maybe every 2 months, or quarterly, or something

as in 1 sprint/month is too frequent?

if we're going to do it, I think they should be scheduled at fixed times, like our meeting slots, so people can play way ahead

or planning them each month is too frequent?

maybe, maybe not

I was thinking holding them would be too much but I could be persuaded otherwise

one thing I will add:

I think good outcome right now is to plan to plan at the next planning meeting

I know that many employers give 20% time and they are happy for people to 'save it' and use it in a block

so e.g. I've talked to employers who say that they'd be happy to have people spend 1 week / month on open source

or no, wait

we don't have a planning meeting in decebmer

we cancelled it yesterday

:lol:

lol

okay

um

nikomatsakis said:

so e.g. I've talked to employers who say that they'd be happy to have people spend 1 week / month on open source

some kind of sprint-like structure could really optimize for this :)

we'll make a concrete plan at some point this month

I think we can maybe just, uh, schedule last week of Jan or something

for when we will focus on each of these two bullets

yeah

it won't take long to decide

I think it should be a schedule like "last week of every month" except Dec

either at a triage meeting or at one of the friday meetings

but we can start with Jan :)

ideally we'd have people sign up, and we would obviously not expect 100% participation

anyway I'm getting ahead of myself but I'm excited :)

/me wonders why "last week of month" is better than "first week of month"

it doesn't matter, but first week of jan is bad :)

well, this Jan anyway

clearly "second week of month" is the ideal compromise

sure, but you've already said we'd cancel last week of december

last week may be bad because of how the human brain works

in particular:

I would like to suggest that we try to do the planning for the sprint at least 2 weeks in advance. That way, if need be, we can use up to two Friday meetings to finalize the details. I think going into the week with things still undecided would likely hurt productivity a lot.

actually i'm now wondering if 2nd week of month is ideal

I can imagine people have something due in February (or whatever)

and they have to cancel sprint to finish it

Wesley Wiser said:

I would like to suggest that we try to do the planning for the sprint at least 2 weeks in advance. That way, if need be, we can use up to two Friday meetings to finalize the details. I think going into the week with things still undecided would likely hurt productivity a lot.

yes, this was a thought I had about last week

i.e., that gives you time in that month to plan

clearly 3rd week is ideal :)

I see

Okay then maybe 3rd week of month

heh

you start planning or the Jan sprint in Jan

people will definitely fail to plan to Feb sprint in Jan

etc

s/people/Niko/

ok I have to go :)

alright. we'll finalize all this later

thanks to everyone in @T-compiler/meeting for attending!!!

simulacrum said:

This is sort of "just" rustbuild

well, for tools specifically I think it would be a giant boost not to have to require building rustc first, cc https://github.com/rust-lang/rust/pull/79540

simulacrum said:

separate components already cuts build times strongly for a good portion of people (e.g., a rustdoc dev only changing rustdoc)

oh oops I didn't read far enough :laughing:

another thing is that the conversation has focused really strongly on incremental builds - which definitely do matter - but not talked much about full/first-time builds

for rustdoc specifically I guess that will matter less once rustc is downloaded from CI

but in general I think it's an area we should work on, and it would help incremental builds too

nikomatsakis said:

to the point, I'd like us to have a way to signal when an ABI-breaking change occurs and track that

this would be amazing :heart: I really hate having to tell people "sometimes incremental just breaks when bootstrapping"