Stream: t-compiler/wg-learning

Topic: organization of the guide


mark-i-m (Mar 20 2020 at 16:37, on Zulip):

Hi all, I hope everyone is staying well.

mark-i-m (Mar 20 2020 at 16:38, on Zulip):

I've been thinking about the overall organization of the guide (especially "part 2")

mark-i-m (Mar 20 2020 at 16:38, on Zulip):

and in conjunction with our overview chapter

mark-i-m (Mar 20 2020 at 16:38, on Zulip):

and especially all of the type-system related stuff, which is pretty scattered atm

mark-i-m (Mar 20 2020 at 16:39, on Zulip):

I'm thinking we should try to reorganize as follows:

mark-i-m (Mar 20 2020 at 16:39, on Zulip):

Part A

    - [High-level overview of the compiler source](./high-level-overview.md)
    - [The Rustc Driver and Interface](./rustc-driver.md)
        - [Rustdoc](./rustdoc.md)
    - [Queries: demand-driven compilation](./query.md)
        - [The Query Evaluation Model in Detail](./queries/query-evaluation-model-in-detail.md)
        - [Incremental compilation](./queries/incremental-compilation.md)
        - [Incremental compilation In Detail](./queries/incremental-compilation-in-detail.md)
        - [Debugging and Testing](./incrcomp-debugging.md)
        - [Profiling Queries](./queries/profiling.md)
        - [Salsa](./salsa.md)
    - [Memory Management in Rustc](./memory.md)
mark-i-m (Mar 20 2020 at 16:40, on Zulip):

This part would be primarily about the high-level architecture of the compiler

mark-i-m (Mar 20 2020 at 16:40, on Zulip):

Part B

    - [Lexing and Parsing](./the-parser.md)
    - [`#[test]` Implementation](./test-implementation.md)
    - [Panic Implementation](./panic-implementation.md)
    - [Macro expansion](./macro-expansion.md)
    - [Name resolution](./name-resolution.md)
    - [The HIR (High-level IR)](./hir.md)
        - [Lowering AST to HIR](./lowering.md)
        - [Debugging](./hir-debugging.md)
    - [The MIR (Mid-level IR), Part 1](./mir/index.md)
        - [MIR construction](./mir/construction.md)
        - [MIR visitor and traversal](./mir/visitor.md)
        - [MIR passes: getting the MIR for a function](./mir/passes.md)
    - [Closure expansion](./closure.md)
mark-i-m (Mar 20 2020 at 16:40, on Zulip):

This part would be mostly about front-end-y stuff

mark-i-m (Mar 20 2020 at 16:40, on Zulip):

and the various IRs

mark-i-m (Mar 20 2020 at 16:43, on Zulip):

Part C

    - [The `ty` module: representing types](./ty.md)
        - [Generics and substitutions](./generics.md)
        - [`TypeFolder` and `TypeFoldable`](./ty-fold.md)
    - [Generic arguments](./generic_arguments.md)
    - [Type inference](./type-inference.md)
    - [Trait solving (old-style)](./traits/resolution.md)
        - [Higher-ranked trait bounds](./traits/hrtb.md)
        - [Caching subtleties](./traits/caching.md)
        - [Specialization](./traits/specialization.md)
    - [Trait solving (new-style)](./traits/index.md)
        - [Lowering to logic](./traits/lowering-to-logic.md)
            - [Goals and clauses](./traits/goals-and-clauses.md)
            - [Equality and associated types](./traits/associated-types.md)
            - [Implied bounds](./traits/implied-bounds.md)
            - [Region constraints](./traits/regions.md)
            - [The lowering module in rustc](./traits/lowering-module.md)
            - [Lowering rules](./traits/lowering-rules.md)
            - [Well-formedness checking](./traits/wf.md)
        - [Canonical queries](./traits/canonical-queries.md)
            - [Canonicalization](./traits/canonicalization.md)
        - [The SLG solver](./traits/slg.md)
        - [An Overview of Chalk](./traits/chalk-overview.md)
        - [Bibliography](./traits/bibliography.md)
    - [Type checking](./type-checking.md)
        - [Method Lookup](./method-lookup.md)
        - [Variance](./variance.md)
        - [Opaque Types](./opaque-types-type-alias-impl-trait.md)
    - [The borrow checker](./borrow_check.md)
        - [Tracking moves and initialization](./borrow_check/moves_and_initialization.md)
            - [Move paths](./borrow_check/moves_and_initialization/move_paths.md)
        - [MIR type checker](./borrow_check/type_check.md)
        - [Region inference](./borrow_check/region_inference.md)
            - [Constraint propagation](./borrow_check/region_inference/constraint_propagation.md)
            - [Lifetime parameters](./borrow_check/region_inference/lifetime_parameters.md)
            - [Member constraints](./borrow_check/region_inference/member_constraints.md)
            - [Placeholders and universes][pau]
            - [Closure constraints](./borrow_check/region_inference/closure_constraints.md)
            - [Error reporting](./borrow_check/region_inference/error_reporting.md)
        - [Two-phase-borrows](./borrow_check/two_phase_borrows.md)
    - [Parameter Environments](./param_env.md)

This part would talk about all the type-system related things. Currently they are very scattered and the chapters are mostly Niko's thoughts at various times when he was working on stuff, so they don't cohere together very well. I think pulling these together and making an effort to make them coherent might making things much easier to understand. These chapters would come after parts (a) and (b) above and would make use of all the machinery introduced there.

mark-i-m (Mar 20 2020 at 16:44, on Zulip):

Also, it might make sense to move the chalk parts elsewhere (perhaps to the chalk book as previously proposed by niko and tmandry) since that stuff is apparently not coming soon?

mark-i-m (Mar 20 2020 at 16:44, on Zulip):

Part D

    - [Constant evaluation](./const-eval.md)
        - [miri const evaluator](./miri.md)
    - [The MIR (Mid-level IR), Part 2](./mir/index.md)
        - [MIR optimizations](./mir/optimizations.md)
        - [Debugging](./mir/debugging.md)
    - [Compiler Backend](./backend/backend.md)
        - [Monomorphization](./backend/monomorph.md)
        - [Lowering MIR](./backend/lowering-mir.md)
        - [Code Generation](./backend/codegen.md)
            - [Updating LLVM](./backend/updating-llvm.md)
            - [Debugging LLVM](./backend/debugging.md)
            - [Backend Agnostic Codegen](./backend/backend-agnostic.md)
        - [Profile-guided Optimization](./profile-guided-optimization.md)
        - [Sanitizers Support](./sanitizers.md)
        - [Debugging Support in Rust Compiler](./debugging-support-in-rustc.md)

This part is all the backend stuff

mark-i-m (Mar 20 2020 at 16:45, on Zulip):

I'm curious what are people's thought on this organization

LeSeulArtichaut (Mar 20 2020 at 17:00, on Zulip):

This organization makes a lot of sense to me. I think it follows the "what the compiler does to my code" logic

LeSeulArtichaut (Mar 20 2020 at 17:00, on Zulip):

I mean, you need to do Part B to do Part C, and Part C to do Part D

mark-i-m (Mar 20 2020 at 17:02, on Zulip):

Well, very nearly... Part C is kind of spread out a bit. For example type checking happens on the HIR while borrow checking happens on the MIR just before monomorphization

mark-i-m (Mar 20 2020 at 17:02, on Zulip):

But logically, I tend to think of all of that as "the type system"

mark-i-m (Mar 20 2020 at 17:05, on Zulip):

I guess type systems are like that in general... they are mostly "unnecessary" computation that doesn't end up represented that much in the output binary (I say mostly, because choosing which trait to codegen does matter).

centril (Mar 20 2020 at 17:05, on Zulip):

Missing bits:

mark-i-m (Mar 20 2020 at 17:05, on Zulip):

But that means that it doesn't easily fit into a flow of "how my code got generated"

LeSeulArtichaut (Mar 20 2020 at 17:06, on Zulip):

Thanks @centril :+1:

mark-i-m (Mar 20 2020 at 17:06, on Zulip):

@centril ast_validation and pattern/exhaustiveness checking are generally missing from the guide

mark-i-m (Mar 20 2020 at 17:07, on Zulip):

But we do have a bit on HAIR

centril (Mar 20 2020 at 17:07, on Zulip):

@mark-i-m I would add stubs for now

centril (Mar 20 2020 at 17:07, on Zulip):

Lowering to MIR should come before monomorphization?

centril (Mar 20 2020 at 17:07, on Zulip):

also it's not part of the backend

mark-i-m (Mar 20 2020 at 17:07, on Zulip):

The HAIR is here (hehe) https://rustc-dev-guide.rust-lang.org/mir/construction.html?highlight=HAIR#mir-construction

centril (Mar 20 2020 at 17:08, on Zulip):

would be good to lift out a separate topic for HAIR and then link to it from MIR construction

mark-i-m (Mar 20 2020 at 17:15, on Zulip):

Lowering to MIR should come before monomorphization?

We both misread: it is about Lowering MIR [to LLVM IR] and this is part of the backend

mark-i-m (Mar 20 2020 at 17:15, on Zulip):

MIR construction is up above in part B

centril (Mar 20 2020 at 17:15, on Zulip):

oh

centril (Mar 20 2020 at 17:16, on Zulip):

yeah that could be clarified in the title :P

mark-i-m (Mar 20 2020 at 17:21, on Zulip):

@centril added/updated ast validation and pattern/exhaustiveness checking in Part B. Is it in the right place? Do you have any other thoughts on this organization?

centril (Mar 20 2020 at 17:22, on Zulip):

@mark-i-m ast validation is not part of HIR / lowering

centril (Mar 20 2020 at 17:22, on Zulip):

so it should be just before nameres

centril (Mar 20 2020 at 17:23, on Zulip):

@mark-i-m pattern and exhaustiveness checking is part of "the type system", the former is done on HIR, the latter on HAIR

centril (Mar 20 2020 at 17:23, on Zulip):

but pattern checking calls exhaustiveness checking

centril (Mar 20 2020 at 17:24, on Zulip):

its in the same query

centril (Mar 20 2020 at 17:24, on Zulip):

So I would put it in the same chapter before borrow checking

mark-i-m (Mar 20 2020 at 17:26, on Zulip):

Thanks @centril updated

mark-i-m (Mar 21 2020 at 02:32, on Zulip):

I just implemented some helpful features for mdbook to support the above: https://github.com/rust-lang/mdBook/pull/1171

mark-i-m (Mar 21 2020 at 02:32, on Zulip):

Hopefully it can be in the next version of mdbook

Chris Simpkins (Mar 22 2020 at 02:38, on Zulip):

LGTM Mark. I'm not sure if you mean to eliminate Part 1 based on the comments above? I think that the information there is very helpful to new contributors. Maybe it could be moved to the end of the guide as a Part E: Contributor Guide section that keeps existing/expands upon information like (1) compiling the compiler - tricks/tips; (2) the x.py guide - how x.py can help you to contribute more easily to the compiler; (3) the RFC process and how to contribute new features; (4) how to report bugs; (5) how to participate in bug diagnostics; (5) how to contribute bug fixes; (6) how to submit a good compiler test with your pull request, how various areas of the compiler source are tested; (7) code conventions (fmt in particular has been a pain point recently and likely warrants a bit of detail)?

mark-i-m (Mar 22 2020 at 22:40, on Zulip):

Oh yeah, sorry, I meant all of that to be a reorganization of what is currently part 2. I very much agree that we should keep all that info, though we might want to reorganize it somehow?

Santiago Pastorino (Mar 24 2020 at 18:56, on Zulip):

just read the organization and all the comments here

Santiago Pastorino (Mar 24 2020 at 18:56, on Zulip):

the organization looks great in my opinion

Santiago Pastorino (Mar 24 2020 at 18:56, on Zulip):

@mark-i-m are you writing this down somewhere?

Santiago Pastorino (Mar 24 2020 at 18:56, on Zulip):

how do you think we can implement this on our current structure?

mark-i-m (Mar 24 2020 at 23:36, on Zulip):

Well, part of it is just reorganizing chapters

mark-i-m (Mar 24 2020 at 23:37, on Zulip):

I've also made a pr to mdbook to make it easier to split the book into parts, but it doesn't seem to be getting attention

mark-i-m (Mar 24 2020 at 23:38, on Zulip):

This one: https://github.com/rust-lang/mdBook/pull/1171

mark-i-m (Mar 24 2020 at 23:38, on Zulip):

Does anyone know who could review such a PR? (e.g. @Eric Huss would you be able to?)

Eric Huss (Mar 24 2020 at 23:59, on Zulip):

I'll try to get to it when I can, but there is a very long queue of reviews. There are other PRs that will probably conflict, like https://github.com/rust-lang/mdBook/pull/1153. It also looks like it changes some APIs, so it may need to roll into 0.4. Just to set expectations, it may take a while to get through.

mark-i-m (Mar 25 2020 at 01:23, on Zulip):

@Eric Huss Ok, thanks for the heads up

mark-i-m (Mar 25 2020 at 01:24, on Zulip):

@Santiago Pastorino In that case, I would recommend doing it as we are already doing parts: we make them very large top-level chapters.

Santiago Pastorino (Mar 25 2020 at 14:58, on Zulip):

@mark-i-m can you explain more about your strategy? I feel like I get what you meant but just in case :)

mark-i-m (Mar 26 2020 at 04:29, on Zulip):

@Santiago Pastorino we already have almost all of those chapters. I recommend we just reorganize the table of contents at first. Then, we can work on making each part "flow" better...

mark-i-m (Mar 26 2020 at 04:30, on Zulip):

And in terms of mdbook, for now we can just have each part be a mdbook chapter with all of the actually chapters as subchapters of the part (like we are doing today with parts 1 and 2)

Santiago Pastorino (Mar 26 2020 at 11:56, on Zulip):

@mark-i-m :+1:

mark-i-m (Apr 01 2020 at 03:19, on Zulip):

PR is up

Last update: Apr 03 2020 at 17:45UTC