Hi all, I hope everyone is staying well.
I've been thinking about the overall organization of the guide (especially "part 2")
and in conjunction with our overview chapter
and especially all of the type-system related stuff, which is pretty scattered atm
I'm thinking we should try to reorganize as follows:
- [High-level overview of the compiler source](./high-level-overview.md) - [The Rustc Driver and Interface](./rustc-driver.md) - [Rustdoc](./rustdoc.md) - [Queries: demand-driven compilation](./query.md) - [The Query Evaluation Model in Detail](./queries/query-evaluation-model-in-detail.md) - [Incremental compilation](./queries/incremental-compilation.md) - [Incremental compilation In Detail](./queries/incremental-compilation-in-detail.md) - [Debugging and Testing](./incrcomp-debugging.md) - [Profiling Queries](./queries/profiling.md) - [Salsa](./salsa.md) - [Memory Management in Rustc](./memory.md)
This part would be primarily about the high-level architecture of the compiler
- [Lexing and Parsing](./the-parser.md) - [`#[test]` Implementation](./test-implementation.md) - [Panic Implementation](./panic-implementation.md) - [Macro expansion](./macro-expansion.md) - [Name resolution](./name-resolution.md) - [The HIR (High-level IR)](./hir.md) - [Lowering AST to HIR](./lowering.md) - [Debugging](./hir-debugging.md) - [The MIR (Mid-level IR), Part 1](./mir/index.md) - [MIR construction](./mir/construction.md) - [MIR visitor and traversal](./mir/visitor.md) - [MIR passes: getting the MIR for a function](./mir/passes.md) - [Closure expansion](./closure.md)
This part would be mostly about front-end-y stuff
and the various IRs
- [The `ty` module: representing types](./ty.md) - [Generics and substitutions](./generics.md) - [`TypeFolder` and `TypeFoldable`](./ty-fold.md) - [Generic arguments](./generic_arguments.md) - [Type inference](./type-inference.md) - [Trait solving (old-style)](./traits/resolution.md) - [Higher-ranked trait bounds](./traits/hrtb.md) - [Caching subtleties](./traits/caching.md) - [Specialization](./traits/specialization.md) - [Trait solving (new-style)](./traits/index.md) - [Lowering to logic](./traits/lowering-to-logic.md) - [Goals and clauses](./traits/goals-and-clauses.md) - [Equality and associated types](./traits/associated-types.md) - [Implied bounds](./traits/implied-bounds.md) - [Region constraints](./traits/regions.md) - [The lowering module in rustc](./traits/lowering-module.md) - [Lowering rules](./traits/lowering-rules.md) - [Well-formedness checking](./traits/wf.md) - [Canonical queries](./traits/canonical-queries.md) - [Canonicalization](./traits/canonicalization.md) - [The SLG solver](./traits/slg.md) - [An Overview of Chalk](./traits/chalk-overview.md) - [Bibliography](./traits/bibliography.md) - [Type checking](./type-checking.md) - [Method Lookup](./method-lookup.md) - [Variance](./variance.md) - [Opaque Types](./opaque-types-type-alias-impl-trait.md) - [The borrow checker](./borrow_check.md) - [Tracking moves and initialization](./borrow_check/moves_and_initialization.md) - [Move paths](./borrow_check/moves_and_initialization/move_paths.md) - [MIR type checker](./borrow_check/type_check.md) - [Region inference](./borrow_check/region_inference.md) - [Constraint propagation](./borrow_check/region_inference/constraint_propagation.md) - [Lifetime parameters](./borrow_check/region_inference/lifetime_parameters.md) - [Member constraints](./borrow_check/region_inference/member_constraints.md) - [Placeholders and universes][pau] - [Closure constraints](./borrow_check/region_inference/closure_constraints.md) - [Error reporting](./borrow_check/region_inference/error_reporting.md) - [Two-phase-borrows](./borrow_check/two_phase_borrows.md) - [Parameter Environments](./param_env.md)
This part would talk about all the type-system related things. Currently they are very scattered and the chapters are mostly Niko's thoughts at various times when he was working on stuff, so they don't cohere together very well. I think pulling these together and making an effort to make them coherent might making things much easier to understand. These chapters would come after parts (a) and (b) above and would make use of all the machinery introduced there.
Also, it might make sense to move the chalk parts elsewhere (perhaps to the chalk book as previously proposed by niko and tmandry) since that stuff is apparently not coming soon?
- [Constant evaluation](./const-eval.md) - [miri const evaluator](./miri.md) - [The MIR (Mid-level IR), Part 2](./mir/index.md) - [MIR optimizations](./mir/optimizations.md) - [Debugging](./mir/debugging.md) - [Compiler Backend](./backend/backend.md) - [Monomorphization](./backend/monomorph.md) - [Lowering MIR](./backend/lowering-mir.md) - [Code Generation](./backend/codegen.md) - [Updating LLVM](./backend/updating-llvm.md) - [Debugging LLVM](./backend/debugging.md) - [Backend Agnostic Codegen](./backend/backend-agnostic.md) - [Profile-guided Optimization](./profile-guided-optimization.md) - [Sanitizers Support](./sanitizers.md) - [Debugging Support in Rust Compiler](./debugging-support-in-rustc.md)
This part is all the backend stuff
I'm curious what are people's thought on this organization
This organization makes a lot of sense to me. I think it follows the "what the compiler does to my code" logic
I mean, you need to do Part B to do Part C, and Part C to do Part D
Well, very nearly... Part C is kind of spread out a bit. For example type checking happens on the HIR while borrow checking happens on the MIR just before monomorphization
But logically, I tend to think of all of that as "the type system"
I guess type systems are like that in general... they are mostly "unnecessary" computation that doesn't end up represented that much in the output binary (I say mostly, because choosing which trait to codegen does matter).
check_match.rs) and exhaustiveness checking (
But that means that it doesn't easily fit into a flow of "how my code got generated"
Thanks @centril :+1:
@centril ast_validation and pattern/exhaustiveness checking are generally missing from the guide
But we do have a bit on HAIR
@mark-i-m I would add stubs for now
Lowering to MIR should come before monomorphization?
also it's not part of the backend
The HAIR is here (hehe) https://rustc-dev-guide.rust-lang.org/mir/construction.html?highlight=HAIR#mir-construction
would be good to lift out a separate topic for HAIR and then link to it from MIR construction
Lowering to MIR should come before monomorphization?
We both misread: it is about
Lowering MIR [to LLVM IR] and this is part of the backend
MIR construction is up above in part B
yeah that could be clarified in the title :P
@centril added/updated ast validation and pattern/exhaustiveness checking in Part B. Is it in the right place? Do you have any other thoughts on this organization?
@mark-i-m ast validation is not part of HIR / lowering
so it should be just before nameres
@mark-i-m pattern and exhaustiveness checking is part of "the type system", the former is done on HIR, the latter on HAIR
but pattern checking calls exhaustiveness checking
its in the same query
So I would put it in the same chapter before borrow checking
Thanks @centril updated
I just implemented some helpful features for mdbook to support the above: https://github.com/rust-lang/mdBook/pull/1171
Hopefully it can be in the next version of mdbook
LGTM Mark. I'm not sure if you mean to eliminate Part 1 based on the comments above? I think that the information there is very helpful to new contributors. Maybe it could be moved to the end of the guide as a Part E: Contributor Guide section that keeps existing/expands upon information like (1) compiling the compiler - tricks/tips; (2) the x.py guide - how x.py can help you to contribute more easily to the compiler; (3) the RFC process and how to contribute new features; (4) how to report bugs; (5) how to participate in bug diagnostics; (5) how to contribute bug fixes; (6) how to submit a good compiler test with your pull request, how various areas of the compiler source are tested; (7) code conventions (fmt in particular has been a pain point recently and likely warrants a bit of detail)?
Oh yeah, sorry, I meant all of that to be a reorganization of what is currently part 2. I very much agree that we should keep all that info, though we might want to reorganize it somehow?
just read the organization and all the comments here
the organization looks great in my opinion
@mark-i-m are you writing this down somewhere?
how do you think we can implement this on our current structure?
Well, part of it is just reorganizing chapters
I've also made a pr to mdbook to make it easier to split the book into parts, but it doesn't seem to be getting attention
Does anyone know who could review such a PR? (e.g. @Eric Huss would you be able to?)
I'll try to get to it when I can, but there is a very long queue of reviews. There are other PRs that will probably conflict, like https://github.com/rust-lang/mdBook/pull/1153. It also looks like it changes some APIs, so it may need to roll into 0.4. Just to set expectations, it may take a while to get through.
@Eric Huss Ok, thanks for the heads up
@Santiago Pastorino In that case, I would recommend doing it as we are already doing parts: we make them very large top-level chapters.
@mark-i-m can you explain more about your strategy? I feel like I get what you meant but just in case :)
@Santiago Pastorino we already have almost all of those chapters. I recommend we just reorganize the table of contents at first. Then, we can work on making each part "flow" better...
And in terms of mdbook, for now we can just have each part be a mdbook chapter with all of the actually chapters as subchapters of the part (like we are doing today with parts 1 and 2)
PR is up