Stream: rustdoc

Topic: Building rustdoc out of tree #79540


view this post on Zulip Joshua Nelson (Nov 29 2020 at 19:53):

one issue is that the rust-toolchain file will be constantly out of date, since rustc is always breaking :/

view this post on Zulip Joshua Nelson (Nov 29 2020 at 19:53):

not much I can do about that though, other than toolstate

view this post on Zulip Robin Moussu (Nov 29 2020 at 20:07):

Does this mean that we just have to rebuild rustdoc each time we update nightly?

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:07):

yes

view this post on Zulip Robin Moussu (Nov 29 2020 at 20:08):

That's seems reasonable (even if slightly annoying). If someone is developing on rustdoc, I don't think he will need to update nightly every day

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:08):

well, almost every day :sweat_smile:

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:08):

see above about toolstate

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:08):

rustc changes frequently

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:09):

also when rebasing it will download a new toolchain too, since rebasing changes the version of rustc you're building against

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:15):

Joshua Nelson said:

also when rebasing it will download a new toolchain too, since rebasing changes the version of rustc you're building against

the reason that doesn't normally rebuild is x.py has some logic that say "just pretend all master toolchains are the same" :upside_down: which has its own issues https://github.com/rust-lang/rust/issues/76720

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:17):

ok, so here are the flags x.py invokes compiletest with:

running: "/home/joshua/rustc2/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/home/joshua/rustc2/build/x86_64-unknown-linux-gnu/stage1/lib" "--run-lib-path" "/home/joshua/rustc2/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/home/joshua/rustc2/build/x86_64-unknown-linux-gnu/stage1/bin/rustc" "--rustdoc-path" "/home/joshua/rustc2/build/x86_64-unknown-linux-gnu/stage1/bin/rustdoc" "--src-base" "/home/joshua/rustc2/src/test/rustdoc" "--build-base" "/home/joshua/rustc2/build/x86_64-unknown-linux-gnu/test/rustdoc" "--stage-id" "stage1-x86_64-unknown-linux-gnu" "--suite" "rustdoc" "--mode" "rustdoc" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/home/joshua/rustc2/build/x86_64-unknown-linux-gnu/ci-llvm/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/home/joshua/rustc2/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/home/joshua/rustc2/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python" "--lldb-python" "/usr/bin/python" "--gdb" "/usr/bin/gdb" "self" "--verbose" "--quiet" "--llvm-version" "11.0.0-rust-1.49.0-nightly" "--llvm-components" "aarch64 aarch64asmparser aarch64codegen aarch64desc aarch64disassembler aarch64info aarch64utils aggressiveinstcombine all all-targets analysis arm armasmparser armcodegen armdesc armdisassembler arminfo armutils asmparser asmprinter avr avrasmparser avrcodegen avrdesc avrdisassembler avrinfo binaryformat bitreader bitstreamreader bitwriter cfguard codegen core coroutines coverage debuginfocodeview debuginfodwarf debuginfogsym debuginfomsf debuginfopdb demangle dlltooldriver dwarflinker engine executionengine extensions frontendopenmp fuzzmutate globalisel hexagon hexagonasmparser hexagoncodegen hexagondesc hexagondisassembler hexagoninfo instcombine instrumentation interpreter ipo irreader jitlink libdriver lineeditor linker lto mc mca mcdisassembler mcjit mcparser mips mipsasmparser mipscodegen mipsdesc mipsdisassembler mipsinfo mirparser msp430 msp430asmparser msp430codegen msp430desc msp430disassembler msp430info native nativecodegen nvptx nvptxcodegen nvptxdesc nvptxinfo objcarcopts object objectyaml option orcerror orcjit passes powerpc powerpcasmparser powerpccodegen powerpcdesc powerpcdisassembler powerpcinfo profiledata remarks riscv riscvasmparser riscvcodegen riscvdesc riscvdisassembler riscvinfo riscvutils runtimedyld scalaropts selectiondag sparc sparcasmparser sparccodegen sparcdesc sparcdisassembler sparcinfo support symbolize systemz systemzasmparser systemzcodegen systemzdesc systemzdisassembler systemzinfo tablegen target textapi transformutils vectorize webassembly webassemblyasmparser webassemblycodegen webassemblydesc webassemblydisassembler webassemblyinfo windowsmanifest x86 x86asmparser x86codegen x86desc x86disassembler x86info xray" "--cc" "" "--cxx" "" "--cflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" ""

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:17):

I'm hoping not all of those are actually necessary

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:18):

let me see how clippy does this

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:20):

oh boy https://github.com/rust-lang/rust-clippy/blob/master/tests/compile-test.rs

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:25):

ok, I'm going to steal a page from clippy's book and use the stable version instead of the in-tree compile test

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:26):

it will rebuild a little more but I don't expect anyone to use both

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:26):

the in-tree one is only meant to be used as a binary and I would rather not put a build system in for the tests themselves

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:31):

@Manish Goregaokar does compiletest_rs have a way to run ui tests for rustdoc?

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:31):

telling it rustdoc is actually rustc seems doomed to failure

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:34):

... is it still maintained? april was a while ago ...

view this post on Zulip Robin Moussu (Nov 29 2020 at 20:35):

Joshua Nelson said:

well, almost every day :sweat_smile:

Do you really update your nightly every day?

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:35):

actually I just realized this will be missing all the diff stuff I just added to rustdoc which is really unfortunate https://github.com/rust-lang/rust/pull/78752 - does compiletest_rs sync with the upstream version at all?

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:36):

Robin Moussu said:

Joshua Nelson said:

well, almost every day :sweat_smile:

Do you really update your nightly every day?

I don't use nightly at all, but you'd have to to build rustdoc without x.py

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:36):

you need the version of nightly that matches the source you have in-tree

view this post on Zulip Robin Moussu (Nov 29 2020 at 20:37):

ah, yes. I was in the shoes of someone who build rustdoc, not use it for documenting rustc. Then yes, it's kind of anoying.

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:37):

building is what matters, not documenting

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:38):

rustdoc is using rustc's internal APIs

view this post on Zulip Robin Moussu (Nov 29 2020 at 20:39):

but why do you need to build rustdoc as frequently? The CI nighly build, but no humans. Or am I missing something obvious?

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:39):

I don't understand - are you asking whether I build rustdoc almost every day?

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:39):

(yes)

view this post on Zulip Robin Moussu (Nov 29 2020 at 20:42):

Usually when I work on something, I fetch the master, work on it for as much time as needed, then (if needed) fetch again master, and rebase my branch. If I was working on rustdoc, I don't think I would fetch the master branch every day, unless I know something I'm working on depends on someone else work. I would fetch more realistically about one time a week I think. (this assume a somewhat stable codebase).

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:43):

oh I see what you mean - you're asking if I rebase every day

view this post on Zulip Robin Moussu (Nov 29 2020 at 20:43):

are you asking whether I build rustdoc almost every day?

rustdoc with up-to-date source fresh from the day

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:43):

it depends on the PRs I'm working on, I probably rebase a lot more than most contirbutors

view this post on Zulip Robin Moussu (Nov 29 2020 at 20:43):

Ah, then yes, it would be really annoying for you.

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:43):

I'd say probably almost every day though, yeah

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:43):

I got a new computer just so it wouldn't take as long to build rustc :sweat_smile:

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:44):

especially for big PRs that need changes to rustc internals, and I need to compare before and after

view this post on Zulip Joshua Nelson (Nov 29 2020 at 20:45):

(note that I'm being very careful throughout this whole discussion to not change the process of building with x.py at all)

view this post on Zulip Robin Moussu (Nov 29 2020 at 20:46):

Anyway, it's time for me to eat, have a great day

view this post on Zulip Joshua Nelson (Nov 29 2020 at 21:53):

https://github.com/rust-lang/rust/pull/79540

view this post on Zulip Manish Goregaokar (Nov 30 2020 at 01:44):

Joshua Nelson said:

Manish Goregaokar does compiletest_rs have a way to run ui tests for rustdoc?

no idea. IMO we should actually move compiletest out of tree and stop maintaining it in tree

view this post on Zulip Joshua Nelson (Nov 30 2020 at 01:45):

oh I was actually thinking the other way, turning compiletest in-tree into a library and changing clippy to use a git dependency

view this post on Zulip Joshua Nelson (Nov 30 2020 at 01:46):

that's less work for both sides: no more syncs, and it's easy to modify in-tree if needed

view this post on Zulip Joshua Nelson (Nov 30 2020 at 01:46):

(although I was even thinking of cargo changes, but we don't need to wait for those: https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/Depend.20on.20a.20binary.3F)

view this post on Zulip Joshua Nelson (Nov 30 2020 at 01:48):

the current plan for rustdoc is actually to use x.py still though https://github.com/rust-lang/rust/pull/79540

view this post on Zulip Joshua Nelson (Dec 02 2020 at 20:42):

this is currently blocked on me figuring out how to slot things into rustbuild's caching, which is based on TypeId of all things: https://github.com/rust-lang/rust/pull/79540#issuecomment-735942001

view this post on Zulip Joshua Nelson (Dec 03 2020 at 22:31):

@simulacrum do you happen to have time to suggest an approach? no rush, but I don't want to jump into coding when I don't know how you want it tackled

view this post on Zulip simulacrum (Dec 03 2020 at 22:32):

uh not right this minute but if you want to assign me and set waiting-on-review I hope to schedule some review time this weekend, can engage more then (leave a comment too)

view this post on Zulip Joshua Nelson (Dec 03 2020 at 22:33):

sounds great, thanks!

view this post on Zulip Joshua Nelson (Dec 27 2020 at 05:26):

@GuillaumeGomez I have a christmas present for you :) https://github.com/rust-lang/rust/pull/79540 is working

view this post on Zulip GuillaumeGomez (Dec 27 2020 at 14:55):

Awesome! \o/

view this post on Zulip Joshua Nelson (Dec 27 2020 at 15:22):

lol I tried to rebase this over a 6-month old commit to see what would break and even git got confused - it was a commit from rustc_codegen_cranelift before the submodule sync :laughing:

view this post on Zulip GuillaumeGomez (Dec 27 2020 at 15:22):

Don't try stuff like that! :o

view this post on Zulip GuillaumeGomez (Dec 27 2020 at 15:23):

when I have too much problems, I simply switch to master and then either port commits by hand (sometimes it works better) or cherry-pick them

view this post on Zulip Joshua Nelson (Dec 27 2020 at 15:23):

well in practice it probably doesn't matter, but if someone tries this with an old enough commit it will break:

This will break if you rebase an old enough branch (I think commits are kept at most 160 days?). This should fail gracefully and build from source in that case, possibly suggesting to rebase to a later commit.

view this post on Zulip GuillaumeGomez (Dec 27 2020 at 15:23):

my oldest rebase was years late

view this post on Zulip GuillaumeGomez (Dec 27 2020 at 15:23):

it was horrible haha

view this post on Zulip Joshua Nelson (Dec 27 2020 at 15:27):

well the difference is this will break before the rebase

view this post on Zulip Joshua Nelson (Dec 27 2020 at 15:27):

just by the passage of time

view this post on Zulip Joshua Nelson (Dec 27 2020 at 15:45):

Since tracing is loaded from sysroot, it will be compiled with static_max_level_info even for rustdoc's own debugging.

by the way, if someone has ideas for fixing this, please let me know

view this post on Zulip Noah Lev (Dec 28 2020 at 01:17):

Joshua Nelson said:

Since tracing is loaded from sysroot, it will be compiled with static_max_level_info even for rustdoc's own debugging.

by the way, if someone has ideas for fixing this, please let me know

Ask in #t-compiler/help? Probably unlikely someone will find the question here :)

view this post on Zulip Joshua Nelson (Dec 28 2020 at 17:57):

https://rust-lang.zulipchat.com/#narrow/stream/182449-t-compiler.2Fhelp/topic/Rustc.20loads.20sysroot.20artifacts.20before.20cargo.20ones.3F

view this post on Zulip Joshua Nelson (Dec 28 2020 at 18:39):

and already solved! :tada:

view this post on Zulip Joshua Nelson (Dec 28 2020 at 18:39):

down to 3 issues, only one of which I consider blocking

view this post on Zulip Joshua Nelson (Jan 08 2021 at 00:11):

@simulacrum do you have a minute to look at https://github.com/rust-lang/rust/pull/79540#issuecomment-751953277 ?

view this post on Zulip simulacrum (Jan 08 2021 at 00:11):

not right now, it's in my queue of prs. hopefully tomorrow or this weekend.

view this post on Zulip Joshua Nelson (Jan 08 2021 at 00:11):

sounds great, thanks!

view this post on Zulip Joshua Nelson (Jan 08 2021 at 17:04):

@simulacrum I saw you left a review, thanks. It was mostly focused on the implementation - could you take a look at https://github.com/rust-lang/rust/pull/79540#issuecomment-751953277 ? I think it's a better approach but it means a lot of the current code will change.

view this post on Zulip simulacrum (Jan 08 2021 at 17:04):

yes I am coming back to that

view this post on Zulip simulacrum (Jan 08 2021 at 20:48):

(probably tomorrow morning)


Last updated: Oct 11 2021 at 22:34 UTC