Stream: rustdoc

Topic: calculate the doc coverage without generating HTMLs


view this post on Zulip Siwei (Apr 08 2021 at 11:16):

I'd like to calculate the doc coverage for each of crates in a large workspace without generating the HTML outputs. ( for the process is really slow , takes more than 1 hours ). After searching the documents I can't find a way to do it. Is there a dry-run mode in the rustdoc ?

view this post on Zulip Joshua Nelson (Apr 08 2021 at 11:35):

--generate-coverage on it's own should skip html outputs

view this post on Zulip Joshua Nelson (Apr 08 2021 at 11:35):

Can you paste the command you're running?

view this post on Zulip Siwei (Apr 08 2021 at 11:44):

RUSTDOCFLAGS="-Z unstable-options --show-coverage" cargo doc --no-deps --offline --workspace
Which command should I pass the --generate-coverage flag to ? I can't find it in the help output neither rustdoc nor cargo doc.
toolchain version: rustdoc 1.52.0-nightly (07194ffcd 2021-02-10)

view this post on Zulip Joshua Nelson (Apr 08 2021 at 11:45):

--show-coverage is what I meant

view this post on Zulip Joshua Nelson (Apr 08 2021 at 11:46):

yeah that command doesn't generate docs for me

$ RUSTDOCFLAGS="-Z unstable-options --show-coverage" cargo +nightly doc --no-deps  --offline  --workspace
    Checking inner v0.1.0 (/home/joshua/src/rust/test-rustdoc/wildcard/inner)
 Documenting wildcard v0.1.0 (/home/joshua/src/rust/test-rustdoc/wildcard)
+-------------------------------------+------------+------------+------------+------------+
| File                                | Documented | Percentage |   Examples | Percentage |
+-------------------------------------+------------+------------+------------+------------+
| src/lib.rs                          |          0 |       0.0% |          0 |       0.0% |
+-------------------------------------+------------+------------+------------+------------+
| Total                               |          0 |       0.0% |          0 |       0.0% |
+-------------------------------------+------------+------------+------------+------------+
    Finished dev [unoptimized + debuginfo] target(s) in 1.34s
$ ls doc
ls: cannot access 'doc': No such file or directory

view this post on Zulip Joshua Nelson (Apr 08 2021 at 11:46):

why do you think it does?

view this post on Zulip Siwei (Apr 08 2021 at 11:58):

It costs too much time I guess it may spend on generating outputs.
Thank you.

view this post on Zulip Joshua Nelson (Apr 08 2021 at 11:58):

you can measure the time with -Z self-profile or -Z time-passes

view this post on Zulip Joshua Nelson (Apr 08 2021 at 19:23):

btw @GuillaumeGomez what do you think of renaming --show-coverage to --output-format coverage? since it skips generating HTML?

view this post on Zulip GuillaumeGomez (Apr 08 2021 at 19:25):

I'm fine with it. I'm just afraid it might make it more "tricky" to find. Do you think we can have both to do the same thing or would it overkill?

view this post on Zulip Joshua Nelson (Apr 08 2021 at 19:25):

I think we should only have one

view this post on Zulip Joshua Nelson (Apr 08 2021 at 19:25):

we can make it warn for a few releases or whatever

view this post on Zulip GuillaumeGomez (Apr 08 2021 at 19:26):

I don't think warnings are needed, it's a nightly option after all. Let's just maybe write a technical blog post about it (so we can also use it to sum up what the option do).

view this post on Zulip GuillaumeGomez (Apr 08 2021 at 19:26):

oh, just thought that it might be an issue

view this post on Zulip GuillaumeGomez (Apr 08 2021 at 19:27):

--show-coverage has also a json output

view this post on Zulip GuillaumeGomez (Apr 08 2021 at 19:27):

by using the --output-format XD

view this post on Zulip GuillaumeGomez (Apr 08 2021 at 19:27):

so I don't think it can work

view this post on Zulip Joshua Nelson (Apr 08 2021 at 19:27):

oh oof

view this post on Zulip GuillaumeGomez (Apr 08 2021 at 19:27):

We use it on docs.rs ;)

view this post on Zulip Joshua Nelson (Apr 08 2021 at 19:27):

I wonder if we should rename --output-format to --backend or something

view this post on Zulip GuillaumeGomez (Apr 08 2021 at 19:28):

Hmmm... Originally it was because we had both json and HTML. The idea remains the same

view this post on Zulip GuillaumeGomez (Apr 08 2021 at 19:28):

And it's a stable option, so we can't rename it easily

view this post on Zulip Joshua Nelson (Apr 08 2021 at 19:28):

oh right

view this post on Zulip Joshua Nelson (Apr 08 2021 at 19:28):

oh well

view this post on Zulip simulacrum (Apr 08 2021 at 19:29):

Generally what other tooling does is just have a coverage and coverage-json mode for output-format, which seems not bad.

view this post on Zulip GuillaumeGomez (Apr 08 2021 at 19:29):

you mean having two different outputs?

view this post on Zulip GuillaumeGomez (Apr 08 2021 at 19:29):

outputs => options

view this post on Zulip GuillaumeGomez (Apr 08 2021 at 19:30):

like --output-format coverage-json and --output-format coverage?

view this post on Zulip Joshua Nelson (Apr 08 2021 at 19:30):

I think simulacrum means --output-format coverage and --output-format json-coverage

view this post on Zulip GuillaumeGomez (Apr 08 2021 at 19:30):

ok so I understood correctly

view this post on Zulip GuillaumeGomez (Apr 08 2021 at 19:30):

that could work too

view this post on Zulip GuillaumeGomez (Apr 08 2021 at 19:30):

I like the idea, it's more explicit

view this post on Zulip GuillaumeGomez (Apr 08 2021 at 19:31):

Since we use it on docs.rs, we'll need a little bit of time before removing the old way (so the warning idea for one release is mandatory here I think)

view this post on Zulip simulacrum (Apr 08 2021 at 19:32):

if you mean release=nightly, then yes, but otherwise I'm not sure I understand :)

view this post on Zulip GuillaumeGomez (Apr 08 2021 at 19:33):

@simulacrum for docs.rs, we would just need a few days of having both options, so not an issue. I was simply thinking about the end users that might use this option (maybe they exist haha)

view this post on Zulip simulacrum (Apr 08 2021 at 19:34):

Sure. I just wanted to check that my model of docs.rs was accurate.

view this post on Zulip GuillaumeGomez (Apr 08 2021 at 19:34):

Unless I'm mistaken, docs.rs updates the nightly daily

view this post on Zulip GuillaumeGomez (Apr 08 2021 at 19:34):

so just like you said, not an issue :)

view this post on Zulip GuillaumeGomez (Apr 08 2021 at 19:35):

@Joshua Nelson Can you open an issue about this please? And I guess you want to do the PR so I'll let you handle that ;)

view this post on Zulip GuillaumeGomez (Apr 08 2021 at 19:35):

I can make the update on docs.rs once it's merged and in the nightly we use

view this post on Zulip Joshua Nelson (Apr 08 2021 at 19:35):

sure :)

view this post on Zulip GuillaumeGomez (Apr 08 2021 at 19:36):

Thanks for bringing this up, it's a really good idea. It simplifies the arguments a lot and I like that :) (and thanks for the suggestion @simulacrum !)

view this post on Zulip Joshua Nelson (Apr 08 2021 at 20:11):

https://github.com/rust-lang/rust/issues/84012

view this post on Zulip Joshua Nelson (Apr 08 2021 at 20:11):

(sorry, I took half an hour to argue with friends about how no actually using dynamic linking for rust is hard)

view this post on Zulip GuillaumeGomez (Apr 08 2021 at 21:03):

I was reading a (bad) book, so it's fine. You can take a few more days :)

view this post on Zulip Siwei (Apr 09 2021 at 10:34):

Joshua Nelson said:

you can measure the time with -Z self-profile or -Z time-passes

It seems that the 'get_blanket_impls' is the most time-consuming one ( costs more than half of total time).Is the process necessary for calc coverage?

view this post on Zulip Joshua Nelson (Apr 09 2021 at 11:57):

Ok yeah that's what I expected

view this post on Zulip Joshua Nelson (Apr 09 2021 at 11:58):

Hmm it shouldn't be necessary - I wonder if we can rip the coverage pass out of everything else and just run it as a HIR pass

view this post on Zulip Joshua Nelson (Apr 09 2021 at 11:58):

That should be basically instantaneous

view this post on Zulip Joshua Nelson (Apr 09 2021 at 12:09):

hmm it looks like running this is intentional? https://github.com/rust-lang/rust/blob/bc66b92f7f1114a78b98504a927ca9c0aa12c71a/src/librustdoc/passes/mod.rs#L113

view this post on Zulip Joshua Nelson (Apr 09 2021 at 12:11):

hmm well removing it doesn't cause the test suite to fail

view this post on Zulip Joshua Nelson (Apr 09 2021 at 12:12):

ok so I think what's going on is rustdoc wants to look at inherent and trait impls but not blanket impls https://github.com/rust-lang/rust/blob/bc66b92f7f1114a78b98504a927ca9c0aa12c71a/src/librustdoc/passes/calculate_doc_coverage.rs#L209-L212

view this post on Zulip Joshua Nelson (Apr 09 2021 at 12:13):

oh wait what it ignores those too lol https://github.com/rust-lang/rust/blob/bc66b92f7f1114a78b98504a927ca9c0aa12c71a/src/librustdoc/passes/calculate_doc_coverage.rs#L214-L237

view this post on Zulip Joshua Nelson (Apr 09 2021 at 12:15):

@Siwei could you confirm whether RUSTDOCFLAGS='--no-defaults --passes=strip_hidden,strip_private,calculate_doc_coverage -Z unstable-options --show-coverage' cargo +nightly doc --no-deps --offline --workspace is faster?

view this post on Zulip Joshua Nelson (Apr 09 2021 at 13:11):

https://github.com/rust-lang/rust/pull/84030

view this post on Zulip Nemo157 (Apr 09 2021 at 22:03):

GuillaumeGomez said:

Unless I'm mistaken, docs.rs updates the nightly daily

not quite daily, with how often rustfmt is broken, in fact currently we're stuck on rustc 1.53.0-nightly (07e0e2ec2 2021-03-24) still

view this post on Zulip Noah Lev (Apr 11 2021 at 01:54):

Wait, why does docs.rs need rustfmt? Why not only install rustc/rustdoc?

view this post on Zulip Joshua Nelson (Apr 11 2021 at 02:02):

some crates depend on it for formatting generated code

view this post on Zulip Joshua Nelson (Apr 11 2021 at 02:03):

https://github.com/rust-lang/docs.rs/issues/357

view this post on Zulip Noah Lev (Apr 11 2021 at 02:07):

Ah, I see.

view this post on Zulip Siwei (Apr 11 2021 at 04:08):

Joshua Nelson said:

Siwei could you confirm whether RUSTDOCFLAGS='--no-defaults --passes=strip_hidden,strip_private,calculate_doc_coverage -Z unstable-options --show-coverage' cargo +nightly doc --no-deps --offline --workspace is faster?

This command reduces the total time from more than 10 hours to less than 10 minutes. But it doesn't output anything about the coverage.

view this post on Zulip Joshua Nelson (Apr 11 2021 at 04:10):

oh huh, that's odd

view this post on Zulip Joshua Nelson (Apr 11 2021 at 04:11):

anyway, https://github.com/rust-lang/rust/pull/84030 should have landed - @Siwei could you try again with the latest nightly? without the special flags, just the ones from before: RUSTDOCFLAGS='-Z unstable-options --show-coverage' cargo +nightly doc --no-deps --offline --workspace

view this post on Zulip Joshua Nelson (Apr 11 2021 at 04:11):

also 10 minutes without get_blanket_impls ??? what sort of crate are you building?

view this post on Zulip Siwei (Apr 11 2021 at 05:18):

Joshua Nelson said:

anyway, https://github.com/rust-lang/rust/pull/84030 should have landed - Siwei could you try again with the latest nightly? without the special flags, just the ones from before: RUSTDOCFLAGS='-Z unstable-options --show-coverage' cargo +nightly doc --no-deps --offline --workspace

A dependency fails to compile with the latest nightly toolchain. I'm trying to fix it. Our workspace contains about 120 crates, 12000 pub structs and 160000 pub functions. (counts by rg)

view this post on Zulip Siwei (Apr 11 2021 at 06:18):

With the latest toolchain ( 2021-04-11 ) everything goes well and the total time cost reduces to 9m50s.It's extremely fast!

view this post on Zulip Joshua Nelson (Apr 11 2021 at 11:29):

Awesome! Glad I could help :) and wow that is a lot of crates

view this post on Zulip Joshua Nelson (Apr 11 2021 at 11:29):

Is your workspace public? It might be useful for working on rustdoc's perf

view this post on Zulip Siwei (Apr 12 2021 at 05:26):

Our projects are private. But maybe I can share some profiling data.

view this post on Zulip Joshua Nelson (Apr 12 2021 at 08:15):

You can skip most of the 10 hours by using --no-default-passes and adding back all the passes in https://github.com/rust-lang/rust/blob/d68f7a2f50958d632ef8dc79212481e4f183ed36/src/librustdoc/passes/mod.rs#L97 manually with --passes. I already know get_blanket_impls is slow, I don't need to know exactly how slow.

view this post on Zulip Siwei (Apr 16 2021 at 01:44):

Maybe something goes wrong with my laptop the first run (which costs more than 10 hours).
image.png

Here're my test results. It seems that I could't make it the same slow when puts everything (collect_trait_impls,STRIP_HIDDEN,STRIP_PRIVATE,CALCULATE_DOC_COVERAGE) to the --passes.
RUSTDOCFLAGS=' -Z unstable-options --show-coverage' cargo doc --no-deps 255min
--no-defaults 12m55s
--no-defaults --passes collect_trait_impls 14m35s
--no-defaults --passes collect_trait_impls,STRIP_HIDDEN,STRIP_PRIVATE
-no-defaults --passes collect_trait_impls,STRIP_HIDDEN,STRIP_PRIVATE,CALCULATE_DOC_COVERAGE 12m

view this post on Zulip Joshua Nelson (Apr 16 2021 at 02:33):

oh sorry, leave out collect_trait_impls from the passes

view this post on Zulip Noah Lev (Apr 17 2021 at 01:20):

Isn't there a -Z time-passes option or something that will show you the times for each pass?

view this post on Zulip Joshua Nelson (Apr 17 2021 at 01:34):

yes, -Z time-passes should work


Last updated: Oct 11 2021 at 22:34 UTC