Stream: t-compiler/wg-self-profile

Topic: expected usage: should we emit the PID to stdout/stderr?


pnkfelix (Sep 17 2019 at 11:10, on Zulip):

quick question: How are people using this typically observing the PID that the rustc invocation used for its files? I personally ended up making a wrapper script that runs rustc ... & in the background and calls jobs -p; wait so that I can observe it, but I'm wondering if it might make sense for -Z self-profile to print the PID for the user?

simulacrum (Sep 17 2019 at 11:13, on Zulip):

I personally feel like we should think about moving to a scheme where we put the crate name, not the pid, in the file

simulacrum (Sep 17 2019 at 11:13, on Zulip):

or maybe both, I guess, though I personally don't really like having the pid name in the files as it doesn't add much

pnkfelix (Sep 17 2019 at 11:13, on Zulip):

or both crate name and pid. Or provide a way for the user to specify file prefix.

simulacrum (Sep 17 2019 at 11:15, on Zulip):

yeah, IIRC problem with file prefix is that a lot of the time there'll be multiple rustc invocations (e.g., with RUSTFLAGS="-Zself-profile" cargo build)

pnkfelix (Sep 17 2019 at 11:17, on Zulip):

right. One could attempt to address that with some sort of variable substitution mechanism.

pnkfelix (Sep 17 2019 at 11:18, on Zulip):

or just tell users to do that themselves

simulacrum (Sep 17 2019 at 11:18, on Zulip):

I think for the most part I've long really wanted a RUSTFLAGS_CRATE_NAME in Cargo for this sort of thing

simulacrum (Sep 17 2019 at 11:19, on Zulip):

maybe we can have measureme have something that runs atop cargo and sets up a "server" or something and store events into one file from a cargo run

simulacrum (Sep 17 2019 at 11:19, on Zulip):

though that seems more complicated

pnkfelix (Sep 17 2019 at 11:19, on Zulip):

... well there is RUSTC_FLAGS=--crate-name _, right?

pnkfelix (Sep 17 2019 at 11:19, on Zulip):

but I guess that cargo needs to know about this too

simulacrum (Sep 17 2019 at 11:19, on Zulip):

er, that doesn't really help though right? Since you want it to run for a particular crate, not override crate names for all crates

simulacrum (Sep 17 2019 at 11:20, on Zulip):

currently the best way is cargo build then cargo clean -p <crate> then cargo build if it's the "last" crate

pnkfelix (Sep 17 2019 at 11:20, on Zulip):

ah right, you want this to just override the name for the immediate crate being built, not its dependencies, right?

simulacrum (Sep 17 2019 at 11:20, on Zulip):

otherwise that'll not work too well, particularly in today's world where we have builds happening simultaneously cross-crate

simulacrum (Sep 17 2019 at 11:21, on Zulip):

no, no, I want something like RUSTFLAGS_serde="-Zself-profile" RUSTFLAGS_serde_derive="..." cargo build

pnkfelix (Sep 17 2019 at 11:21, on Zulip):

oh oh oh

pnkfelix (Sep 17 2019 at 11:22, on Zulip):

RUSTFLAGS_${CRATE_NAME} now I see

pnkfelix (Sep 17 2019 at 11:22, on Zulip):

that sounds useful indeed

simulacrum (Sep 17 2019 at 11:24, on Zulip):

It might be a bit painful to support since IIRC we're struggling to get env-parsing in cargo working with matching "parts" (serde vs. serde_derive) but since we know a priori exactly the env variable we expect maybe that's not too bad

simulacrum (Sep 17 2019 at 11:24, on Zulip):

It does seem like a more generally useful feature (certainly, perf wants it, I could imagine e.g. crater runs wanting it)

Wesley Wiser (Sep 17 2019 at 13:27, on Zulip):

Maybe there's a cargo (debugging) feature here as well? If there was a -Z process-info flag you could pass to cargo and have it output a file like:

pid      process
9382   rustc --crate-name "serde" ....
122    rustc --crate-name "bin_utils" ....
4983   rustc --crate-name "foo" ....

that might be useful as well.

simulacrum (Sep 17 2019 at 13:28, on Zulip):

eh, perhaps, but I personally don't like the tying to pids regardless (it seems a bit obscure, and that output could plausibly have the same pid for all three crates, right?)

Wesley Wiser (Sep 17 2019 at 13:30, on Zulip):

At least on Linux and Windows, that seems unlikely

Wesley Wiser (Sep 17 2019 at 13:31, on Zulip):

I'm not sure what macOS does

simulacrum (Sep 17 2019 at 13:32, on Zulip):

Yeah, it does seem unlikely -- I think it circles back during numbering, right? but still, pids are a bit unfriendly from a usability perspective, though they do seem 'state of the art' to an extent (e.g., valgrind also uses them)

Wesley Wiser (Sep 17 2019 at 13:33, on Zulip):

Agreed! I think it would be great to include both the crate name and the pid in the self-profile filename. Usually, crate name should be enough.

Wesley Wiser (Sep 17 2019 at 13:46, on Zulip):

This seemed slightly familiar to me so I tested it on nightly and it appears that try to include the crate name in the output file name already.

pnkfelix (Sep 17 2019 at 13:46, on Zulip):

oh yeah, that is true

pnkfelix (Sep 17 2019 at 13:47, on Zulip):

my own specific use case was that I wanted to just have a good pattern for knowing what pid was assigned to rustc I just ran

mw (Sep 17 2019 at 13:47, on Zulip):

I think there should be a wrapper tool at some point that handles all of this for you

Wesley Wiser (Sep 17 2019 at 13:47, on Zulip):

If I do RUSTFLAGS="-Z self-profile" cargo build on a fresh checkout of regex, I get this:

___-3764.{events,string_data,string_index}
___-5580.{events,string_data,string_index}
___-24468.{events,string_data,string_index}
regex_syntax-11648.{events,string_data,string_index}
regex-32204.{events,string_data,string_index}
mw (Sep 17 2019 at 13:47, on Zulip):

maybe a cargo subcommand

pnkfelix (Sep 17 2019 at 13:47, on Zulip):

I would imagine e.g. that using jobs -p the way I described above is vulnerable to a race with other processes being created

pnkfelix (Sep 17 2019 at 13:48, on Zulip):

(in terms of being able to subsequently automatically run measureme without a human attempting to infer the correct PID)

Wesley Wiser (Sep 17 2019 at 13:49, on Zulip):

If we have this info (https://github.com/rust-lang/measureme/issues/15) then such a subcommand could get the header info for all the profile traces in the current directory and offer to process the most recent one.

Wesley Wiser (Sep 17 2019 at 13:49, on Zulip):

Or some other heuristic

pnkfelix (Sep 17 2019 at 13:51, on Zulip):

my specific use case is that I'm running the compiler three times: once without incremental, once with incremental with no incrdata, and once with incremental with incrdata. So its important, in that context, for me to have some way of observing the three PIDs that were created.

pnkfelix (Sep 17 2019 at 13:52, on Zulip):

Note in particular that in order to get use of the incrdata that was created, it is not ideal to take the approach of using distinct --crate-name for each run

pnkfelix (Sep 17 2019 at 13:52, on Zulip):

(I could do it, by including an extra fourth non-self-profiled run to generate the incrdata before the third run I mentioned above)

Wesley Wiser (Sep 17 2019 at 13:54, on Zulip):

Are you invoking rustc directly or going through cargo?

pnkfelix (Sep 17 2019 at 13:54, on Zulip):

concrete gist of my wrapper script

pnkfelix (Sep 17 2019 at 13:55, on Zulip):

in this case I'm invoking rustc directly

pnkfelix (Sep 17 2019 at 13:55, on Zulip):

which is why I am not crazy worried about how I'm using jobs -p

pnkfelix (Sep 17 2019 at 13:55, on Zulip):

but it still seems suboptimal to me.

Wesley Wiser (Sep 17 2019 at 13:55, on Zulip):

Can you use $!?

Wesley Wiser (Sep 17 2019 at 13:55, on Zulip):

https://unix.stackexchange.com/questions/30370/how-to-get-the-pid-of-the-last-executed-command-in-shell-script

pnkfelix (Sep 17 2019 at 13:56, on Zulip):

I experimented with that but it didn't seem to give me the right number

pnkfelix (Sep 17 2019 at 13:56, on Zulip):

it was always off by one or so

pnkfelix (Sep 17 2019 at 13:56, on Zulip):

i'm not 100% sure why

Wesley Wiser (Sep 17 2019 at 13:56, on Zulip):

Weird

pnkfelix (Sep 17 2019 at 13:56, on Zulip):

it might have been because I was using time in a different way when I was trying to use $!

pnkfelix (Sep 17 2019 at 13:57, on Zulip):

/me goes to give $! a try again

Wesley Wiser (Sep 17 2019 at 13:58, on Zulip):

It might be a good idea to have rustc print something out anyway when -Z self-profile is turned on. Even a short message would be a good indication that it's running and could include the pid.

pnkfelix (Sep 17 2019 at 13:59, on Zulip):

yeah okay echo $! does seem to do the right thing now that I time the wait at the end instead of the initial call...

pnkfelix (Sep 17 2019 at 14:00, on Zulip):

(and moving the time to the initial invocation causes the aforementioned off-by-one error)

pnkfelix (Sep 17 2019 at 14:01, on Zulip):

It might be a good idea to have rustc print something out anyway when -Z self-profile is turned on. Even a short message would be a good indication that it's running and could include the pid.

right, this is what I was wondering about at the outset: Whether there is a good reason not to do this.

Wesley Wiser (Sep 17 2019 at 14:02, on Zulip):

I can't think of one.

mw (Sep 19 2019 at 09:32, on Zulip):

one reason might be to make the output the same, regardless of whether self-profiling is on or off

pnkfelix (Sep 19 2019 at 09:54, on Zulip):

okay well maybe I'll add a different -Z flag to make it print the process id. ;)

pnkfelix (Sep 19 2019 at 09:55, on Zulip):

(i do appreciate the benefit of keeping stdout/stderr invariant, at least absent other instrumentation like RUSTC_LOG)

Last update: Nov 17 2019 at 06:55UTC