Stream: t-compiler/rust-analyzer

Topic: loading rustc sources in RA


Xavier Denis (Nov 10 2020 at 18:56, on Zulip):

Hi, is there a better solution than cfg(NOT_A_PLATFORM) to load rustc sources in RA? It makes things non-portable and hard to share. If there's a config file i where i could give RA additional loadpaths that would solve all my problems (I could keep it out of git)

Xavier Denis (Nov 10 2020 at 18:56, on Zulip):

also thanks for the work! I'm a recent convert but it's been a great experience so far!

Xavier Denis (Nov 10 2020 at 18:59, on Zulip):

since my project is a rustc driver, I really need RA to find those crates (or it becomes useless) but having them in cargo.toml means I can't use CI or really ever have other contributors

matklad (Nov 10 2020 at 19:17, on Zulip):

I am not aware about any hacks for supporting rustc driver use-case

matklad (Nov 10 2020 at 19:17, on Zulip):

but that should be moderately easy to support

matklad (Nov 10 2020 at 19:18, on Zulip):

in project_model, we need to have a sepaate codepath to run cargo metadata on rustc's checkout, and plug that into crate graph

Xavier Denis (Nov 10 2020 at 19:22, on Zulip):

I think my usecase is the same as anyone that wants to refer to an extern crate honestly

matklad (Nov 10 2020 at 19:24, on Zulip):

I mean, for typical use-cases cargo metadata just works

Xavier Denis (Nov 10 2020 at 19:24, on Zulip):

hmm right but that's derived from the Cargo.toml

Xavier Denis (Nov 10 2020 at 19:25, on Zulip):

i wish that having the rustc-dev component would make those crates visible like normal ones...

Xavier Denis (Nov 10 2020 at 19:26, on Zulip):

matklad said:

in project_model, we need to have a sepaate codepath to run cargo metadata on rustc's checkout, and plug that into crate graph

this still comes back to how do you tell RA where the checkout is

Xavier Denis (Nov 10 2020 at 19:26, on Zulip):

I tried using the linkedProjects functionality but it seems like it treats all the linked projects as actual parts of the project

Xavier Denis (Nov 10 2020 at 19:27, on Zulip):

so the indexed crates went up from ~500 to 1700

matklad (Nov 10 2020 at 19:27, on Zulip):

I mean, you need to write some code inside rust-analyzer to support this use-case

matklad (Nov 10 2020 at 19:27, on Zulip):

it's not only the matter of telling ra where the crates are, you also need to explain that your crate depend on rustc's crate

matklad (Nov 10 2020 at 19:28, on Zulip):

there's no sysroot in ra, all deps are explicit

Xavier Denis (Nov 10 2020 at 19:28, on Zulip):

im happy to write code to make this happen!

matklad (Nov 10 2020 at 19:30, on Zulip):

So, there are two bits where this needs to be handled

matklad (Nov 10 2020 at 19:30, on Zulip):

(beyond threading various configs)

matklad (Nov 10 2020 at 19:30, on Zulip):

This thing runs cargo metadata on your project, to learn about deps: https://github.com/rust-analyzer/rust-analyzer/blob/5c06e820fa02b47a1550576f2a7071ff94fb0c64/crates/project_model/src/lib.rs#L157-L161

matklad (Nov 10 2020 at 19:30, on Zulip):

You need to teach it to also run metadata on rustc_private crates

matklad (Nov 10 2020 at 19:31, on Zulip):

I think the way to do that is to add a rustc_source: Option<AbsPathBuf> field to CargoConfig and then do the stuff

Xavier Denis (Nov 10 2020 at 19:31, on Zulip):

k seems reasonable

matklad (Nov 10 2020 at 19:32, on Zulip):

The second bit is this 120-lines long match arm :D

https://github.com/rust-analyzer/rust-analyzer/blob/5c06e820fa02b47a1550576f2a7071ff94fb0c64/crates/project_model/src/lib.rs#L368-L486

matklad (Nov 10 2020 at 19:32, on Zulip):

Here, we lower cargo-metadata representation to our internal CrateGraph, which makes all the deps explicit

matklad (Nov 10 2020 at 19:33, on Zulip):

Here, you'd need to graft rustc's crate onto vanila cargo metadata

Xavier Denis (Nov 10 2020 at 19:35, on Zulip):

k well i'll go work on that right now

Xavier Denis (Nov 10 2020 at 19:39, on Zulip):

hmm what's the best way to discover the path for the rustc-dev component...

Xavier Denis (Nov 10 2020 at 19:39, on Zulip):

RA doesn't assume things were installed from rustup does it?

matklad (Nov 10 2020 at 19:40, on Zulip):

hmm what's the best way to discover the path for the rustc-dev component...

I'd go just for a value in configuration file

matklad (Nov 10 2020 at 19:40, on Zulip):

You need some config to opt-in into this value anyway, so might as well require the user to supply the full path

Xavier Denis (Nov 10 2020 at 19:41, on Zulip):

yea, that's fine but that value can't be in Cargo.toml

Xavier Denis (Nov 10 2020 at 19:41, on Zulip):

or it's self-defeating

matklad (Nov 10 2020 at 19:41, on Zulip):

I mean, it should work just all the other configs

matklad (Nov 10 2020 at 19:41, on Zulip):

Take a look how CargoConfig is populated

matklad (Nov 10 2020 at 19:41, on Zulip):

(in config.rs)

Xavier Denis (Nov 10 2020 at 19:41, on Zulip):

yea that's what im reading rn

Xavier Denis (Nov 10 2020 at 19:41, on Zulip):

RA works fine on this project ;)

matklad (Nov 10 2020 at 19:42, on Zulip):

(yet again I need someone to implement https://github.com/rust-analyzer/rust-analyzer/issues/2549 :D )

Xavier Denis (Nov 10 2020 at 19:42, on Zulip):

aha yea i was just thinking how useful that would be

Xavier Denis (Nov 10 2020 at 21:53, on Zulip):

hmm is it wrong of me to load the rustc compiler into a CargoWorkspace ?

Xavier Denis (Nov 10 2020 at 21:54, on Zulip):

or should I really manually iterate over the crates in the compiler/ directory and manually build the graph entries?

Xavier Denis (Nov 10 2020 at 21:55, on Zulip):

it seems like running cargo metadata anywhere in the workspace returns the same info (which I guess is expected)

matklad (Nov 10 2020 at 21:56, on Zulip):

Loading it into CargoWOrkspace seems correct

matklad (Nov 10 2020 at 21:57, on Zulip):

So, you'll have something like

#[derive(Clone, Eq, PartialEq)]
pub enum ProjectWorkspace {
    /// Project workspace was discovered by running `cargo metadata` and `rustc --print sysroot`.
    Cargo { cargo: CargoWorkspace, sysroot: Sysroot, rustc: Option<CargoWorkspace> },
    /// Project workspace was manually specified using a `rust-project.json` file.
    Json { project: ProjectJson, sysroot: Option<Sysroot> },
}
Xavier Denis (Nov 10 2020 at 22:00, on Zulip):

yep!

Xavier Denis (Nov 10 2020 at 22:00, on Zulip):

i got all of that working but it seems to load WAY too many packages now

Xavier Denis (Nov 10 2020 at 22:00, on Zulip):

~2307

matklad (Nov 10 2020 at 22:00, on Zulip):

Well, rustc is big :-)

Xavier Denis (Nov 10 2020 at 22:00, on Zulip):

previously using the NOT_A_TARGET hack it was around ~400

matklad (Nov 10 2020 at 22:00, on Zulip):

Although, yeah, 2307 seems a bit too much

Joshua Nelson (Nov 10 2020 at 22:00, on Zulip):

around 400 sounds closer to right, yeah

Joshua Nelson (Nov 10 2020 at 22:00, on Zulip):

is it maybe loading some packages twice?

Xavier Denis (Nov 10 2020 at 22:01, on Zulip):

which is weird cuz the cargo metadata is ~400

Xavier Denis (Nov 10 2020 at 22:01, on Zulip):

Joshua Nelson said:

is it maybe loading some packages twice?

sounds probable... what's the best way to see all the packages that were loaded?

Joshua Nelson (Nov 10 2020 at 22:02, on Zulip):

I like jq :P

matklad (Nov 10 2020 at 22:02, on Zulip):

sounds probable... what's the best way to see all the packages that were loaded?

probably just dbg!(workspace)?

matklad (Nov 10 2020 at 22:03, on Zulip):

there's also some logging this this module, but I don't think we dump the whole projst structure anywahre

Xavier Denis (Nov 10 2020 at 22:04, on Zulip):

yea the logs just go by so fast

matklad (Nov 10 2020 at 22:05, on Zulip):

You need to filter by module to get any kind of useful logs

matklad (Nov 10 2020 at 22:06, on Zulip):

otherwise, you'll be swamped with chalk/salsa output

matklad (Nov 10 2020 at 22:07, on Zulip):

Just saw the PR, its on the right track!

Xavier Denis (Nov 10 2020 at 22:08, on Zulip):

just need to get it working!

matklad (Nov 10 2020 at 22:09, on Zulip):

The tricky part, yeah

Xavier Denis (Nov 10 2020 at 22:11, on Zulip):

interesting.... the workspace has the correct number of entries (488)

Xavier Denis (Nov 10 2020 at 22:12, on Zulip):

anyways... time to debug!

Xavier Denis (Nov 10 2020 at 22:23, on Zulip):

yea that's what I thought the problem is the crate x target multiplication

Xavier Denis (Nov 10 2020 at 22:24, on Zulip):

i just need to rule out non-current targets and it should be good

matklad (Nov 10 2020 at 22:26, on Zulip):

@Xavier Denis yeah, there's similar logic in that we only create library targets for non-current-ws crates

Xavier Denis (Nov 10 2020 at 22:31, on Zulip):

yea im skipping all targets other than lib and that seems to fix it

Xavier Denis (Nov 10 2020 at 22:32, on Zulip):

I'll refactor everything tomorrow morning and push a revised PR

Xavier Denis (Nov 10 2020 at 23:11, on Zulip):

ok i have a _janky_ version that works locally! i'll clean it up in the AM but hey! it works!

matklad (Nov 10 2020 at 23:24, on Zulip):

:tada:

Xavier Denis (Nov 11 2020 at 12:06, on Zulip):

ok, i tried cleaning it up a bit, not sure about how testing works for something like this

Last update: Jul 27 2021 at 21:45UTC