Stream: t-compiler/help

Topic: How would I serialize a rustc data structure?


Manish Goregaokar (Jun 05 2020 at 23:48, on Zulip):

They use references.

Traits aren't the important part. The work is in creating a data strcture, we'll figure out how it can go into emtadata later.

Joshua Nelson (Jun 06 2020 at 00:02, on Zulip):

Do you have some other examples of serializable structures you can point me to?

Joshua Nelson (Jun 06 2020 at 00:29, on Zulip):

My idea for making parent non-referential was to have one structure that stored all modules and store an index into that structure

Joshua Nelson (Jun 06 2020 at 00:30, on Zulip):

something like this:

/// A `ModuleData` that is non self-referential
struct OwnedModuleData {
    parent: Option<ModuleIndex>,
    ...
}

/// An opaque reference to an `OwnedModuleData`. To be used in conjunction with a `ModuleStore`.
struct ModuleIndex(usize);

/// A list of all modules needed to serialize a specific module
struct ModuleStore(Vec<OwnedModuleData>);
Joshua Nelson (Jun 06 2020 at 00:31, on Zulip):

and all the structs with lifetime parameters aren't a big deal, it's only the references that matter

Joshua Nelson (Jun 06 2020 at 00:32, on Zulip):

so that leaves

    glob_importers: RefCell<Vec<&'a Import<'a>>>,
    globs: RefCell<Vec<&'a Import<'a>>>,

    // Used to memoize the traits in this module for faster searches through all traits in scope.
    traits: RefCell<Option<Box<[(Ident, &'a NameBinding<'a>)]>>>
Joshua Nelson (Jun 06 2020 at 00:32, on Zulip):

From the comment it seems like traits might not need to be stored? Since it's only a cache and can be recalculated?

Joshua Nelson (Jun 06 2020 at 00:33, on Zulip):

Do you know why glob_importers and globs are RefCells?

Joshua Nelson (Jun 06 2020 at 00:39, on Zulip):

update: this is why globs is a RefCell: https://hastebin.com/gorakaqiha.coffeescript

Joshua Nelson (Jun 06 2020 at 01:31, on Zulip):

and all the structs with lifetime parameters aren't a big deal, it's only the references that matter

This appears to be incorrect, I don't see any easy way to make any of these owned.

Joshua Nelson (Jun 06 2020 at 01:32, on Zulip):

What if I narrowed the scope and instead only tried to serialize Resolutions? I think that's all rustdoc really needs

Joshua Nelson (Jun 06 2020 at 01:32, on Zulip):

unless i'm misunderstanding glob imports

Joshua Nelson (Jun 06 2020 at 01:34, on Zulip):

oh well that has a &'a Import<'a> anyway so I'd have to solve this problem sooner or later

Joshua Nelson (Jun 06 2020 at 01:39, on Zulip):

I'm finding it really hard to understand the lifetimes because of all the RefCell

Joshua Nelson (Jun 06 2020 at 01:40, on Zulip):

The type Module = &ModuleData alias is convenient but means that all the mutability is forced to use refcell

Joshua Nelson (Jun 06 2020 at 01:54, on Zulip):

I'm seeing something called ResolverOutputs which has no borrows and looks like basically what we'd need, can I reuse that?

Manish Goregaokar (Jun 07 2020 at 18:03, on Zulip):

@Joshua Nelson Given your comments here and in the github thread I think you're overfocusing on the "serialize" part of it. The high level thing is that rustdoc needs to be able to resolve reexport documentation against their original module map, which is not available in cross-crate scenarios. Serializing is necessary to stick it into the rlib so we can get it back.

Manish Goregaokar (Jun 07 2020 at 18:04, on Zulip):

I don't think we actually should be trying to serialize the _entire_ Resolver. Rather, we should be reading it and producing a new type that _can_ be serialized. The traits field is unnecessary. globs kind of is, sadly.

Manish Goregaokar (Jun 07 2020 at 18:12, on Zulip):

You really need to talk to compiler team members about this, though.

Joshua Nelson (Jun 07 2020 at 20:07, on Zulip):

I'm really confused how serialization is related at all, to be honest

Joshua Nelson (Jun 07 2020 at 20:08, on Zulip):

The reason I keep focusing on it is I'm trying to find out how it's related

Joshua Nelson (Jun 07 2020 at 20:08, on Zulip):

none of the changes I made in https://github.com/rust-lang/rust/pull/73101/ were related to serialization

Joshua Nelson (Jun 07 2020 at 20:10, on Zulip):

also I might be confused - I thought you were a compiler team member? @Manish Goregaokar

Joshua Nelson (Jun 07 2020 at 20:14, on Zulip):

ah ok you're part of the clippy team, that's why I was confused (plus core of course)

Manish Goregaokar (Jun 07 2020 at 21:22, on Zulip):

@Joshua Nelson I mentioned serialization because this is information downstream crates do not have, and the way for them to get it is through metadata, and to put something in metadata we need to be able to serialize it cross crate (so, no self references)

Manish Goregaokar (Jun 07 2020 at 21:24, on Zulip):

@Joshua Nelson it's not clear to me how that PR fixes this bug. The test case is:

// crate a:
pub struct Foo;

/// Link to [Foo]
pub struct Bar;

// crate b:
pub use a::Bar;

the docs for b::Bar should link correctly to a::Foo.

Manish Goregaokar (Jun 07 2020 at 21:25, on Zulip):

Yeah I've never been compiler team. I'm capable of reviewing for most of the compiler, and I understand the resolver, but not well enough for something like this.

Joshua Nelson (Jun 07 2020 at 21:27, on Zulip):

this is what my PR outputs for that test case: image.png

Joshua Nelson (Jun 07 2020 at 21:27, on Zulip):

the link is to /a/struct.Foo.html.

Joshua Nelson (Jun 07 2020 at 21:28, on Zulip):

I'm not sure how to add test cases with multiple crates, I can look into that

Joshua Nelson (Jun 07 2020 at 21:28, on Zulip):

the reason my PR works is because it resolves types relative to the other crate (a), not to the current crate (b)

Joshua Nelson (Jun 07 2020 at 21:29, on Zulip):

which is correct because the docs were written in a, not in b. So if we try to resolve relative to b none of the same items will be in scope.

Manish Goregaokar (Jun 08 2020 at 03:07, on Zulip):

@Joshua Nelson huh, that surprises me. Does that even work when A and B are within some deeper modules (all public)?

Manish Goregaokar (Jun 08 2020 at 03:07, on Zulip):

I think I have a guess here as to what's happening: the ExportMap already has what we needed to some extent

Manish Goregaokar (Jun 08 2020 at 03:08, on Zulip):

@Joshua Nelson another test case would be to stick Bar in a private module, and re-export it from the top level, and have Foo in a public module with Bar using Foo in docs with a use foomod::Foo in scope.

Joshua Nelson (Jun 08 2020 at 03:09, on Zulip):

Sounds good, I'll try that tomorrow

Joshua Nelson (Jun 08 2020 at 03:13, on Zulip):

Also --no-deps breaks things for reasons I don't understand

Joshua Nelson (Jun 08 2020 at 03:14, on Zulip):

https://github.com/rust-lang/rust/pull/73101#issuecomment-640330018

Joshua Nelson (Jun 08 2020 at 03:21, on Zulip):

Only if the item wasn't inlined though? That's weird, that means it can be fixed in rustdoc though

Joshua Nelson (Jun 08 2020 at 03:21, on Zulip):

Only if the item wasn't inlined though? That's weird, that means it can be fixed in rustdoc though

Manish Goregaokar (Jun 08 2020 at 16:33, on Zulip):

@Joshua Nelson oh btw i am not watching htis channel, i won't see things you don't ping me on

Last update: Jan 22 2021 at 13:30UTC