Stream: t-compiler/rust-analyzer

Topic: Attributes are large


Jonas Schievink [he/him] (May 24 2021 at 22:47, on Zulip):

According to some memory profiles I took a while ago, attributes stored in ItemTrees might take up significant space.

I suspect that this is in part due to token trees not using interning or well-known symbols (which would eliminate ~all copies of builtin attribute names).

What would be the right approach to improving this? It seems to me that using hir_def::Name instead of SmolStrs in tt::Ident isn't semantically right. rustc just has Symbols, which are interned strings without much semantic attached to them.

Jonas Schievink [he/him] (May 24 2021 at 22:48, on Zulip):

Also I'm wondering if we could do something about all those duplicated documentation strings here

Jonas Schievink [he/him] (May 24 2021 at 22:49, on Zulip):

Interning AttrInput alone saves over 10 MB already, probably because there are a lot of similar-enough attributes like #[cfg(test)]

matklad (May 25 2021 at 09:19, on Zulip):

One fun thing to try would be to replace

    pub(crate) path: Interned<ModPath>,
    pub(crate) input: Option<AttrInput>,

with

    pub(crate) raw_text: String,

and see how that affects memory usage. I don't have a good intuition about the difference in memory usage between a string and a token tree. I suspect strings might be significantly more memory-efficient.

matklad (May 25 2021 at 09:33, on Zulip):

@Jonas Schievink [he/him] what are you using for heap-profiling, mastiff? Do you think it makes sense to add an xtask for that? Or maybe some docs to the dev/README? I'd love to take a look at profiles myself, but I want to re-use your hard-earned knowledge if possible :)

matklad (May 25 2021 at 11:26, on Zulip):

image.png

matklad (May 25 2021 at 11:27, on Zulip):

new hack to quickly estimate the impact of sizer of things

matklad (May 25 2021 at 11:28, on Zulip):

This changes Item Collection: 1.48s, 15ginstr, 73mb to Item Collection: 1.45s, 15ginstr, 78mb for rowan, so RawAttrs themselves are 5-10% of total memory

matklad (May 25 2021 at 11:28, on Zulip):

which does sound like a lot

Jonas Schievink [he/him] (May 25 2021 at 11:29, on Zulip):

matklad said:

Jonas Schievink [he/him] what are you using for heap-profiling, mastiff? Do you think it makes sense to add an xtask for that? Or maybe some docs to the dev/README? I'd love to take a look at profiles myself, but I want to re-use your hard-earned knowledge if possible :)

nowadays I mostly use heaptrack since it's a lot faster than valgrind

Last update: Jul 27 2021 at 22:15UTC