Stream: t-compiler/wg-rls-2.0

Topic: Memory usage


Sydney Acksman (May 29 2020 at 20:34, on Zulip):

rust-analyzer is eating all my goddamn memory

Sydney Acksman (May 29 2020 at 20:35, on Zulip):

Who wants a 23 GB core dump file from it running on Ubuntu?

std::Veetaha (May 29 2020 at 20:36, on Zulip):

What size of a project have you opened with RA?

Sydney Acksman (May 29 2020 at 20:38, on Zulip):

It's not too big, it's just lol-html https://github.com/cloudflare/lol-html

std::Veetaha (May 29 2020 at 20:39, on Zulip):

Are you using vscode?

Sydney Acksman (May 29 2020 at 20:40, on Zulip):

Yep

std::Veetaha (May 29 2020 at 20:41, on Zulip):

Try running Rust Analyzer: Status command
image.png

Sydney Acksman (May 29 2020 at 20:41, on Zulip):

Alright I'll try it once it happens again. Shouldn't be too long...

std::Veetaha (May 29 2020 at 20:41, on Zulip):

<kbd>Ctrl+Shift+P</kbd>

Sydney Acksman (May 29 2020 at 20:42, on Zulip):

Oh wait it has live updates, nice

Sydney Acksman (Jun 01 2020 at 20:05, on Zulip):

Caught it using almost 15 GB

workspaces:
140 packages loaded

analysis:
2429 (33mb) files
154143 (14mb) symbols
1495 trees, 159 retained
1052527 trees, 186 retained (macros)


memory:
0b allocated 0b resident
gc 16227 seconds ago

requests:
 12488 textDocument/codeAction             2ms
*12473 textDocument/codeAction             2ms
 12474 textDocument/documentHighlight      0ms
 12475 textDocument/codeAction             2ms
 12477 textDocument/codeAction             2ms
 12478 textDocument/codeAction             5ms
 12481 textDocument/codeAction             2ms
 12482 textDocument/codeAction             2ms
 12483 textDocument/codeAction             2ms
 12486 textDocument/codeAction             2ms
std::Veetaha (Jun 01 2020 at 20:17, on Zulip):

@Edwin Cheng could this be something with interning the proc macros output?
1052527 trees, 186 retained (macros) seems ridiculous, but I may be wrong...

Edwin Cheng (Jun 01 2020 at 20:18, on Zulip):

proc macor enabled ?

Sydney Acksman (Jun 01 2020 at 20:19, on Zulip):

image.png
nope

std::Veetaha (Jun 01 2020 at 20:20, on Zulip):

@Sydney Acksman, also, what if you run Rust Analyzer: run garbage collection when you catch such a spike? (I am really just blindly guessing, I'd love if we could get some kind of stable reproduction....)

Sydney Acksman (Jun 01 2020 at 20:20, on Zulip):

Sometimes it collects, but if the garbage becomes too much it just doesn't work and I have to kill it.

std::Veetaha (Jun 01 2020 at 20:22, on Zulip):

Collects as in "all the gigabytes are freed"?

Sydney Acksman (Jun 01 2020 at 20:22, on Zulip):

Yep.

std::Veetaha (Jun 01 2020 at 20:22, on Zulip):

Hmm, that's at least something...

Edwin Cheng (Jun 01 2020 at 20:24, on Zulip):

Proc macro is not gc-abled. So I think it is not related. What platform you are in ? Are you working on Win32 related API? (winapi crate is one of the crate used macro heavily)

std::Veetaha (Jun 01 2020 at 20:24, on Zulip):

Also, what version are you running?
image.png

Sydney Acksman (Jun 01 2020 at 20:25, on Zulip):

Ubuntu with VS Code
Version is 05-25

Sydney Acksman (Jun 01 2020 at 20:26, on Zulip):

Cores are throwing work at each other non-stop. image.png

std::Veetaha (Jun 01 2020 at 20:27, on Zulip):

Wow, that's hot :D

Sydney Acksman (Jun 01 2020 at 20:28, on Zulip):

Plugged in laptop knows no bounds (except max operating temp)

std::Veetaha (Jun 01 2020 at 20:29, on Zulip):

Is there any output in here (there should be Rust Analyzer entry, without the Language Server Trace channel)?
image.png

Sydney Acksman (Jun 01 2020 at 20:30, on Zulip):
[ERROR rust_analyzer::main_loop] overly long loop turn: 1.68208508s

thread '<unknown>' has overflowed its stack
fatal runtime error: stack overflow
[Info  - 3:21:16 PM] Connection to server got closed. Server will restart.

thread '<unknown>' has overflowed its stack
fatal runtime error: stack overflow
[Info  - 3:25:20 PM] Connection to server got closed. Server will restart.

thread '<unknown>' has overflowed its stack
fatal runtime error: stack overflow
[Info  - 3:28:49 PM] Connection to server got closed. Server will restart.
Sydney Acksman (Jun 01 2020 at 20:31, on Zulip):

There's a lot of "long loop turn" entries. Some as long as 479 seconds (which may be my computer going to sleep).

std::Veetaha (Jun 01 2020 at 20:31, on Zulip):
thread '<unknown>' has overflowed its stack
fatal runtime error: stack overflow
[Info  - 3:28:49 PM] Connection to server got closed. Server will restart.

Are there many more entires of this?

Sydney Acksman (Jun 01 2020 at 20:32, on Zulip):

Yep, 2 more above the loop turns

Sydney Acksman (Jun 01 2020 at 20:32, on Zulip):
thread '<unknown>' has overflowed its stack
fatal runtime error: stack overflow
[Info  - 3:06:07 PM] Connection to server got closed. Server will restart.

thread '<unknown>' has overflowed its stack
fatal runtime error: stack overflow
[Info  - 3:09:08 PM] Connection to server got closed. Server will restart.
std::Veetaha (Jun 01 2020 at 20:33, on Zulip):

Hm, I would expect there to be hundreds of such log entries. I am not sure how we handle the stack overflow of proc macros (or maybe it is our bad? cc @Edwin Cheng ), but maybe this could lead to some memory leak?

std::Veetaha (Jun 01 2020 at 20:34, on Zulip):

Also, the fact that we run proc macro server when proc macros are disabled is strange...

Edwin Cheng (Jun 01 2020 at 20:34, on Zulip):

@Veetaha Why you think it is proc-macro?

Edwin Cheng (Jun 01 2020 at 20:35, on Zulip):

Maybe https://github.com/rust-analyzer/rust-analyzer/issues/4453

std::Veetaha (Jun 01 2020 at 20:36, on Zulip):

Hmm, yes I'm wrong, it seems like vscode-lanugageserver restarts rust-analyzer

Jeremy Kolb (Jun 01 2020 at 21:26, on Zulip):

Are you doing anything... interesting when it occurs? or editing a particular file?

Jeremy Kolb (Jun 01 2020 at 21:26, on Zulip):

I'm on windows and memory usage is fairly low

Sydney Acksman (Jun 01 2020 at 21:50, on Zulip):

I've found a file that causes it: https://github.com/cloudflare/lol-html/blob/master/src/rewriter/mod.rs

Sydney Acksman (Jun 01 2020 at 21:51, on Zulip):

Just holding this file open leaks memory like crazy.

Edwin Cheng (Jun 01 2020 at 21:58, on Zulip):

What happend you comment this line ? https://github.com/cloudflare/lol-html/blob/344063e669731f8151a4b506d85f3fa4cc0fbd8a/src/rewriter/mod.rs#L533

Sydney Acksman (Jun 01 2020 at 22:08, on Zulip):

Nothing, it's still leaking

Edwin Cheng (Jun 01 2020 at 22:11, on Zulip):

One more question, you use using cargo nightly ?

Sydney Acksman (Jun 01 2020 at 22:15, on Zulip):

No, cargo is 1.43.0

Edwin Cheng (Jun 01 2020 at 22:19, on Zulip):

I guess it is related to proc-macro ? But I can't confirm that, maybe need time to investigate. But recently I don't have much time available ..

std::Veetaha (Jun 01 2020 at 22:48, on Zulip):

I have the same behavior:
image.png
What is weird is that closing vscode doesn't help. For some reason there is a stray rust-analyzer process left. I wonder who doesn't cleanup vscode-languageclient or we?

std::Veetaha (Jun 01 2020 at 22:49, on Zulip):

@Sydney Acksman thanks for the repo!

std::Veetaha (Jun 01 2020 at 22:58, on Zulip):

Openned an issue here

Laurențiu Nicola (Jun 02 2020 at 06:47, on Zulip):

Why would it be related to proc macros? They're disabled and parse_macro seems to be related to MBE? And it seemed to me that MBE tends to expand or parse too much.

Jeremy Kolb (Jun 02 2020 at 12:23, on Zulip):

I can reproduce it too

Last update: Sep 27 2020 at 13:45UTC