Stream: t-compiler/rust-analyzer

Topic: running RA on docs.rs


Florian Diebold (Jan 18 2021 at 15:44, on Zulip):

Joshua Nelson said:

Jonas Schievink said:

If sorting issues by number of :+1: reactions is anything to go by, reducing memory usage and supporting inner items properly are high on the list

big +1 to less memory usage, I don't even use RA on docs.rs because it crashes my laptop

is docs.rs that huge? maybe there's something else going on, it might be worth running analysis-stats on it and/or creating an issue

Joshua Nelson (Jan 18 2021 at 16:03, on Zulip):

I did: https://github.com/rust-analyzer/rust-analyzer/issues/7052

Joshua Nelson (Jan 18 2021 at 16:03, on Zulip):

the debug tools didn't work for me: https://github.com/rust-analyzer/rust-analyzer/issues/7052#issuecomment-751499782
and no one else was able to reproduce the memory usage: https://github.com/rust-analyzer/rust-analyzer/issues/7052#issuecomment-751518257

Joshua Nelson (Jan 18 2021 at 16:04, on Zulip):

and then it got marked as unactionable

Joshua Nelson (Jan 18 2021 at 16:04, on Zulip):

so I don't know what the next steps are

Edwin Cheng (Jan 18 2021 at 16:07, on Zulip):

Do you see the panic now ?

Joshua Nelson (Jan 18 2021 at 16:08, on Zulip):

I see a panic:

[Info  - 11:07:36 AM] Connection to server got closed. Server will restart.
[Error - 11:07:36 AM] Request textDocument/semanticTokens/full failed.
Error: Connection got disposed.
    at Object.dispose (/home/joshua/.vscode/extensions/matklad.rust-analyzer-0.2.457/out/src/main.js:5534:27)
    at Object.dispose (/home/joshua/.vscode/extensions/matklad.rust-analyzer-0.2.457/out/src/main.js:11546:35)
    at LanguageClient.handleConnectionClosed (/home/joshua/.vscode/extensions/matklad.rust-analyzer-0.2.457/out/src/main.js:13897:42)
    at LanguageClient.handleConnectionClosed (/home/joshua/.vscode/extensions/matklad.rust-analyzer-0.2.457/out/src/main.js:16803:15)
    at closeHandler (/home/joshua/.vscode/extensions/matklad.rust-analyzer-0.2.457/out/src/main.js:13884:18)
    at CallbackList.invoke (/home/joshua/.vscode/extensions/matklad.rust-analyzer-0.2.457/out/src/main.js:3527:39)
    at Emitter.fire (/home/joshua/.vscode/extensions/matklad.rust-analyzer-0.2.457/out/src/main.js:3589:36)
    at closeHandler (/home/joshua/.vscode/extensions/matklad.rust-analyzer-0.2.457/out/src/main.js:4747:26)
    at CallbackList.invoke (/home/joshua/.vscode/extensions/matklad.rust-analyzer-0.2.457/out/src/main.js:3527:39)
    at Emitter.fire (/home/joshua/.vscode/extensions/matklad.rust-analyzer-0.2.457/out/src/main.js:3589:36)
    at StreamMessageWriter.fireClose (/home/joshua/.vscode/extensions/matklad.rust-analyzer-0.2.457/out/src/main.js:4009:27)
    at Socket.<anonymous> (/home/joshua/.vscode/extensions/matklad.rust-analyzer-0.2.457/out/src/main.js:4042:42)
    at Socket.emit (events.js:223:5)
    at Pipe.<anonymous> (net.js:664:12)
Joshua Nelson (Jan 18 2021 at 16:08, on Zulip):

I never saw the original panic even 3 weeks ago, maybe I wasn't looking at the logs

Joshua Nelson (Jan 18 2021 at 16:09, on Zulip):

the server just crashed I think because it's back to 55/369 roots, it was at 200 just a second ago

Jonas Schievink [he/him] (Jan 18 2021 at 16:09, on Zulip):

I think running the memory usage command now (intentionally) breaks salsa invariants, so a panic is expected

Joshua Nelson (Jan 18 2021 at 16:09, on Zulip):

not sure if that's an OOM or not

Joshua Nelson (Jan 18 2021 at 16:09, on Zulip):

but it happens pretty consistently

Joshua Nelson (Jan 18 2021 at 16:09, on Zulip):

Jonas Schievink said:

I think running the memory usage command now (intentionally) breaks salsa invariants, so a panic is expected

this is without running any commands at all

Joshua Nelson (Jan 18 2021 at 16:09, on Zulip):

I just have it open to a file without making any changes

Joshua Nelson (Jan 18 2021 at 16:10, on Zulip):

ok the server keeps getting OOM'd so I'm going to close RA now

Jonas Schievink [he/him] (Jan 18 2021 at 16:10, on Zulip):

on linux dmesg will tell you if something gets OOM-killed

Joshua Nelson (Jan 18 2021 at 16:11, on Zulip):

I'm using earlyoom, let me see where it sends its logs

Joshua Nelson (Jan 18 2021 at 16:11, on Zulip):

(without earlyoom my computer will freeze up completely instead of killing a process)

Laurențiu (Jan 18 2021 at 16:14, on Zulip):

Did anyone else try it on Windows?

Joshua Nelson (Jan 18 2021 at 16:14, on Zulip):

yup it was killed:

Jan 18 11:13:42 joshua-ThinkPad-P51s earlyoom[988]: mem avail:   834 of  7597 MiB (10 %), swap free:   77 of  975 MiB ( 7 %)
Jan 18 11:13:42 joshua-ThinkPad-P51s earlyoom[988]: low memory! at or below SIGTERM limits: mem 10 %, swap 10 %
Jan 18 11:13:42 joshua-ThinkPad-P51s earlyoom[988]: sending SIGTERM to process 51844 "rust-analyzer-x": badness 216, VmRSS 1850 MiB
Jan 18 11:13:42 joshua-ThinkPad-P51s earlyoom[988]: process exited after 0.1 seconds
Jan 18 11:14:07 joshua-ThinkPad-P51s earlyoom[988]: mem avail:  2206 of  7597 MiB (29 %), swap free:   77 of  975 MiB ( 7 %)
Joshua Nelson (Jan 18 2021 at 16:15, on Zulip):

I can close some tabs and see how much memory it will take if I let it run long enough

Joshua Nelson (Jan 18 2021 at 16:16, on Zulip):

it got up to 2.7 GB this time and seems to be working reasonably well

Joshua Nelson (Jan 18 2021 at 16:16, on Zulip):

but I had to close pretty much everything except the zulip tab

Joshua Nelson (Jan 18 2021 at 16:18, on Zulip):

so it's not a bug in RA, it just uses too much memory for me to use it normally

Joshua Nelson (Jan 18 2021 at 16:21, on Zulip):

I tried the 'clear database' command again and it has the same issue as before, it shows -3.2 GB which I think means it allocated memory while the command was running

Joshua Nelson (Jan 18 2021 at 16:21, on Zulip):

top still shows RA using 2.7 GB RSS

Joshua Nelson (Jan 18 2021 at 16:21, on Zulip):

anyway, let me know if I can help more with this

Edwin Cheng (Jan 18 2021 at 16:22, on Zulip):

Here is my machine stats from top:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 109309 edwin     20   0 1920708   1.4g  15504 S   0.0   8.8   0:35.15 rust-analyzer
 109872 edwin     20   0  153216  79268  69088 S   0.0   0.5   0:02.13 /home/edwin/.cargo/bin/rust-analyzer proc-macro
Edwin Cheng (Jan 18 2021 at 16:23, on Zulip):

So it is around 1.5g and no panic occuried..

Joshua Nelson (Jan 18 2021 at 16:23, on Zulip):

:shrug: that is not my experience

Joshua Nelson (Jan 18 2021 at 16:23, on Zulip):

I'm on debian 10 with rustc 1.49.0 (e1884a8e3 2020-12-29) if it helps

Edwin Cheng (Jan 18 2021 at 16:25, on Zulip):

x86 ?

Joshua Nelson (Jan 18 2021 at 16:25, on Zulip):

x86_64, yeah. This is an 7th-gen intel i5 with 4 cores (2 physical cores) and 8 GB memory

Edwin Cheng (Jan 18 2021 at 16:26, on Zulip):

Iam still on 1.48 , let me update and try

Joshua Nelson (Jan 18 2021 at 16:27, on Zulip):

hmm, I might have some RA settings that aren't the default? How do I find those?

Edwin Cheng (Jan 18 2021 at 16:27, on Zulip):

If you are using vscode, just copy the json settings ?

oliver (Jan 18 2021 at 16:28, on Zulip):

do you only have the one Rust extension running?

Joshua Nelson (Jan 18 2021 at 16:28, on Zulip):
    "rust-analyzer.checkOnSave.overrideCommand": null,
    "rust-analyzer.checkOnSave.allTargets": false,
    "rust-analyzer.checkOnSave.enable": false
Joshua Nelson (Jan 18 2021 at 16:29, on Zulip):

oliver said:

do you only have the one Rust extension running?

yes, RA works fine on other projects, and works fine on this one other than the memory usage

Joshua Nelson (Jan 18 2021 at 16:29, on Zulip):

and top specifically shows rust-analyzer-x, not VSCode, as using the memory

Joshua Nelson (Jan 18 2021 at 16:30, on Zulip):

Joshua Nelson said:

    "rust-analyzer.checkOnSave.overrideCommand": null,
    "rust-analyzer.checkOnSave.allTargets": false,
    "rust-analyzer.checkOnSave.enable": false

(I have a bunch more things in settings, but none of them start with rust-analyzer)

oliver (Jan 18 2021 at 16:31, on Zulip):

me

    "rust-analyzer.updates.channel": "nightly",
    "rust-analyzer.updates.askBeforeDownload": false,
    "rust-analyzer.cargo.allFeatures": true,
    "rust-analyzer.assist.importMergeBehaviour": "full",
    "rust-analyzer.cargo.target": null,
    "rust-analyzer.cargoRunner": null,
Laurențiu (Jan 18 2021 at 16:32, on Zulip):

Does rust-analyzer-x analysis-stats . work?

oliver (Jan 18 2021 at 16:33, on Zulip):

as well ofc "rust-client.rustupPath": "~/.cargo/bin/rustup",

Joshua Nelson (Jan 18 2021 at 16:33, on Zulip):

hmm I need to find where in PATH rust-analyzer is, one sec

Laurențiu (Jan 18 2021 at 16:34, on Zulip):

Why does it say -x?

Joshua Nelson (Jan 18 2021 at 16:34, on Zulip):

oh it must be /home/joshua/.config/Code/User/globalStorage/matklad.rust-analyzer/rust-analyzer-x86_64-unknown-linux-gnu

Joshua Nelson (Jan 18 2021 at 16:34, on Zulip):

I guess top cut it off because it was too long lol

Laurențiu (Jan 18 2021 at 16:34, on Zulip):

Right

Joshua Nelson (Jan 18 2021 at 16:34, on Zulip):

Failed to create perf counter: Operation not permitted (os error 1)

do I need to run it with sudo?

Joshua Nelson (Jan 18 2021 at 16:35, on Zulip):

(does RA try to modify things in the home directory?)

Laurențiu (Jan 18 2021 at 16:35, on Zulip):

Maybe try installing GNU time and running command time ~/.config/Code/User/globalStorage/matklad.rust-analyzer/rust-analyzer-x86_64-unknown-linux-gnu analysis-stats .

Laurențiu (Jan 18 2021 at 16:35, on Zulip):

For the perf counters yes, but you don't _need_ them, I suppose

Joshua Nelson (Jan 18 2021 at 16:35, on Zulip):

oh sorry it got OOM killed again, let me finish the other thing I'm doing

> /home/joshua/.config/Code/User/globalStorage/matklad.rust-analyzer/rust-analyzer-x86_64-unknown-linux-gnu analysis-stats .
Failed to create perf counter: Operation not permitted (os error 1)
Terminated
Laurențiu (Jan 18 2021 at 16:35, on Zulip):

It runs cargo metadata so it might write some files as root if you use sudo

Laurențiu (Jan 18 2021 at 16:36, on Zulip):

Cool, then it's easier to reproduce it

Laurențiu (Jan 18 2021 at 16:37, on Zulip):
Database loaded:     2.28s, 516minstr
  crates: 9, mods: 102, decls: 900, fns: 623
Item Collection:     10.07s, 106ginstr
  exprs: 48928, ??ty: 676 (1%), ?ty: 797 (1%), !ty: 93
Inference:           6.56s, 59ginstr
Total:               16.63s, 165ginstr
17.83user 0.41system 0:19.51elapsed 93%CPU (0avgtext+0avgdata 1206296maxresident)k
179344inputs+16520outputs (94major+382095minor)pagefaults 0swaps
Laurențiu (Jan 18 2021 at 16:37, on Zulip):

So it's using.. 1.2G on my system

Laurențiu (Jan 18 2021 at 16:38, on Zulip):

What distro are you running? Maybe it's allocator-specific?

Edwin Cheng (Jan 18 2021 at 16:40, on Zulip):

Joshua Nelson said:

Joshua Nelson said:

    "rust-analyzer.checkOnSave.overrideCommand": null,
    "rust-analyzer.checkOnSave.allTargets": false,
    "rust-analyzer.checkOnSave.enable": false

(I have a bunch more things in settings, but none of them start with rust-analyzer)

So you do not enable proc-macro support ...? Um..

Laurențiu (Jan 18 2021 at 16:40, on Zulip):

analysis-stats also doesn't enable proc macros by default

Laurențiu (Jan 18 2021 at 16:40, on Zulip):

If you're using a pre-built binary, I guess it's not CentOS 7, so the oldest one that works is Ubuntu 16.04 with GLIBC 2.23 and the new allocator got merged in 2.26..

Joshua Nelson (Jan 18 2021 at 16:42, on Zulip):

this is ubuntu 20.04

Florian Diebold (Jan 18 2021 at 16:42, on Zulip):

there's also the --memory-usage argument to analysis-stats

Florian Diebold (Jan 18 2021 at 16:42, on Zulip):
Database loaded:     1.26s, 512minstr, 129mb
  crates: 9, mods: 102, decls: 900, fns: 623
Item Collection:     25.28s, 136ginstr, 755mb
  exprs: 49526, ??ty: 246 (0%), ?ty: 455 (0%), !ty: 10
Inference:           16.16s, 76ginstr, 306mb
Total:               41.44s, 212ginstr, 1062mb
43.09user 0.69system 0:43.95elapsed 99%CPU (0avgtext+0avgdata 1226628maxresident)k
0inputs+8outputs (0major+373318minor)pagefaults 0swaps
Laurențiu (Jan 18 2021 at 16:42, on Zulip):

Not if it OOMs :smiley:

Joshua Nelson (Jan 18 2021 at 16:43, on Zulip):

Edwin Cheng said:

Joshua Nelson said:

Joshua Nelson said:

    "rust-analyzer.checkOnSave.overrideCommand": null,
    "rust-analyzer.checkOnSave.allTargets": false,
    "rust-analyzer.checkOnSave.enable": false

(I have a bunch more things in settings, but none of them start with rust-analyzer)

So you do not enable proc-macro support ...? Um..

docs.rs doesn't use a ton of proc-macros I don't think

Florian Diebold (Jan 18 2021 at 16:43, on Zulip):

in any case, it seems to OOM without proc-macros anyway

Joshua Nelson (Jan 18 2021 at 16:43, on Zulip):

still waiting for the other build to finish, sorry

Joshua Nelson (Jan 18 2021 at 16:44, on Zulip):

clearly the solution is for rust to just be a simpler language to compile :laughing:

Edwin Cheng (Jan 18 2021 at 16:44, on Zulip):

Okay, rust 1.49 build and disabled proc-macro in my machine:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 130314 edwin     20   0 1645292   1.1g  16156 S   0.0   7.3   0:29.51 rust-analyzer
Edwin Cheng (Jan 18 2021 at 16:46, on Zulip):

@Joshua Nelson what's your rust-analyzer version ?

Joshua Nelson (Jan 18 2021 at 16:47, on Zulip):

--version tells me rust-analyzer fde4a86

Joshua Nelson (Jan 18 2021 at 16:47, on Zulip):

not sure how to get a date from that

Laurențiu (Jan 18 2021 at 16:47, on Zulip):

It's today's

Edwin Cheng (Jan 18 2021 at 16:48, on Zulip):

You build from source ? but your path is vscode extension one ?

Joshua Nelson (Jan 18 2021 at 16:49, on Zulip):

I did not build from source. This is from the 'rust-analyzer server is not installed' button in vscode

Joshua Nelson (Jan 18 2021 at 16:50, on Zulip):

(the other build is still running, sorry)

Laurențiu (Jan 18 2021 at 16:51, on Zulip):

Tried it in Docker:

Failed to create perf counter: Operation not permitted (os error 1)
Database loaded:     374.68ms, 131mb
Failed to create perf counter: Operation not permitted (os error 1)
  crates: 9, mods: 102, decls: 900, fns: 623
Item Collection:     10.05s, 771mb
Failed to create perf counter: Operation not permitted (os error 1)
  exprs: 48928, ??ty: 676 (1%), ?ty: 797 (1%), !ty: 93
Inference:           6.43s, 301mb
Total:               16.48s, 1072mb
Laurențiu (Jan 18 2021 at 16:53, on Zulip):

Next step would be a live USB :smile:

Joshua Nelson (Jan 18 2021 at 16:55, on Zulip):

I wouldn't expect it to be related to the distro

oliver (Jan 18 2021 at 16:55, on Zulip):

I'm on the same thing as you and using 935Mb

Joshua Nelson (Jan 18 2021 at 16:57, on Zulip):

Laurențiu Nicola said:

Maybe try installing GNU time and running command time ~/.config/Code/User/globalStorage/matklad.rust-analyzer/rust-analyzer-x86_64-unknown-linux-gnu analysis-stats .

ok the other build finally finished, running this now

Joshua Nelson (Jan 18 2021 at 16:59, on Zulip):
> command time -v /home/joshua/.config/Code/User/globalStorage/matklad.rust-analyzer/rust-analyzer-x86_64-unknown-linux-gnu analysis-stats .
Database loaded:     45.09s
  crates: 9, mods: 102, decls: 900, fns: 623
Item Collection:     17.72s
92/623 14% processing: web::crate_details::tests::test_last_successful_build_with_intermittent_releases_failed_or_ya3/623 14% processing: web::crate_details::tests::test_last_successful_build_with_intermittent_releases_failed_or_yan  exprs: 49528, ??ty: 246 (0%), ?ty: 455 (0%), !ty: 11
Inference:           11.38s
Total:               29.09s
    Command being timed: "/home/joshua/.config/Code/User/globalStorage/matklad.rust-analyzer/rust-analyzer-x86_64-unknown-linux-gnu analysis-stats ."
    User time (seconds): 33.97
    System time (seconds): 10.80
    Percent of CPU this job got: 59%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 1:15.48
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 2867164
Joshua Nelson (Jan 18 2021 at 17:01, on Zulip):

oh whoops let me run that again with --memory-usage

Joshua Nelson (Jan 18 2021 at 17:01, on Zulip):
> command time -v /home/joshua/.config/Code/User/globalStorage/matklad.rust-analyzer/rust-analyzer-x86_64-unknown-linux-gnu analysis-stats --memory-usage .
Database loaded:     10.98s, 1682mb
  crates: 9, mods: 102, decls: 900, fns: 623
Item Collection:     18.26s, -3332mb
92/623 14% processing: web::crate_details::tests::test_last_successful_build_with_intermittent_releases_failed_or_ya3/623 14% processing: web::crate_details::tests::test_last_successful_build_with_intermittent_releases_failed_or_yan  exprs: 49528, ??ty: 246 (0%), ?ty: 455 (0%), !ty: 11
Inference:           12.36s, 307mb
Total:               30.63s, -3025mb
    Command being timed: "/home/joshua/.config/Code/User/globalStorage/matklad.rust-analyzer/rust-analyzer-x86_64-unknown-linux-gnu analysis-stats --memory-usage ."
    User time (seconds): 33.40
    System time (seconds): 3.47
    Percent of CPU this job got: 85%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:42.93
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 2872688
Joshua Nelson (Jan 18 2021 at 17:02, on Zulip):

Total: 30.63s, -3025mb

this looks very wrong

oliver (Jan 18 2021 at 17:02, on Zulip):

like corrupted memory wrong...

Daniel Silverstone (Jan 18 2021 at 17:02, on Zulip):

Is that using an i32 somewhere to count memory usage?

Jonas Schievink [he/him] (Jan 18 2021 at 17:02, on Zulip):

what, with the cli?? that should never happen

Joshua Nelson (Jan 18 2021 at 17:03, on Zulip):

that's what it says

Jonas Schievink [he/him] (Jan 18 2021 at 17:04, on Zulip):

we use isize to count this

Joshua Nelson (Jan 18 2021 at 17:05, on Zulip):

hmm, isize is 8 bytes on x64

Jonas Schievink [he/him] (Jan 18 2021 at 17:05, on Zulip):

why is the actual per-query memory usage missing? did this break recently?

oliver (Jan 18 2021 at 17:06, on Zulip):

you need to wait a bit for that

Jonas Schievink [he/him] (Jan 18 2021 at 17:06, on Zulip):

it happens locally too, it just exits without outputting it

Joshua Nelson (Jan 18 2021 at 17:07, on Zulip):

Daniel Silverstone said:

Is that using an i32 somewhere to count memory usage?

hmm, casting that to u32 isn't very informative: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=75edeec5350fc6b12e9f50b2a64def8d

Jonas Schievink [he/him] (Jan 18 2021 at 17:07, on Zulip):

ah, now you have to pass --verbose too

Joshua Nelson (Jan 18 2021 at 17:07, on Zulip):

sure, running that

Jonas Schievink [he/him] (Jan 18 2021 at 17:07, on Zulip):

can't wait for this to be a bug in glibc's memory allocator

Joshua Nelson (Jan 18 2021 at 17:08, on Zulip):

hmm, this output is weird:

/home/joshua/src/rust/docs.rs/src/web/mod.rs 372:22-372:25: Expected str, got &str
/home/joshua/src/rust/docs.rs/src/web/rustdoc.rs 388:25-388:40: Expected &str, got &String
/home/joshua/src/rust/docs.rs/src/web/builds.rs 269:20-269:22: Expected &(dyn ToSql + Sync), got &_
/home/joshua/src/rust/docs.rs/src/web/metrics.rs 41:21-41:38: Expected Box<dyn Handler, Global>, got Box<impl Handler, Global>
/home/joshua/src/rust/docs.rs/src/test/fakes.rs 303:40-303:52: Expected _, got &str
/home/joshua/src/rust/docs.rs/src/test/fakes.rs 305:36-305:42: Expected &str, got &<[&String] as Join<&str>>::Output
/home/joshua/src/rust/docs.rs/src/test/fakes.rs 314:56-314:70: Expected Option<&str>, got Option<&String>
/home/joshua/src/rust/docs.rs/src/db/migrate.rs 64:8-180:9: Expected Vec<Box<dyn PostgresMigration, Global>>, got Vec<Box<Amigration, Global>>
Joshua Nelson (Jan 18 2021 at 17:10, on Zulip):

anyway, here's with --verbose: https://gist.github.com/jyn514/01a0f789a2464d116d5c3974d4b1f52f

Joshua Nelson (Jan 18 2021 at 17:11, on Zulip):

Total: 29.23s, -3025mb

smh

Joshua Nelson (Jan 18 2021 at 17:11, on Zulip):

the culprit seems to be

 -3280mb FileTextQuery (purge)
Jonas Schievink [he/him] (Jan 18 2021 at 17:12, on Zulip):

oh

int uordblks;

Jonas Schievink [he/him] (Jan 18 2021 at 17:12, on Zulip):

int is i32 on x86_64

Joshua Nelson (Jan 18 2021 at 17:12, on Zulip):

that would do it

Joshua Nelson (Jan 18 2021 at 17:12, on Zulip):

where is uordblks from?

Jonas Schievink [he/him] (Jan 18 2021 at 17:12, on Zulip):

https://man7.org/linux/man-pages/man3/mallinfo.3.html

Jonas Schievink [he/him] (Jan 18 2021 at 17:12, on Zulip):

that's what we use to figure out the memory usage via mallinfo()

Joshua Nelson (Jan 18 2021 at 17:13, on Zulip):

looks like malloc_info() handles this: https://stackoverflow.com/questions/40878169/64-bit-capable-alternative-to-mallinfo

Joshua Nelson (Jan 18 2021 at 17:13, on Zulip):

https://linux.die.net/man/3/malloc_info

Joshua Nelson (Jan 18 2021 at 17:13, on Zulip):

lol wtf why is it XML

Jonas Schievink [he/him] (Jan 18 2021 at 17:13, on Zulip):

XML?!

Laurențiu (Jan 18 2021 at 17:14, on Zulip):

What's a block in mallinfo?

Laurențiu (Jan 18 2021 at 17:14, on Zulip):

If they're not bytes, it might still fit in an int

Joshua Nelson (Jan 18 2021 at 17:16, on Zulip):

oh great, there's malloc_stats but it only prints to stdout :joy: https://man7.org/linux/man-pages/man3/malloc_stats.3.html

Joshua Nelson (Jan 18 2021 at 17:16, on Zulip):

what clown designed this

Jonas Schievink [he/him] (Jan 18 2021 at 17:17, on Zulip):

urgh this sucks, we've wanted a high-performance pure-Rust allocator for a long time now

Jonas Schievink [he/him] (Jan 18 2021 at 17:17, on Zulip):

it would be much easier to support this then

Joshua Nelson (Jan 18 2021 at 17:18, on Zulip):

I think jemalloc has support for introspection

Jonas Schievink [he/him] (Jan 18 2021 at 17:18, on Zulip):

I think we used to use that, even

Jonas Schievink [he/him] (Jan 18 2021 at 17:18, on Zulip):

but that required passing a cargo feature

Joshua Nelson (Jan 18 2021 at 17:21, on Zulip):

Jonas Schievink said:

but that required passing a cargo feature

what's wrong with that?

Jonas Schievink [he/him] (Jan 18 2021 at 17:24, on Zulip):

minor annoyance, not sure if there was anything else...

Jonas Schievink [he/him] (Jan 18 2021 at 17:24, on Zulip):

https://github.com/rust-analyzer/rust-analyzer/pull/5479

Jonas Schievink [he/him] (Jan 18 2021 at 17:25, on Zulip):

removing jemalloc entirely was probably not the right move then

Jonas Schievink [he/him] (Jan 18 2021 at 17:25, on Zulip):

mallinfo is also way way slower than jemalloc's API

Joshua Nelson (Jan 18 2021 at 17:25, on Zulip):

if features are annoying, you could set jemalloc on by default

Joshua Nelson (Jan 18 2021 at 17:26, on Zulip):

and then only disable it if someone really cares

Jonas Schievink [he/him] (Jan 18 2021 at 17:26, on Zulip):

I can't find what removed it entirely though

Joshua Nelson (Jan 18 2021 at 17:27, on Zulip):

Jonas Schievink said:

can't wait for this to be a bug in glibc's memory allocator

I guess in a sense it was lmao

Daniel Silverstone (Jan 18 2021 at 17:28, on Zulip):

What's surprising is how much more RSS you seem to need over other people though

Joshua Nelson (Jan 18 2021 at 17:28, on Zulip):

yeah :( IDK what's up with that

Joshua Nelson (Jan 18 2021 at 17:29, on Zulip):

could someone else post the --verbose output for the query memory usage?

Joshua Nelson (Jan 18 2021 at 17:29, on Zulip):

maybe @Jonas Schievink or @Laurențiu Nicola since you had about ~1.3 GB used?

oliver (Jan 18 2021 at 17:33, on Zulip):

https://paste.debian.net/1181653/

Laurențiu (Jan 18 2021 at 17:34, on Zulip):

https://bpa.st/JHYA

Edwin Cheng (Jan 18 2021 at 17:34, on Zulip):

https://gist.github.com/edwin0cheng/82663ac774ae14aa362236fadfce52e3

matklad (Jan 18 2021 at 17:35, on Zulip):

urgh this sucks, we've wanted a high-performance pure-Rust allocator for a long time now

this needs #[thread_local] sadly

Jonas Schievink [he/him] (Jan 18 2021 at 17:39, on Zulip):

looks like the regression was due to the hygiene frame query then

Jonas Schievink [he/him] (Jan 18 2021 at 17:41, on Zulip):

low-effort improvement: https://github.com/rust-analyzer/rust-analyzer/pull/7331

Joshua Nelson (Jan 18 2021 at 17:53, on Zulip):

Wow, FileTextQuery is using significantly less memory for you

Joshua Nelson (Jan 18 2021 at 17:53, on Zulip):

I don't know why that would be different

Edwin Cheng (Jan 18 2021 at 17:55, on Zulip):

Do you have any text files generated from the project ? We all tested from a fresh github pull

Jonas Schievink [he/him] (Jan 18 2021 at 17:55, on Zulip):

it's not actually using over 3 GB, that's the mallinfo wraparound

Jonas Schievink [he/him] (Jan 18 2021 at 17:56, on Zulip):

FileTextQuery really is just the raw file contents of on-disk files, you'd notice if docs.rs was more than 3 GB in size

Jonas Schievink [he/him] (Jan 18 2021 at 17:56, on Zulip):

well, that plus dependencies

cynecx (Jan 18 2021 at 17:57, on Zulip):

perhaps its the allocator not releasing memory to the system?

Jonas Schievink [he/him] (Jan 18 2021 at 17:57, on Zulip):

60 MB sounds about right for FileTextQuery

Jonas Schievink [he/him] (Jan 18 2021 at 17:57, on Zulip):

cynecx said:

perhaps its the allocator not releasing memory to the system?

we measure what the allocator reports, not what the kernel reports

cynecx (Jan 18 2021 at 18:11, on Zulip):

(deleted)

cynecx (Jan 18 2021 at 18:11, on Zulip):

isn't vfs unsually high? 60mb vs 813mb.

Jonas Schievink [he/him] (Jan 18 2021 at 18:18, on Zulip):

currently adding back jemalloc support for more accurate stats

Florian Diebold (Jan 18 2021 at 18:29, on Zulip):

/me waits for the bug to disappear with jemalloc

Joshua Nelson (Jan 18 2021 at 18:37, on Zulip):

Edwin Cheng said:

Do you have any text files generated from the project ? We all tested from a fresh github pull

hmm I do have a bunch of untracked files, let me try deleting those

Joshua Nelson (Jan 18 2021 at 18:38, on Zulip):

oh I bet I know what it is - does rust-analyzer try to look for recursive projects? It might be looking in .rustwide even though it shouldn't

Jonas Schievink [he/him] (Jan 18 2021 at 18:39, on Zulip):

hmm, that could be it

cynecx (Jan 18 2021 at 18:40, on Zulip):

well that actually explains why vfs is so high lol

Joshua Nelson (Jan 18 2021 at 18:40, on Zulip):

ok yeah I removed those and it's down to 1.9 GB

Joshua Nelson (Jan 18 2021 at 23:20, on Zulip):

(continued in https://rust-lang.zulipchat.com/#narrow/stream/185405-t-compiler.2Fwg-rls-2.2E0/topic/VFS.20eagerly.20loads.20all.20.2Ers.20files.20in.20the.20project.20directory)

Laurențiu (Feb 20 2021 at 14:26, on Zulip):

Jonas Schievink [he/him] said:

int is i32 on x86_64

The mallinfo2 function is added to report statistics as per mallinfo,
but with larger field widths to accurately report values that are
larger than fit in an integer.

Laurențiu (Feb 20 2021 at 14:29, on Zulip):

At least if you're running Arch or Gentoo

Last update: Jul 24 2021 at 20:00UTC