Stream: t-compiler/help

Topic: no space left on device (tmpfs) when compiling rustc


Santiago Pastorino (Aug 14 2019 at 18:28, on Zulip):

I was getting the following error recently ...

Santiago Pastorino (Aug 14 2019 at 18:28, on Zulip):
   Compiling rustc_driver v0.0.0 (/home/santiago/src/oss/rust1/src/librustc_driver)
error: failed to write version script: No space left on device (os error 28)
Santiago Pastorino (Aug 14 2019 at 18:29, on Zulip):

and was due to tmpfs being automatically configured by ArchLinux to use half of my RAM (on my local machine 8GB right now) so that meant 4GB of tmpfs

Santiago Pastorino (Aug 14 2019 at 18:29, on Zulip):

and was running out of space when compiling

Santiago Pastorino (Aug 14 2019 at 18:30, on Zulip):

I ended wondering ... given that tmpfs is usually resides in memory and/or swap partition, is it the best idea to have the compilation (at least of rustc) using /tmp ?

Santiago Pastorino (Aug 14 2019 at 18:30, on Zulip):

did someone else hitted something like this?

Santiago Pastorino (Aug 14 2019 at 18:30, on Zulip):

am I wrong? (probably :joy:)

nagisa (Aug 14 2019 at 19:30, on Zulip):

rustc puts some temporary stuff into tmpfs afaik

Santiago Pastorino (Aug 14 2019 at 19:58, on Zulip):

@nagisa yeah, but it was filling up 4GB of stuff, which could mean 4GB of memory for intermediate compilation artifacts

Santiago Pastorino (Aug 14 2019 at 19:58, on Zulip):

besides from the RAM used by the process to compile

Santiago Pastorino (Aug 15 2019 at 15:58, on Zulip):

to add more info about this ...

Santiago Pastorino (Aug 15 2019 at 15:59, on Zulip):
[santiago@archlinux rust1 (master)]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
dev             3.8G     0  3.8G   0% /dev
run             3.8G  1.7M  3.8G   1% /run
/dev/nvme0n1p2  234G  149G   73G  68% /
tmpfs           3.8G  179M  3.6G   5% /dev/shm
tmpfs           3.8G     0  3.8G   0% /sys/fs/cgroup
tmpfs           3.8G  3.8G     0 100% /tmp
/dev/nvme0n1p1  511M   68M  444M  14% /boot
tmpfs           771M  8.0M  763M   2% /run/user/1000
Santiago Pastorino (Aug 15 2019 at 15:59, on Zulip):

rustc files took up to 4GB on my machine and that's why it gave that error

nikomatsakis (Aug 15 2019 at 15:59, on Zulip):

It does sound to me (based on some links that @Santiago Pastorino sent me) that /tmp is not meant to be used for larger files

Santiago Pastorino (Aug 15 2019 at 15:59, on Zulip):

yep

Santiago Pastorino (Aug 15 2019 at 15:59, on Zulip):

/var/tmp should be used instead

nikomatsakis (Aug 15 2019 at 15:59, on Zulip):

There must be some system API that we can use to get a "suitable path for temporary, large files" right?

nikomatsakis (Aug 15 2019 at 16:00, on Zulip):

like, rather than hardcode /var/tmp

nikomatsakis (Aug 15 2019 at 16:00, on Zulip):

I'm not sure where we're selecting those paths

nikomatsakis (Aug 15 2019 at 16:00, on Zulip):

I suspect I know the file though

Santiago Pastorino (Aug 15 2019 at 16:00, on Zulip):

can you point more or less to where?

Santiago Pastorino (Aug 15 2019 at 16:00, on Zulip):

I can investigate further

Santiago Pastorino (Aug 15 2019 at 16:01, on Zulip):

for more info about this issue

Santiago Pastorino (Aug 15 2019 at 16:01, on Zulip):

As mentioned, this is a tmpfs on many Linuxes/Unixes (and most likely will be for most soon), and hence should be used only for small files. It's generally a shared namespace, hence the only APIs for using it should be mkstemp()

Santiago Pastorino (Aug 15 2019 at 16:01, on Zulip):

http://0pointer.net/blog/projects/tmp.html

Santiago Pastorino (Aug 15 2019 at 16:01, on Zulip):

My CD burning application writes huge .iso files to /tmp, and this breaks on tmpfs!

The application should be fixed to use /var/tmp.

Santiago Pastorino (Aug 15 2019 at 16:01, on Zulip):

https://fedoraproject.org/wiki/Features/tmp-on-tmpfs

nikomatsakis (Aug 15 2019 at 16:02, on Zulip):

I think @Santiago Pastorino it winds up coming from the OutputFilenames struct

nikomatsakis (Aug 15 2019 at 16:02, on Zulip):

which is configured from some out_directory field

nikomatsakis (Aug 15 2019 at 16:02, on Zulip):

although that seems unlikely to be /tmp

nikomatsakis (Aug 15 2019 at 16:03, on Zulip):

I guess I don't know for sure

nikomatsakis (Aug 15 2019 at 16:03, on Zulip):

which is the file that you saw in /tmp?

nikomatsakis (Aug 15 2019 at 16:03, on Zulip):

something about metadata?

Santiago Pastorino (Aug 15 2019 at 16:05, on Zulip):

yes but others too

Santiago Pastorino (Aug 15 2019 at 16:05, on Zulip):

let me try to reproduce it

Santiago Pastorino (Aug 15 2019 at 16:06, on Zulip):

it may take a while though :)

Santiago Pastorino (Aug 15 2019 at 18:25, on Zulip):

ok, did a quick repro ...

Santiago Pastorino (Aug 15 2019 at 18:26, on Zulip):

first, my /tmp is of type tmpfs and uses up to 4GB

Santiago Pastorino (Aug 15 2019 at 18:27, on Zulip):
[santiago@archlinux rustcaSq42I]$ mount | grep /tmp
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime,size=4194304k)
Santiago Pastorino (Aug 15 2019 at 18:27, on Zulip):

when I try to build from scratch, I get ...

Santiago Pastorino (Aug 15 2019 at 18:27, on Zulip):
   Compiling rustc_save_analysis v0.0.0 (/home/santiago/src/oss/rust2/src/librustc_save_analysis)
   Compiling rustc_interface v0.0.0 (/home/santiago/src/oss/rust2/src/librustc_interface)
LLVM ERROR: IO failure on output stream: No space left on device
error: Could not compile `rustc_driver`.

To learn more, run the command again with --verbose.
command did not execute successfully: "/home/santiago/src/oss/rust2/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "-j" "4" "--release" "--features" "" "--manifest-path" "/home/santiago/src/oss/rust2/src/rustc/Cargo.toml" "--message-format" "json-render-diagnostics"
expected success, got: exit code: 101
failed to run: /home/santiago/src/oss/rust2/build/bootstrap/debug/bootstrap build -i --stage 1 src/libstd
Build completed unsuccessfully in 0:41:50
Santiago Pastorino (Aug 15 2019 at 18:27, on Zulip):

then I see ...

Santiago Pastorino (Aug 15 2019 at 18:28, on Zulip):
[santiago@archlinux rust2 (master)]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
...
tmpfs           4.0G  4.0G     0 100% /tmp
Santiago Pastorino (Aug 15 2019 at 18:28, on Zulip):

and then on /tmp I see the following ...

Santiago Pastorino (Aug 15 2019 at 18:28, on Zulip):
[santiago@archlinux rustcaSq42I]$ pwd
/tmp/rustcaSq42I
[santiago@archlinux rustcaSq42I]$ ls -lh
total 4.0G
-rw-r--r-- 1 santiago santiago  6.9M Aug 15 14:44 libcc-580cb3ec679a1ef0.rlib
-rw-r--r-- 1 santiago santiago  243K Aug 15 14:44 libcrc32fast-df4710f26ff3874f.rlib
-rw-r--r-- 1 santiago santiago   746 Aug 15 14:44 libeither-be66c1a9963b273c.rlib
-rw-r--r-- 1 santiago santiago  2.1M Aug 15 14:44 libenv_logger-39812d84bbfe2b11.rlib
-rw-r--r-- 1 santiago santiago  595K Aug 15 14:44 libflate2-31c9dbe4de46ec52.rlib
-rw-r--r-- 1 santiago santiago  414K Aug 15 14:44 libhumantime-d1a9d86e783663fa.rlib
-rw-r--r-- 1 santiago santiago   742 Aug 15 14:44 libitoa-cafbf9ae61da85d2.rlib
-rw-r--r-- 1 santiago santiago  202K Aug 15 14:44 liblog_settings-2620b484081ec8f2.rlib
-rw-r--r-- 1 santiago santiago   64K Aug 15 14:44 libminiz_sys-829f35f627be716a.rlib
-rw-r--r-- 1 santiago santiago  584K Aug 15 14:44 libpunycode-bc98388f5faa9996.rlib
-rw-r--r-- 1 santiago santiago   764 Aug 15 14:44 libquick_error-b52ae6e62fe18a00.rlib
-rw-r--r-- 1 santiago santiago   770 Aug 15 14:44 libremove_dir_all-a10b98c02f91914a.rlib
-rw-r--r-- 1 santiago santiago  666M Aug 15 14:44 librustc-300974c3ebad3d8b.rlib
-rw-r--r-- 1 santiago santiago  139M Aug 15 14:44 librustc_ast_borrowck-064564744d0b66e4.rlib
-rw-r--r-- 1 santiago santiago  158M Aug 15 14:44 librustc_codegen_ssa-0b961b0d1ebdaa34.rlib
-rw-r--r-- 1 santiago santiago  141M Aug 15 14:44 librustc_codegen_utils-3c4fec2d7a4aaa58.rlib
-rw-r--r-- 1 santiago santiago  215M Aug 15 14:44 librustc_incremental-413cebfc5fe72a28.rlib
-rw-r--r-- 1 santiago santiago  163M Aug 15 14:44 librustc_interface-4a187b71ebecda3f.rlib
-rw-r--r-- 1 santiago santiago  128M Aug 15 14:44 librustc_lint-f5d7618b6d033ce8.rlib
-rw-r--r-- 1 santiago santiago  315M Aug 15 14:44 librustc_metadata-a179352ab6d85b74.rlib
-rw-r--r-- 1 santiago santiago  822M Aug 15 14:44 librustc_mir-232a965ccb3e9077.rlib
-rw-r--r-- 1 santiago santiago  143M Aug 15 14:44 librustc_passes-c08b598d7f5694c2.rlib
-rw-r--r-- 1 santiago santiago   36M Aug 15 14:44 librustc_plugin-8f9749116c8b6ad6.rlib
-rw-r--r-- 1 santiago santiago  138M Aug 15 14:44 librustc_privacy-1eb27d780fbbcb81.rlib
-rw-r--r-- 1 santiago santiago   76M Aug 15 14:44 librustc_resolve-23cfc118665ed372.rlib
-rw-r--r-- 1 santiago santiago  138M Aug 15 14:44 librustc_save_analysis-970ae44e6e8aee42.rlib
-rw-r--r-- 1 santiago santiago  267M Aug 15 14:44 librustc_traits-1f50191ae0f8c9ba.rlib
-rw-r--r-- 1 santiago santiago  469M Aug 15 14:44 librustc_typeck-0246067c1ba94397.rlib
-rw-r--r-- 1 santiago santiago  298K Aug 15 14:44 libryu-17527a943b47993b.rlib
-rw-r--r-- 1 santiago santiago  3.6M Aug 15 14:44 libserde_json-f180caae537f064f.rlib
-rw-r--r-- 1 santiago santiago   71M Aug 15 14:44 libsyntax_ext-c3562645f3bbdcaa.rlib
-rw-r--r-- 1 santiago santiago 1006K Aug 15 14:44 libtempfile-7dcc5500a694e84f.rlib
-rw-r--r-- 1 santiago santiago  1.1M Aug 15 14:44 list
Santiago Pastorino (Aug 15 2019 at 18:30, on Zulip):

and ...

Santiago Pastorino (Aug 15 2019 at 18:30, on Zulip):
[santiago@archlinux rustcaSq42I]$ du -sh .
4.0G    .
Santiago Pastorino (Aug 15 2019 at 18:30, on Zulip):

@nikomatsakis ^^^, in particular start here

Santiago Pastorino (Aug 15 2019 at 21:28, on Zulip):

/cc @Alex Crichton

Santiago Pastorino (Aug 26 2019 at 19:14, on Zulip):

@nagisa unsure if you ended reading this since you commented in the beginning

Santiago Pastorino (Aug 26 2019 at 19:15, on Zulip):

maybe another person that may know about it is @Zoxc

Santiago Pastorino (Aug 26 2019 at 19:15, on Zulip):

I guess this https://github.com/rust-lang/rust/issues/61978 is unrelated but just in case, bringing it up

Santiago Pastorino (Aug 26 2019 at 19:16, on Zulip):

I meant, that's not related per se, what I meant given that @Zoxc worked on that, they may know about it

Zoxc (Aug 26 2019 at 19:31, on Zulip):

I don't like that rustc puts anything in a temp dir at all. It tends to leave a lot of stuff there due to crashes, etc.

Santiago Pastorino (Aug 26 2019 at 19:33, on Zulip):

should we at least have an issue to track this?

Santiago Pastorino (Aug 26 2019 at 19:34, on Zulip):

I don't like that rustc puts anything in a temp dir at all. It tends to leave a lot of stuff there due to crashes, etc.

what would you do btw?

Zoxc (Aug 26 2019 at 19:40, on Zulip):

Let cargo tell rustc about the build dir so rustc can use that

nagisa (Aug 26 2019 at 20:46, on Zulip):

This is the first time I see full rlibs in there.

nagisa (Aug 26 2019 at 20:46, on Zulip):

but if we do put them there then its just a genuine bug in rustc.

Santiago Pastorino (Aug 26 2019 at 20:54, on Zulip):

@nagisa yes, that's what I'm saying :)

Santiago Pastorino (Aug 26 2019 at 20:55, on Zulip):

I think it's easy to reproduce on Linux, just set your tmpfs to be 4GB or 2GB and try to do a fresh compile

Last update: Nov 11 2019 at 22:50UTC