Stream: general

Topic: working directory management

davidtwco (Mar 08 2019 at 10:03, on Zulip):

I recently made a small bash script that I've been using to manage my working directories and keep them up-to-date. Thought I'd drop a link here in case anyone is interested.

In my workflow, I have a projects/rust directory with my config.toml in it. I then have a projects/rust/workdirs directory containing my working copies of rust-lang/rust (around nine at the moment), in directories rust0 through rust8. Each working directory symlinks the config.toml from the parent directory. When I start on a new issue, I pick a working directory, check out a branch in it, and leave it assigned to that issue until the PR lands - that way I can always go back and respond to feedback quickly.

With this script, I define a .workman_config file alongside my config.toml which contains the project name, git urls (both upstream and my fork) and commands to run. After that, I can run..

It also will look for a file in the working directory that will allow it to update the directory, but not assign it. I use this to keep a unoptimized build directory that takes longer to build but I can use with a debugger better.

It can share the same workdirs directory for multiple projects. I currently use it with rust, rust-analyzer and compiler-team, each with their own config file and a zsh alias for workman that loads the right config file.

RalfJ (Mar 08 2019 at 10:09, on Zulip):

@davidtwco do you know "git worktree"? that way you can have multiple check-outs of the same repo, sharing repo configuration and git object store. (unfortunately, submodules are not shared though.)

davidtwco (Mar 08 2019 at 10:11, on Zulip):

That won't affect needing to rebuild when I checkout other branches though, right?

davidtwco (Mar 08 2019 at 10:11, on Zulip):

Huh, maybe it would.

davidtwco (Mar 08 2019 at 10:13, on Zulip):

Oh well, figured it was likely that there'd be a far easier way, particularly after I'd put the effort into writing that script.

RalfJ (Mar 08 2019 at 10:13, on Zulip):

I don't think it changes much, and your script does way more

RalfJ (Mar 08 2019 at 10:14, on Zulip):

but it'd save you some cloning and pulling -- which I guess doesn't matter much for you because of the cron job. I like that idea but my machine is usually turned off at night, so I can't copy it :/

davidtwco (Mar 08 2019 at 10:14, on Zulip):

I could probably make the script use worktrees instead of the cloning and pulling. But still use this to handle the resetting and updating of working directories.

ange (Mar 08 2019 at 15:32, on Zulip):

@davidtwco thanks for sharing. do you use something to keep track of changes to your config.toml too?

davidtwco (Mar 08 2019 at 15:32, on Zulip):

I don't, I just copy it from machine to machine.

ange (Mar 08 2019 at 15:36, on Zulip):

I wish more people would share tips and tricks they're using in their setup. I'm sure I'm not the first person who cooked up a script to normalize away insignificant differences in debug logs, or wanted to grep and then do a vimdiff of pretty-printed ({:#?}) structures that extend over multiple lines

ange (Mar 08 2019 at 15:37, on Zulip):


pnkfelix (Mar 12 2019 at 15:20, on Zulip):

I really should be using git-worktree (or at least investigating it)

DPC (Mar 12 2019 at 22:41, on Zulip):

Just curious, is there a reason why you are creating new projects instead of creating new branches every time? It might make things easier

pnkfelix (Mar 12 2019 at 23:14, on Zulip):

I don't know if the question was addressed to me, but: if our build times were instantaneous, then maybe I could imagine just keeping work-in-progress on different branches and switching between branches as necessary

pnkfelix (Mar 12 2019 at 23:15, on Zulip):

but since it takes a long time to bootstrap a rustc build, i tend to make separate clones of the repo, so that the work-in-progress on that project is kept along side its most recent build.

DPC (Mar 13 2019 at 04:30, on Zulip):

Addressed if to David but it's fine that you answered. Yeah makes sense if you build rustc in parallel

davidtwco (Mar 13 2019 at 07:12, on Zulip):

Exactly what @pnkfelix said.

Vadim Petrochenkov (Mar 13 2019 at 10:43, on Zulip):

Hm, I tried git worktrees but didn't find them useful.
First, multiple builds eat all the space on my SSD.
Second, I usually start any work with rebase on master so I have to build anyway, and the build from some rustc or std crate changed during rebase to test --stage 1 src/test/ui --test-args something is only few minutes.
After that I usually build multiple times anyway to test intermediate changes, so the initial build time is amortized.

Vadim Petrochenkov (Mar 13 2019 at 10:45, on Zulip):

So, branches all the way.

pnkfelix (Mar 26 2019 at 11:44, on Zulip):

Just curious, is there a reason why you are creating new projects instead of creating new branches every time? It might make things easier

one thing I will admit that is bad about creating new projects rather than creating branches in the same working directory: I'm pretty sure it makes ccache much less effective. (I am currently assuming there are absolute paths ending up in the intermediate build products for LLVM, which makes the ccache unable to be shared between projects. If this assumption is incorrect, though, then I do not know why I am observing such a low hit rate for my set up.)

davidtwco (May 24 2019 at 09:41, on Zulip):

For anyone who uses workman (cc @oli), I've just added experimental git-worktree support (it's off by default as there's no nice way to migrate to it) and a remove subcommand:

oli (May 24 2019 at 13:17, on Zulip):

perfect timing, I was just about to start setting up workman in a new environment

davidtwco (May 24 2019 at 13:35, on Zulip):

git-worktree is strange in that it doesn't allow a branch to be checked out in more than one working directory, so workman keeps the unassigned with a detached HEAD which should be updated with master. There might be a bug or two to work out around that.

davidtwco (May 24 2019 at 13:36, on Zulip):

Or rather, it isn't strange, it's just one place where it doesn't quite work how workman would like it to.

RalfJ (Jun 08 2019 at 20:42, on Zulip):

While I am for the third time in 2 days waiting for LLVM to compile, which takes forever... is there a way to share the compiled LLVM across working directories? Do you usually do that?

RalfJ (Jun 08 2019 at 20:42, on Zulip):

alternatively if I can make it not build 3 dozens backends but only x86, that might also save some time^^

oli (Jun 08 2019 at 20:43, on Zulip):

technically there is ccache (you can enable it in config.toml), but it doesn't seem to work across working dirs

simulacrum (Jun 08 2019 at 20:57, on Zulip):

theoretically you can copy/symlink LLVM directory but... no idea if that'll actually work

nagisa (Jun 09 2019 at 01:28, on Zulip):

you can easily change targets built for llvm

nagisa (Jun 09 2019 at 01:29, on Zulip):

but that doesn’t really reduce the llvm build time that much

Vadim Petrochenkov (Jun 09 2019 at 07:08, on Zulip):

ccache itself supports single cache for multiple build directories and multiple source directories (with some restrictions on their relative paths).
I didn't try to use it with LLVM specifically, but used it with some other projects.
Rustbuild is probably missing the necessary configuration though (CCACHE_HASHDIR/CCACHE_BASEDIR).

oli (Jun 09 2019 at 07:08, on Zulip):

@eddyb ^

simulacrum (Jun 09 2019 at 12:50, on Zulip):

sccache which we use on CI doesn't support BASEDIR, so that does seem likely. PRs welcome!

Last update: Aug 13 2020 at 09:15UTC