We'd want a custom jobserver protocol which can share tokens equally between processes. We want the jobserver to be able to ask processes to give back tokens when new processes are spawned. We'd also want cargo to reserve tokens if it wants to spawn a number of rustc instances so it knows each can get one.
For Rayon, it would be helpful if we could spawn threads one at a time. So a thread spawns, it blocks on a jobserver token, once it gets it, it spawns the next thread and so on. Once rustc is done, we want to be able to exit without waiting for all the threads to spawn and then terminate.
One thing to note is that unlike regular Rayon, only threads which are not waiting on a jobserver token will wake up when there's more work. So if jobserver tokens are spread more equally between processes you won't see the behavior where Rayon wakes up all the threads. If there's just a few rustc processes you can still get issues with the jobserver tokens shuffling between processes. Hopefully those can be fixed by making sure there's more work to be done in parallel, adding more idling time before falling asleep and using fibers / stackful coroutines to avoid blocking on queries.