Stream: t-cargo/PubGrub

Topic: public/private


view this post on Zulip Matthieu Pizenberg (Aug 04 2021 at 08:34):

By the way, I just had a look at the issue for tracking public/private. What is the core idea/problem with "scoped goals" and "public dependencies"? (I know nothing about the RFC)

view this post on Zulip Matthieu Pizenberg (Aug 04 2021 at 09:53):

I just read the public/private RFC. So if I understand well, there are two goals:

  1. catch shared usage of a type at different versions at dependency resolution instead of at compilation
  2. possibly enable more situations for private dependencies (they can basically be any version they want)

(1) would require a constraint preventing from having two different versions of a crate connected via a chain of public edges (dependencies) in the dependency graph. One remark I have about this RFC is that it will also result in false positives. It could prevent a crate from compiling because in theory two versions of a public dependency are available, but in practice they are not used together.

(2) I suppose this could even enable situations where we have "a" at version 1.1 and 1.2 in the solution if they are both private dependencies? or is that stretching the concept of private too far? because how does that play with a public "p" within the private "a". I'm not sure I've read in the RFC what is the situations for public deps of private deps.

view this post on Zulip Eh2406 (Aug 04 2021 at 14:11):

That all sounds right. Your point about false positives is entirely correct. That is one of the reasons the RFC cannot be implemented as described. It will need some syntax in Cargo.toml for "I don't care about this conflict".

view this post on Zulip Eh2406 (Aug 04 2021 at 14:16):

have "a" at version 1.1 and 1.2 in the solution

That is going further then the RFC does. (Some think it is a better design, but it would be to big a braking change.)
The RFC only adds new conflicts/incompatibilities, it does not relax any existing conflicts/incompatibilities.
If the pre-RFC resolver thinks there are no solutions, then there are no solutions with the post-RFC resolver.

view this post on Zulip Matthieu Pizenberg (Aug 09 2021 at 08:43):

hum I thought that private/public would be relatively easy to implement but now that I'm thinking more about it, it does not seem easy at all ^^.

view this post on Zulip Matthieu Pizenberg (Aug 09 2021 at 08:45):

If public was always public, it would be easy I think, but public is only public within its subgraph of publicly connected nodes.

view this post on Zulip Matthieu Pizenberg (Aug 09 2021 at 08:52):

hum, but one advantage is that it's a directed graph, which goes in only one direction, and any private link cuts its current publicly connected subgraph. So each public subgraph can be uniquely identified by its private root, which can be identified by its package name and version. So maybe we can have a "private_root" field on the package type handled by the dependency provider.

view this post on Zulip Matthieu Pizenberg (Aug 09 2021 at 09:01):

I'll have a try writing a section in the guide about what I have in mind, see how that goes while I try to explain it clearly

view this post on Zulip Matthieu Pizenberg (Aug 09 2021 at 10:35):

New guide section started: https://github.com/pubgrub-rs/guide/pull/4

view this post on Zulip Eh2406 (Aug 09 2021 at 14:05):

It ends up being horrendously complicated. Remember that V1 of A may export B as public, but the V1.1 may have it private. For my own sanity, I will focus on getting parity with cargo features before doing any kind of unimplemented or future RFC's. But I certainly don't mind you spending time thinking about it. :-)

view this post on Zulip Matthieu Pizenberg (Aug 09 2021 at 15:06):

Alright, re-reading what I wrote in the guide, I realize I have an "off-by-one" error in my chaining of dependencies, but I have a clear path to solving it. I'm pretty confident it's gonna work, and not be more complex than the "multiple versions" scheme with buckets.

view this post on Zulip Matthieu Pizenberg (Aug 10 2021 at 00:11):

Actually, it was not off by one, it was ok, but not sufficient. Then I realized I need more than one marker. I basically need to add a seed marker to the public path everytime there exist another private path.

view this post on Zulip Matthieu Pizenberg (Aug 10 2021 at 00:12):

Sorry for the confusing note ^^. It's mainly for my future self. With a picture to help me remember correctly

view this post on Zulip Matthieu Pizenberg (Aug 10 2021 at 00:12):

IMG_20210810_020722.jpg

view this post on Zulip Eh2406 (Aug 10 2021 at 00:13):

I have made many such pictures in my day.

view this post on Zulip Eh2406 (Aug 10 2021 at 00:14):

The end-to-end prop tests where written because p/p is stuff is hard.

view this post on Zulip Matthieu Pizenberg (Aug 10 2021 at 00:15):

Ahah I can believe it ;)

view this post on Zulip Matthieu Pizenberg (Aug 10 2021 at 14:33):

Alright, I think I've nailed it, at least in theory since I haven't completed the implementation. You can read my thoughts here: https://github.com/pubgrub-rs/guide/blob/b7726517446c39f29d16be3d69ee819925cedc7a/src/limitations/public_private.md

view this post on Zulip Matthieu Pizenberg (Aug 10 2021 at 14:34):

The two missing images are the following

view this post on Zulip Matthieu Pizenberg (Aug 10 2021 at 14:34):

image.png

view this post on Zulip Matthieu Pizenberg (Aug 10 2021 at 14:34):

image.png

view this post on Zulip Matthieu Pizenberg (Aug 10 2021 at 18:21):

And I think the code in PR https://github.com/pubgrub-rs/advanced_dependency_providers/pull/9 is now correct. I'll have to add other edge cases tests to be sure, and to add documentation. And finally to finish the joint explanation in the guide.

view this post on Zulip Eh2406 (Aug 10 2021 at 20:43):

I will read it over! I will not be convinced that we have nailed it, until we have time prop-tests checking it. I thought I had this working several times, but a new attempt is very exciting.

view this post on Zulip Matthieu Pizenberg (Aug 11 2021 at 16:17):

I have a question regarding the solver/compiler interface. If there are two places with a dependency to "a" and the solver's solutions contains two different versions of "a", say V1 and V2. How does the compiler knows which one to use where ?

view this post on Zulip Eh2406 (Aug 11 2021 at 16:33):

For the details you will have to ask Alex or Eric. But the general idea is that package versions is a cargo concept. So A v1 gets built into a directory (or file) like like A-{a arbitrary hash value} and A v2 in A-{a different hash value}. Then Cargo can tell Rustc "the files related to what you know as A can be found at A-{the correct hash}". Rust also uses the hash to mangle the the symbols so that the linker also treats them as totally different things.

view this post on Zulip Matthieu Pizenberg (Aug 11 2021 at 17:00):

Ok, I was thinking of that because I'm realizing that we don't put any traceability in the solution. So if "a" depends on "b" 1.0..3.0 and "b" is in the solution at both versions 1 and 2, how does the compiler would know which version/directory to use?

view this post on Zulip Eh2406 (Aug 11 2021 at 19:02):

Whatever code lowered the problem to PubGrubs input will need to convert the output. So if it uses Proxies&Buckets and there are more then one compatible Buckets that match and are selected then it will need to look at the Proxies that represent that connection. It will need to walk the implied and unambiguous graph from PubGrubs output and shrink package->Proxies->Buckets->version into a package -> version graph. This may be tricky to get right, but it should be O(n).

view this post on Zulip Matthieu Pizenberg (Aug 11 2021 at 19:17):

Whatever code lowered the problem to PubGrubs input will need to convert the output

Seems reasonable!

view this post on Zulip Eh2406 (Aug 11 2021 at 19:27):

It would probably be good/completionist of us to add the code for that into the examples in advanced_dependency_providers. But that feels like extra credit.

view this post on Zulip Matthieu Pizenberg (Aug 11 2021 at 19:28):

"... will be left as an exercise to the reader" XD


Last updated: Oct 21 2021 at 20:33 UTC