Stream: t-cargo/PubGrub

Topic: root package


view this post on Zulip MrGreenTea (Dec 03 2020 at 12:36):

Hi, thanks for the great work!
I am currently experimenting with the pubgrub-rs crate and hope to integrate it with pypi.org and especially with poetry.

Currently I am struggling with resolving dependencies for a package because I want to only look up packages that are not my root package on the remote.

For example let's say I have root at version 1.0.0 and that has not been released to pypi.org.
I don't want to look up root at pypi.org, but all other packages.
Because choose_package_version get's an impl Iterator I am currently struggling with that.
What I tried to do is show if (T, U) == ("root", _) and if so return ("root", 1.0.0) but if it's not root i need to provide the iterator to my remote dependency provider. Even then I can't really get it to work.

This is the current implementation that does not work.

fn choose_package_version<T: Borrow<String>, U: Borrow<Range<PEP440Version>>>(
        &self,
        potential_packages: impl Iterator<Item = (T, U)>,
    ) -> Result<(T, Option<PEP440Version>), Box<dyn Error>> {
        let vec = potential_packages.collect();
        match vec.iter().find(|(p, _)| p.borrow() == &self.root.package) {
            Some((p, _)) => {
                Ok(
                    (p, Some(self.root.version.clone()))
                )
            }
            None => self.remote.choose_package_version(vec.iter())
        }
    }

Would appreciate any and every help :)

view this post on Zulip Matthieu Pizenberg (Dec 03 2020 at 13:31):

Hi @MrGreenTea , welcome! If I get the problem right, you are interested in an example of dealing with dependencies of a project being worked on instead of dependencies of an already published package.
Your code logic seems fine, is the issue a compilation issue?
Do you have a repository we could clone to test things?

PS: I'm currently working on integrating pubgrub-rs into a dependency provider for elm projects so I'll have exactly the same problematic soon and will be able to share a more complete example use case. But that may take few weeks.

view this post on Zulip Matthieu Pizenberg (Dec 03 2020 at 13:35):

By the way, the landing page for poetry is magnificent :) and I very much appreciate improvements in managing python packages ^^

view this post on Zulip Matthieu Pizenberg (Dec 03 2020 at 13:55):

Just a heads up, looking at the dependency specifications of poetry, it seems like you have a lot of features for which pubgrub is not ready yet. Things like environment markers, multiple constraints, pre-releases are currently in an exploratory phase: https://github.com/pubgrub-rs/advanced_dependency_providers/issues

view this post on Zulip Eh2406 (Dec 03 2020 at 14:06):

To be clear we would love help figuring out how to get them to work, but we don't have an answer to point you to.

view this post on Zulip Eh2406 (Dec 03 2020 at 14:08):

for the root problem we can collect into a vector.

view this post on Zulip Matthieu Pizenberg (Dec 03 2020 at 14:11):

indeed, maybe doing vec: Vec<_> = potential_packages.collect(); will help. Then in the None case use vec.into_iter().

view this post on Zulip MrGreenTea (Dec 03 2020 at 15:40):

That's exactly right, I want to resolve packages of the project being worked on.

view this post on Zulip MrGreenTea (Dec 03 2020 at 15:41):

That pubgrub-rs might not support everything is totally okay, we're in the experimental phase too and kinda new to rust as well :)
I hope to contribute to these things soon if possible.

I will share the repository tomorrow with the current experiments.

view this post on Zulip MrGreenTea (Dec 03 2020 at 15:42):

I tried collecting into a vector but ran into borrow checker issues I could not resolve yet. I hoped perhaps someone here had a simple solution to this already. Thanks for the quick reply!

view this post on Zulip MrGreenTea (Dec 03 2020 at 15:43):

If you ever work in python you really should use poetry, it's a joy to work with and we use it for all our python projects. I'm not affiliated with them (yet ;) ) though, this is something I'm doing with a colleague for now.

view this post on Zulip Matthieu Pizenberg (Dec 03 2020 at 15:44):

I hoped perhaps someone here had a simple solution to this already. Thanks for the quick reply!

I'll try to prioritize the things I have to do in that elm dependency provider in order to get as soon as possible to an example related to your use case now that I now that, but might still take some time ^^

view this post on Zulip Eh2406 (Dec 03 2020 at 15:54):

When the code can be shared, I will be happy to look at the borrow checker issues. They are particularly narly to diagnose without looking at the code.

view this post on Zulip MrGreenTea (Dec 04 2020 at 20:43):

fn choose_package_version<T: Borrow<String>, U: Borrow<Range<PEP440Version>>>(
        &self,
        potential_packages: impl Iterator<Item = (T, U)>,
    ) -> Result<(T, Option<PEP440Version>), Box<dyn Error>> {
        let vec: Vec<(T, U)> = potential_packages.collect();
        match vec.iter().find(|(p, _)| p.borrow() == &self.root.package) {
            Some((p, _)) => {
                Ok(
                    (p, Some(self.root.version.clone()))
                )
            }
            None => self.remote.choose_package_version(vec.into_iter())
        }
    }

This is the part I am struggling with.
I am actually unsure if it's the borrow checker or something else I am clueless about here.
In the Some((p, _)) case p: &T but I need T and can't move it.
This is the error I get:

error[E0308]: mismatched types
  --> src/poetry_provider.rs:41:22
   |
33 |     fn choose_package_version<T: Borrow<String>, U: Borrow<Range<PEP440Version>>>(
   |                               - this type parameter
...
41 |                     (p, Some(self.root.version.clone()))
   |                      ^ expected type parameter `T`, found `&T`
   |
   = note: expected type parameter `T`
                   found reference `&T`

view this post on Zulip MrGreenTea (Dec 04 2020 at 20:44):

I don't feel comfortable making the repo public yet, but if you send me your github username I'll add you if you are interested to see the current experiments ;)

view this post on Zulip Eh2406 (Dec 04 2020 at 20:46):

I am Eh2406 on github, and I am very interested!

view this post on Zulip Eh2406 (Dec 04 2020 at 20:47):

I would like to try to get things to work with the api we have, but it may be that for 0.3 the api may need to be more flexible.

view this post on Zulip MrGreenTea (Dec 04 2020 at 20:50):

I've sent you an invite

view this post on Zulip Eh2406 (Dec 04 2020 at 21:12):

I am getting Error: "Python 3.x interpreter not found" I think I need to set a PYO3_PYTHON but I don't know to what.

view this post on Zulip MrGreenTea (Dec 04 2020 at 21:14):

Hm weird, I did not have to do that. I assume set it to the path of your python3 binary?

view this post on Zulip Eh2406 (Dec 04 2020 at 21:18):

With some tinkering I got it to work. That is what I get for having multiple pythons installed.

view this post on Zulip Eh2406 (Dec 04 2020 at 21:24):

https://doc.rust-lang.org/std/iter/trait.Iterator.html#method.partition
may be a way to get this to work.

view this post on Zulip MrGreenTea (Dec 04 2020 at 21:32):

He cool yeah I got it working with that:

let (root, other): (Vec<_>, Vec<_>) = potential_packages.partition(|(p, _)| p.borrow() == &self.root.package);
match root.into_iter().next() {
    Some((p, _)) => Ok((p, Some(self.root.version.clone()))),
    None => self.remote.choose_package_version(other.into_iter())
}

view this post on Zulip Eh2406 (Dec 04 2020 at 21:34):

If the allocations are in the profile, we can optimize with filter map. But we got something working.

view this post on Zulip Eh2406 (Dec 04 2020 at 21:45):

        let mut root = None;
        let other = self
            .remote
            .choose_package_version(potential_packages.filter_map(|(t, u)| {
                if t.borrow() == &self.root.package {
                    root = Some((t, Some(self.root.version.clone())));
                    None
                } else {
                    Some((t, u))
                }
            }));
        if let Some(root) = root {
            Ok(root)
        } else {
            other
        }

view this post on Zulip Eh2406 (Dec 04 2020 at 21:54):

But let's stick with the 5 line version until we need to make things more complicated.

view this post on Zulip MrGreenTea (Dec 04 2020 at 22:35):

I agree, it works for now on the rust side, thanks a lot!
I'll have to do some work on the python side now to integrate the solution correctly.

view this post on Zulip MrGreenTea (Dec 04 2020 at 22:40):

I think we'll rewrite this entirely after we're over this initial exploratory phase.

view this post on Zulip Eh2406 (Dec 05 2020 at 01:02):

The way I am reading the code, this may also be a good place to try and work on https://github.com/pubgrub-rs/advanced_dependency_providers/issues/6 when we have full examples that can be tested.

view this post on Zulip MrGreenTea (Dec 05 2020 at 09:04):

What place exactly do you think would benefit from async? The reqwest calls?

view this post on Zulip Eh2406 (Dec 05 2020 at 13:36):

Yes, in theory it can be requesting meny deps in parallel.

view this post on Zulip Matthieu Pizenberg (Dec 05 2020 at 13:49):

I haven't read @MrGreenTea code, but from pubgrub point of view, only choose_package_version has an opportunity of parallelism/concurency. The get_dependencies method will only be called one at a time when evaluating a chosen package. So depending on the registry architecture and API, it may have an opportunity for parallelism or not in choose_package_version. For example in Elm, there is no API to ask for existing versions of only one package. We can only ask for all existing versions (without their dependencies) or all recent versions (again without their dependencies). So only one network request is made in choose_package_version (actually 0 request since its better to do that request once before solving) and there is no opportunity to do parallel things with elm packages.

view this post on Zulip Eh2406 (Dec 05 2020 at 13:59):

You are correct. I ment requesting the versions of many packages in parallel.


Last updated: Oct 21 2021 at 21:32 UTC