Stream: t-cargo/PubGrub

Topic: random


view this post on Zulip Matthieu Pizenberg (Nov 23 2020 at 11:10):

image.png

Is it normal if I feel like I need to improve the package manager of my distrib? (just kidding)

view this post on Zulip Alex Tokarev (Nov 23 2020 at 11:14):

I use Arch btw :sunglasses:

view this post on Zulip Matthieu Pizenberg (Nov 27 2020 at 16:01):

Hey I'm trying to make some rust code compile for my work, but can't figure out how to deal with the lifetimes. I've done a minimal playground example with the same computation flow: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=abc1720627025aea6245fccebdb90a9c

view this post on Zulip Matthieu Pizenberg (Nov 27 2020 at 16:02):

In case you have time to have a look, any idea on how to fix that lifetime issue?

view this post on Zulip Matthieu Pizenberg (Nov 27 2020 at 16:27):

Alternative version with generics instead of box, but does not compile either ^^: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=2ab0302c665221fb907d6f0f000c4041

view this post on Zulip Eh2406 (Nov 27 2020 at 16:30):

Trying to figure out what is important and what is the example.
You want F to borrow from self and be stored in self?

view this post on Zulip Matthieu Pizenberg (Nov 27 2020 at 16:34):

full is a utility function, which depends on a many things. Inside the update I'm only interested by the partial signature. Ideally I'd like it to be a closure depending on values inside the State. So yes, in theory F should not escape self

view this post on Zulip Matthieu Pizenberg (Nov 27 2020 at 16:36):

The thing is that the update modifies self and thus I'd need to update the partial closure to reflect that

view this post on Zulip Eh2406 (Nov 27 2020 at 16:36):

I don't think this can work as there is nothing stopping you from adding this method

    fn invalidate_pointer(&mut self) {
        self.data = "diffrent".to_string();
        // Note how self.partial is still using a
        // &str that pintes into freead memory
       self.partial(1)
    }

view this post on Zulip Matthieu Pizenberg (Nov 27 2020 at 16:37):

hum

view this post on Zulip Eh2406 (Nov 27 2020 at 16:39):

which workarounds make sense depends on why you want this structure.

view this post on Zulip Matthieu Pizenberg (Nov 27 2020 at 16:40):

makes sense

view this post on Zulip Matthieu Pizenberg (Nov 27 2020 at 16:40):

I want to be able to switch different full functions with different parameter types

view this post on Zulip Matthieu Pizenberg (Nov 27 2020 at 16:41):

And inside the update only use the usize -> String part of the signature

view this post on Zulip Matthieu Pizenberg (Nov 27 2020 at 16:42):

I also need that partial function to be updated since it depends on values changing in the update usually.

view this post on Zulip Eh2406 (Nov 27 2020 at 16:43):

One solution is to use a RC<String> So it won't be freed until both have changed. If the ref counting and cache miss is not perf critical.

view this post on Zulip Matthieu Pizenberg (Nov 27 2020 at 16:44):

I'm making an example with String but basically those are images so big objects and I don't think Rc would matter

view this post on Zulip Eh2406 (Nov 27 2020 at 16:45):

Or have data be a &'a ___ so it can guaranteed to live langer then the struct.

view this post on Zulip Matthieu Pizenberg (Nov 27 2020 at 16:46):

That may not be possible since some of that data is computed within the update loop, thus owned by the State

view this post on Zulip Matthieu Pizenberg (Nov 27 2020 at 16:47):

One solution is to use a RC<String>

Ok, will try

view this post on Zulip Eh2406 (Nov 27 2020 at 16:47):

It took me awhile to realize that all my python vars have tipe Rc<RefCell<T>> and it never bothered me then. So if it is not per critical and it make the code ezere to read, an RCis not bad.

view this post on Zulip Matthieu Pizenberg (Nov 27 2020 at 16:50):

Does not seem to make the trick :( https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=8b36ef03b7e5dd44144803b7c9103ff0
or maybe I've still something wrong

view this post on Zulip Eh2406 (Nov 27 2020 at 16:53):

https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=23295052051c77a3181a7da9bf70bb20

view this post on Zulip Matthieu Pizenberg (Nov 27 2020 at 16:56):

Ah thank you! So I basically need to Rc all the parameters of my full function and use move for the closure

view this post on Zulip Eh2406 (Nov 27 2020 at 16:59):

Yes, you want the closure to own the variables it references (so move) but share the ownership with the struct (so Rc).

view this post on Zulip Eh2406 (Nov 27 2020 at 17:01):

and the bad method from before becomes:

    fn invalidate_pointer(&mut self) {
        self.data = Rc::new("diffrent".to_string());
        // Note how self.partial has not been updated so is still using
       // the old string, which may be a bug, but it is not UB.
       self.partial(1)
    }

view this post on Zulip Matthieu Pizenberg (Dec 05 2020 at 22:54):

I've made some progress with the dependency provider for elm but I'm stuck with a compilation error :( https://github.com/mpizenberg/elm_dependency_provider/blob/master/examples/solve_pkg_deps.rs#L19

With the offline provider it works smoothly, but with the online one resolve says it does not implement DependencyProvider, I'm not figuring it out

view this post on Zulip Matthieu Pizenberg (Dec 05 2020 at 22:55):

If one of you have some time checking it out, let me know if you find the issue

view this post on Zulip Eh2406 (Dec 06 2020 at 00:20):

ElmPackageProviderOnline::new returns a result.

view this post on Zulip Matthieu Pizenberg (Dec 10 2020 at 17:28):

I'm having a weird compilation issue, related to musl. If I compile this binary https://github.com/mpizenberg/elm-test-rs with cargo build --release --target x86_64-unknown-linux-musl Cargo tells me this:

  running: "musl-g++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "src" "-Wall" "-Wextra" "-o" "/home/matthieu/git/mpizenberg/elm-test-rs_musl/target/x86_64-unknown-linux-musl/release/build/tree-sitter-elm-9d31565a479bf4e7/out/src/scanner.o" "-c" "src/scanner.cc"

  --- stderr

  error occurred: Failed to find tool. Is `musl-g++` installed?

view this post on Zulip Matthieu Pizenberg (Dec 10 2020 at 17:30):

And it is right that musl-g++ does not exist, even though I've installed community/musl and community/kernel-headers-musl packages on my archlinux system. They don't seem to contain a musl-g++ binary, only musl-gcc.

view this post on Zulip Eh2406 (Dec 10 2020 at 17:40):

Sorry don't know about musl

view this post on Zulip Matthieu Pizenberg (Jan 07 2021 at 08:21):

This might be relevant? (don't know I haven't understood in details what this is) https://github.com/rust-lang/cargo/issues/8088 "Features 2.0 meta tracking issue"

view this post on Zulip Eh2406 (Jan 07 2021 at 15:03):

It is really cool and good work. It adds a second pass over the resolution graph ( after the resolver ) that reduces featuresand optional dependencies that are not needed for the exact build that is happening. Pubgrub does not need to handle this step itself in order to be useful in Cargo. There will always need to be a separate pass. For example the lock file needs to know if your dependencies are incompatible on windows even if you are currently building on linux. But it may be an interesting stretch goal to try and incorporate it into a Pubgrub-based resolver, so that it can allow conflicting versions as long as there is no system that will require them both.

view this post on Zulip Matthieu Pizenberg (Feb 04 2021 at 13:23):

I'm currently working on some web stuff for work and I'm wondering how I could retrieve the stderr output of some rust wasm and put it in a dedicated panel of the web gui. This seems rather not possible. But I've seen the log crate that could be somewhat useful I guess. And it seems like something that could be useful to pugrub too (the logging). Have you used the crate already? is it good/bad practice?

view this post on Zulip Eh2406 (Feb 04 2021 at 13:58):

I have used the log crate in general. It is widely used and very nice.
I think there are adapters to use it in wasm, but I don't recall trying them.

view this post on Zulip Matthieu Pizenberg (Feb 15 2021 at 19:05):

I've tried the log crate and like it! thanks for confirming it's wide adoption!

view this post on Zulip Matthieu Pizenberg (Feb 15 2021 at 19:06):

Also sorry I've been underwater with finishing version 1.0.0 of elm-test-rs (just released 1.0.0-beta this weekend) + some things I needed to do for the exercism project, so couldn't find time to review your arena PR @Eh2406

view this post on Zulip Eh2406 (Feb 15 2021 at 19:12):

No rush! I am working on simplifying the code around find_satisfier_and_previous_satisfier_levelnext. Mostly so I can understand it better.

view this post on Zulip Matthieu Pizenberg (Feb 15 2021 at 19:16):

I think I explained a bit the backtracking logic in that part of the guide: https://pubgrub-rs-guide.netlify.app/internals/conflict_resolution.html Don't hesitate to have a look at it while working on that

view this post on Zulip Eh2406 (Feb 15 2021 at 19:20):

I will definitely reread that and pubgrub#57 when I have simplified the code.

view this post on Zulip Eh2406 (Feb 15 2021 at 19:23):

So far I am not making big structure/algorithmic changes. Just some return type and order of operation changes.

view this post on Zulip Matthieu Pizenberg (Feb 15 2021 at 19:29):

By the way, one way of improving this in complexity would be to turn that search into a O(M) instead of O(N) where N is the total number of assignment while M is the number of assign related to that conflict (with package terms appearing in the conflict). That could be done by changing the data structure used to search those satisfiers. But it would be quite a big architectural change, and would require using a hashmap of some sort that would have impact to other things in the profile of the algorithm as a whole. But I wanted to mention that since it would be linked to a different approach for storing the partial solution.

view this post on Zulip Eh2406 (Feb 15 2021 at 19:31):

Yes, indeed! That is one of the things I was thinking of heading towards, not this PR but over the next few PRs.

view this post on Zulip Matthieu Pizenberg (Feb 26 2021 at 07:15):

Hey, I finally fixed all the hairy stuff and weird windows bug I wanted to fix for elm-test-rs. I didn't thought I'd encounter so many problems last weekend. So I'll have a look at the arena PR tonight. I agree having to rebase on long architecture changes would be annoying.

view this post on Zulip Matthieu Pizenberg (Apr 15 2021 at 14:03):

Small random note. Today I needed to load an image in wasm but I was getting stuck with an error message saying rayon cannot be used, even though I had only added the feature jpeg and not jpeg/rayon in my dependencies

view this post on Zulip Matthieu Pizenberg (Apr 15 2021 at 14:05):

Turns out, this was because my lowrr-wasm crate was in the same workspace that my lowrr-lib crate, which depends on the image crate with the default features. So the resolver compiled my wasm lib also with default features.

view this post on Zulip Matthieu Pizenberg (Jun 10 2021 at 21:45):

Today I saw a new release of anyhow due to a dependency problem (cf issue https://github.com/dtolnay/anyhow/pull/155). I hope once PubGrub is advanced enough, solving dependencies with minimal versions will be easy and efficient enough that a simple option could be added to cargo test like --minimal-versions so that people add this to their CI.

view this post on Zulip Eh2406 (Jun 11 2021 at 13:44):

It exists. https://doc.rust-lang.org/cargo/reference/unstable.html#minimal-versions . It is a global check, it will also fail if any transitive dep is not right. Meany projects don't think it is a problem. Anything that depend on them cant use it. So we are going to need a more local tool.

Also sorry for disappearing this week. My benchmarking computer is having an issue, and I got distracted. I will come back.

view this post on Zulip Matthieu Pizenberg (Jun 27 2021 at 18:09):

Hey, just to let you know I might not answer on github during approximately 3 weeks, except maybe for the 0.2.1 release and to write a little post about it on reddit. Otherwise, I'll occasionally read but not do any concrete progress for the upcoming month, roughly.

view this post on Zulip Matthieu Pizenberg (Jun 27 2021 at 18:11):

Nothing important, it's just that I'll be on holiday starting 1st of July and won't be on my computer much.

view this post on Zulip Eh2406 (Jun 27 2021 at 18:12):

Enjoy your holiday!

view this post on Zulip Eh2406 (Jun 27 2021 at 18:16):

I am giving up on a replacement fan showing up, and trying to figure out how to make the gpu fans (witch are right there) run all the time.

view this post on Zulip Eh2406 (Jun 27 2021 at 18:18):

Hopefully, I can get back to "full cratis.io generation" timings for various branches.

view this post on Zulip Matthieu Pizenberg (Jun 27 2021 at 18:20):

view this post on Zulip Matthieu Pizenberg (Jun 27 2021 at 18:27):

I was looking for this complete meme when you mentioned repairing your computer but unfortunately this one is very low res and not complete :(. Here is what I meant to post :) https://www.youtube.com/watch?v=AbSehcT19u0

view this post on Zulip Eh2406 (Jun 27 2021 at 18:52):

So last time I ran it, it took ~24h. I am now trying with 0.2.1, I want to see how the new partial_solution changes things.
Then I want to try the priority_queue and some other branches.

view this post on Zulip Eh2406 (Jun 28 2021 at 14:28):

It took 20h, a ~16% improvement! All the more impressive because our slowest crates are releasing new versions eavry few days.

view this post on Zulip Matthieu Pizenberg (Jun 29 2021 at 10:25):

Nice!

view this post on Zulip Eh2406 (Jun 29 2021 at 16:32):

Using https://github.com/pubgrub-rs/pubgrub/commit/5802cca3ba5a354a4540d83d33508609af6f14e8 on it's own made things slow down to 22.5h. I don't know if that is mostly because:

view this post on Zulip Eh2406 (Jun 30 2021 at 01:57):

Using the priority_queue branch (this to be practice) it ran in ... way to fast ... possible explanations:

view this post on Zulip Josh Triplett (Jun 30 2021 at 04:01):

@Eh2406 For the purposes of iterating on optimization, would it make sense to grab a snapshot of a subset of the index, something that takes around 20-30 minutes at most, and use that?

view this post on Zulip Matthieu Pizenberg (Jun 30 2021 at 14:29):

Eh2406 said:

It took 20h, a ~16% improvement! All the more impressive because our slowest crates are releasing new versions every few days.

I'd really love to see a histogram of the timings distribution if that's something doable (maybe measure total time for a given crate, divided by the number of versions to normalize by solver call) both with 0.2.1 and with the priority_queue branch.
If you make a csv, we can visualize it easily with data voyager for example: https://vega.github.io/voyager/

view this post on Zulip Eh2406 (Jun 30 2021 at 14:32):

I was just going to say, the next step is to re-run with the stats being sent to a csv then we can analyze however we want to.

view this post on Zulip Eh2406 (Jun 30 2021 at 14:34):

One stat to collect (that I have not so far) is the type of the output from resolve, I.E. did it find a solution. If the two versions find a solution for the same crate/vers, then I will feal better about "I there's a bug and the new version, it is getting the wrong answer fast"

view this post on Zulip Eh2406 (Jun 30 2021 at 14:38):

I may spend some time on "There was something very slow I was doing in the old code". if there is and I can do the 0.2.1 experiment in say 10h, then collecting the stats will be a lot more pleasant.

view this post on Zulip Matthieu Pizenberg (Jun 30 2021 at 21:13):

I've release 0.2.1 and documented what I did in that new PR: https://github.com/pubgrub-rs/pubgrub/pull/100/files

view this post on Zulip Matthieu Pizenberg (Jun 30 2021 at 21:15):

I'll write a small post on reddit, and maybe should update the release notes on github with the summary in the changelog

view this post on Zulip Eh2406 (Jun 30 2021 at 21:16):

Yay, thank you!

view this post on Zulip Matthieu Pizenberg (Jun 30 2021 at 21:30):

Here we go :) https://www.reddit.com/r/rust/comments/ob721q/announcing_version_021_of_pubgrub_a_dependency/

view this post on Zulip Eh2406 (Jun 30 2021 at 21:33):

Nicely written!

view this post on Zulip Eh2406 (Jul 02 2021 at 00:19):

Here is the results. out_que.zip
No attempt to speed things up, just ran it and collected the output.

view this post on Zulip Eh2406 (Jul 02 2021 at 02:20):

scatter.png
Less correlation in time then I was expecting.

view this post on Zulip Eh2406 (Jul 02 2021 at 02:21):

image.png
But there are some times where they get a different outcome.

view this post on Zulip Eh2406 (Jul 02 2021 at 02:26):

The problematic cases:
name ver time_que_s succeeded_que time succeeded
spongedown 0.4.0 0.01 true 0 false
spongedown 0.4.1 0.01 true 0 false
spongedown 0.4.2 0.01 true 0 false
oni_simulator 0.1.0 0 false 0 true
tinki 0.1.0 0.03 true 0.01 false
slotmap 0.2.1 0 false 0.01 true
slotmap 0.4.0 0 false 0.01 true
slotmap 1.0.3 0 false 0.01 true
oni_simulator 0.1.2 0 false 0.01 true
slotmap 1.0.1 0 false 0.02 true
slotmap 1.0.2 0 false 0.02 true

view this post on Zulip Eh2406 (Jul 02 2021 at 14:11):

To check if the differences are new versions being published or yanked, I need to change the code to look at a unchanging snapshot of the index.

view this post on Zulip Matthieu Pizenberg (Jul 04 2021 at 00:19):

Eh2406 said:

Here is the results. out_que.zip
No attempt to speed things up, just ran it and collected the output.

Awesome, It's weird indeed that there are differences

view this post on Zulip Matthieu Pizenberg (Jul 04 2021 at 00:20):

Eh2406 said:

scatter.png
Less correlation in time then I was expecting.

Do we see things better for all the small timings if we use a logarithmic scale on both axes?

view this post on Zulip Eh2406 (Jul 04 2021 at 00:29):

out.zip
Here is the joined data. Dig in as you wish!

view this post on Zulip Eh2406 (Jul 04 2021 at 00:31):

I did find a way to speed up the 0.2.1 based run from 22 -> 8.6

view this post on Zulip Eh2406 (Jul 04 2021 at 00:35):

Code before:

    fn choose_package_version<
        T: std::borrow::Borrow<Intern<Names>>,
        U: std::borrow::Borrow<Range<Semantic>>,
    >(
        &self,
        potential_packages: impl Iterator<Item = (T, U)>,
    ) -> Result<(T, Option<Semantic>), Box<dyn Error>> {
        Ok(choose_package_with_fewest_versions(
            |p| match &**p {
                Names::Links(name) => Either::Left(
                    (0..self.links.borrow().get(name).cloned().unwrap_or(0))
                        .map(|i| Semantic::new(&format!("{}.0.0", i)).unwrap()),
                ),
                Names::Wide(_, req, _, _) | Names::WideFeatures(_, req, _, _, _) => {
                    Either::Right(Either::Left(
                        // one version for each bucket that match req
                        self.get_versions(p.crate_())
                            .into_iter()
                            .filter(|v| req.matches(&v.semver))
                            .map(|v| SemverCompatibility::from(v))
                            .collect::<std::collections::BTreeSet<_>>()
                            .into_iter()
                            .rev()
                            .map(|v| Semantic::from(&v)),
                    ))
                }
                _ => Either::Right(Either::Right(
                    self.get_versions(p.crate_()).into_iter().cloned(),
                )),
            },
            potential_packages,
        ))
    }

code affter:

    fn choose_package_version<
        T: std::borrow::Borrow<Intern<Names>>,
        U: std::borrow::Borrow<Range<Semantic>>,
    >(
        &self,
        potential_packages: impl Iterator<Item = (T, U)>,
    ) -> Result<(T, Option<Semantic>), Box<dyn Error>> {
        let out = {
            let (pkg, range) = potential_packages
                .min_by_key(|(p, range): &(T, U)| {
                    match &**p.borrow() {
                        Names::Links(name) => {
                            (0..self.links.borrow().get(name).cloned().unwrap_or(0))
                                .map(|i| Semantic::new(&format!("{}.0.0", i)).unwrap())
                                .filter(|v| range.borrow().contains(v))
                                .count()
                        }

                        Names::Wide(_, req, _, _) | Names::WideFeatures(_, req, _, _, _) => {
                            // one version for each bucket that match req
                            self.get_versions(p.borrow().crate_())
                                .into_iter()
                                .filter(|v| req.matches(&v.semver))
                                .map(|v| SemverCompatibility::from(v))
                                .collect::<std::collections::BTreeSet<_>>()
                                .into_iter()
                                .map(|v| Semantic::from(&v))
                                .filter(|v| range.borrow().contains(v))
                                .count()
                        }
                        _ => self
                            .get_versions(p.borrow().crate_())
                            .into_iter()
                            .filter(|v| range.borrow().contains(v))
                            .count(),
                    }
                })
                .expect("potential_packages gave us an empty iterator");
            let version = match &**pkg.borrow() {
                Names::Links(name) => (0..self.links.borrow().get(name).cloned().unwrap_or(0))
                    .map(|i| Semantic::new(&format!("{}.0.0", i)).unwrap())
                    .find(|v| range.borrow().contains(v)),

                Names::Wide(_, req, _, _) | Names::WideFeatures(_, req, _, _, _) => {
                    // one version for each bucket that match req
                    self.get_versions(pkg.borrow().crate_())
                        .into_iter()
                        .filter(|v| req.matches(&v.semver))
                        .map(|v| SemverCompatibility::from(v))
                        .collect::<std::collections::BTreeSet<_>>()
                        .into_iter()
                        .rev()
                        .map(|v| Semantic::from(&v))
                        .find(|v| range.borrow().contains(v))
                }
                _ => self
                    .get_versions(pkg.borrow().crate_())
                    .into_iter()
                    .find(|v| range.borrow().contains(v))
                    .cloned(),
            };
            (pkg, version)
        };
        Ok(out)
    }

Not enterally shuoor why.

view this post on Zulip Eh2406 (Jul 04 2021 at 14:53):

So looking at that code, there are a lot of ways to make things faster:

This will make both versions a lot faster.

view this post on Zulip Eh2406 (Jul 04 2021 at 15:01):

So we are now at 21,648s for 0.2.1 and 4,542s for Que. (yes that is 1 hour 16 minutes)

view this post on Zulip Eh2406 (Jul 04 2021 at 15:08):

out.zip the joined data, comfortingly they always get the same pass/fail!

view this post on Zulip Matthieu Pizenberg (Jul 05 2021 at 00:21):

Eh2406 said:

So we are now at 21,648s for 0.2.1 and 4,542s for Que. (yes that is 1 hour 16 minutes)

wow! such a long time just for one package!

view this post on Zulip Eh2406 (Jul 05 2021 at 00:34):

Sorry, no, that's all versions of all packages.

view this post on Zulip Eh2406 (Jul 05 2021 at 00:35):

One is based on the 0.2.1 branch of Pub grub. The other one is the priority queue branch.

view this post on Zulip Matthieu Pizenberg (Jul 18 2021 at 17:31):

@Eh2406 I'd love if we could make a call some day in the coming weeks when you are free. So that we can talk about our recent experiments (intervals, priority queues, pre-release for cargo etc.). Let me know if that's ok with you and when you can be available. This may have to be on a weekend to accommodate with the time difference. Personally, I'm fairly free until mid august I think

view this post on Zulip Eh2406 (Jul 18 2021 at 17:37):

I would love to!
May not be what you meant, but I am available for the next few hours.
Or we can try to find a time slot that works for both of us!

view this post on Zulip Eh2406 (Jul 18 2021 at 18:18):

@Matthieu Pizenberg Looking at a calendar, my weekends are surprisingly booked for the next 5 weeks. But my work schedule is pretty flexible. I may be able to take an hour or shift things around, for a time that is after work for you.

view this post on Zulip Matthieu Pizenberg (Jul 18 2021 at 18:43):

Can't tonight ^^. An evening for me on a day where you can shift things could work too.

view this post on Zulip Matthieu Pizenberg (Jul 18 2021 at 18:45):

But if that's a weekend after the summer that's ok for me too. I'm in no hurry and still have plenty of documentation to do meanwhile

view this post on Zulip Eh2406 (Jul 18 2021 at 18:47):

BTW thank you for the documentation! It is very good. I understand this stuff better having read your documentation, even when I came up with it and have implemented it!

view this post on Zulip Matthieu Pizenberg (Jul 18 2021 at 19:02):

I'm glad you like it :smiling_face:

view this post on Zulip Eh2406 (Aug 02 2021 at 17:40):

We should publicize the new guide, and not weigh for 0.2.2. It is too good to not be talking about it!

view this post on Zulip Matthieu Pizenberg (Aug 04 2021 at 10:47):

I hesitated between a public announcement and just telling it to people concerned by those limitations in github issues. I did the latter because I thought new users will not be aware of the old guide without the new section anyway. But if you think it's worth an announcement, I could prepare something and share it here for review before public share

view this post on Zulip Matthieu Pizenberg (Aug 04 2021 at 12:37):

draft:

New section in pugrub guide (dependency solver) about advanced usage and limitations

PubGrub is an algorithm to solve dependency requirements, and pubgrub-rs/pubgrub is a Rust implementation of that algorithm.

In November 2020, we released version 0.2 of pubgrub with sound foundations and an API enabling solving dependencies of simple systems. This enabled for example solving package dependencies of the Elm ecosystem for the alternative tests runner elm-test-rs.

Since then, we focused on performances and released in June this year (2021) version 0.2.1 with speed improvements ranging from 3x to 8x faster resolutions.

During all that time, many people have tried pubgrub and reached the limitations imposed by its current design. Some of these limitations can be circumvented, like enabling optional dependencies, or being able to pick multiple versions of the same package. Some however, run deep within the knowledge mechanisms of the algorithm and cannot be overcome with the current design. This is the case of pre-releases when those are coupled with "impure" requirements. By that we mean requirements that do not follow the mathematical properties of the order of elements in the corresponding sets. For example saying that "1.0 <= v < 3.0" should not contain version 2.0-alpha is problematic.

In order to help pubgrub users, including ourselves, we thus wrote a new section in the pubgrub guide to address this very topic of advanced usage and limitations. We hope that it explains clearly the challenges faced, and that it will help us develop the new APIs to overcome those in future versions of pubgrub.

view this post on Zulip Eh2406 (Aug 04 2021 at 13:36):

I like that! The guide is amazing, the more it gets talked up the better!

view this post on Zulip Matthieu Pizenberg (Aug 04 2021 at 14:48):

Here we go: https://www.reddit.com/r/rust/comments/oxu88c/new_section_in_pugrub_guide_dependency_solver/

view this post on Zulip Matthieu Pizenberg (Aug 05 2021 at 12:42):

I don't know if this is legit, but if it is, there really is conference for everything ^^ http://packaging-con.org/

view this post on Zulip Eh2406 (Aug 05 2021 at 12:46):

It does seem interesting, I signed up for their email list.

view this post on Zulip Eh2406 (Aug 05 2021 at 12:55):

Do we want to submit a talk?

view this post on Zulip Matthieu Pizenberg (Aug 06 2021 at 17:44):

could but what would it be? would be nice to have a demo of cargo using pubgrub, but that's a bit far off

view this post on Zulip Eh2406 (Aug 06 2021 at 18:37):

I think the work you have done on PubGrub as a reusable library is prite cool.

view this post on Zulip Eh2406 (Aug 08 2021 at 01:29):

Thanks for the good work you did! I'm on vacation for the next week, so I will only have limited time to review changes and no time to test perf work. Sorry.

view this post on Zulip Matthieu Pizenberg (Aug 08 2021 at 08:10):

No worry, enjoy your holiday! we can discuss all that when you're back

view this post on Zulip Eh2406 (Aug 23 2021 at 01:00):

I hit a small snag with our idea for comparing the logs of the priority queue branch. After running for about 2 hours the log file filled up my ssd.

view this post on Zulip Matthieu Pizenberg (Aug 26 2021 at 13:22):

Seems like this will need a more custom approach

view this post on Zulip Eh2406 (Aug 26 2021 at 14:02):

As a start I recorded for each package if it Err or Ok and if it Ok how big was the solution. I have not had time to look at the data.

view this post on Zulip Eh2406 (Aug 26 2021 at 18:43):

I took a quick look. The size of the solutions are the same.

view this post on Zulip Eh2406 (Oct 01 2021 at 15:07):

Just saw your presentation! That is so cool!
I have bean into Medieval History for a long time. I made a chain maille hauberk. I got a facsimile of the Bayeux as a 16th birthday gift.
It is so cool that you got to do original research on it!

view this post on Zulip Matthieu Pizenberg (Oct 02 2021 at 01:03):

Thanks! Yes it's a very interesting project and the discussions afterwards were also really nice but wasn't recorded. I hope we can continue with those CV meetups once a month

view this post on Zulip Matthieu Pizenberg (Oct 02 2021 at 01:06):

Now I need to see you in medieval wears!


Last updated: Oct 21 2021 at 21:46 UTC