Stream: rustdoc

Topic: Binary search format


view this post on Zulip Joshua Nelson (Jan 22 2021 at 18:46):

So I was looking at perf for https://github.com/microsoft/windows-rs/issues/420#issuecomment-765549226 and one of the issues is the search index is almost 60 MB (!!). There's an issue open for fix it (for almost 5 years now): https://github.com/rust-lang/rust/issues/31387. Some of the suggestions were:

I think "use a binary format" is the simplest, and should also have a big effect on the file size and load time. @GuillaumeGomez what do you think? Do you have suggestions for a format?

view this post on Zulip GuillaumeGomez (Jan 22 2021 at 18:58):

Your guess is as good as mine. However, please keep in mind that we need to be able to manipulate it easily to add or update a crate in the search index.

view this post on Zulip Dániel Buga (Jan 22 2021 at 19:05):

Would a compressed json (using gz or something) be terrible?

view this post on Zulip Joshua Nelson (Jan 22 2021 at 19:06):

hmm, I'm not sure if that works without a web server

view this post on Zulip Joshua Nelson (Jan 22 2021 at 19:06):

doing all this locally makes it hard :/

view this post on Zulip Dániel Buga (Jan 22 2021 at 19:07):

I'm not sure if it's any more complicated than with a server that serves a .gz, but I'm not sure if js implementations of any inflate algo would be fast enough.

view this post on Zulip Dániel Buga (Jan 22 2021 at 19:08):

anyway, probably a bad idea :smile:

view this post on Zulip Joshua Nelson (Jan 22 2021 at 19:08):

I would be interested in seeing numbers for gzip

view this post on Zulip Joshua Nelson (Jan 22 2021 at 19:08):

are you interested in writing that up? :)

view this post on Zulip Dániel Buga (Jan 22 2021 at 19:09):

unfortunately I'm more interested in figuring out why the keccak benchmark borrow-checks 3300 times :frown:

view this post on Zulip GuillaumeGomez (Jan 22 2021 at 19:10):

To be perfectly honest: I think that if it's not something handled by the web-browser directly, the size will reduce, but the performance will drop too.

view this post on Zulip GuillaumeGomez (Jan 22 2021 at 19:11):

We can add a compressed JSON that we decompress in the browser, but that will very likely be horribly slow

view this post on Zulip GuillaumeGomez (Jan 22 2021 at 19:11):

our only viable options are the ones provided by the browsers' API

view this post on Zulip GuillaumeGomez (Jan 22 2021 at 19:11):

Take a look at my blog post I posted today to give you a little idea of the things I went through already ;)

view this post on Zulip GuillaumeGomez (Jan 22 2021 at 19:12):

but since I'm not a front-end engineer, it's very possible that I missed a way to do things better, which is why I'd be glad if someone came with someone I never heard of (it'd be really wonderful)


Last updated: Oct 21 2021 at 19:46 UTC