Stream: t-compiler/rust-analyzer

Topic: Wasm version of Rust Analyzer


Achim Schneider (Jun 01 2021 at 13:41, on Zulip):

Hello,

I'm experimenting with the WASM version of Rust Analyzer: https://github.com/rust-analyzer/rust-analyzer-wasm

As far as I can see, it can only receive the content of a single file but not of a full crate. This is since it is only implementing ra_ap_ide::Analysis and not ra_ap_ide::AnalysisHost? If I want it to analyze a complete crate, resolving dependencies from a Cargo.toml file, I assume that I've to add AnalysisHost, create a Change and use AnalysisHost.apply_change, so that I can finally create a Analysis instance from AnaylsisHost for analysis of the current file?

Did someone ever try something like this? Any hints on some sample code? I'm an experienced Typescript dev, but relatively new to Rust. I appreciate any help!

Best,
Achim

Florian Diebold (Jun 01 2021 at 13:48, on Zulip):

I don't think it's about AnalysisHost vs. Analysis; you need an AnalysisHost to get an Analysis, from_single_file just creates one internally. The issue is getting a project model, since you can't run cargo. You could just look at what from_single_file does and add more files and even a more complicated CrateGraph if you want

Florian Diebold (Jun 01 2021 at 13:50, on Zulip):

I see that the WASM demo actually recreates the analysis whenever the code changes; I don't think that's really the intended way to do it, not sure why it's doing that

Florian Diebold (Jun 01 2021 at 13:51, on Zulip):

usually you have to hold on to the AnalysisHost and then apply changes to the files as you get them

Florian Diebold (Jun 01 2021 at 13:52, on Zulip):

also, Analysis::from_single_file has a doc comment that isn't a doc comment...

bjorn3 (Jun 01 2021 at 13:53, on Zulip):

The comment on impl Analysis can also be turned into a doc comment.

Florian Diebold (Jun 01 2021 at 13:54, on Zulip):

that one is more a notice for people changing the code though, so it kind of makes sense to not make it a doc comment

Achim Schneider (Jun 01 2021 at 13:55, on Zulip):

Yes, I was also thinking that it is a very preliminary version!

But investigating the from_single_file method in the Rust Analyzer sourcecode is already a big help. That's the sample creation of an AnalysisHost from some file, that I was searching for. Will try to understand and use it. Thanks!

Achim Schneider (Jun 02 2021 at 10:04, on Zulip):

Is there any sample code available which converts a Cargo.toml file into a CrateGraph that I can pass to AnalysisHost.

I found this guide: https://github.com/rust-analyzer/rust-analyzer/blob/guide-2019-01/, but I think it's outdated.

Florian Diebold (Jun 02 2021 at 10:07, on Zulip):

you need Cargo to do that. Then it's basically this

Florian Diebold (Jun 02 2021 at 10:08, on Zulip):

but you're not going to be able to do it in wasm, unless you compile Cargo to wasm (which may be possible? I don't know how well-abstracted the I/O parts are in Cargo)

bjorn3 (Jun 02 2021 at 10:09, on Zulip):

The I/O parts are not abstracted at all. It would probably compile for wasm32-wasi though.

Achim Schneider (Jun 02 2021 at 10:23, on Zulip):

But if most of the files I want to edit online are using the same dependencies, then I could create the CrateGraph once and parse it as a JSON to the Wasm version of Rust Analyzer?

Florian Diebold (Jun 02 2021 at 10:33, on Zulip):

that should be possible. Note that you still need to somehow bundle and provide the code for all those dependencies

Florian Diebold (Jun 02 2021 at 10:33, on Zulip):

including the standard library

Florian Diebold (Jun 02 2021 at 10:34, on Zulip):

or I guess you could also retrieve it

Achim Schneider (Jun 03 2021 at 12:30, on Zulip):

There seems to be even an implementation of a .json format for CrateGraphs: https://github.com/rust-analyzer/rust-analyzer/blob/dbdfeeeff91b5e42d8687df09dda1d29f99b34f8/crates/project_model/src/project_json.rs. But within the code I can only find functions which are converting json to CrateGraphs. Do you have any advice of how I can get such a rust-project.json file from either a crate.graph or Cargo.toml file / cargo metadata output?

Florian Diebold (Jun 03 2021 at 12:31, on Zulip):

no, they're intended to be written by hand or other build systems

Achim Schneider (Jun 03 2021 at 12:33, on Zulip):

But it should be possible to write a script which converts cargo metadata to a rust-project.json file?

Florian Diebold (Jun 03 2021 at 12:34, on Zulip):

yes, that should be possible. It could even be useful as a rust-analyzer command

Florian Diebold (Jun 03 2021 at 12:35, on Zulip):

which would probably be pretty straightforward, since it could just get the CrateGraph and then convert back to the project json format

Achim Schneider (Jun 11 2021 at 06:43, on Zulip):

I implemented a serialization and deserialization of the CrateGraph as well as of the Change object in my own version of RustAnalyzer. I will see if I can provide a PR which implements a CLI command for creating a rust-project.json file! Many thanks so far!

Achim Schneider (Jun 11 2021 at 06:45, on Zulip):

I'm currently running into problems when I try to compile any version of rust-analyzer to WebAssembly up from 0.0.48.

Here is the compiler error after changing the rust version to 0.0.48. in Cargo.toml of the wasm version of rust-analyzer:

error[E0690]: transparent struct needs exactly one non-zero-sized field, but has 0
--> /home/achim/.cargo/registry/src/github.com-1ecc6299db9ec823/ra_ap_stdx-0.0.48/src/lib.rs:182:1
|
182 | pub struct JodChild(pub process::Child);
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ needs exactly one non-zero-sized field, but has 0

error: aborting due to previous error

For more information about this error, try rustc --explain E0690.
error: could not compile ra_ap_stdx

Achim Schneider (Jun 11 2021 at 06:45, on Zulip):

Any ideas what could fix this?

Daniel Mcnab (Jun 11 2021 at 08:16, on Zulip):

Ah, so that's because wasm can't have child processes
The easiest way would be to make it be not repr(transparent) on wasm and panic before trying to transmute into it

Florian Diebold (Jun 11 2021 at 08:41, on Zulip):

it kind of seems that if we're trying to create child processes in wasm, it's already too late? i.e. maybe the type just shouldn't exist in wasm?

Florian Diebold (Jun 11 2021 at 08:42, on Zulip):

but maybe that goes against the normal approach

Florian Diebold (Jun 11 2021 at 08:44, on Zulip):

it's used by flycheck, which probably isn't in your dependency tree, and proc_macro_api, which... probably also shouldn't be

Florian Diebold (Jun 11 2021 at 08:46, on Zulip):

I wonder if we should have a WASM build in CI

Florian Diebold (Jun 11 2021 at 08:47, on Zulip):

of the crates that are supposed to work in WASM

Florian Diebold (Jun 11 2021 at 08:47, on Zulip):

i.e. everything behind ide

Florian Diebold (Jun 11 2021 at 08:49, on Zulip):

I guess the other question is why JodChild needs to be repr(transparent) at all :thinking:

Florian Diebold (Jun 11 2021 at 08:51, on Zulip):

@matklad why can't JodChild::into_inner not simply return self.0?

Florian Diebold (Jun 11 2021 at 08:55, on Zulip):

oh, because it implements Drop

bjorn3 (Jun 11 2021 at 09:35, on Zulip):

It could contain an Option<Child> and do nothing in the Drop impl if it is None.

matklad (Jun 11 2021 at 09:39, on Zulip):

It'd be fine to make repr transparent conditional, and just panic in the into on wasm

Achim Schneider (Jun 11 2021 at 10:19, on Zulip):

While trying to compile the most recent version of rust-analyzer to wasm, I'm facing even more errors:

Achim Schneider (Jun 11 2021 at 10:19, on Zulip):

error[E0433]: failed to resolve: use of undeclared crate or module imp
--> /home/achim/.cargo/registry/src/github.com-1ecc6299db9ec823/ra_ap_stdx-0.0.59/src/process.rs:25:9
|
25 | imp::read2(out, err, &mut |is_out, data, eof| {
| ^^^ use of undeclared crate or module imp

error[E0690]: transparent struct needs exactly one non-zero-sized field, but has 0
--> /home/achim/.cargo/registry/src/github.com-1ecc6299db9ec823/ra_ap_stdx-0.0.59/src/lib.rs:114:1
|
114 | pub struct JodChild(pub std::process::Child);
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ needs exactly one non-zero-sized field, but has 0

error[E0658]: use of unstable library feature 'partition_point': new API
--> /home/achim/.cargo/registry/src/github.com-1ecc6299db9ec823/ra_ap_stdx-0.0.59/src/lib.rs:95:23
|
95 | let start = slice.partition_point(|it| key(it) == Ordering::Less);
| ^^^^^^^^^^^^^^^
|
= note: see issue #73831 <https://github.com/rust-lang/rust/issues/73831> for more information
= help: add #![feature(partition_point)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'partition_point': new API
--> /home/achim/.cargo/registry/src/github.com-1ecc6299db9ec823/ra_ap_stdx-0.0.59/src/lib.rs:96:30
|
96 | let len = slice[start..].partition_point(|it| key(it) == Ordering::Equal);
| ^^^^^^^^^^^^^^^
|
= note: see issue #73831 <https://github.com/rust-lang/rust/issues/73831> for more information
= help: add #![feature(partition_point)] to the crate attributes to enable

error: aborting due to 4 previous errors

Some errors have detailed explanations: E0433, E0658, E0690.
For more information about an error, try rustc --explain E0433.
error: could not compile ra_ap_stdx

Lukas Wirth (Jun 11 2021 at 10:21, on Zulip):

Two of those are due to you not running the latest stable toolchain, RA usually only builds on the latest rust version

Achim Schneider (Jun 11 2021 at 10:32, on Zulip):

Ah, thanks! The toolchain is set to nightly-2021-02-11
in the rust-toolchain file in the wasm GitHub Repo. After setting it to stable, two errors remain:

error[E0433]: failed to resolve: use of undeclared crate or module imp
--> /home/achim/.cargo/registry/src/github.com-1ecc6299db9ec823/ra_ap_stdx-0.0.59/src/process.rs:25:9
|
25 | imp::read2(out, err, &mut |is_out, data, eof| {
| ^^^ use of undeclared crate or module imp

error[E0690]: transparent struct needs exactly one non-zero-sized field, but has 0
--> /home/achim/.cargo/registry/src/github.com-1ecc6299db9ec823/ra_ap_stdx-0.0.59/src/lib.rs:114:1
|
114 | pub struct JodChild(pub std::process::Child);
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ needs exactly one non-zero-sized field, but has 0

Achim Schneider (Jun 11 2021 at 10:33, on Zulip):

I'm guessing that E0433 is due to trying some fs access, which is not available is wasm?

Jonas Schievink [he/him] (Jun 11 2021 at 10:35, on Zulip):

it's because our process.rs only has implementations for unix and windows

Jonas Schievink [he/him] (Jun 11 2021 at 10:35, on Zulip):

https://github.com/rust-analyzer/rust-analyzer/blob/master/crates/stdx/src/process.rs

Achim Schneider (Jun 11 2021 at 12:01, on Zulip):

Okay, I see the point. I think I can probably fix that JodChild issue. Is it possible to implement a wasm version for process.rs? Problably by preceding imp with #[cfg(target_arch = "wasm32")]? Sorry for all the noob questions, really very new to rust!

Jonas Schievink [he/him] (Jun 11 2021 at 12:04, on Zulip):

yeah, presumably the WASM implementation would just return an error unconditionally (since the Command API also doesn't work there)

Achim Schneider (Jun 11 2021 at 12:06, on Zulip):

I was assuming that it shouIdn't get called! I tried to add a dummy implementation. Compilation passes beyond this point! Now new errors are showing up :/

Compiling ide v0.0.0 (/home/achim/Projects/rust-analyzer/crates/ide)
error[E0277]: the trait bound std::string::String: From<Url> is not satisfied
--> /home/achim/Projects/rust-analyzer/crates/ide/src/doc_links.rs:289:24
|
289 | .map(|url| url.into())
| ^^^^ the trait From<Url> is not implemented for std::string::String
|
= help: the following implementations were found:
<std::string::String as From<&'a js_sys::JsString>>
<std::string::String as From<&mut str>>
<std::string::String as From<&std::string::String>>
<std::string::String as From<&str>>
and 10 others
= note: required because of the requirements on the impl of Into<std::string::String> for Url

error[E0277]: the trait bound std::string::String: From<Url> is not satisfied
--> /home/achim/Projects/rust-analyzer/crates/ide/src/doc_links.rs:328:19
|
328 | Some((new_url.into(), strip_prefixes_suffixes(title).to_string()))
| ^^^^ the trait From<Url> is not implemented for std::string::String
|
= help: the following implementations were found:
<std::string::String as From<&'a js_sys::JsString>>
<std::string::String as From<&mut str>>
<std::string::String as From<&std::string::String>>
<std::string::String as From<&str>>
and 10 others
= note: required because of the requirements on the impl of Into<std::string::String> for Url

error[E0277]: the trait bound std::string::String: From<Url> is not satisfied
--> /home/achim/Projects/rust-analyzer/crates/ide/src/doc_links.rs:348:24
|
348 | .map(|url| url.into())
| ^^^^ the trait From<Url> is not implemented for std::string::String
|
= help: the following implementations were found:
<std::string::String as From<&'a js_sys::JsString>>
<std::string::String as From<&mut str>>
<std::string::String as From<&std::string::String>>
<std::string::String as From<&str>>
and 10 others
= note: required because of the requirements on the impl of Into<std::string::String> for Url

error: aborting due to 3 previous errors

Achim Schneider (Jun 21 2021 at 11:06, on Zulip):

Just in case that someone is interested: I managed to serialize/deserialize a Change object. Now my WASM Version of rust analyzer is loading the dependencies for ink Smart Contracts (https://github.com/paritytech/ink), allowing the online editing of a Smart Contract file.

You can test the current version here:

https://friendly-ramanujan-7bfd78.netlify.app/

Please note that rust analyzer only becomes available after being loaded, which takes up to 20 seconds.

Achim Schneider (Jun 21 2021 at 11:07, on Zulip):

@Florian Diebold @matklad Do you have any estimate how long it might take until the current version of rust analyzer becomes compilable to wasm again? I'm currently using version 0.45.

matklad (Jun 21 2021 at 11:09, on Zulip):

In "CPU time", I'd estimate that as half-a-day worth of work. In "wall-clock time", well, when someone invests this half a day!

Achim Schneider (Jun 21 2021 at 11:12, on Zulip):

I can do my best trying to support. It's just that my background is being a TypeScript developer so far, this is the first time for me working with rust. Will probably find support from some colleagues within my company too ;)

bjorn3 (Jun 21 2021 at 13:19, on Zulip):

I am getting the following error:

DataCloneError: Worker.postMessage: The WebAssembly.Memory object cannot be serialized. The Cross-Origin-Opener-Policy and Cross-Origin-Embedder-Policy HTTP headers can be used to enable this.

bjorn3 (Jun 21 2021 at 13:19, on Zulip):

This is using firefox.

Achim Schneider (Jun 21 2021 at 13:30, on Zulip):

Ah yes! Currently it is only working for Chrome. It's just a very early WIP...

Florian Diebold (Jun 21 2021 at 13:34, on Zulip):

it's working for me, very cool :+1:

Florian Diebold (Jun 22 2021 at 19:54, on Zulip):

https://github.com/rust-analyzer/rust-analyzer/pull/9381

Achim Schneider (Jun 29 2021 at 12:58, on Zulip):

@Florian Diebold Thanks a lot for fixing the compilation to WASM! Unfortunately, wasm-pack build still fails to build a valid .wasm file:

[INFO]: Checking for the Wasm target... [INFO]: Compiling to Wasm... Compiling wasm_demo v0.1.0 (/home/achim/Projects/wasm/rust-analyzer-react-typescript) error: linking with rust-lld` failed: exit status: 1

...

= note: rust-lld: error: mutable global exported but 'mutable-globals' feature not present in inputs: __tls_base. Use --no-check-features to suppress.
rust-lld: error: mutable global exported but 'mutable-globals' feature not present in inputs: __tls_size. Use --no-check-features to suppress.
rust-lld: error: mutable global exported but 'mutable-globals' feature not present in inputs: __tls_align. Use --no-check-features to suppress.

`

Any suggestions what I could do to fix this?

bjorn3 (Jun 29 2021 at 13:03, on Zulip):

What is the exact wasm-pack command you tried? Did you enable any target-features?

Florian Diebold (Jun 29 2021 at 13:04, on Zulip):

looks like https://github.com/rustwasm/wasm-bindgen/issues/2487 ?

Florian Diebold (Jun 29 2021 at 13:05, on Zulip):

so updating wasm-bindgen may help?

bjorn3 (Jun 29 2021 at 13:06, on Zulip):

That only applies to when using wasm shared memory for multi threading and the change in that PR is in the command the user uses, not what wasm-bindgen uses.

Achim Schneider (Jun 29 2021 at 13:19, on Zulip):

I use wasm-pack build --target web for compiling. I tried a variety of wasm-bindgen versions and toolchains, it fails with wasm-bindgen = "0.2.74" too...

Achim Schneider (Jun 29 2021 at 13:20, on Zulip):

wasm-bingen-rayon requires to set some addiaitonal parameters, like nightly toolchain and some extra flags in .cargo/config:

`[target.wasm32-unknown-unknown]
rustflags = ["-C", "target-feature=+atomics,+bulk-memory"]

[unstable]
build-std = ["panic_abort", "std"]`

See: https://github.com/GoogleChromeLabs/wasm-bindgen-rayon

bjorn3 (Jun 29 2021 at 13:21, on Zulip):

You have to add +mutable-globals: https://github.com/rustwasm/wasm-bindgen/pull/2488/files#diff-b4df63c506d6bfc7e14d93aeb1fe129c3ad10b3f6b1df7a1f4cc6ebff862d059R15

Achim Schneider (Jun 29 2021 at 13:37, on Zulip):

I tried this flag before, since the error message also suggested it. But I either had a typo or something else was wrongly configured. However, it compiles now :) Thanks a lot!

Achim Schneider (Jun 30 2021 at 11:33, on Zulip):

I'm currently using my own modified version of rust analyzer in which I implemented the serialization and deserialization of the Change object, which is send to the AnalysisHost. I also added a CLI command to generate a .json file for a rust project. This way I'm able to load the dependencies for the rust analyzer into the version which I compiled to WASM. Would this feature be of general interest? Should I create a PR for it? Or would there be a better way to load the dependencies of a rust project into the WASM version?

Florian Diebold (Jun 30 2021 at 11:52, on Zulip):

hmm, maybe it'd be best to make a draft PR so we have something concrete to talk about

Achim Schneider (Jul 15 2021 at 07:32, on Zulip):

Hey ;)
I would still like to do this PR.

However, I was serializing/deserializing the Change object without the proc-macros. I was planning to add this before I publish a PR.

But I realized that the proc-macros are handled by ProcMacroSrv which is not used by the IDE crate? So I guess they won't be available in a wasm version of rust-analyzer, anyway? Or at least not until someone implements a wasm version of ProcMacroSrv? Since it is using threads it probably wouldn't compile to wasm?

Laurențiu (Jul 15 2021 at 07:35, on Zulip):

Do you have a compiler? You'd need to compile the proc macros before running them anyway.

Achim Schneider (Jul 15 2021 at 07:49, on Zulip):

I don't have a compiler in WASM, unfortunately. But it might be possible to use pre-compiled proc-macros? Can you tell me where rust analyzer handles the compilation of proc macros?

bjorn3 (Jul 15 2021 at 09:15, on Zulip):

It is not possible to compile proc-macros for wasm. They are compiled by rustc as dylibs (invoked by cargo check). None of the wasm targets don't support dylibs yet.

Florian Diebold (Jul 15 2021 at 10:34, on Zulip):

well, there's nothing in rust-analyzer that requires that to be the case. for the hir / ide crates, a proc macro is just a dyn ProcMacroExpander. it'd be possible to build in precompiled proc macros, for example. it wouldn't be trivial, though

Florian Diebold (Jul 15 2021 at 10:36, on Zulip):

it might even be possible to precompile the proc macros to wasm using watt and load that

Florian Diebold (Jul 15 2021 at 10:42, on Zulip):

nevertheless, I would recommend leaving that out of the PR for now :sweat_smile:

Last update: Jul 24 2021 at 21:00UTC