Stream: t-compiler/wg-rls-2.0

Topic: reusable code analyzer

pksunkara (Apr 15 2020 at 13:13, on Zulip):

So, I have been recently thinking of trying to come up with a rust code analyzer for one of my projects and I am wondering if I could adopt rust-analyzer code. I need to analyze only the actual code (not deps), I probably want to edit some stuff and reconstruct the code back again (so need to preserve whitespace in the AST), I need the types of stuff so that analysis can be done on types too.

matklad (Apr 15 2020 at 13:34, on Zulip):

I suggest trying to fork rust-analyzer, and implement the analysis as an assist

matklad (Apr 15 2020 at 13:34, on Zulip):

That's the easiest way to experiment with analysis and transformation

pksunkara (Apr 15 2020 at 13:36, on Zulip):

Thanks @matklad . Will do that.

pksunkara (Apr 20 2020 at 15:24, on Zulip):

@matklad I started trying to do this and as far as I understand, assists need the cursor to be somewhere. I want to do a visitor kind of thing for the tree. Do I continue with this part of the code by changing stuff? Or should I explore other way I can do this?

matklad (Apr 20 2020 at 15:27, on Zulip):

You are correct, that assist starts from a cursor.

Still, I would start with an assist, to see if the underlying analysis/editing API is enough for your use-case. If it is enough, it should be possible to re-rail the implementation onto a visitor. But we currently don't have a vistor-based infra, so it would be significantly more work to add it

pksunkara (Apr 20 2020 at 15:28, on Zulip):

Yeah, that is exactly what I was thinking. Looks like the editing API looks enough (or easy to enhance). I will work on this a bit more. And this is also making me understand RA code more which means I can contribute some fixes to IDE issues that were annoying me :D

matklad (Apr 20 2020 at 15:32, on Zulip):

Actually, I think our assist infra could use some improvement.

Running assists/inspections in batch on the code base is something we want to do eventually. And for that, we need an API which can work both from text cursor and from tree visitor. Good food for thought, thanks!

pksunkara (Apr 20 2020 at 15:34, on Zulip):

I agree. Basically, we are working on releasing Clap v3. And I want to create a tool that would automate the upgrade from v2 to v3 so that users can just run one command and their code is now upgraded to use clap v3. And this can be made generic for all crates

matklad (Apr 20 2020 at 15:36, on Zulip):

Ah yeah, long term I'd really love to see kotlin-style reprecated, replaceWith to automate library upgrades.

matklad (Apr 20 2020 at 15:39, on Zulip):

At the moment, I think it would be ok to pin rust-analyzer to a specific commit and implement such a tool on top of a "frozen" API. Sadly, I think there currently isn't an API stable enough to guaranteed that the tool would work across different ra versions.

You might also take a look at ssr, which might be relevant to your use-case

matklad (Apr 20 2020 at 15:51, on Zulip):

Another exapmle of tree-walking is here:

pksunkara (Apr 24 2020 at 11:17, on Zulip):

I was talking about this to someone else and wanted to also paste this here.

So, I am trying to build a small visitor for code analysis. Visitor will be working on code that is written as is. So, macros will not be expanded. But you will also have access to the types of every expression, item, etc.. (which is resolved after expanding macros). rust-analyzer have the syntax AST ready, and it also have types information for the whole project.

Why the type information is important for the analysis? Regarding the cargo-up I linked above, let's assume clap::Arg has changed it's method help to about. If we are running the tool on the following code:

let a = clap::Arg::new("a");"help message");

Without the type information, you can't recognise that help needs to be changed to about. With type info, we know that a is of type clap::Arg. So, the cargo-up visitor when visiting method_call can be something like this:

if method == "help" && expr.type == "clap::Arg"  {
    method = "about";

I am currently trying to get a small MVP of the visitor working

Florian Diebold (Apr 24 2020 at 11:34, on Zulip):

we also provide the actual resolved method, using that would be more accurate than checking name and type in some edge cases

Florian Diebold (Apr 24 2020 at 11:40, on Zulip):

actually not even edge cases, your approach would already have problems with autoref/deref (i.e. what if it's a &self method and the expression type is &clap::Arg? what if the expression type is &&&clap::Arg or Arc<clap::Arg>?)

pksunkara (Apr 24 2020 at 11:42, on Zulip):

That's actually really helpful. I am still going through the rust-analyzer code. I would appreciate it if you could point that stuff out

Florian Diebold (Apr 24 2020 at 11:47, on Zulip):

in the Semantics API, you've got resolve_method_call, which will give you the resolved Function, so then you'd just have to have some way of getting the Function you actually want to replace

pksunkara (May 04 2020 at 22:03, on Zulip):

@matklad I got a basic visitor working, but unfortunately I am stuck trying to load the semantics. Looks like load_cargo which I am using to load the code isn't loading library_roots. Really appreciate some direction here

matklad (May 05 2020 at 08:38, on Zulip):

@pksunkara I'd probably start with looking analysis_stats -- it also uses load_cargo, and it does work with dependencies

pksunkara (May 05 2020 at 08:41, on Zulip):

Okay, thanks

pksunkara (May 06 2020 at 07:06, on Zulip):

So, load_cargo does load the dependency source roots, but it doesn't set them as library_roots which led me to the original confusion. library_roots is only set in the WorldState related logic. I got the semantics working though, turns out I need to do an additional semantics.parse for the file I am visiting.

pksunkara (May 06 2020 at 08:11, on Zulip):

@Florian Diebold Hey, thanks for the help. I was able to get the module name clap::arg and fn name help from the Function, but I am stumped on how to get the struct Arg for it, since the function is a method call on clap::arg::Arg. Am I missing something?

Florian Diebold (May 06 2020 at 08:13, on Zulip):

@pksunkara is this while looking at the method call, or at the definition? i.e. why do you need to get the struct?

pksunkara (May 06 2020 at 08:13, on Zulip):

While looking at the method call. I want to get the full path of the definition

pksunkara (May 06 2020 at 08:14, on Zulip):

There might be clap::arg::Arg::help(&self) and clap::arg::Arg2::help(&self). How would I differentiate?

pksunkara (May 06 2020 at 08:16, on Zulip):

I am looking at how the hover hints in vscode work. If I hover on method_call, they have the full path (including the struct) and then the function docs.

Florian Diebold (May 06 2020 at 08:19, on Zulip):

my suggestion would be to find the Function you want to replace first, for example by walking down the modules in the clap crate to the impl and looking for the function in the impl by name. And then you can just compare the resolved Function with the one you looked up

Florian Diebold (May 06 2020 at 08:20, on Zulip):

but you can also go from the Function to the containing ImplDef using as_assoc_item and then container

Florian Diebold (May 06 2020 at 08:21, on Zulip):

and then the ImplDef has a target_ty

pksunkara (May 06 2020 at 08:21, on Zulip):

walking down the modules in the clap crate to the impl and looking for the function in the impl by name

I would prefer not to do it because we are not actually changing the libraries code, but only the user written code

pksunkara (May 06 2020 at 08:21, on Zulip):

But if I need a reference to the original implementation so that I can compare them, I guess I can do it.

Florian Diebold (May 06 2020 at 08:21, on Zulip):

I don't mean changing that code, just finding the right Function so you know what you're looking for

Florian Diebold (May 06 2020 at 08:23, on Zulip):

if you go from the resolved Function to the impl to the type, you still have to determine somehow whether that's the type you were looking for

pksunkara (May 06 2020 at 08:24, on Zulip):

you still have to determine somehow whether that's the type you were looking for

I don't think that's an issue, because I am doing it inside a visitor pattern and this feels correct way to me

Florian Diebold (May 06 2020 at 08:26, on Zulip):

I mean, finding the type to compare to is just about as complicated as finding the function, you've just taken more steps to get there

pksunkara (May 06 2020 at 08:29, on Zulip):

Definitely, both approaches work. I will keep it in mind. Thanks for the help.

pksunkara (May 06 2020 at 11:33, on Zulip):

@matklad Got this successfully working with minor visibility changes in rust_analyzer. What do you want me to do? Should I send PRs to add ra_visitor?

Last update: Sep 30 2020 at 16:45UTC