Stream: t-compiler/rust-analyzer

Topic: Quasy quoting for assists


Lukas Wirth (Nov 05 2020 at 19:34, on Zulip):

So I started tinkering around with a proc-macro for this and I made some good progress, though I do not know whether the approach I have so far will work out for everything in the end. A snippet I was able to replace with it now:

fn extract_struct_def(
    rewriter: &mut SyntaxRewriter,
    enum_: &ast::Enum,
    variant_name: ast::Name,
    variant_list: &ast::TupleFieldList,
    start_offset: &SyntaxElement,
    visibility: Option<ast::Visibility>,
) -> Option<()> {
-    let variant_list = make::tuple_field_list(
-        variant_list
-            .fields()
-            .flat_map(|field| Some(make::tuple_field(Some(make::visibility_pub()), field.ty()?))),
-    );
-
-    rewriter.insert_before(
-        start_offset,
-        make::struct_(visibility, variant_name, None, variant_list.into()).syntax(),
-    );
+    let tys = variant_list.fields().flat_map(|field| field.ty());
+    let struct_ = make_quote::quote! {
+        #visibility struct #variant_name( #(pub #tys),* );
+    };
+
+    rewriter.insert_before(start_offset, struct_.syntax());
    rewriter.insert_before(start_offset, &make::tokens::blank_line());

    if let indent_level @ 1..=usize::MAX = IndentLevel::from_node(enum_.syntax()).0 as usize {
        rewriter
            .insert_before(start_offset, &make::tokens::whitespace(&" ".repeat(4 * indent_level)));
    }
    Some(())
}

The invocation there specifically:

make_quote::quote! {
    #visibility struct #variant_name( #(pub #tys),* );
}

expands to

make::struct_(
    visibility,
    variant_name,
    None,
    ast::FieldList::TupleFieldList(make::tuple_field_list(
        ::itertools::__std_iter::IntoIterator::into_iter(tys)
            .map(|(tys)| make::tuple_field(Some(make::visibility_pub()), tys)),
    )),
);
Lukas Wirth (Nov 05 2020 at 20:24, on Zulip):

So this currently relies on make to typecheck things, but i imagine ideally this should typecheck on its own and then just emit a string that gets parsed to an ast?

matklad (Nov 06 2020 at 11:08, on Zulip):

So this currently relies on make to typecheck things, but i imagine ideally this should typecheck on its own and then just emit a string that gets parsed to an ast?

Not really, ideally, we should remove strings from make

matklad (Nov 06 2020 at 11:08, on Zulip):

Strings are just a way to quicly implement AST construction without too much work

matklad (Nov 06 2020 at 11:10, on Zulip):

Ideally, we should build AST directly from tree pieces, bypassing the parser

matklad (Nov 06 2020 at 11:13, on Zulip):

I am not entirely sure how to square this with quasi-quoting :-) An austronautical approach would be to run rust-analyzer's parser at compile time to allow building anything

matklad (Nov 06 2020 at 11:13, on Zulip):

A more down-to-earth approach would be to special-case everything. Basically, the whole of make API, but with a nicer syntax

matklad (Nov 06 2020 at 11:16, on Zulip):

Both approaches, however, are compatible with building AST from trees.

@veykril you might also look at https://github.com/apple/swift/tree/main/lib/Syntax for inpsiration

Lukas Wirth (Nov 06 2020 at 15:48, on Zulip):

Just took a look at the swift stuff. Were there any big differences between swifts syntax trees and rowan?

Lukas Wirth (Nov 06 2020 at 15:49, on Zulip):

I suppose using the make api is the ideal choice to make here though what I always wondered about it is that internally all make functions just display and then reparse the displayed string, doesn't this go heavily against the tree interning/sharing of what this architecture is kind of about? Since we discard all the parts passed to make by reparsing them from a string

matklad (Nov 06 2020 at 16:04, on Zulip):

Yes, rowan is almost 1-1 copy of libsyntax. There are two big differences

matklad (Nov 06 2020 at 16:04, on Zulip):

well, more than two :)

matklad (Nov 06 2020 at 16:06, on Zulip):

handling of trivia. In swift syntax, trivia are attached to non-trivial tokens.

That is, every node has a statically fixed layout, and things like struct /* many*/ /* separate */ /*comment* F /*nodes*/ /*here*/; are handled by attaching a vec of comments/whitespace to a fixed non-ws token

matklad (Nov 06 2020 at 16:06, on Zulip):

I am actually warming up to this idea, and think that it might make sense to transition rust-analyzer to this representation

matklad (Nov 06 2020 at 16:21, on Zulip):

handling of collections. Because swift supports only nodes with fixed layout, all collection nodes are homogeneous.

Ie, if in Rust, for {foo: i32, bar: f64} we put {} and the fields into tht same node, swift will introduce an intermediate node here

matklad (Nov 06 2020 at 16:21, on Zulip):

one node will hold three elements: two braces and field list

matklad (Nov 06 2020 at 16:21, on Zulip):

the field list node will hold N identical nodes (fields)

matklad (Nov 06 2020 at 16:22, on Zulip):

thread safety swift uses Arcs, which means arc bumps during traversal of sytnax nodes. rust-analyzer uses Rc, which should be much cheaper

Lukas Wirth (Nov 06 2020 at 16:47, on Zulip):

Ah I see, I also just reread https://github.com/rust-analyzer/rust-analyzer/blob/master/docs/dev/syntax.md so I do see the differences you mentioned there as well now.

Lukas Wirth (Nov 06 2020 at 16:48, on Zulip):

Regarding make stringifying and reparsing the string as a node, is it meant to stay like this in the future? Im just asking out of curiosity.

matklad (Nov 06 2020 at 17:01, on Zulip):

Oup, sorry, forgot to answer that one

matklad (Nov 06 2020 at 17:01, on Zulip):

No, just the opposite

matklad (Nov 06 2020 at 17:01, on Zulip):

stringification is a temp hack, and we totally should replace it with assembling trees from bits and pieces

matklad (Nov 06 2020 at 17:02, on Zulip):

The specific case where stringification falls down is that, if a source node has a syntax error, the stringified&reparsed node might not reparse into the desired node type

Lukas Wirth (Nov 06 2020 at 18:22, on Zulip):

Oh that does seem like a problem, so I take the sole reason make works the way it does currently is because it was simple to do it like that for now?

Lukas Wirth (Nov 06 2020 at 18:23, on Zulip):

From the looks of it, emitting make calls seems to be the best choice here nevertheless, since I dont think the make will change much then should the stringify hack disappear at some point, so I guess my proc-macro prototype is already on a good track like this.

matklad (Nov 06 2020 at 19:08, on Zulip):

yeah, make should be phased away, but its ok to use it today

matklad (Nov 06 2020 at 19:08, on Zulip):

However, an alternative would be to use ungrammar directly

matklad (Nov 06 2020 at 19:08, on Zulip):

https://github.com/rust-analyzer/ungrammar/blob/eb7e474d641fdb5210318479696cb7b362190cfd/rust.ungram#L164-L169

matklad (Nov 06 2020 at 19:09, on Zulip):

It contains all the info necessary to build a tree from pieces.

Lukas Wirth (Nov 06 2020 at 22:46, on Zulip):

What would replace make? I would believe the surface api would look the same no?

Lukas Wirth (Nov 06 2020 at 22:47, on Zulip):

I imagine if anything it should in the end be generated to some degree like most of the ast stuff?

matklad (Nov 06 2020 at 22:48, on Zulip):

Yeah, I mean, the impl should change

matklad (Nov 06 2020 at 22:48, on Zulip):

the API I think looks about right, although it's plausible that we can replace all of it with just ast::make! macro

matklad (Nov 06 2020 at 22:48, on Zulip):

And yeah, it should be generated from ungrmmar

Lukas Wirth (Nov 06 2020 at 22:56, on Zulip):

Alright, now I'm grasping the full situation regarding this I believe, thanks for the thorough explanation :thumbs_up:

matklad (Nov 09 2020 at 14:37, on Zulip):

@Lukas Wirth have you seen https://docs.microsoft.com/en-us/dotnet/csharp/roslyn-sdk/get-started/syntax-transformation ? This is an interesting doc to study re tree edits

Lukas Wirth (Nov 09 2020 at 14:47, on Zulip):

No I haven't. Will take a look it later :thumbs_up:

matklad (Nov 17 2020 at 22:20, on Zulip):

@Lukas Wirth just randomly cc'ing you to take a look at this fun API from roslyn: https://github.com/dotnet/roslyn/blob/4d8cada2584dd2e58b391c32c84f10f29442fdde/src/Workspaces/Core/Portable/Editing/SyntaxEditor.cs#L125-L141

I wonder if this "The node passed into the compute function includes changes from prior edits" bit could be used to make SyntaxRewriter infra less horrible..

Lukas Wirth (Nov 17 2020 at 23:28, on Zulip):

That looks does look interesting. Good question whether that could somehow clean up SyntaxRewriter, though I'm not really sure how that could be applied to make it less of a mess

Lukas Wirth (Nov 17 2020 at 23:37, on Zulip):

Maybe I just can't think of any the use cases for this function, though i also don't really have in mind right now how we use SyntaxRewriter, will have to take a look at that again

Last update: Jul 29 2021 at 08:45UTC