Stream: t-lang

Topic: Making new `_#` prefixes not a breaking change


scottmcm (Apr 30 2020 at 19:53, on Zulip):

making raw idents worked only with r# because it was already reserved.

We should, as part of the next edition, make _all_ ${ident}#${stuff} be reserved, so that we have space here to do things.

Without saying what those things would be, it could be s"Hello" for Strings, keyword#yeet, etc.

Sebastian Malton (Apr 30 2020 at 19:59, on Zulip):

I think that this is a good idea

pnkfelix (Apr 30 2020 at 20:00, on Zulip):

scottmcm said:

[...] it could be s"Hello" for Strings, [...]

this was supposed to be s#"Hello", right?

kennytm (Apr 30 2020 at 20:01, on Zulip):

"Hello"s is already a single token.

nikomatsakis (Apr 30 2020 at 20:01, on Zulip):

I think we would want to do two things:

and in particular, whenever we make use of this lexical space in a design, we are effectively encourage people to build and experiment.

nikomatsakis (Apr 30 2020 at 20:02, on Zulip):

I would (personally) be fine with the "must be a proc-macro" version though

pnkfelix (Apr 30 2020 at 20:02, on Zulip):

kennytm said:

"Hello"s is already a single token.

(yes but I think the intent was to enable a kind of easy playground for new language features, in this case literals that take the structure of String rather than &'static str)

pnkfelix (Apr 30 2020 at 20:03, on Zulip):

((at least that's my inference...))

Josh Triplett (Apr 30 2020 at 20:04, on Zulip):

Here's what I just posted to #20760:

Josh Triplett (Apr 30 2020 at 20:04, on Zulip):

We'd also like to standardize a syntax for "non-breaking syntax/keywords", so that it'd be possible to introduce new keywords understood by the compiler in a namespace that nobody especially wants to type in their code regularly, but that people can experiment with surface syntax for in macros or proc macros. This syntax would be stabilized for a given feature, and continue working even after we introduce some new surface syntax (e.g. at an edition boundary). One such proposal would be r#$token, where token will always get interpreted as a keyword. Then, we could have e.g. r#$raw_ref(...) and r#$raw_mut_ref(...), and decide what the surface syntax should look like later, deferring that either to an edition (with a new keyword) or a point where we find backward-compatible syntax.

nikomatsakis (Apr 30 2020 at 20:05, on Zulip):

Seems good. I'm definitely into the idea. But I agree that it's a good idea to have #20760 on our list of "edition things to think about"

simulacrum (Apr 30 2020 at 20:05, on Zulip):

@scottmcm so if I follow correctly, we would essentially make #$ not allowed within identifiers?

nikomatsakis (Apr 30 2020 at 20:06, on Zulip):

BTW, just to spell out what I was saying about the "special span" idea --

nikomatsakis (Apr 30 2020 at 20:06, on Zulip):

This honestly seems much nicer to me than some random set of tokens

nikomatsakis (Apr 30 2020 at 20:07, on Zulip):

I think the ideal is that we then tie these experiments into a running project group

nikomatsakis (Apr 30 2020 at 20:07, on Zulip):

That is pursuing some design goals

simulacrum (Apr 30 2020 at 20:07, on Zulip):

an interesting detail there is (in theory at least) we could make such a crate have a "stability hole" in it or whatever

Josh Triplett (Apr 30 2020 at 20:07, on Zulip):

So, this would be going the "you can only generate this via macro" route?

nikomatsakis (Apr 30 2020 at 20:07, on Zulip):

simulacrum said:

an interesting detail there is (in theory at least) we could make such a crate have a "stability hole" in it or whatever

what do you mean by stability hole?

nikomatsakis (Apr 30 2020 at 20:07, on Zulip):

Josh Triplett said:

So, this would be going the "you can only generate this via macro" route?

right

simulacrum (Apr 30 2020 at 20:08, on Zulip):

so basically the programmatic way to use the syntax isn't actually stable, but the crate can use it

pnkfelix (Apr 30 2020 at 20:08, on Zulip):

for writing compiler unit tests, I suspect I would prefer some token syntax nonetheless

Josh Triplett (Apr 30 2020 at 20:08, on Zulip):

@nikomatsakis What would be the advantage of having these be macro-generated-only, rather than having special lexer syntax?

simulacrum (Apr 30 2020 at 20:08, on Zulip):

(e.g. maybe we just put this crate in the sysroot)

pnkfelix (Apr 30 2020 at 20:08, on Zulip):

but that wouldn't have to be stable

pnkfelix (Apr 30 2020 at 20:08, on Zulip):

(I just mention my own desires because I don't want to get into a "its good for me but not for thee" situation, as pcwalton used to say...)

nikomatsakis (Apr 30 2020 at 20:09, on Zulip):

pnkfelix said:

for writing compiler unit tests, I suspect I would prefer some token syntax nonetheless

that's an interesting point, presumably those unit tests could use experiments::x from sysroot or whatever?

Josh Triplett (Apr 30 2020 at 20:09, on Zulip):

simulacrum said:

(e.g. maybe we just put this crate in the sysroot)

If we're going to go the macro-only route, I would definitely want to see this crate in the sysroot. Perhaps in meta?

nikomatsakis (Apr 30 2020 at 20:09, on Zulip):

@Josh Triplett there is no real advantage apart from not having to define something like r#$

simulacrum (Apr 30 2020 at 20:09, on Zulip):

one disadvantage to the sysroot is we can't issue breaking changes

nikomatsakis (Apr 30 2020 at 20:09, on Zulip):

what sort of breaking change would we issue?

simulacrum (Apr 30 2020 at 20:10, on Zulip):

well, e.g., say we "finish" experimentation and want to remove support

nikomatsakis (Apr 30 2020 at 20:10, on Zulip):

we never do that

nikomatsakis (Apr 30 2020 at 20:10, on Zulip):

that's the whole point :)

nikomatsakis (Apr 30 2020 at 20:10, on Zulip):

the experiments are stable

nikomatsakis (Apr 30 2020 at 20:10, on Zulip):

they're just "deprecated"

Josh Triplett (Apr 30 2020 at 20:10, on Zulip):

Yeah, the proposal here was that this is stable, and you just stop using it because it looks awful.

nikomatsakis (Apr 30 2020 at 20:10, on Zulip):

in favor of the new approach

nikomatsakis (Apr 30 2020 at 20:10, on Zulip):

or maybe not, who cares

simulacrum (Apr 30 2020 at 20:10, on Zulip):

well yes I meant that we experimented and found it wanting

simulacrum (Apr 30 2020 at 20:10, on Zulip):

like, we want to remove it as a feature entirely

nikomatsakis (Apr 30 2020 at 20:10, on Zulip):

I don't think we get that option

nikomatsakis (Apr 30 2020 at 20:10, on Zulip):

if we take this approach

nikomatsakis (Apr 30 2020 at 20:10, on Zulip):

we might never make a nice syntax on top

nikomatsakis (Apr 30 2020 at 20:11, on Zulip):

but we never get to remove the core primitive

simulacrum (Apr 30 2020 at 20:11, on Zulip):

yeah, that makes sense

nikomatsakis (Apr 30 2020 at 20:11, on Zulip):

that said, we probably do want a way to have the core primitive on nightly only

nikomatsakis (Apr 30 2020 at 20:11, on Zulip):

i.e., so we can tinker a bit with how the core primitive should work before we stabilize it

Josh Triplett (Apr 30 2020 at 20:11, on Zulip):

@nikomatsakis There's a straightforward way to do that.

nikomatsakis (Apr 30 2020 at 20:11, on Zulip):

which might also address your point, @simulacrum, since maybe we uncover in that time that the primitive is broken

Josh Triplett (Apr 30 2020 at 20:11, on Zulip):

We can decide when to stabilize the r#$ version, and until it's stable it's like any nightly feature.

Lokathor (Apr 30 2020 at 20:11, on Zulip):

I see more of what you mean by the macro only thing now. as long as the playground can use the experiments crate that'd work.

nikomatsakis (Apr 30 2020 at 20:12, on Zulip):

Josh Triplett said:

We can decide when to stabilize the r#$ version, and until it's stable it's like any nightly feature.

yes

Josh Triplett (Apr 30 2020 at 20:12, on Zulip):

The advantage of this would be that we can independently make and stabilize two decisions: "should we have this feature" and "what syntax should we use for this feature".

Josh Triplett (Apr 30 2020 at 20:12, on Zulip):

Right now we have to tie the two together.

nikomatsakis (Apr 30 2020 at 20:12, on Zulip):

anyway @pnkfelix raised a good point about compiler unit tests, that's more convincing to me than playground

simulacrum (Apr 30 2020 at 20:12, on Zulip):

I'd personally be on board with either a meta or experiments crate in sysroot or just tokens with #$ in them (or whatever)

nikomatsakis (Apr 30 2020 at 20:12, on Zulip):

yes I don't care very much

simulacrum (Apr 30 2020 at 20:13, on Zulip):

we can also do something in between full on sysroot, e.g. you need to rustup component add experiments to get it

simulacrum (Apr 30 2020 at 20:13, on Zulip):

but I don't think there's much value in that

Lokathor (Apr 30 2020 at 20:13, on Zulip):

It may be a style issue or who you know or something, but almost all the experimental rust collaboration i see is via playground links

nikomatsakis (Apr 30 2020 at 20:13, on Zulip):

I think it depends on the scale

nikomatsakis (Apr 30 2020 at 20:13, on Zulip):

for raw_ref... yeah

Josh Triplett (Apr 30 2020 at 20:13, on Zulip):

I can only think of one advantage to making this macro-only rather than special "future keyword" syntax:

nikomatsakis (Apr 30 2020 at 20:13, on Zulip):

but I could see us using this to e.g. experiment with generators or who knows at some point

nikomatsakis (Apr 30 2020 at 20:14, on Zulip):

or e.g. stuff like fehler

Josh Triplett (Apr 30 2020 at 20:15, on Zulip):

If it's macro-only, we can give it a fully descriptive name, like ReturnError, because proc macros are already quite verbose. If it's syntax-like, like r#$foo, it kinda wants to be shorter.

Josh Triplett (Apr 30 2020 at 20:15, on Zulip):

We could say r#$return_error e;, nothing stops us. Or we could say r#$yeet e;. :)

Josh Triplett (Apr 30 2020 at 20:16, on Zulip):

We could even say r#$12345 where 12345 is the tracking issue. ;)

Lokathor (Apr 30 2020 at 20:16, on Zulip):

So for clarification, would be experiments proc-macro crate be specially blessed, or would all proc macros have a way to emit these special keywords and the experiments crate is just the easy way for outside users to access each special keyword one at a time?

Josh Triplett (Apr 30 2020 at 20:16, on Zulip):

@Lokathor The latter, if we go that route.

nikomatsakis (Apr 30 2020 at 20:17, on Zulip):

the only reason to have the experiments crate would be for playground

nikomatsakis (Apr 30 2020 at 20:17, on Zulip):

and tests etc

nikomatsakis (Apr 30 2020 at 20:17, on Zulip):

i.e., not special, just a convenience

simulacrum (Apr 30 2020 at 20:17, on Zulip):

if we have the special keywords at all (vs programmatic access) I don't know that we'd want a "special crate"

Lokathor (Apr 30 2020 at 20:18, on Zulip):

r#$tracker_id is a very good way to make it work and strongly encourage people to update their code as soon as the thing gets a real stable syntax, because it's even more ugly than the other proposals.

simulacrum (Apr 30 2020 at 20:18, on Zulip):

I do like the tracking issue number thing, though I'm sure it'll be the new thing on the "obfuscated rust" block :)

Josh Triplett (Apr 30 2020 at 20:18, on Zulip):

@Lokathor True. I'm still somewhat leaning towards r#$descriptive_verbose_name, though, for the sake of not making people carry around an index. :)

Josh Triplett (Apr 30 2020 at 20:19, on Zulip):

I'd rather people say "wow that's long" than "wow that's obscure".

Josh Triplett (Apr 30 2020 at 20:19, on Zulip):

That holds doubly true for things we stabilize, and thus things that will permanently work.

Lokathor (Apr 30 2020 at 20:20, on Zulip):

that's the policy that the tinyvec crate has to avoid name clashes with actual vec method names, such as ArrayVec::grab_spare_slice

Josh Triplett (Apr 30 2020 at 20:20, on Zulip):

@Lokathor Nice precedent.

Lokathor (Apr 30 2020 at 20:21, on Zulip):

One advantage that maybe hasn't been said out loud yet: it will probably be extremely easy for rustfix to apply the "convert to stabilized syntax" changes

Josh Triplett (Apr 30 2020 at 20:21, on Zulip):

Yeah. Even if the syntax isn't just a keyword.

Josh Triplett (Apr 30 2020 at 20:22, on Zulip):

It's easy to convert r#$raw_ref(foo) to an arbitrary raw reference syntax.

Josh Triplett (Apr 30 2020 at 20:22, on Zulip):

Likewise r#$raw_mut_ref, or r#$asm().

nikomatsakis (Apr 30 2020 at 20:24, on Zulip):

what I'm imagining is that most folks wouldn't use the syntax directly anyway but rather some macro that wraps it, so rustfmt rewriting might not be super useful, but that's ok.

simulacrum (Apr 30 2020 at 20:25, on Zulip):

for e.g. packed struct fields I'm guessing using the syntax directly would be common

simulacrum (Apr 30 2020 at 20:25, on Zulip):

seems hard to do better than the r#$raw_reference (especially without postfix macros)

Lokathor (Apr 30 2020 at 20:31, on Zulip):

can a macro even return a place?

kennytm (Apr 30 2020 at 20:35, on Zulip):
macro_rules! x {
    ($x:ident) => { $x }
}
fn main() {
    let x!(y) = 5;
    println!("{}", y);
}
scottmcm (Apr 30 2020 at 20:38, on Zulip):

simulacrum said:

scottmcm so if I follow correctly, we would essentially make #$ not allowed within identifiers?

We couldn't do raw#match because that lexes as three things in macros today, and thus was a breaking change. So what I meant is that we'd change that lexing rule to be one token (in an edition) and thus could define meaning for it later if we wanted. And one could continue to call a macro that wanted it as three tokens by just adding spaces.

simulacrum (Apr 30 2020 at 20:41, on Zulip):

ah, okay, sure

Josh Triplett (Apr 30 2020 at 23:01, on Zulip):

So, if we want to move forward with r#$descriptive_name syntax, is anyone interested in writing that up and proposing it for review?

Lokathor (Apr 30 2020 at 23:03, on Zulip):

Suppose I could do that, i already have the repo cloned

Lokathor (Apr 30 2020 at 23:03, on Zulip):

Seems like a very simple proposal to write.

Josh Triplett (Apr 30 2020 at 23:04, on Zulip):

Thank you. Please feel free to note that this was discussed in a lang team meeting and subsequent Zulip thread.

Josh Triplett (Apr 30 2020 at 23:05, on Zulip):

The most important justification is that it lets us separately review and approve proposals for new functionality and proposals for syntax for that functionality.

Josh Triplett (Apr 30 2020 at 23:05, on Zulip):

And let people experiment with the latter.

scottmcm (Apr 30 2020 at 23:08, on Zulip):

@Josh Triplett do we want r#$name or something#name? I guess it's the former if we need it _now_, but I would kinda rather the latter even though it'd only be usable in the next edition.

Josh Triplett (Apr 30 2020 at 23:08, on Zulip):

I'd like something that's usable in any Rust edition, going all the way back to 2015.

Josh Triplett (Apr 30 2020 at 23:09, on Zulip):

That doesn't prevent us from using something#name in new editions for other purposes, such as new kinds of literals.

Josh Triplett (Apr 30 2020 at 23:09, on Zulip):

But if we're developing something that's intended as "raw syntaxless language constructs", tying it to an edition seems counterproductive.

Josh Triplett (Apr 30 2020 at 23:13, on Zulip):

As an example, if we wanted some new syntax for literal String values or literal Path values, we could prototype it as r#$string_literal("foo") or r#$path_literal("foo") and later on introduce something like s#"foo"# or s"foo" or p#"foo"#.

Lokathor (May 01 2020 at 00:34, on Zulip):

https://github.com/rust-lang/rfcs/pull/2919

Josh Triplett (May 01 2020 at 00:46, on Zulip):

@Lokathor Nice work!

Josh Triplett (May 01 2020 at 00:46, on Zulip):

In the last section, you might mention that such a crate may be provided via the sysroot. (And also, s/whitelist/safelist/ please.)

Josh Triplett (May 01 2020 at 00:48, on Zulip):

Another alternative:
We could wait to implement and stabilize features until we've determined their final surface syntax. This would make it more difficult for people to experiment with surface syntax.

Josh Triplett (May 01 2020 at 00:52, on Zulip):

You might also add that this will allow usage of new features on old editions. Using the new syntax for those features may require moving to a new edition, but using the raw keyword syntax doesn't.

Lokathor (May 01 2020 at 01:28, on Zulip):

I'm not clear on what "available via the sysroot means". Does that mean it's like alloc and proc_macro where you never actually list it in your Cargo.toml file?

Josh Triplett (May 01 2020 at 01:46, on Zulip):

Right.

nikomatsakis (May 01 2020 at 21:15, on Zulip):

Lokathor said:

https://github.com/rust-lang/rfcs/pull/2919

woah, that was fast ... heh...

Last update: Jun 05 2020 at 23:15UTC