Stream: t-compiler/wg-learning

Topic: Proposal: New Rustc Guide Diagrams


Chris Simpkins (Mar 01 2020 at 21:34, on Zulip):

I am working on new graphviz diagrams for the Rustc Overview and pulled together a hackmd document for discussion about other areas of the Guide where we might support the text with additional visuals: https://hackmd.io/24S8FCKTS2mp8zVOdrDjww

The graphviz sources and local build instructions are included. You can also render / explore in real time right in hackmd or on https://sketchviz.com.

Please don't hesitate to edit the current examples, add new proposals, or recommend additions that might be helpful in this thread.

LeSeulArtichaut (Mar 01 2020 at 21:42, on Zulip):

I have recently tried to modify the method resolution part of the compiler, and was really confused, because I had no overall idea of how the system works, especially for diagnostics. So I think it deserves a schema

Santiago Pastorino (Mar 02 2020 at 17:41, on Zulip):

agreed on everything :)

Santiago Pastorino (Mar 02 2020 at 17:42, on Zulip):

the diagrams look great

Santiago Pastorino (Mar 02 2020 at 17:42, on Zulip):

and yeah as @LeSeulArtichaut says this deserves a schema

Chris Simpkins (Mar 02 2020 at 17:44, on Zulip):

@Santiago Pastorino How do you suggest that we approach the work? Edit in that document, reach consensus on how the diagram / visual should appear, then someone submit PR with it on the rustc-guide repo at which point we mark it as done in the hackmd document?

Santiago Pastorino (Mar 02 2020 at 17:47, on Zulip):

yeah I would totally do that

Santiago Pastorino (Mar 02 2020 at 17:48, on Zulip):

first reach consensus, edit the document and send a PR for wider reviews :)

Chris Simpkins (Mar 02 2020 at 17:52, on Zulip):

@LeSeulArtichaut Want to add the schema that you recommended yesterday?

LeSeulArtichaut (Mar 02 2020 at 17:53, on Zulip):

I don't think I have enough knowledge on the topic for that

LeSeulArtichaut (Mar 02 2020 at 17:54, on Zulip):

But I wouldn't mind diving deeper into the code to finally find out

LeSeulArtichaut (Mar 02 2020 at 17:56, on Zulip):

Btw @Chris Simpkins How did you plan to integrate the schemas to the book when they are ready? You will convert them to the SVG format, is this right? If so I think we should keep the source Graphviz graph somewhere on the repo to be able to easily edit them

Chris Simpkins (Mar 02 2020 at 18:01, on Zulip):

LeSeulArtichaut said:

Btw Chris Simpkins How did you plan to integrate the schemas to the book when they are ready? You will convert them to the SVG format, is this right? If so I think we should keep the source graph somewhere on the repo to be able to easily edit them

Haven't planned that far ahead :slight_smile: But yes, that will be straightforward if .DOT sources are acceptable in repo. The sources are currently linked in github gists on the hackmd document. Is there time to discuss this briefly as part of the WG-Learning meeting agenda this week @Santiago Pastorino @mark-i-m ? If this seems like a good objective for one of our initial sprints, perhaps @LeSeulArtichaut (if interested) and I can work out details and pitch them to you

Chris Simpkins (Mar 02 2020 at 18:06, on Zulip):

@LeSeulArtichaut

You will convert them to the SVG format, is this right?

To directly answer your question, graphviz supports many formats (png, jpg, svg, pdf). This is something that we should definitely discuss before the first PR on the repository. I haven't looked through other image types to see what is currently used, file size impact / image quality, raster vs. vector...

LeSeulArtichaut (Mar 02 2020 at 20:21, on Zulip):

I see no reason to not accept .dot files in the repo. Those are text files, which do not carry much information (in the sense that it is Graphviz which converts them to an image), so they are very lightweight.
For example, the Stages of the Compile Process diagram is only 913 chars long, which is more then 7 times lighter than the SUMMARY.md.

LeSeulArtichaut (Mar 02 2020 at 20:22, on Zulip):

In fact, I think the question that stands now is, how do we convert the .dots to exploitable images. Can we do it when building the book, for example? Would it be OK that way? etc...

Chris Simpkins (Mar 02 2020 at 20:46, on Zulip):

LeSeulArtichaut said:

I see no reason to not accept .dot files in the repo. Those are text files, which do not carry much information (in the sense that it is Graphviz which converts them to an image), so they are very lightweight.
For example, the Stages of the Compile Process diagram is only 913 chars long, which is more then 7 times lighter than the SUMMARY.md.

In fact, I think the question that stands now is, how do we convert the .dots to exploitable images. Can we do it when building the book, for example? Would it be OK to add that dependency? etc...

agree with all of these comments.

Chris Simpkins (Mar 02 2020 at 20:50, on Zulip):

we need to know who approves this

LeSeulArtichaut (Mar 02 2020 at 20:52, on Zulip):

I'll dive into some CI research and file size investigations

Chris Simpkins (Mar 02 2020 at 20:53, on Zulip):

I think that the rustc-guide is being manually built and pushed now?

LeSeulArtichaut (Mar 02 2020 at 20:53, on Zulip):

I haven't ever dived into those details actually :slight_smile:

LeSeulArtichaut (Mar 02 2020 at 20:54, on Zulip):

But as this is hosted on GitHub pages I think there is an automated process...?

Chris Simpkins (Mar 02 2020 at 20:54, on Zulip):

It looks like there is a Github Action https://github.com/rust-lang/rustc-guide/deployments

LeSeulArtichaut (Mar 02 2020 at 20:55, on Zulip):

Mhh, this is what I thought

Chris Simpkins (Mar 02 2020 at 21:02, on Zulip):

I don't see any static site build scripting though. There is a CI dir with a shell script for line length linting in MD files

LeSeulArtichaut (Mar 02 2020 at 21:03, on Zulip):

The MD files are transformed into HTML/CSS by mdbook

LeSeulArtichaut (Mar 02 2020 at 21:05, on Zulip):

(https://github.com/rust-lang/mdBook)

Chris Simpkins (Mar 02 2020 at 21:06, on Zulip):

I wonder where the mdbook build workflow lives

LeSeulArtichaut (Mar 02 2020 at 21:07, on Zulip):

You could probably find an answer here

LeSeulArtichaut (Mar 02 2020 at 21:08, on Zulip):

I'm actually peeling the docs too :big_smile:

LeSeulArtichaut (Mar 02 2020 at 21:08, on Zulip):
The build command is used to render your book:

mdbook build

It will try to parse your SUMMARY.md file to understand the structure of your book and fetch the corresponding files.

The rendered output will maintain the same directory structure as the source for convenience. Large books will therefore remain structured when rendered.
Chris Simpkins (Mar 02 2020 at 21:09, on Zulip):

Here are the docs on the book.toml config file: https://rust-lang.github.io/mdBook/format/config.html

LeSeulArtichaut (Mar 02 2020 at 21:09, on Zulip):

Idk if you can launch external processes though

Chris Simpkins (Mar 02 2020 at 21:10, on Zulip):

But I think that the image builds will live outside of the MD -> html builds. We would just need to point to the proper image file paths in the MD sources

LeSeulArtichaut (Mar 02 2020 at 21:10, on Zulip):

I think that's correct

LeSeulArtichaut (Mar 02 2020 at 21:11, on Zulip):

I'll do some local tests

LeSeulArtichaut (Mar 02 2020 at 21:15, on Zulip):

I think I have a plan in mind

LeSeulArtichaut (Mar 02 2020 at 21:15, on Zulip):

We already have src/img

LeSeulArtichaut (Mar 02 2020 at 21:17, on Zulip):

So we basically have two options:
1) We "compile" the .dots manually on every change and store them here
or
2) We "compile" the .dots on the CI build before running mdbook so we don't need to actually store them in the repo

LeSeulArtichaut (Mar 02 2020 at 21:17, on Zulip):

@Santiago Pastorino Thoughts?

Chris Simpkins (Mar 02 2020 at 21:22, on Zulip):

Does rebuilding image files that don't need to be rebuilt because there were no changes in the images interfere with browser image caching?

Santiago Pastorino (Mar 02 2020 at 21:25, on Zulip):

I guess @mark-i-m or @simulacrum may know better here

LeSeulArtichaut (Mar 02 2020 at 21:26, on Zulip):

@Chris Simpkins I don't think mdbook ever deals with the server, thus the cache. It probably is the GitHub server realm, which I think should be OK with proper caching

LeSeulArtichaut (Mar 02 2020 at 21:29, on Zulip):

@Chris Simpkins Heh... https://crates.io/crates/mdbook-graphviz

Chris Simpkins (Mar 02 2020 at 21:31, on Zulip):

@LeSeulArtichaut Ah! Very nice. So we wouldn't need a separate build process for images. We can just use the code in the hackmd Markdown source. That would be ideal. I believe that the only non-graphviz default is the typeface that is used for the nodes at the moment. We would just need to choose one that is supported in the env where the build occurs.

simulacrum (Mar 02 2020 at 21:32, on Zulip):

I would not worry about browser image caches

simulacrum (Mar 02 2020 at 21:32, on Zulip):

(though I have no idea if our caching headers are good enough in practice)

Chris Simpkins (Mar 02 2020 at 21:34, on Zulip):

@simulacrum Thoughts about an automated build approach for new diagram img files in the Rustc Guide? We are pulling together graphviz .dot source files to support it in https://hackmd.io/24S8FCKTS2mp8zVOdrDjww. @LeSeulArtichaut and I are interested in whether it would be appropriate to commit these img sources to the repository and create an automated guide img build approach with them.

simulacrum (Mar 02 2020 at 21:37, on Zulip):

I would prefer the raw sources, though we should avoid if possible a mandatory dependency on non-rust tools (e.g. graphviz) to avoid making contribution on (especially) non Linux platforms harder

simulacrum (Mar 02 2020 at 21:38, on Zulip):

Requiring graphviz or whatever to edit the raw sources is fine though - we may want to commit both the inputs and outputs if that's the case

Chris Simpkins (Mar 02 2020 at 21:39, on Zulip):

That seems to argue for support via CI/CD? Contributors could just edit the graphviz source files and push for updates?

LeSeulArtichaut (Mar 02 2020 at 21:39, on Zulip):

So committing both the raw .dot and the output .svg or .jpg or whatever

LeSeulArtichaut (Mar 02 2020 at 21:39, on Zulip):

Chris Simpkins said:

That seems to argue for support via CI/CD? Contributors could just edit the graphviz source files and push for updates?

You mean, automatically added in a PR?

Chris Simpkins (Mar 02 2020 at 21:42, on Zulip):

Platforms without graphviz availability might have more difficulty generating new images (though there are tools like https://sketchviz.com/ available as web apps to support views of edits). But editing existing ones would simply involve a text editor and PR. The build happens in the CD pipeline

LeSeulArtichaut (Mar 02 2020 at 22:15, on Zulip):

@Chris Simpkins Maybe add those concerns to the proposal HackMD?

mark-i-m (Mar 02 2020 at 23:29, on Zulip):

I'm a bit torn. I don't like committing artifacts, but I agree that we should not make graphviz mandatory to contribute...

mark-i-m (Mar 02 2020 at 23:29, on Zulip):

Regarding the technical aspects, the .travis.yml can be used to invoke a post-test-success script before Travis pushes artifacts to gh pages

Chris Simpkins (Mar 03 2020 at 04:51, on Zulip):

LeSeulArtichaut said:

Chris Simpkins Maybe add those concerns to the proposal HackMD?

Added to this section https://hackmd.io/24S8FCKTS2mp8zVOdrDjww?both#Potential-Issues-with-this-Approach

Last update: Apr 03 2020 at 17:55UTC