rust-lang/rustc-dev-guide

A guide to how rustc works and how to contribute to it.

This is a collaborative effort to build a guide that explains how rustc works. The aim of the guide is to help new contributors get oriented to rustc, as well as to help more experienced folks in figuring out some new part of the compiler that they haven't worked on before.

You can read the latest version of the guide here.

You may also find the rustdocs for the compiler itself useful. Note that these are not intended as a guide; it's recommended that you search for the docs you're looking for instead of reading them top to bottom.

For documentation on developing the standard library, see std-dev-guide.

Contributing to the guide

The guide is useful today, but it has a lot of work still to go.

If you'd like to help improve the guide, we'd love to have you! You can find plenty of issues on the issue tracker. Just post a comment on the issue you would like to work on to make sure that we don't accidentally duplicate work. If you think something is missing, please open an issue about it!

In general, if you don't know how the compiler works, that is not a problem! In that case, what we will do is to schedule a bit of time for you to talk with someone who does know the code, or who wants to pair with you and figure it out. Then you can work on writing up what you learned.

In general, when writing about a particular part of the compiler's code, we recommend that you link to the relevant parts of the rustc rustdocs.

Build Instructions

To build a local static HTML site, install mdbook with:

> cargo install mdbook mdbook-linkcheck mdbook-toc mdbook-mermaid

and execute the following command in the root of the repository:

> mdbook build --open

The build files are found in the book/html directory.

Link Validations

We use mdbook-linkcheck to validate URLs included in our documentation. linkcheck will be run automatically when you build with the instructions in the section above.

Table of Contents

We use mdbook-toc to auto-generate TOCs for long sections. You can invoke the preprocessor by including the <!-- toc --> marker at the place where you want the TOC.

How to fix toolstate failures

NOTE: Currently, we do not track the rustc-dev-guide toolstate due to spurious failures, but we leave these instructions for when we do it again in the future.

1. You will get a ping from the toolstate commit. e.g. https://github.com/rust-lang-nursery/rust-toolstate/commit/8ffa0e4c30ac9ba8546b7046e5c4ccc2b96ebdd4

2. The commit contains a link to the PR that caused the breakage. e.g. rust-lang/rust#64321

3. If you go to that PR's thread, there is a post from bors with a link to the CI status: rust-lang/rust#64321 (comment)

4. Follow the check-actions link to get to the Actions page for that build

5. There will be approximately 1 billion different jobs for the build. They are for different configurations and platforms. The rustc-dev-guide build only runs on the Linux x86_64-gnu-tools job. So click on that job in the list, which is about 60% down in the list.

6. Click the Run build step in the job to get the console log for the step.

7. Click on the log and Ctrl-f to get a search box in the log

8. Search for rustc-dev-guide. This gets you to the place where the links are checked. It is usually ~11K lines into the log.

9. Look at the links in the log near that point in the log

10. Fix those links in the rustc-dev-guide (by making a PR in the rustc-dev-guide repo)

11. Make a PR on the rust-lang/rust repo to update the rustc-dev-guide git submodule in src/docs/rustc-dev-guide. To make a PR, the following steps are useful.

# Assuming you already cloned the rust-lang/rust repo and you're in the correct directory
git submodule update --remote src/doc/rustc-dev-guide
git add -u
git commit -m "Update rustc-dev-guide"
# Note that you can use -i, which is short for --incremental, in the following command
./x test --incremental src/doc/rustc-dev-guide # This is optional and should succeed anyway
# Open a PR in rust-lang/rust

12. Wait for PR to merge

Voilà!

rustc-dev-guide's Issues

'print this book' button doesn't work

Hi, I believe the 'print this book' button in the top right corner of the documentation doesn't work. It only prints a screenshot of the current page instead of producing a concatenated document of all the pages in the book.

there is some misunderstood parts

In https://github.com/rust-lang-nursery/rustc-guide/blob/master/src/hir.md there is a sentence about "It is a desugared version of the abstract syntax tree"

I am not a native English speaker. So I just understand it in my mother language. When I tried to translate this sentence into Chinese which is my mother language. This looks like saying that HIR is a kind of abstract syntax tree.

If this sentence does not cause ambiguity in English, please close and ignore this issue.

"name resolution"

We should be talking about the librustc_resolve crate works here. Some topics of interest:

• Handling scopes within the crate
  • What is a rib? (#22)
  • Where do ribs get added / removed, what happens if I need to introduce a binder? (#22)
  • How does pub use and stuff like that work
  • Globs and ambiguity
• How do names from external crates fit in?

Writing mentored issues

Not sure if this is the right fit for this book, it relates more to the process of contributing rather than the technical details. Would it be useful to have a section in the guide for writing mentored issues?

The intention being that experienced contributors who are just getting started writing mentored issues would be able to review it and structure their mentoring instructions in as helpful and useful a way as possible and that the quality of the instruction would then be consistent?

It would contain things that mentors have found beneficial to include in mentoring instuctions; or things those that have done some mentored issues found useful or not so useful; examples of good and bad instruction?

Edit: only just now noticed that mentoring is mentioned in #25 so this might be a dupe.

Line wrap convention?

What's the line wrap convention? I initially guessed 80 chars max, but then it seems like a) a few lines overflow (I'd guess these were oversights), b) many lines wrap before they need to (I'd guess intentionally). I can't figure out the logic for b) in particular, since no consistent scheme seems to be used.

Personally, I'm not a big fan of hard wrapping in markup files (or code, to a lesser extent, though I know that's more contentious!). Should we be doing wrapping at all? If yes, let's document a consistent style, I say.

Bisecting errors against PRs

One of the best way of figuring out the cause of a regression is bisecting against the PRs that have landed recently. There are a number of folks (cc'd below) who do this from time to time -- I'm not really sure what wizardry they use to do it, but I'd love to see it documented!

cc @Mark-Simulacrum, @kennytm, @TimNN, @est31

Link to internal rustdocs

When rust-lang/rust-central-station#40 lands we'll have a stable location which contains internal API documentation (i.e. the equivalent of ./x.py doc with crate-docs = true) 🎉

This means when referring to a particular struct or function we'll be able too link to its API docs instead of linking to the original source file and hoping for the best.

Once that PR is merged we'll need to:

• Update the code index to point at the API docs instead of source code
• Emphasise linking to API docs where possible when people are contributing new content
• Update the "contributing" section of the README appropriately (#208)
• Update the "How to build the compiler and run what you built" to mention building docs and where they can be found online?

CI using old cache of mdbook

I'm not 100% certain without access to the Travis cache, but I believe that it is using an old version of mdbook (0.0.28) from the cache. This was noticed because Travis is not running mdbook-linkcheck, and the output in the Travis logs looks like the old version.

In the Travis logs, it looks like PRs are picking up this old cache from the master branch:

fetching master/cache-linux-trusty-e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855--cargo-stable.tgz
found cache

Due to the way ci/install.sh is written, it will skip if mdbook is already installed, but it does not check which version is already installed.

A quick short-term solution is to delete the travis caches.

I can make a PR to change the install.sh script to force a reinstall if there is a version mismatch. Alternatively, you can just manually delete the travis caches whenever you upgrade mdbook.

"Standard library (libstd, liballoc, libcore, libtest) guide"

Some main points raised by rust-lang/rust#50466:

• How to build and test the library without needing to rebuild the compiler every time.

  • developing from stage0: one could use ./x.py test --stage 0 --no-doc src/libstd to quickly build and test the standard library, if features from a pre-built compiler is sufficient.
  • developing from stage1 (e.g. a new intrinsic is added): Suggest the proper --keep-stage syntax so the compiler will only be built once and then kept frozen?

• Code arrangement

  • For libcore the tests must be doc tests or integrated tests in src/libcore/tests/

./x.py check

So I've recently discovered that ./x.py check is actually really dang fast -- we should add a note about it in the "how to build compiler" section. It's in particular very useful when you're doing some kind of "type-based refactoring", like renaming a method, or changing the signature of some function.

More information on attributes

I recently wrote a clippy lint that deals with attributes (mostly outer attributes) and found very little information on how they are implemented. Most of the information I gathered through PR comments and issues in the rust repo. The reference only explains how to use the different attributes.

Do you think it would make sense to have a chapter on attributes in the guide?

Some weird things and questions I came across:

• The AST seems to contain attributes that are not in the actual source code (Is this because of macro expansion?) (rust-lang/rust-clippy#2340 (comment))
• some expanded attributes seem to have an empty TokenStream and some not
• cfg_attr is special somehow
• In general lots of inconsistencies in the data from the syntax crate?

I would be happy to start something myself, but would need some pointers as to where the attribute stuff happens.

advice for debugging incremental compilation hashing failures etc

Over in rust-lang/rust#47935, @michaelwoerister wrote:

The [dependency tracking] readme is largely outdated. The "Testing the dependency graph" and "Debugging the dependency graph" sections are still correct, I think, but the rest is probably just confusing at this point.

I think we ought o move some of those sections over, and/or add some new material on how to debug incremental compilation failures. This is always something I have to kind of rediscover.

setup repo to automatically render

It'd be nice if, after each push, you could go visit a GH pages or something and see the output, so that people can go read the current contents. If you're interested in doing this, leave a comment! (If you need collaborator status, I can add you.)

Switching Out LLVM Backends

Recently support landed in rustc for dynamically loading LLVM. It would be nice if the rustc-guide documented how to switch to a custom LLVM backend. I'm not sure if this was planned to be included in the trans chapter(#6), as it does not yet have a tracking issue I could comment on.

Sorry if this is out of scope for the rustc-guide!

An example where people ask for this documentation.

Cheers

document environment variables used when building the compiler

I am a newbie to compiler development and came across a pretty big documentation gap: environment variables.

I ran the following script to try and pull out all the existing environment variables. I can run it on whatever directories that project contributors want me to. I would like to document these if possible!

Action items for someone more knowledgeable than me:

• I am running against src/lib* src/bootstrap. Are there other directories I should be searching?
• I am searching for env::var(_os) (rust) and os.environ.get (python). Are there other regexes that should be searched?
• Please provide feedback on which of these should be better documented, and if you have a one line "off the top of my head" documentation feel free to write it here.
• Does stage0 complicate things, and we need to be mindful of what environment variables are set when compiling the compiler?
• I think these should probably go in the Configuration section. Let me know if you have a better spot in mind.

rg '^.*?((env::var(_os)?)|(os\.environ\.get))\(([\w"]+)\).*?$' \
    -r '$5' -N --no-filename \
    src/lib* src/bootstrap \
    | sort -u

Output

"APPVEYOR"                                                                                                                                                                                                 
"AR"                    
"BOOTSTRAP_PARENT_ID"
"BOOTSTRAP_PYTHON"     
"BUILD"                                
"CARGO_CFG_TARGET_ARCH"
"CARGO_CFG_TARGET_ENV"
"CARGO_CFG_TARGET_OS"
"CARGO_CFG_TARGET_VENDOR"
"CARGO_TARGET_DIR"     
"CFG_COMPILER_HOST_TRIPLE"
"CFLAGS"            
"DESTDIR"                       
"HOME"                        
"HOST"               
"JEMALLOC_OVERRIDE"
key            
"L4RE_LIBDIR"        
libdir         
"LLVM_CONFIG"  
"LLVM_LINK_SHARED"
"LLVM_RUSTLLVM"        
"LLVM_STATIC_STDCPP"   
"MACOSX_DEPLOYMENT_TARGET"
"MACOSX_STD_DEPLOYMENT_TARGET"
"MAKEFLAGS"     
"MSYSCON"          
"MUSL_ROOT"             
"NUM_JOBS"                 
"OUT_DIR"              
"PAGER"                                
"PATH"                
"RUST_BACKTRACE"   
rustc              
"RUSTC_BOOTSTRAP"
"RUSTC_CODEGEN_UNITS"  
"RUSTC_COLOR"            
"RUSTC_CRT_STATIC"
"RUSTC_DEBUG_ASSERTIONS"        
"RUSTC_DEBUGINFO"
"RUSTC_DEBUGINFO_LINES"
"RUSTC_FORCE_INCR_COMP_ARTIFACT_HEADER"
"RUSTC_FORCE_UNSTABLE"
"RUSTC_HOST_LINKER"  
"RUSTC_INCREMENTAL"
"RUSTC_LIBDIR" 
"RUSTC_METADATA_SUFFIX"
"RUSTC_NO_PREFER_DYNAMIC"
"RUSTC_ON_FAIL"        
"RUSTC_RETRY_LINKER_ON_SEGFAULT"
"RUSTC_RPATH"
"RUSTC_SAVE_ANALYSIS"
"RUSTC_STAGE"
"RUSTC_SYSROOT"
"RUSTC_TARGET_LINKER"
"RUSTC_THINLTO"
"RUSTC_VERBOSE"
"RUST_DEP_GRAPH"
"RUST_DEP_GRAPH_FILTER"
"RUSTDOC_CRATE_VERSION"
"RUSTDOC_REAL"   
"RUST_FORBID_DEP_GRAPH_EDGE"
"RUST_MIN_STACK"
"RUST_REGION_GRAPH"
"RUST_REGION_GRAPH_NODE"
"RUST_SAVE_ANALYSIS_CONFIG"
"RUST_TARGET_PATH"     
"RUST_TEST_NOCAPTURE"                  
"RUST_TEST_STRESS"    
"RUST_TEST_THREADS"
"SCCACHE_ERROR_LOG"
"SCCACHE_PATH"
"SCCACHE_TARGET"       
"SRC"                    
"SUDO_USER"    
"TARGET"                        
"TERM"       
"TERMINFO"           
"TERMINFO_DIRS"
"TEST_DEVICE_ADDR"
"TEST_MIRI"          
"TMPDIR"       
"TRAVIS"       
"USER"          
"USERPROFILE"          
var                    
"WIX"       

Glossary alphabetical

I just noticed that the glossary is not perfectly in alpha order... Note to self: fix it when I get back to a computer...

"Navigating the compiler from the outside"

I'm currently working on a project which uses rustc internals to analyse a crate so I can then generate FFI bindings. Kinda like cbindgen, except where they focus on static linking at compile time or dynamically when your program gets loaded into memory, I'm trying to load a dynamic library at runtime using dlopen (*nix) and LoadLibrary (windows).

Seeing as this is a guide on how rustc works under the hood, would it be worth adding a chapter that mentions:

• how people can use rustc internals as a library,
• how you ask rustc_driver to run some code after a particular phase, and
• how you can extract information from an analysed crate?

My code is on GitLab so people in a similar situation or writing this chapter can use that as a reference.

---

This is slightly off topic, but I'm really impressed with how easy the compiler internals (especially the HIR and Session) are to work with once you understand how they're designed! It was initially a bit confusing because there are loads of data structures (unfortunately, not many have doc-comments) and lots of complexity, but after an hour or two of jumping around the crate docs I was able to be quite proficient in finding the information I want.

"Conventions used in the compiler"

Some of this material exists already scattered about, and/or maybe this overlaps with the glossary, but I think it'd be helpful to try and capture some common conventions.

Here are some things I'm thinking about:

• How to format code (#47)
  • Um... kind of a mess. Hopefully we'll move to rustfmt.
• Naming conventions (maybe this should just be in the glossary, actually?)
  • the cx suffix -- short for "context"
  • 'tcx and 'gcx -- mostly direct elsewhere for detailed explanation?
• When to add/avoid adding dependencies from crates.io? What is the vetting process?
  • Whitelisting?
• What else?

move over the rustc README

There is a lot of existing content scattered in READMEs through rustc. We should move over that content. To start, the rustc README is a good candiate. It desribes the layout of the crates and contains a glossary (oh, we should probably add the glossary to its own chapter).

EDIT(@mark-i-m): Added TODOs

EDIT2(@mark-i-m): A lot of the content of the READMEs is redundant with content we already have, so please check that the content you are moving over is not already in the appropriate chapter. If it is, please post here, so we can check it off the list 😄

TODOs:

• Overall arch: https://github.com/rust-lang/rust/blob/master/src/librustc/README.md
• Dependency tracking/Incr. Compl.: https://github.com/rust-lang/rust/blob/master/src/librustc/dep_graph/README.md
  • Actually, I think the chapter in this book is more up-to-date than the README, which doesn't have red-green in it... There is some info in the READMEs, such as testing and debugging, that is not TMK in the book yet. - @mark-i-m
  • See rust-lang/rust#47935
  • See #87
  • See #92
• HIR: https://github.com/rust-lang/rust/blob/master/src/librustc/hir/README.md
• Type Inference: https://github.com/rust-lang/rust/blob/master/src/librustc/infer/README.md (#31)
• MIR: https://github.com/rust-lang/rust/blob/master/src/librustc/mir/README.md (#36)
• Trait resolution: https://github.com/rust-lang/rust/blob/master/src/librustc/traits/README.md
• The ty module: https://github.com/rust-lang/rust/blob/master/src/librustc/ty/README.md
• ./src/librustc/infer/region_constraints/README.md
  • I'm not sure we want to move this one over since it will soon be obsolete. #67 describes the new system (NLL). (Agreed -@nikomatsakis)
• ./src/librustc/infer/lexical_region_resolve/README.md
  • Same here (Agreed -@nikomatsakis)
• ./src/librustc/infer/higher_ranked/README.md
  • Skolemization (#67) -- This material is different from what is covered in the rustc-guide, because the rustc-guide is describing the newer system we are aiming to add, and this is covering the older system we currently use. (-@nikomatsakis)
  • LUB/GLB -- same here (-@nikomatsakis)
• ./src/librustc/ty/maps/README.md
• ./src/librustc_data_structures/obligation_forest/README.md
  • (I probably wouldn't move this over either. -@nikomatsakis)
• ./src/librustc_typeck/variance/README.md (#85)
• ./src/librustc_typeck/check/method/README.md (#85)
• ./src/librustc_typeck/README.md (#85)
• ./src/librustc_borrowck/borrowck/README.md
  • This too is basically obsolete. The NLL RFC offers better coverage of the new-and-shiny borrow checker, and we should add more of that material, I think. -@nikomatsakis
• Rustdoc README from rust-lang/rust#48283 (#86 )

Expand descriptions of the various compilation stages

This is what we currently have: https://github.com/rust-lang-nursery/rustc-guide/blob/46864ca9714707f18e658afba9d2ea62afbe2374/src/how-to-build-and-run.md#running-xpy-and-building-a-stage1-compiler

There's a relevant thread on reddit about what these stages all mean and I think the descriptions in this answer could be incorporated into this section in the rustc guide

Profiling the Compiler

It would be nice to have a section on how to profile the compiler, what tools to use, how to invoke them, and maybe a section on how to interpret results.

include a section on how the compiler team functions

We should talk about how the compiler team itself functions:

• Chatting typically in #rustc on IRC
• Open-to-public triage meetings in #rustc on IRC at 10am Boston time on Thursday
  • Careful around US daylight savings time :/
• How to find open issues
• Mentoring and how to ask for it (#64)

Parallel rustc

As @Zoxc's work to [parallelize rustc] starts to land, we need to start documenting it!

Some example topics:

• Copy some of the text from the internals thread as to the overall strategy
• What the heck is Lrc? When should I be using it?
• What do I use when I want a ref cell?

Chapter on MIRI?

... perhaps as a subchapter under the MIR chapter?

"macro expansion"

Talk about where macro expansion happens. There is lots of other stuff we should cover here eventually:

• Hygiene
• Custom derive and procedural macros and how those work

cc @jseyfried -- can you help on elaborating the list of things that belong here?

"Debugging the compiler"

This section should talk about the various tips and tricks for debugging the compiler, as well as perhaps some amount of "debug log conventions". It doesn't necessarily have to be a super long section, but here are some of the things I can think of:

• How to use RUST_LOG and config.toml first and foremost (overlaps with #8)
• Some of the handy -Zflags:
  • example: -Ztreat-err-as-bug, which causes errors to be reported as bugs. Useful when you are debugging an error that should not be happening because you can use RUST_BACKTRACE=1 then to get a backtrace of where the error was reported; also avoids dumping out more logs than you need.
  • example: -Zunpretty=hir-tree, which will dump out the HIR
  • example: -Zverbose, which dumps out extra
  • see -Zhelp for more
• Conventions for logs ("loosely followed")
  • Use {:?} most of the time in debug logs
  • I tend to use debug!("foo(...)") at the start of a function foo and debug!("foo: ...") or logs within that function (- nikomatsakis)
• Options to try and correlate node-ids and def-ids with the source
  • e.g., --pretty expanded,identified
• Options to dump the MIR
  • -Zdump-mir and -Zdump-mir-graphviz

What are "filemaps"?

They're mentioned on https://rust-lang-nursery.github.io/rustc-guide/the-parser.html, but not defined. I think it would be good to explain what they are first.

chapter explaining how `Kind` works

We don't explain Kind at all!

cc @varkor -- want to write it =)

Finish chapters on macro expansion?

Could we get someone to do this? I'm not sure who would be most appropriate. @jseyfried knows it better than most, but he's been absent lately.

"Using the compiler testing framework"

This section should describe how to compiler is tested. Substantial work has been done but there remains more to document:

• discuss bors, travis, platform emulation (#66 )
• crater: what is it and why is it useful? How do you request a crater run and when should you to do so? (#151)
• maybe talk about "perf runs" too (#151)
• rustdoc tests (cc @QuietMisdreavus)
• talk about unit tests from libs and things (#62)
• Doctests: How to run the doc tests in the standard library. Other doc testing steps like linkchecker. (#62)
• distcheck (#62)
• mention that tests also get run for individual packages, such as bootstrap, cargo, rls, rustfmt, miri, clippy, cargotest. (#62)

documentation about feature gates

I find myself referencing this forge article a lot:

https://forge.rust-lang.org/feature-guide.html
EDIT (mark-i-m): this moved to https://rust-lang.github.io/rustc-guide/implementing_new_features.html

It's ok, but not great. Some things I would like to see covered in rustc-guide instead (and then perhaps remove that article):

• how to add a feature gate in the code
  • how to add a test covering your feature gate
• how to change for simple syntactic things
• how to do more advanced gates by looking at sess.features.your_feature_here as a boolean

master tracking issue

This tracking issue contains links to the issues for all the chapters and subsections of the book. It is also the main place to come to if you're looking to get started on contributing.

How to contribute

Skim down the list below and find a chapter that looks interesting to you. It should have a link to an associated tracking issue. Click on that link and leave a comment there saying you'd like to help out (and/or start writing something).

In general, if you don't know how the compiler works, that is not a problem! In that case, what we will do is to schedule a bit of time for you to talk with someone who does know the code, or who wants to pair with you and figure it out. Then you can work on writing up what you learned. Just cc @nikomatsakis for help.

If you think there is a missing section please file an issue and we will consider how to fit into this outline.

The outline

• How to build the compiler and run what you built (#8)
• Walkthrough: a typical contribution (#237)
• Using the compiler testing framework (#10)
• Debugging the compiler (#11)
• Conventions used in the compiler (#12)
• The parser (#13)
• Diagnostics: reporting errors and lints (#14)
• Macro expansion (#15)
• Name resolution (#16)
• HIR and HIR lowering (#17)
• Representing types (ty module in depth)
• Type inference
• Trait resolution (#52, #70 )
• Type checking
• MIR construction
• MIR borrowck (#67 )
• MIR optimizations
• MIRI (#30, #45)
• trans: generating LLVM IR (#89)
• Glossary

deployment stopped working

I don't really know why but travis is failing, giving a kind of generic message:

Couldn't push the build to github.com/rust-lang-nursery/rustc-guide.git:gh-pages

Bah humbug. I hate this stuff. Anybody got a clue? I can give permissions as needed if think you know how to fix =)

How to use CTAGS

One of the challenges with rustc is that the RLS can't handle it, making code navigation difficult. I personally use ctags; I use this script to set it up:

https://github.com/nikomatsakis/rust-etags

CTAGS integrate into emacs + vim quite easily, I don't know about other editors. I typically have this common as my standard "build rustc" command:

rust-ctags src/lib* && ./x.py build ...

This allows me to do "jump-to-def" with whatever functions were around when I last built .This is ridiculously useful.

Some setup instructions should be added to the "getting started" chapter I think.

"HIR and HIR Lowering"

Talk about the HIR:

• how it is structured,
• how to dump it and inspect it,
• the code that creates it (#148)
• the HIR map

I can not get the hir-tree

When I:

cargo rustc -- -Zunpretty=hir-tree

I got:

error: unknown debugging option: `unpretty`

Glossary

Are there any other terms that should be added to the glossary? Please make suggestions.

• filemap (#35)
• soundness (#54)
• completeness (#54)
• skolemization (#67)
• sigil (#57)
• LTO, ThinLTO, FatLTO (#56)
• TLS (#56)
• LLVM (#56)
• IR (#56)
• Fat pointer (#112)
• region (#57)
• HIR map (#63)
• CTFE (compile-time function evaluation)... This just needs a link to the const eval/miri sections (#152)
• rib (see name resolution chapter) (#152)
• upvar (I believe an "upvar" is just a variable captured by a closure) (#152)
• early-bound lifetime (#281)
• interned (#281)
• item (#281)
• late-bound lifetime (#281)
• lang item (#281)
• memoized (#281)
• string interning (#281)
• drop glue (#281)
• late-bound regions/lifetimes (#281)
• gensym
• landing pad
• skolemization leak

Automated Link Checking

I'm assuming this guide is going to contain lots of links to various bits and pieces around the rust-lang/rust repository, so it's probably a good idea to set up some sort of link checking in CI.

I've made a backend for mdbook which will read your book and check all the links are valid called mdbook-linkcheck. We'll be making another release in the next couple days (rust-lang/mdBook#558) which will include alternate backends and allow us to use mdbook-linkcheck as part of the usual dev process.

This issue is mostly just a reminder for myself. When the next version of mdbook is up on crates.io I'll come back and make a PR including mdbook-linkcheck in CI.

Add ICH to the glossary

Follow-up from rust-lang/rust#47782. @avborhanian could you submit a PR here as well? Thanks!

Chapter 2 "How to build the compiler..." Assumes Knowledge of Compiler Stages

Chapter 2 assumes you understand why there are multiple compiler stages and what each is used for. A short section explaining this would make the chapter much clearer.

"The parser"

It'd be worth covering

• where the lexer is
• how the parser works
• conventions around error recovery
• - What is this eat function?
  • It consumes a token, and queues up the next token as lookahead
• How do you handle identifiers and keywords?
• What about constructing spans?
  • (quoting Niko from below) I have to bring this back into cache, but in general there is this last_span thing that stores the span of the last token, and many things follow the idiom of saving the "lo" point of the span, parsing some stuff, then extracting the "hi" point and combining them. This would be used to make a span that encompasses, for example, an entire trait definition (the lo point would come from the trait keyword, but the end point comes after having parsed a bunch of other things).
• What is this crazy pretty printer?
  • It dumps out the AST in human readable form.
  • We have tests that will parse each file in run-pass, pretty print it, and then test that the resulting pretty printed file type checks, so you have to implement this.
• Something like this: #13 (comment) and #13 (comment)

document where various bits of rust are desugared

It's been mentioned to me that it would be useful to have a list of all the "desugarings" that occur during the compilation process. In particular, which bits of Rust disappear during e.g. HIR conversion, HAIR conversion, MIR conversion. Here are some notes, I'd love to see someone turn some of this into a more detailed writeup! If you're interested, ping me here or (better) on gitter/IRC.

Desugaring that occur during HIR lowering

While constructing the HIR, we perform the following desugarings. It seems worth documenting these at a high-level (e.g, showing the equivalent Rust code), as well as covering some of the mechanisms we use to do it (e.g., the special spans we construct).

• the ? operator and catch blocks
  • desugared during HIR lowering, as explained here
• the for loop
  • desugared during HIR lowering
• the if let and while let forms
  • desugared during HIR lowering (if let, while let

Desugaring that occur during MIR lowering

While lowering to MIR, we perform the following desugarings:

• match expressions
  • patterns are desugared during HAIR construction into simpler form
    • this for example introduces explicit & patterns to implement default binding modes
  • then, desugared during MIR lowering
• calls
• control-flow into control-flow-graph

Document common manipulations and operations

While playing around using rustc internals as a library I found it was pretty difficult to discover how to do a bunch of "common" operations. We might be able to improve discoverability by documenting these in an appendix.

Off the top of my head, it'd be nice to know how to:

• Get the fully qualified name of a type given its HIR representation and location (e.g. c_char -> libc::c_char or std::os::raw::c_char, depending on context)
• Get a Path for a given Item
• Find the DefId of a function or struct declaration
• How to pretty-print a function declaration

Feel free to add other "common" questions or shortcuts in the comments and I'll update this list accordingly.

CC: #74

"Diagnostics: reporting errors and lints"

We should talk first about:

• Guidelines for emitting errors
  • How to find an error code and add an extended error
  • Conventions are how the error should be formatted (lowercase letter, etc)
    • link or maybe just inline the RFC with guidelines on how extended errors should look
  • when to use span_label etc
• How to emit lints
  • When to use a lint
  • If you have a tcx
  • Before you have a tcx
  • Some tips here at the end of Rustc bug fix procedure

EDIT(mark-i-m): See #14 (comment)

cc @estebank

"How to build the compiler and run what you built"

This section should describe how to build the compiler and run what you built. There is plenty of existing material that we can draw on. For example, the build system section of the CONTRIBUTING.md file covers similar material.

One thing that I think is missing from existing material is a focus on "practical recipes". Here are the highlights I would mention:

config.toml

First thing first, create a config.toml with the following options enabled:
- debug-assertions = true -- enables debug logs (as well as additional assertions)
- debuginfo-lines = true -- get line-number information
- use-jemalloc = false -- this means that tools like valgrind work better; it also decreases peak memory usage
- (any other suggestions? perhaps something to enable LLVM assertions, if that is needed now?)

• Some options you may want:
  • ccache = true -- ❓ helps speed up builds, you may need to install ccache first though
  • debug = true -- ❓ I turn this one, but really all that is useful I think is line number information

How to run compiler and enable debug logs

• I would recommend setting up a rustup toolchain for stage1 and stage2
  • that means you can do rustc +stage1 ... and rustc +stage2 ... to run the compiler you've built
• Talk about how to use the RUST_LOG environment variable
  • example: RUST_LOG=rustc::infer rustc +stage1 foo.rs >& killme will dump all the debug logs from the the infer module of the rustc crate.
• Mention that there are -Z flags useful for debugging and others things (and direct to #11)

Useful commands and targets for ./x.py:

./x.py build

Builds everything. This is very slow, and you won't want to run it a lot. Ultimately this builds the compiler with the beta compiler -- kind of roughly giving you the nightly compiler. It then builds the compiler again, using that nightly compiler (to check that it can build itself, and to resolve some annoying issues around linking).

./x.py build --stage 1 src/libstd or ./x.py build --incremental src/libstd

These commands will build a working stage1 rustc and standard library. The output can be found in build/<target>/stage1/bin/rustc. It's the fastest way to get a working compiler.

It may be somewhat surprising that this command builds the compiler, since it says to build src/libstd (the standard library). The reason for this is that we are building the compiler once (the stage1 compiler) and then using that compiler to build the stage1 standard library. This will be the standard library that your compiled programs will use.

I guess we can omit the ./x.py test routines for the next section.

What else?

Maybe leave some thoughts in the comments below on things you would have liked to know?

add notes about `rustdoc-ui` and `rustdoc-js` tests

While reviewing rust-lang/rust#50302 i noticed that these test folders aren't mentioned in the testing chapter at all. Since the linked PR adds a special configuration flag to the rustdoc-js tests, it would be worth mentioning it in the testing guide.

Codegen: LLVM IR, Monomorphization, Codegen Units, Partitioning, Symbol Linkage and Visibility

Quite a few complicated things are going on during trans. Would be great to have it documented.

• Generating LLVM IR
  • Monomorphization
• Codegen Units
• Partitioning
• Symbol Linkage and Visibiliy

Code index

One thing that could be helpful is an index of important data structures, functions, methods, traits, etc in the compiler. I propose we add a chapter after the glossary with a big table (sorted alphabetically by item name) that looks something like this:

Item	Kind	Short description	Chapter	Declaration
foo	function	used by the parser to keep track bar	Parsing	src/libsyntax/foo.rs

TODO:

• TyCtxt (#55)
• Ty (#85)
• DefId, HirId, NodeId, BodyId (#153 )
• CodeMap (#69)
• StringReader (#72)
• ParseSess (#68)
• Crate (#76)
• TokenStream (#153)
• Rib (#153)
• ParamEnv (#153)
• TraitRef (#153)
• Span (#139)
• P<T> (as mentioned in #182) (#275)
• FileMap
• Parser

Downloading artifacts from Rust's CI

There is an awesome utility by @kennytm with the short and pithy name rustup-toolchain-install-master. What this does is to download the artifacts produced by Rust's CI for a specific SHA1 -- this basically corresponds to the successful landing of some PR -- and then sets them up for your local use. (@kennytm -- does this include @bors try artifacts as well?) This is helpful when you want to examine the resulting build of a PR without doing the build yourself.

We should mention this tool somewhere, it's super useful!

Links don't work on github

as pointed out in #100 (comment)

We could make all of the links in the book absolute (always point to the guide) or we could leave them relative and just tell people to use the hosted mdbook version, rather than github... My personal leaning is towards the latter because it seems cleaner (easier to move the book later, easier to write, etc.)...

@nikomatsakis @Michael-F-Bryan Any thoughts?

cc @rajcspsg