rust-lang / rustc-dev-guide Goto Github PK
View Code? Open in Web Editor NEWA guide to how rustc works and how to contribute to it.
Home Page: https://rustc-dev-guide.rust-lang.org
License: Apache License 2.0
A guide to how rustc works and how to contribute to it.
Home Page: https://rustc-dev-guide.rust-lang.org
License: Apache License 2.0
This is what we currently have: https://github.com/rust-lang-nursery/rustc-guide/blob/46864ca9714707f18e658afba9d2ea62afbe2374/src/how-to-build-and-run.md#running-xpy-and-building-a-stage1-compiler
There's a relevant thread on reddit about what these stages all mean and I think the descriptions in this answer could be incorporated into this section in the rustc guide
This section should describe how to compiler is tested. Substantial work has been done but there remains more to document:
It'd be worth covering
eat
function?
last_span
thing that stores the span of the last token, and many things follow the idiom of saving the "lo" point of the span, parsing some stuff, then extracting the "hi" point and combining them. This would be used to make a span that encompasses, for example, an entire trait
definition (the lo point would come from the trait
keyword, but the end point comes after having parsed a bunch of other things).I find myself referencing this forge article a lot:
https://forge.rust-lang.org/feature-guide.html
EDIT (mark-i-m): this moved to https://rust-lang.github.io/rustc-guide/implementing_new_features.html
It's ok, but not great. Some things I would like to see covered in rustc-guide instead (and then perhaps remove that article):
sess.features.your_feature_here
as a booleanI am a newbie to compiler development and came across a pretty big documentation gap: environment variables.
I ran the following script to try and pull out all the existing environment variables. I can run it on whatever directories that project contributors want me to. I would like to document these if possible!
Action items for someone more knowledgeable than me:
src/lib* src/bootstrap
. Are there other directories I should be searching?env::var(_os)
(rust) and os.environ.get
(python). Are there other regexes that should be searched?rg '^.*?((env::var(_os)?)|(os\.environ\.get))\(([\w"]+)\).*?$' \
-r '$5' -N --no-filename \
src/lib* src/bootstrap \
| sort -u
"APPVEYOR"
"AR"
"BOOTSTRAP_PARENT_ID"
"BOOTSTRAP_PYTHON"
"BUILD"
"CARGO_CFG_TARGET_ARCH"
"CARGO_CFG_TARGET_ENV"
"CARGO_CFG_TARGET_OS"
"CARGO_CFG_TARGET_VENDOR"
"CARGO_TARGET_DIR"
"CFG_COMPILER_HOST_TRIPLE"
"CFLAGS"
"DESTDIR"
"HOME"
"HOST"
"JEMALLOC_OVERRIDE"
key
"L4RE_LIBDIR"
libdir
"LLVM_CONFIG"
"LLVM_LINK_SHARED"
"LLVM_RUSTLLVM"
"LLVM_STATIC_STDCPP"
"MACOSX_DEPLOYMENT_TARGET"
"MACOSX_STD_DEPLOYMENT_TARGET"
"MAKEFLAGS"
"MSYSCON"
"MUSL_ROOT"
"NUM_JOBS"
"OUT_DIR"
"PAGER"
"PATH"
"RUST_BACKTRACE"
rustc
"RUSTC_BOOTSTRAP"
"RUSTC_CODEGEN_UNITS"
"RUSTC_COLOR"
"RUSTC_CRT_STATIC"
"RUSTC_DEBUG_ASSERTIONS"
"RUSTC_DEBUGINFO"
"RUSTC_DEBUGINFO_LINES"
"RUSTC_FORCE_INCR_COMP_ARTIFACT_HEADER"
"RUSTC_FORCE_UNSTABLE"
"RUSTC_HOST_LINKER"
"RUSTC_INCREMENTAL"
"RUSTC_LIBDIR"
"RUSTC_METADATA_SUFFIX"
"RUSTC_NO_PREFER_DYNAMIC"
"RUSTC_ON_FAIL"
"RUSTC_RETRY_LINKER_ON_SEGFAULT"
"RUSTC_RPATH"
"RUSTC_SAVE_ANALYSIS"
"RUSTC_STAGE"
"RUSTC_SYSROOT"
"RUSTC_TARGET_LINKER"
"RUSTC_THINLTO"
"RUSTC_VERBOSE"
"RUST_DEP_GRAPH"
"RUST_DEP_GRAPH_FILTER"
"RUSTDOC_CRATE_VERSION"
"RUSTDOC_REAL"
"RUST_FORBID_DEP_GRAPH_EDGE"
"RUST_MIN_STACK"
"RUST_REGION_GRAPH"
"RUST_REGION_GRAPH_NODE"
"RUST_SAVE_ANALYSIS_CONFIG"
"RUST_TARGET_PATH"
"RUST_TEST_NOCAPTURE"
"RUST_TEST_STRESS"
"RUST_TEST_THREADS"
"SCCACHE_ERROR_LOG"
"SCCACHE_PATH"
"SCCACHE_TARGET"
"SRC"
"SUDO_USER"
"TARGET"
"TERM"
"TERMINFO"
"TERMINFO_DIRS"
"TEST_DEVICE_ADDR"
"TEST_MIRI"
"TMPDIR"
"TRAVIS"
"USER"
"USERPROFILE"
var
"WIX"
There is a lot of existing content scattered in READMEs through rustc. We should move over that content. To start, the rustc README is a good candiate. It desribes the layout of the crates and contains a glossary (oh, we should probably add the glossary to its own chapter).
EDIT(@mark-i-m): Added TODOs
EDIT2(@mark-i-m): A lot of the content of the READMEs is redundant with content we already have, so please check that the content you are moving over is not already in the appropriate chapter. If it is, please post here, so we can check it off the list ๐
TODOs:
ty
module: https://github.com/rust-lang/rust/blob/master/src/librustc/ty/README.mdTalk about the HIR:
One thing that could be helpful is an index of important data structures, functions, methods, traits, etc in the compiler. I propose we add a chapter after the glossary with a big table (sorted alphabetically by item name) that looks something like this:
Item | Kind | Short description | Chapter | Declaration |
---|---|---|---|---|
foo |
function | used by the parser to keep track bar | Parsing | src/libsyntax/foo.rs |
TODO:
Quite a few complicated things are going on during trans. Would be great to have it documented.
I recently wrote a clippy lint that deals with attributes (mostly outer attributes) and found very little information on how they are implemented. Most of the information I gathered through PR comments and issues in the rust repo. The reference only explains how to use the different attributes.
Do you think it would make sense to have a chapter on attributes in the guide?
Some weird things and questions I came across:
I would be happy to start something myself, but would need some pointers as to where the attribute stuff happens.
What's the line wrap convention? I initially guessed 80 chars max, but then it seems like a) a few lines overflow (I'd guess these were oversights), b) many lines wrap before they need to (I'd guess intentionally). I can't figure out the logic for b) in particular, since no consistent scheme seems to be used.
Personally, I'm not a big fan of hard wrapping in markup files (or code, to a lesser extent, though I know that's more contentious!). Should we be doing wrapping at all? If yes, let's document a consistent style, I say.
Some of this material exists already scattered about, and/or maybe this overlaps with the glossary, but I think it'd be helpful to try and capture some common conventions.
Here are some things I'm thinking about:
cx
suffix -- short for "context"'tcx
and 'gcx
-- mostly direct elsewhere for detailed explanation?As @Zoxc's work to [parallelize rustc] starts to land, we need to start documenting it!
Some example topics:
Lrc
? When should I be using it?I'm assuming this guide is going to contain lots of links to various bits and pieces around the rust-lang/rust
repository, so it's probably a good idea to set up some sort of link checking in CI.
I've made a backend for mdbook
which will read your book and check all the links are valid called mdbook-linkcheck. We'll be making another release in the next couple days (rust-lang/mdBook#558) which will include alternate backends and allow us to use mdbook-linkcheck
as part of the usual dev process.
This issue is mostly just a reminder for myself. When the next version of mdbook
is up on crates.io I'll come back and make a PR including mdbook-linkcheck
in CI.
This tracking issue contains links to the issues for all the chapters and subsections of the book. It is also the main place to come to if you're looking to get started on contributing.
Skim down the list below and find a chapter that looks interesting to you. It should have a link to an associated tracking issue. Click on that link and leave a comment there saying you'd like to help out (and/or start writing something).
In general, if you don't know how the compiler works, that is not a problem! In that case, what we will do is to schedule a bit of time for you to talk with someone who does know the code, or who wants to pair with you and figure it out. Then you can work on writing up what you learned. Just cc @nikomatsakis for help.
If you think there is a missing section please file an issue and we will consider how to fit into this outline.
ty
module in depth)This section should talk about the various tips and tricks for debugging the compiler, as well as perhaps some amount of "debug log conventions". It doesn't necessarily have to be a super long section, but here are some of the things I can think of:
RUST_LOG
and config.toml
first and foremost (overlaps with #8)-Zflags
:
-Ztreat-err-as-bug
, which causes errors to be reported as bugs. Useful when you are debugging an error that should not be happening because you can use RUST_BACKTRACE=1
then to get a backtrace of where the error was reported; also avoids dumping out more logs than you need.-Zunpretty=hir-tree
, which will dump out the HIR-Zverbose
, which dumps out extra-Zhelp
for more{:?}
most of the time in debug logsdebug!("foo(...)")
at the start of a function foo
and debug!("foo: ...")
or logs within that function (- nikomatsakis)--pretty expanded,identified
-Zdump-mir
and -Zdump-mir-graphviz
It'd be nice if, after each push, you could go visit a GH pages or something and see the output, so that people can go read the current contents. If you're interested in doing this, leave a comment! (If you need collaborator status, I can add you.)
It would be nice to have a section on how to profile the compiler, what tools to use, how to invoke them, and maybe a section on how to interpret results.
I don't really know why but travis is failing, giving a kind of generic message:
Couldn't push the build to github.com/rust-lang-nursery/rustc-guide.git:gh-pages
Bah humbug. I hate this stuff. Anybody got a clue? I can give permissions as needed if think you know how to fix =)
When rust-lang/rust-central-station#40 lands we'll have a stable location which contains internal API documentation (i.e. the equivalent of ./x.py doc
with crate-docs = true
) ๐
This means when referring to a particular struct or function we'll be able too link to its API docs instead of linking to the original source file and hoping for the best.
Once that PR is merged we'll need to:
One of the challenges with rustc is that the RLS can't handle it, making code navigation difficult. I personally use ctags; I use this script to set it up:
https://github.com/nikomatsakis/rust-etags
CTAGS integrate into emacs + vim quite easily, I don't know about other editors. I typically have this common as my standard "build rustc" command:
rust-ctags src/lib* && ./x.py build ...
This allows me to do "jump-to-def" with whatever functions were around when I last built .This is ridiculously useful.
Some setup instructions should be added to the "getting started" chapter I think.
Over in rust-lang/rust#47935, @michaelwoerister wrote:
The [dependency tracking] readme is largely outdated. The "Testing the dependency graph" and "Debugging the dependency graph" sections are still correct, I think, but the rest is probably just confusing at this point.
I think we ought o move some of those sections over, and/or add some new material on how to debug incremental compilation failures. This is always something I have to kind of rediscover.
Could we get someone to do this? I'm not sure who would be most appropriate. @jseyfried knows it better than most, but he's been absent lately.
So I've recently discovered that ./x.py check
is actually really dang fast -- we should add a note about it in the "how to build compiler" section. It's in particular very useful when you're doing some kind of "type-based refactoring", like renaming a method, or changing the signature of some function.
Chapter 2 assumes you understand why there are multiple compiler stages and what each is used for. A short section explaining this would make the chapter much clearer.
They're mentioned on https://rust-lang-nursery.github.io/rustc-guide/the-parser.html, but not defined. I think it would be good to explain what they are first.
This section should describe how to build the compiler and run what you built. There is plenty of existing material that we can draw on. For example, the build system section of the CONTRIBUTING.md file covers similar material.
One thing that I think is missing from existing material is a focus on "practical recipes". Here are the highlights I would mention:
First thing first, create a config.toml
with the following options enabled:
- debug-assertions = true
-- enables debug logs (as well as additional assertions)
- debuginfo-lines = true
-- get line-number information
- use-jemalloc = false
-- this means that tools like valgrind work better; it also decreases peak memory usage
- (any other suggestions? perhaps something to enable LLVM assertions, if that is needed now?)
ccache = true
-- โ helps speed up builds, you may need to install ccache first thoughdebug = true
-- โ I turn this one, but really all that is useful I think is line number informationrustup toolchain
for stage1 and stage2
rustc +stage1 ...
and rustc +stage2 ...
to run the compiler you've builtRUST_LOG
environment variable
RUST_LOG=rustc::infer rustc +stage1 foo.rs >& killme
will dump all the debug logs from the the infer
module of the rustc
crate.-Z
flags useful for debugging and others things (and direct to #11)./x.py
:./x.py build
Builds everything. This is very slow, and you won't want to run it a lot. Ultimately this builds the compiler with the beta compiler -- kind of roughly giving you the nightly compiler. It then builds the compiler again, using that nightly compiler (to check that it can build itself, and to resolve some annoying issues around linking).
./x.py build --stage 1 src/libstd
or ./x.py build --incremental src/libstd
These commands will build a working stage1 rustc and standard library. The output can be found in build/<target>/stage1/bin/rustc
. It's the fastest way to get a working compiler.
It may be somewhat surprising that this command builds the compiler, since it says to build src/libstd
(the standard library). The reason for this is that we are building the compiler once (the stage1 compiler) and then using that compiler to build the stage1 standard library. This will be the standard library that your compiled programs will use.
I guess we can omit the ./x.py test
routines for the next section.
Maybe leave some thoughts in the comments below on things you would have liked to know?
While reviewing rust-lang/rust#50302 i noticed that these test folders aren't mentioned in the testing chapter at all. Since the linked PR adds a special configuration flag to the rustdoc-js
tests, it would be worth mentioning it in the testing guide.
I'm currently working on a project which uses rustc
internals to analyse a crate so I can then generate FFI bindings. Kinda like cbindgen
, except where they focus on static linking at compile time or dynamically when your program gets loaded into memory, I'm trying to load a dynamic library at runtime using dlopen
(*nix) and LoadLibrary
(windows).
Seeing as this is a guide on how rustc
works under the hood, would it be worth adding a chapter that mentions:
rustc
internals as a library,rustc_driver
to run some code after a particular phase, andMy code is on GitLab so people in a similar situation or writing this chapter can use that as a reference.
This is slightly off topic, but I'm really impressed with how easy the compiler internals (especially the HIR and Session
) are to work with once you understand how they're designed! It was initially a bit confusing because there are loads of data structures (unfortunately, not many have doc-comments) and lots of complexity, but after an hour or two of jumping around the crate docs I was able to be quite proficient in finding the information I want.
In https://github.com/rust-lang-nursery/rustc-guide/blob/master/src/hir.md there is a sentence about "It is a desugared version of the abstract syntax tree"
I am not a native English speaker. So I just understand it in my mother language. When I tried to translate this sentence into Chinese which is my mother language. This looks like saying that HIR is a kind of abstract syntax tree.
If this sentence does not cause ambiguity in English, please close and ignore this issue.
It's been mentioned to me that it would be useful to have a list of all the "desugarings" that occur during the compilation process. In particular, which bits of Rust disappear during e.g. HIR conversion, HAIR conversion, MIR conversion. Here are some notes, I'd love to see someone turn some of this into a more detailed writeup! If you're interested, ping me here or (better) on gitter/IRC.
While constructing the HIR, we perform the following desugarings. It seems worth documenting these at a high-level (e.g, showing the equivalent Rust code), as well as covering some of the mechanisms we use to do it (e.g., the special spans we construct).
?
operator and catch
blocks
for
loop
if let
and while let
forms
While lowering to MIR, we perform the following desugarings:
match
expressions
&
patterns to implement default binding modesWe should be talking about the librustc_resolve
crate works here. Some topics of interest:
We should talk first about:
span_label
etcEDIT(mark-i-m): See #14 (comment)
cc @estebank
Follow-up from rust-lang/rust#47782. @avborhanian could you submit a PR here as well? Thanks!
Hi, I believe the 'print this book' button in the top right corner of the documentation doesn't work. It only prints a screenshot of the current page instead of producing a concatenated document of all the pages in the book.
Talk about where macro expansion happens. There is lots of other stuff we should cover here eventually:
cc @jseyfried -- can you help on elaborating the list of things that belong here?
One of the best way of figuring out the cause of a regression is bisecting against the PRs that have landed recently. There are a number of folks (cc'd below) who do this from time to time -- I'm not really sure what wizardry they use to do it, but I'd love to see it documented!
cc @Mark-Simulacrum, @kennytm, @TimNN, @est31
Not sure if this is the right fit for this book, it relates more to the process of contributing rather than the technical details. Would it be useful to have a section in the guide for writing mentored issues?
The intention being that experienced contributors who are just getting started writing mentored issues would be able to review it and structure their mentoring instructions in as helpful and useful a way as possible and that the quality of the instruction would then be consistent?
It would contain things that mentors have found beneficial to include in mentoring instuctions; or things those that have done some mentored issues found useful or not so useful; examples of good and bad instruction?
Edit: only just now noticed that mentoring is mentioned in #25 so this might be a dupe.
We should talk about how the compiler team itself functions:
#rustc
on IRC#rustc
on IRC at 10am Boston time on Thursday
I'm not 100% certain without access to the Travis cache, but I believe that it is using an old version of mdbook (0.0.28) from the cache. This was noticed because Travis is not running mdbook-linkcheck, and the output in the Travis logs looks like the old version.
In the Travis logs, it looks like PRs are picking up this old cache from the master branch:
fetching master/cache-linux-trusty-e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855--cargo-stable.tgz
found cache
Due to the way ci/install.sh is written, it will skip if mdbook is already installed, but it does not check which version is already installed.
A quick short-term solution is to delete the travis caches.
I can make a PR to change the install.sh script to force a reinstall if there is a version mismatch. Alternatively, you can just manually delete the travis caches whenever you upgrade mdbook.
We don't explain Kind
at all!
cc @varkor -- want to write it =)
When I:
cargo rustc -- -Zunpretty=hir-tree
I got:
error: unknown debugging option: `unpretty`
Are there any other terms that should be added to the glossary? Please make suggestions.
as pointed out in #100 (comment)
We could make all of the links in the book absolute (always point to the guide) or we could leave them relative and just tell people to use the hosted mdbook version, rather than github... My personal leaning is towards the latter because it seems cleaner (easier to move the book later, easier to write, etc.)...
@nikomatsakis @Michael-F-Bryan Any thoughts?
cc @rajcspsg
I just noticed that the glossary is not perfectly in alpha order... Note to self: fix it when I get back to a computer...
Recently support landed in rustc for dynamically loading LLVM. It would be nice if the rustc-guide documented how to switch to a custom LLVM backend. I'm not sure if this was planned to be included in the trans chapter(#6), as it does not yet have a tracking issue I could comment on.
Sorry if this is out of scope for the rustc-guide!
An example where people ask for this documentation.
Cheers
Some main points raised by rust-lang/rust#50466:
How to build and test the library without needing to rebuild the compiler every time.
./x.py test --stage 0 --no-doc src/libstd
to quickly build and test the standard library, if features from a pre-built compiler is sufficient.--keep-stage
syntax so the compiler will only be built once and then kept frozen?Code arrangement
src/libcore/tests/
There is an awesome utility by @kennytm with the short and pithy name rustup-toolchain-install-master
. What this does is to download the artifacts produced by Rust's CI for a specific SHA1 -- this basically corresponds to the successful landing of some PR -- and then sets them up for your local use. (@kennytm -- does this include @bors try
artifacts as well?) This is helpful when you want to examine the resulting build of a PR without doing the build yourself.
We should mention this tool somewhere, it's super useful!
... perhaps as a subchapter under the MIR chapter?
While playing around using rustc internals as a library I found it was pretty difficult to discover how to do a bunch of "common" operations. We might be able to improve discoverability by documenting these in an appendix.
Off the top of my head, it'd be nice to know how to:
c_char
-> libc::c_char
or std::os::raw::c_char
, depending on context)Path
for a given Item
Feel free to add other "common" questions or shortcuts in the comments and I'll update this list accordingly.
CC: #74
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.