phil-opp/blog_os

Writing an OS in Rust

Blog OS

This repository contains the source code for the Writing an OS in Rust series at os.phil-opp.com.

If you have questions, open an issue or chat with us on Gitter.

Where is the code?

The code for each post lives in a separate git branch. This makes it possible to see the intermediate state after each post.

The code for the latest post is available here.

You can find the branch for each post by following the (source code) link in the post list below. The branches are named post-XX where XX is the post number, for example post-03 for the VGA Text Mode post or post-07 for the Hardware Interrupts post. For build instructions, see the Readme of the respective branch.

You can check out a branch in a subdirectory using git worktree:

git worktree add code post-10

The above command creates a subdirectory named code that contains the code for the 10th post ("Heap Allocation").

Posts

The goal of this project is to provide step-by-step tutorials in individual blog posts. We currently have the following set of posts:

Bare Bones:

• A Freestanding Rust Binary (source code)
• A Minimal Rust Kernel (source code)
• VGA Text Mode (source code)
• Testing (source code)

Interrupts:

• CPU Exceptions (source code)
• Double Faults (source code)
• Hardware Interrupts (source code)

Memory Management:

• Introduction to Paging (source code)
• Paging Implementation (source code)
• Heap Allocation (source code)
• Allocator Designs (source code)

Multitasking:

• Async/Await (source code)

First Edition Posts

The current version of the blog is already the second edition. The first edition is outdated and no longer maintained, but might still be useful. The posts of the first edition are:

Click to expand

Bare Bones:

• A Minimal x86 Kernel (source code)
• Entering Long Mode (source code)
• Set Up Rust (source code)
• Printing to Screen (source code)

Memory Management:

• Allocating Frames (source code)
• Page Tables (source code)
• Remap the Kernel (source code)
• Kernel Heap (source code)

Exceptions:

• Handling Exceptions (source code)
• Double Faults (source code)

Additional Resources:

• Cross Compile Binutils
• Cross Compile libcore
• Set Up GDB
• Handling Exceptions using Naked Functions
  • Catching Exceptions (source code)
  • Better Exception Messages (source code)
  • Returning from Exceptions (source code)

License

This project, with exception of the blog/content folder, is licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT)

at your option.

For licensing of the blog/content folder, see the blog/content/README.md.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

blog_os's Issues

Make `remap_the_kernel` unsafe

It must be called only once.

Why use imul when computing table addresses?

https://github.com/phil-opp/blog_os/blob/entering_longmode/src/arch/x86_64/boot.asm#L57

This line uses imul, when unsigned semantics are required, in general.

Missing/wrong `Writer::new`function in Printing to Screen post

Reported by FreeFull:

When you mention the write!/writeln! macros can be now used, the example uses Writer::new(...) although Writer doesn't have a new method.

Note that the posts need a current nightly…

…and not a nightly from the time the post was written. Probably link the travis build, too.

Also check the linked Vagrantfile.

reported by @mmalone on IRC

Nasm warning on checksum calculation on Mac OS

dd 0x100000000 - (0xe85250d6 + 0 + (header_end - header_start))

produces the warning:

multiboot_header.asm:7: warning: numeric constant 0x100000000 does not fit in 32 bits

The following version compiles without warnings (but causes a warning on Linux):

dd - (0xe85250d6 + 0 + (header_end - header_start))

See comment from Georgiy Slobodenyuk.

Table of Contents

Most of the posts are pretty long, so a table of contents might be a good idea.

Fix warnings

Currently the compiler issues the following warnings:

src/memory/paging/table.rs:90:1: 90:15 warning: enum is never used: `Level3`, #[warn(dead_code)] on by default
src/memory/paging/table.rs:90 enum Level3 {}
                              ^~~~~~~~~~~~~~
src/memory/paging/table.rs:91:1: 91:15 warning: enum is never used: `Level2`, #[warn(dead_code)] on by default
src/memory/paging/table.rs:91 enum Level2 {}
                              ^~~~~~~~~~~~~~
src/memory/paging/table.rs:32:27: 32:44 warning: private trait in public interface (error E0445), #[warn(private_in_public)] on by default
src/memory/paging/table.rs:32 impl<L> Table<L> where L: HierarchicalLevel
                                                        ^~~~~~~~~~~~~~~~~
src/memory/paging/table.rs:32:27: 32:44 warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!

src/memory/paging/mapper.rs:88:5: 93:6 warning: method is never used: `map`, #[warn(dead_code)] on by default
src/memory/paging/mapper.rs:88     pub fn map<A>(&mut self, page: Page, flags: EntryFlags, allocator: &mut A)
src/memory/paging/mapper.rs:89         where A: FrameAllocator
src/memory/paging/mapper.rs:90     {
src/memory/paging/mapper.rs:91         let frame = allocator.allocate_frame().expect("out of memory");
src/memory/paging/mapper.rs:92         self.map_to(page, frame, flags, allocator)
src/memory/paging/mapper.rs:93     }
src/memory/paging/mapper.rs:102:44: 102:53 warning: unused variable: `allocator`, #[warn(unused_variables)] on by default
src/memory/paging/mapper.rs:102     pub fn unmap<A>(&mut self, page: Page, allocator: &mut A)
                                                                           ^~~~~~~~~
src/memory/paging/mapper.rs:112:13: 112:18 warning: unused variable: `frame`, #[warn(unused_variables)] on by default
src/memory/paging/mapper.rs:112         let frame = p1[page.p1_index()].pointed_frame().unwrap();
                                            ^~~~~

The warnings in mapper.rs are ok (we don't use these values yet), but the table.rs warnings need to get fixed.

• make HierachicalLevel, Level3, and Level2 public
• fix the dead code warnings for Level3 and Level2 (why is it even dead code?)

1GB pages are not supported in QEMU<2.1 and Intel CPUs before 2010

Support for 1GB pages was introduced in QEMU 2.1 in 2014. Intel CPUs support it since Westmere (2010).

Either add a test through CPUID 0x80000001, EDX bit 26 or use 2MB pages instead.

Long mode page incorrectly identifies VGA character size

The "Some Tests" section states that a screen character consists of an 8-byte color code and 8-byte character. This should read as 8-bit for both.

Migrate to kramdown

Currenly we use redcarpet, which won't be available after May 1st.

Alternative: switch to hugo.

Replace `*`-wildcards in dependencies

Enable SSE in 32-bit assembly to remove the 64-bit error function

The setup_SSE function can be moved to the boot.asm file. Then we don't need an error function in the long_mode_init.asm file.

Correct information about Linux's internal allocator

Linux uses the buddy allocator for frame allocations, not for virtual memory. Thanks to @DSMan195276 for pointing it out!

We should fix the last paragraph of the introduction and also remove the wrong “kernels normally don't do heavy concurrent allocations” claim.

Mac OS support

Some Mac OS users seem to have tool problems in the first post. So it might be a good idea to create a separate page that lists all needed extra steps (such as updating nasm).

Unfortunately, I have zero Mac experience and can only copy instructions from the post's comments. So if someone got it working, I would really appreciate a list of required commands!

Grammar: `set up` instead of `setup`

Consider dual licensing (apache2/MIT)

See intermezzOS/kernel#8 for advantages of dual licensing.

Are there any drawbacks?

[Page Tables] Should the `unmap` function be public, too?

Flush tlb also on `map_to` for exotic CPUs

Because Intel CPUs don't remember "not present" pages, documentation from Intel may say that you don't need to invalidate when changing a page from "not present" to "present". Intel's documentation is only correct for Intel CPUs. It is not correct for all 80x86 CPUs. Some CPUs (mostly Cyrix) do remember when a page was "not present" and because of those CPUs you do have to invalidate when changing a page from "not present" to "present".

From this stackoverflow answer: http://stackoverflow.com/a/28384866/866447

Update cross-compile-libcore page to use nightly-libcore

Extend some explanations

Some issues that were brought up by rylev on IRC:

• "CPUID check" too fast
• more assembly explanations, especially cmp and jmp

Update: below @mtnygard mentioned some issues with Printing to Screen:

• too quick
• unclear code fragments (which file? function or impl method?)
• better document the lib.rs and Cargo.toml changes required for Unique and spin
• type conflict: byte strings vs definition of write_str

VGA colors: 'variant is never used'

Rust complains that most of the VGA colour constants are dead code. This creates a big pile of compiler warnings to wade through; perhaps this enum should have the #[allow(dead_code)] attribute?

What is a frame?

Your blog series is normally easy to follow, but I got a bit lost when I started reading "Frame Allocation" because you leapt straight in without saying what a frame is. Could you dedicate a paragraph or two to this so the reader knows where they're standing?

test issue

Include changes from blog PR 8

Include changes from phil-opp/phil-opp.github.io#8

[Set Up Rust] Example code does not cause a SSE error anymore

See http://os.phil-opp.com/set-up-rust.html#comment-2425734866

Long mode test is wrong

Should be:

test_long_mode:
    mov eax, 0x80000000    ; Set the A-register to 0x80000000.
    cpuid                  ; CPU identification.
    cmp eax, 0x80000001    ; Compare the A-register with 0x80000001.
    jb .no_long_mode       ; It is less, there is no long mode.
    mov eax, 0x80000001    ; Set the A-register to 0x80000001.
    cpuid                  ; CPU identification.
    test edx, 1 << 29      ; Test if the LM-bit, which is bit 29, is set in the D-register.
    jz .no_long_mode       ; They aren't, there is no long mode.
    ret

But is:

check_long_mode:
    mov eax, 0x80000000    ; Set the A-register to 0x80000000.
    cpuid                  ; CPU identification.
    cmp eax, 0x80000001    ; Compare the A-register with 0x80000001.
    jb .no_long_mode       ; It is less, there is no long mode.
    mov eax, 0x80000000    ; Set the A-register to 0x80000000.
    cpuid                  ; CPU identification.
    cmp eax, 0x80000001    ; Compare the A-register with 0x80000001.
    jb .no_long_mode       ; It is less, there is no long mode.
    ret

Add some notes regarding problems with `grub-mkrescue`

It seems like grub-mkrescue causes some issues. See this comment.

Update: We should add some notes regarding

• the different naming of grub-mkrescue in some distros (grub2-mkrescue)
• the need for mtools (to create an EFI image) or -d /usr/lib/grub/i386-pc (to avoid EFI) on EFI systems
• that xorriso is contained in libisoburn for some distros

Shorter title for page table post

Maybe just Page Tables...

Change `multiboot_header` section to `multiboot` to match git source and later tutorial

On http://blog.phil-opp.com/rust-os/multiboot-kernel.html you have used multiboot_header but in the git source and the page http://blog.phil-opp.com/rust-os/setup-rust.html you refer to the section multiboot . little confusing if following along step by step.

As a foot-note, thank you so much for this series, it is a great educational piece.

Use `-cdrom` for QEMU

Replace

qemu-system-x86_64 -drive format=raw,file=os.iso

with the simpler

qemu-system-x86_64 -cdrom os.iso

See intermezzOS/book@ac18e0d

Page Table image appears to have bits 47 and 48 swapped

http://os.phil-opp.com/modifying-page-tables.html

It looks like the P4 table entry goes to bit 48 and overlaps bit 47, but 48 is a copy of 47 and part of the signed extension.

Use -mno-red-zone

See monocasa's comment on reddit and the OSDev forum.

PhysicalAddress and VirtualAddress types probably shouldn't be interoperable

At the moment you define PhysicalAddress and VirtualAddress as synonyms via the type keyword, but this has the sideeffect that either can be passed to functions that take the other since they're both usize. It might be worth defining them via the newtype pattern (struct PhysicalAddress(usize);) instead.

Set noalloc flag for multiboot_header section

Instead of merging .multiboot_header into . rodata, we could just specify it as noalloc.

[Page Tables] Do we need atomics for `Entry`?

Right now an Entry is just an u64. Maybe it should be an AtomicUsize instead.

The reason is that we're modifying the tables through recursive mapping and thus have some magic addresses. The compiler and the CPU don't know this and could maybe reorder instructions in a wrong way.

For example, it would be bad if the following is reordered (pseudo code):

create_p3_table(p4[0x42]);
modify 0xffffffffffe42000; // the magic address to the created table

If some optimizer would cause modification of the magic address before the p3 is created, it would cause a page fault.

The question is: Is it possible that an optimizer does such things?

Add short introduction on the front page

It would be nice to have some kind of introduction on the front page, which describes the scope of the project and provides some links (e.g. to the github repo). Then we could also remove the ugly Latest Post banner.

vga_buffer.rs: `Unique<>` is unstable

src/vga_buffer.rs:1:5: 1:22 error: use of unstable library feature 'unique': needs an RFC to flesh out design (see issue #27730)
src/vga_buffer.rs:1 use core::ptr::Unique;
                        ^~~~~~~~~~~~~~~~~
src/vga_buffer.rs:1:5: 1:22 help: add #![feature(unique)] to the crate attributes to enable

Link "Set Up GDB" post more prominently

grub-mkrescue needs xorriso

For post 1, Would be nice to point out that grub-mkrescue needs xorriso installed. The command throws a cryptic error about xorriso version otherwise. Would be nice to save a (albeit quick) Google search.

Use official bitflags with no_std feature instead of own fork

@cmr created a bitflags-core fork, so we should switch to it.

Update: It seems like the official bitflags crate has a no_std feature now. See bitflags/bitflags#29.

Rename `RecursivePageTable` to `ActivePageTable`

AreaFrameAllocator no longer safe to use after remapping the kernel

The AreaFrameAllocator is unsafe to use after kernel remapping because it references the multiboot information structure, which is not remapped. I'm not sure what your next steps are (it's possible this is by design) but this took me a hot second to debug once I got through your last post and started off on my own adventures. If other people are having similar problems you have a couple choices:

• Stop using the AreaFrameAllocator and us some other FrameAllocator after the kernel remap
• Remap the multiboot header in remap_the_kernel, like you did the VGA buffer

You already have a reference to the boot_info object in remap_the_kernel, so you can use that for the remap. It's a bit ghetto, but here's a quick implementation of the latter option:

let boot_info_frame = Frame::containing_address(boot_info as *const _ as usize);
mapper.identity_map(boot_info_frame, PRESENT, allocator);

Compiling and running the blog_os

Is this what the start up is supposed to look like when you download and do make run?

It is complaining there is no bootable device.

Link “Virtual Memory with 256bytes of RAM” in page table post

http://blog.robertelder.org/virtual-memory-with-256-bytes-of-ram/

fatal: unrecognised output format 'elf64' on OS X

➜  os-rust  nasm -f elf64 multiboot_header.asm
nasm: fatal: unrecognised output format 'elf64' - use -hf for a list

This is the error I get when going through part one of your tutorial when trying to specify elf64 as the format for the linked executable.

nasm -hf lists:

valid output formats for -f are (`*' denotes default):
  * bin       flat-form binary files (e.g. DOS .COM, .SYS)
    aout      Linux a.out object files
    aoutb     NetBSD/FreeBSD a.out object files
    coff      COFF (i386) object files (e.g. DJGPP for DOS)
    elf       ELF32 (i386) object files (e.g. Linux)
    as86      Linux as86 (bin86 version 0.3) object files
    obj       MS-DOS 16-bit/32-bit OMF object files
    win32     Microsoft Win32 (i386) object files
    rdf       Relocatable Dynamic Object File Format v2.0
    ieee      IEEE-695 (LADsoft variant) object file format
    macho     NeXTstep/OpenStep/Rhapsody/Darwin/MacOS X object files

Add note that `stack_bottom` needs to be above `stack_top` in the source

Add a note in Entering Longmode that stack_bottom needs to be above stack_top in the source code because swapping these symbols causes ugly bugs.

[deleted]

[deleted]

Setting unneeded `write protect` bit when enabling paging

Ahmed Charles noted in a comment that we set the write protect bit when enabling paging. It's bit 16 in the CR0 register, Wikipedia describes it like this:

When set, the CPU can't write to read-only pages when privilege level is 0

The question is wether we should keep it (and add an explaining comment) or remove it (since it's not required for the switch to long mode).

I think it's reasonable to set it but maybe it's better to do it in a later post when it's required. When we remove it, it must not cause problems for people with the old version.

Any opinions are appreciated :).

Compare / contrast my VGA driver

Hey Phil,

Thanks again for a rad tutorial. I've been trying to do stuff on my own, while sort of peeking back at your stuff for hints. Given that 'print to the screen' is mostly Just Rust, I tried doing it myself. I figured opening an issue might be a decent way to discuss the differences, I'm not sure it's the right thing, let me know if I should have emailed you. 😄

Here's where I'm at: https://github.com/steveklabnik/nucleus/blob/cec875df64d988d707537c8058aa4862a532ef36/vga/src/lib.rs

The big difference, looking back, is in the way we built the buffer: I made mine a single, long buffer, and did calculations to get the x/y going. I also wrote a flush() that ends up doing the writing of the buffer to the screen, which is just a copy, since the buffer now has the same representation as the VGA memory itself.

Your code also handles long lines correctly; I'll have to fix mine to take this into account. I also don't need Unique, due to flush() handling it.

And I think I spotted another bug, I'm making the arrays twice as long as they need to be, now that they hold the struct rather than the bytes they did before...

Anyway. Would be interested in hearing your thoughts, or if I've done something else silly somewhere.

might be good to remind users to override rust to nightly

in ubuntu 14.04 after installing multirust and running the command multirust update nightly I ran into the following error:

 #[feature] may not be used on the stable release channel

which suggested I was still using stable. running multirust override nightly did the trick, may be worth throwing into your document for other newbie readers such as myself.

Thanks for making a really excellent tutorial by the way!