open-education-hub/operating-systems

Teaching resources (OER) for Operating Systems

Operating Systems (OER)

These are open educational resources (OER) for Operating Systems classes. They are to be used by teachers, trainers, students and hobbyists who want to learn about modern topics in operating systems.

Using the Content

Content is located in the content/ folder. It currently consists of 5 chapters:

• Software Stack
• Data
• Compute
• Input/Output
• Application Interaction

Each chapter has its own folder. Content for each chapter is split in two subfolders:

• lecture/: content to be presented and discussed as part of lectures
• lab/: content to be worked on as practical activities during labs / seminars

Lecture content is expected to be presented and followed. Lab content is expected to be used as practice work.

Chapter Contents

Lecture content consists of slides and demos. Slides are written in GitHub Markdown and use reveal-md and reveal.js to render HTML output. Lecture slides are built from the slides.md file using the make command (and the Makefile). Demos are snippets of code and support files that showcase concepts and ideas related to the lecture. Demos are located in the demo/ folder. Each demo has its own folder with source code, Makefile or other build files (if required) and support files.

Lab content consists of lab text and lab activities. Lab text is placed in the README.md file. Each lab activity has its own folder with source code, Makefile or other build files (if required) and support files.

Contributing

Contributions are welcome. See the contribution guide on how you could report or fix issues and on how you can improve the content.

Reviewers are requested to follow the reviewing guide.

operating-systems's Issues

Create GIFs with `convert` instead of `ffpmeg`

slides.mk is currently using ffmpeg to create a GIF file from multiple SVG images (see the videos target. This causes the image quality of some resulting GIFs to drop unexpectedly. In addition, text and black lines get some unwanted yellowish shadows. I've included an example GIF and one of the original SVGs used for creating it below.

The GIF:

One of the SVG images:

Using convert instead of ffmpeg seems to result in higher-quality GIFs. An example of how to create a GIF from SVGs using convert is:

student@os:/.../app-interact/lecture$ convert -delay 200 -loop 0 media/synchronization/* out.gif

io/lab: Explain why `read()` and `write()` need to be called in loops

The File Descriptors section only quotes read()'s man page as an argument to why this syscall must be performed inside a loop. This is likely not enough. It should be clearly stated that read() and write() may return fewer bytes than requested, which is why we call them inside loops.

data/lab: Use `BUFSIZE` instead of `BUFSIZ`

These lines define BUFSIZE only in case it's not already defined by libc. In most cases, however, it is defined and students try to redefine it here to no avail. Replace BUFSIZ with BUFSIZE and remove the #ifndef block to avoid this confusion. Update the Markdown support accordingly.

software-stack/libcall-syscall: `ltrace` doesn't show library calls at all

ltrace works by extracting stuff out of the PLT, so if binutils is optimizing a PLT access into a GOT access (by relying on now binding) it's likely that ltrace is going to stop working.

I've tested this on 2 quite different Debian based distros:

Ubuntu 20.04

bin	version
gcc	9.4.0
ld	2.34

The main question is why are binaries linked with the -z now flag (now binding) on Ubuntu.

Even when trying to inspect /bin/ls, there is no output for ltrace:

$ scanelf -a /bin/ls
 TYPE    PAX   PERM ENDIAN STK/REL/PTL TEXTREL RPATH BIND FILE 
ET_DYN PeMRxS 0755 LE RW- R-- RW-    -      -   NOW /bin/ls 

$ ltrace /bin/ls
issues-ltrace.md
+++ exited (status 0) +++

ParrotOS

bin	version
gcc	10.2.1
ld	2.35.2

On my ParrotOS machine (a Debian based rolling distro), binaries are linked by default with -z lazy, ergo, ltrace works like expected:

$ scanelf -a /bin/ls
 TYPE    PAX   PERM ENDIAN STK/REL/PTL TEXTREL RPATH BIND FILE 
ET_DYN PeMRxS 0755 LE RW- R-- RW-    -      -   LAZY /bin/ls

$ ltrace /bin/ls
strrchr("/bin/ls", '/')                          = "/ls"
setlocale(LC_ALL, "")                            = "LC_CTYPE=en_US.UTF-8;LC_NUMERIC="...
bindtextdomain("coreutils", "/usr/share/locale") = "/usr/share/locale"
textdomain("coreutils")                          = "coreutils"
__cxa_atexit(0x56359a85d5a0, 0, 0x56359a873368, 0) = 0
isatty(1)                                        = 1

[...]

+++ exited (status 0) +++

Useful links

Further explaining the issue

• https://stackoverflow.com/questions/43213505/no-output-when-running-ltrace

• https://sourceware.org/legacy-ml/binutils-cvs/2015-05/msg00106.html

• https://bugzilla.redhat.com/show_bug.cgi?id=1333481

• https://stackoverflow.com/questions/58076539/plt-plt-got-what-is-different

Fixing the issue

• namhyung/uftrace#592

This change will enable tracing library calls have no PLT. As PLT is needed to trace those libcalls, uftrace make up a trampoline and fill necessary instructions there.
The code is available at review/libcall-noplt-v1 branch.

• https://stackoverflow.com/questions/61394462/alternative-to-ltrace-that-works-on-binaries-linked-with-z-now?noredirect=1&lq=1

Quick fixes on Ubuntu

I've found 2 alternative hacks:

• Installed and used the uftrace tool:

$ sudo apt install uftrace

$ uftrace --force /usr/bin/pwd
/home/maria/Documents/school-work/SO/my-so-lab-2022/software-stack/libcall-syscall
# DURATION     TID     FUNCTION
   2.766 us [ 26498] | getenv();
   0.160 us [ 26498] | strrchr();
  82.619 us [ 26498] | setlocale();
   0.591 us [ 26498] | bindtextdomain();
   0.742 us [ 26498] | textdomain();
   0.241 us [ 26498] | __cxa_atexit();
   4.511 us [ 26498] | getopt_long();
   7.036 us [ 26498] | getcwd();
  15.686 us [ 26498] | puts();
   0.170 us [ 26498] | free();
   0.241 us [ 26498] | __fpending();
   0.261 us [ 26498] | fileno();
   0.241 us [ 26498] | __freading();
   0.100 us [ 26498] | __freading();
   0.621 us [ 26498] | fflush();
   1.273 us [ 26498] | fclose();
   0.110 us [ 26498] | __fpending();
   0.110 us [ 26498] | fileno();
   0.111 us [ 26498] | __freading();
   0.090 us [ 26498] | __freading();
   0.120 us [ 26498] | fflush();
   0.592 us [ 26498] | fclose();

• Modified the Makefile (for the required exercise) in order to force the lazy binding:

diff --git a/content/chapters/software-stack/lab/support/libcall-syscall/Makefile b/content/chapters/software-stack/lab/support/libcall-syscall/Makefile
index bd91a72..a91df3f 100644
--- a/content/chapters/software-stack/lab/support/libcall-syscall/Makefile
+++ b/content/chapters/software-stack/lab/support/libcall-syscall/Makefile
@@ -1,4 +1,5 @@
 CFLAGS = -Wall
+LDFLAGS = -z lazy
 
 .PHONY: all clean

$ ltrace ./call
fopen("a.txt", "wt")                             = 0x5603b16fb2a0
strlen("Hello, world!\n")                        = 14
fwrite("Hello, world!\n", 1, 14, 0x5603b16fb2a0) = 14
fflush(0x5603b16fb2a0)                           = 0
+++ exited (status 0) +++

The second alternative does half of the work, as we cannot change binaries that are already compiled (e.g. /bin/pwd).

Fix minishell assignment solution

compute/lab: Add a table with differences between threads and processes

This subsection compares threads to processes but fails to specify the differences between them clearly. This table from the lecture needs to be added to the lab as well.

io/lab: Add exercise using `socketpair()`

socketpair() can be used to create a pair of unnamed UNIX sockets (AF_UNIX family). It creates a bidirectional connection between two related processes. It's similar to a a pipe (and to the pipe() call - it returns two file descriptors), except that it's bidirectional.

Add an exercise as part of the I/O lab to use socketpair() to make to related processes communicate. The socket pair can use both SOCK_STREAM and SOCK_DGRAM as socket types.

This is an example of using socketpair(), could also be referenced in the exercise.

data/lab: Create the Memory Security section

The Memory Security section is incomplete. The skeleton needs to be completed with text, quizzes and exercises.

Configuring issue templates

Right now we don't have any issue templates and this leads to the fact that people create issues without all the details and without any labels.

It would be nice to have some templates as we use on Unikraft.
For more details, you can check the documentation from GitHub.

Docusaurus Lab App-Interaction

Capitolu SO-Cloud este vizibil strict din pagina Overview, din pagina generala a laboratorului nu poate fi vazuta.
De asemenea, nu se poate da next din capitolul anterior.

Add GitHub action that publishes updated website when pushing to each pull request

Building the website locally is tedious, takes a very long time, and 20GB+ of disk space. This makes it difficult to test changes to the documentation.

The solution is to add a GitHub action triggered by a new commit to a PR that runs the builder, publishes the rendered content, and then adds a comment to the PR containing a link to the published website that contains the changes within the PR. You can draw inspiration from the GitHub action used by the OS Internals class. An example of a comment created by this action is linux-kernel-labs/linux#316 (comment).

Ideally, this action would only be triggered by changes to the Markdown documentation or quizzes and not just code. For this, PRs should first be labelled automatically as per #89. According to this issue, the action should only be executed when the area/documentation label is present.

software-stack/arena: solution for `AArch64` asm porting task segfaults

After compiling hello.s provided in the solution/ folder and running the executable with

qemu-aarch64 -L /usr/aarch64-linux-gnu ./hello

the program crashes with a SEGFAULT.

Further debugging (and surfing the net) proved that the aforementioned behavior is a bug from qemu as documented here and on the qemu upstream mailing list.

It would be nice to have the app running, so a possible hack would be adding some padding:

.section .bss

    .lcomm buffer, 128
    .p2align 12

Another solution would be getting rid off of the bss section completely as it is not used by the program.

Note: The code is perfectly fine and runs alright on AArch64 machines (without emulation or qemu being involved).

lab: Make the heading of each section `#` instead of `##`

Docusaurus uses the name of the file instead of the first heading as the title page. This way we end up with sections looking like this where the title is set to process-memory instead of Process Memory:

The Markdown files for each section have to be updated and the topmost heading changed from ## to #. The rest of the headings probably need to be bumped one level higher as well since Markdownlint doesn't allow adjacent headings to differ by more than one #. Look into lab content files as well as quizzes.

Create script to generate lab support files from solutions / templates

There is currently plenty of duplicate code between support/ and solution/ directories for labs. The best approach is to only provide the solution/ folder inside the repository. And the support/ files will be generated from that.

This approach is currently used by the linux-kernel-labs GitHub organization, by the generate_skels.py script. The script generates the skels/ directory with the support files from the corresponding tools/labs/templates/ directory.

lab: Overview tabs are inconsistents

The overview tabs for each chapter are inconsistent.
For example, for compute, io and app-interaction contains only a table of content, but the software-stack and data contain beside the table of content some introduction information.
I don't know how it's better, but I think it should be a difference between Overview and Introduction and this should be consistent throughout all chapters.

Also, the setup part for Git should be removed right now since we have the base support directory for every lab.

io/{lab,lecture}: Add demo using `lsof` while spawning multiple browser tabs

The sockets opened by each browser tab can be inspected using lsof. This behaviour is merely mentioned here, but no demo is provided. Both the lab and the lecture would benefit from having a demo where lsof is running while opening several firefox tables.

Reorder chapter order

The chapters are currently displayed in an incorrect order on Docusaurus:

The correct order is:

1. Software stack
2. Data
3. Compute
4. I/O
5. Application Interaction

The order of chapters in config.yaml has te be changed to reflect the one above.

Add script to run `super-linter` locally

During local development is hard to write code following super-linter rules and for each mistake, it will be necessary to run the corresponding action on GitHub, which leads to a waste of time.

Because of this, we need a method to be able to run the super-linter locally.

My suggestion would be to add a script to check this inside util folder. You can see here an example to run super-linter locally.

Add `checkpatch.pl` as GitHub action

checkpatch.pl is a Perl script used for coding style checking in the Linux kernel. We want to use it for C source code.

Add the checkpatch.pl GitHub action to this repository.

io/lecture: Pass interface name as command line argument for `demo/devices/hwattr-ioctl.c`

In the demo/devices/hwattr-ioctl.c demo, the interface name is hardcoded as the IFNAME macro. Update the demo to pass the interface name as a command line argument, making the program easily work on different interface names.

assignment/minishell: Add statement

software-stack/lab: Missing images

For this lab there are 2 images missing in the web interface.

software-stack/lab: Explain syscalls

The Basic Syscalls chapter in the Software Stack lab starts very abruptly by showcasing the x86_64 syscall API without saying what a syscall is, why it needs to be performed etc.

This should be addressed by adding a paragraph (together with some diagrams if possible) stating the reason syscalls are important, what they do and why we need them before the Basic System Calls section. These details should be added to a new section.

Add a GitHub pipeline to build every source file if build system is changed

All Makefiles use this general build system.
If something changes with this we need to check if it will affect some Makefiles.

io/lecture: Make servers run in a loop in `demo/IPC/`

The implementation of servers in demo/IPC/ is: receive connection, receive, send, end. Update the implementation to fit a proper server implementation, where the server runs an infinite loop (while(1)) waiting for connections, receive,s end, end connection, loop.

This goes for all 4 servers in demo/IPC/:

• datagram_socket_server.c
• stream_socket_server.c
• unix_socket_server.c

data/lab: Drop caches before benchmarking `mmap_copy` and `copy`

The buffer cache massively improves the performance of both mmap_copy and copy. Because the buffer cache is taught during the I/O chapter, students might get confused why the running time of each program drops so much. To avoid this, use this script from the I/O lab here as well and update the lab support to briefly explain why it's needed.

data/lab: Add quizzes

In the Data lab there are quizzes only for the Working with Memory section. The others have dedicated quiz sections which are marked with TODO.

compute/lab: Replace Scheduling section with User-level Threads

libult is both buggy as described in #51 and difficult to understand. The purpose of this exercise and section is to visualise user-level threads. This can be done a lot easier using either:

• goroutines in Go
• fibres in D. The advantage of D fibres is that D is similar enough to C so the overhead from switching languages is smaller. Another useful explanation can be found here.

Pick one user-level thread implementation and replace libult with it. The new exercise should focus on the functionality of user-level threads and not on their implementation. The Scheduling section needs to be replaced with "User-level Threads" in accordance with the changes above.

Automate assigning of GitHub labels to pull requests

We use labels to classify pull requests. Labels titled area/... or topic/... could be automatically assigned, according to their contents.

Create a GitHub workflow to automate the assigning of labels to pull requests. Take inspiration from the labeler GitHub action, or from the governance repository for Unikraft.

compute/lab: Explain spurious wakeups

The subsection on conditions does a bad job at explaining why the condition needs to be checked in a while loop:

while len(messages) == 0:
    event.wait()

Concretely, it doesn't mention that a spurious wakeup may happen. This explanation is required to clarify the way in which conditions work.

software-stack/arena: missing skel files for `AArch64` asm task

Porting asm program to AArch64 task (from System Calls) states that we should be using skel files from the arm/ subfolder:

Use the skeleton files in the arm/ folder. Find information about the aarch64 system calls here.

However, this folder does not exist. The only related info that can be found is placed in the solutions/ file, which is not ideal if someone wishes to work on the item without checking the answers.

If, though, the student is expected to resolve the exercise independently, the task should be updated accordingly.

Configuring PR templates

Right now we don't have any PR templates and this leads to the fact that people create PR without a proper description.

It would be nice to have some templates as we use on Unikraft.
For more details, you can check the documentation from GitHub.

io/lecture: Add datagram UNIX domain demo

The demos in demo/IPC/ feature a stream UNIX domain server and client (unix_socket_server.c + unix_socket_client.c). Rename them to unix_stream_socket_server.c and unix_stream_socket_client.c and add a datagram UNIX domain server and client (unix_datagram_socket_server.c and unix_datagram_socket_client.c).

lab: Move thread memory from the Data lab to the Compute lab

This section about multithreaded programs is misplaced in the Data lab because it proceeds the Compute chapters, which is where students first learn about threads. This section together with its supporting code and quiz should be placed in the Compute lab, at the end of the "Threads" section.

compute/lab: Replace `with event` constructs with regular locking

The support code for conditions uses with event constructs. They are confusing for students who don't have a good grasp on synchronisation mechanisms yet. They should be replaced with event.acquire(). and event.release() to make the code clearer. Also, update the corresponding Markdown file to remove the reference to the with statement.

io/lab: Add sample output for "Now We Pipe" exercise

The Now We Pipe exercise asks students to simply send some data from the child process to the parent. Add a sample console output that shows input being written (to the child process) and then echoed by the parent.

data/lab: Explain the `perm` argument of the `open()` syscall

This subsection of the Data lab uses open() for the destination file. Because it might not exist, the perm argument is also given to open(). The lab support contains no explanation about it, which may be confusing. The support needs to be updated to explain why this argument is needed.

`sqlfluff` linter fails on file generated with `mysqldump`

2022-12-15 17:23:51 [ERROR]   Found errors in [sqlfluff] linter!
2022-12-15 17:23:51 [ERROR]   Error code: 2. Command output:
------

User Error: No dialect was specified. You must configure a dialect or specify one on the command line using --dialect after the command. Available dialects:
ansi, athena, bigquery, clickhouse, databricks, db2, exasol, hive, mysql, oracle, postgres, redshift, snowflake, soql, sparksql, sqlite, teradata, tsql

Also, running it locally with sqlfluff parse db.sql --dialect mysql will fail with parse errors. So, even if the above error is solved, the check will still fail.

The .sql file was generated with mysqldump and wasn't modified afterwards in any way.

Intro page on website is empty

Intro page is empty.
It doesn't contain the general information mentioned there.

compute/lab: Replace D code with C

The code of most exercises in the Compute lab is written in D. However, this brings little to no benefit to students. It would be much better if exercises were mostly in C, with a few examples in Python here and there. The lab support (the Markdown content) should be updated accordingly.

app interact lab missing so cloud

Add `cppcheck` static checker as GitHub action

With super-linter as a workflow, support for cpplint is provided. An additional static checker for C/C++ code is cppcheck. There is a GitHub action that could be used.

I'm not certain if adding cppcheck along cpplint provides any extra benefits.

clang-format is currently a no Go due to its strict adherence to non-configurable style rules, as described in PR #24 .

software-stack/lab: Replace or explain the `write()` syscall

The write() syscall is used multiple times in the Software Stack lab [1, 2, 3]. This is problematic because the students don't know what a file descriptor is at this point in the course. We need to add some lines here that briefly explain that file descriptors will be studied later in the course and that for now, it suffices to know what 0, 1 and 2 are.

[1] https://github.com/open-education-hub/operating-systems/blob/master/content/chapters/software-stack/lab/support/basic-syscall/hello.asm
[2] https://github.com/open-education-hub/operating-systems/blob/master/content/chapters/software-stack/lab/support/syscall-wrapper/syscall.s
[3] https://github.com/open-education-hub/operating-systems/blob/master/content/chapters/software-stack/lab/support/common-functions/syscall.s

lab/compute: `libult` can have in some cases non deterministic output

I just compiled and run the code.
As you can see that 2317641 should not be present in the output.

Add instructions on running `markdownlint` locally

The repository features super-linter as a worfklow in the .github/workflow/ directory. This is enabled at each push and it runs markdownlint-cli.

We need to update the README.md file and / or the CONTRIBUTING.md file with information on installing and using markdownlint locally, before creating a PR.

Installing is done using:

$ npm install -g markdownlint-cli

Checking a Markdown file is done using:

$ markdownlint --config .github/workflows/config/config.json --rules .github/workflows/rules/rules.js content/chapters/data/lab/quiz/*.md

io/lecture: Pass partition name as command line argument for `demo/devices/filesystem-size.c`

In the demo/devices/filesystem-size.c demo, the disk / partition name is hardcoded as the FS_PATH macro. Update the demo to pass the disk / partition name as a command line argument, making the program easily work on different disk / partition names names.

Mutex vs Atomic

I think in this section 1 you should explain better this sentence.

So using the hardware support is more efficient, but it can only be leveraged for simple, individual instructions, such as loads and stores.

Hardware support is a bit unclear to me.
A good example is presented in this blog post 2. That example with assembly-code explained very well the difference between Mutex and Atomic and what is with that hardware support.

compute/lab: Rename "Benchmarks" section

The first proper section of the Compute lab is improperly called "Benchmarks". However, it is not about CPU benchmarks per see. It just wants to convey the message that while the hardware (CPU) may be powerful, it is the brain (OS) that needs to know how to make use of this power. A better name for this section may be "Hardware Perspective".

compute/lab: Add exercise with minor and major page faults

This practice item only briefly mentions major and minor page faults without going into detail about them. Create a follow-up exercise in the Arena that showcases the differences between the two. Also add a quiz for this new exercise.

data/lab: Fix `ALLOC_SIZE_KB` value

The demo in the data lab shows allocations of 8KB, but ALLOC_SIZE_KB is set to 256 (corresponding to 256KB allocations), as requested by this practice item. It should be set to 8.