In the next three labs you will implement preemptive multitasking among multiple simultaneously active user-mode environments.
In this lab, you will add multiprocessor support to JOS, implement round-robin scheduling, and add basic environment management system calls (calls that create and destroy environments, and allocate/map memory).
In Lab 6, you will implement a Unix-like fork(),
which allows a user-mode environment to create copies of
itself.
Finally, in Lab 7 you will add support for inter-process communication (IPC), allowing different user-mode environments to communicate and synchronize with each other explicitly. You will also add support for hardware clock interrupts and preemption.
Create a local branch called lab5 based on our lab5 branch, origin/lab5, and then fetch the latest version from the course repository:
$ cd ~/cs134/lab $ git checkout --track origin/lab5 Branch lab5 set up to track remote branch refs/remotes/origin/lab5. Switched to a new branch "lab5" $ git pull upstream lab5 # Pulls any changes I have made in the upstream repository $
You will now need to merge the changes you made in your lab4 branch into the lab5 branch, as follows:
$ git merge lab4 Merge made by the recursive strategy. ... $
In some cases, Git may not be able to figure out how to merge your changes with the new lab assignment (e.g. if you modified some of the code that is changed in the second lab assignment). In that case, the git merge command will tell you which files are conflicted, and you should first resolve the conflict (by editing the relevant files) and then commit the resulting files with git commit -a.
Important note. If your Pull Request for lab4 has finished being reviewed, then you know that lab4 is complete, and you will never need to merge from lab4 again. However, if it is is still being reviewed, then there may be changes required before the review is complete. Those changes will need to be merged into lab5-no-code, which you can do after the Pull Request is complete, by another call to git merge lab4 from lab5-no-code. Then, you would do a git merge lab5-no-code from lab5.You should merge into both labs so that the Pull Request for lab5 does not include the changes from lab4.
At this point, Lab 5 is ready to go. Before making any code changes, do the following:
$ git branch lab5-no-code # creates a branch prior to adding any Lab 5 code $ git push -u origin lab5-no-code # pushes the new branch to the originLab 5 contains a number of new source files, some of which you should browse before you start:
| kern/cpu.h | Kernel-private definitions for multiprocessor support |
| kern/mpconfig.c | Code to read the multiprocessor configuration |
| kern/lapic.c | Kernel code driving the local APIC unit in each processor |
| kern/mpentry.S | Assembly-language entry code for non-boot CPUs |
| kern/spinlock.h | Kernel-private definitions for spin locks, including the big kernel lock |
| kern/spinlock.c | Kernel code implementing spin locks |
| kern/sched.c | Code skeleton of the scheduler that you are about to implement |
In this lab and subsequent labs, do all of the regular exercises described in the lab. You can also do challenge problems. (Some challenge problems are more challenging than others, of course!) Additionally, write up brief answers to any questions posed in the lab and a short (e.g., one or two paragraph) description of what you did to solve each chosen challenge problem. Place the write-up in a file called answers-lab5.txt in the top level of your lab directory before submitting your work. Do not forget to add that file to git.
In this lab, you will first extend JOS to run on a multiprocessor system, and then implement some new JOS kernel system calls to allow user-level environments to create additional new environments. You will also implement cooperative round-robin scheduling, allowing the kernel to switch from one environment to another when the current environment voluntarily relinquishes the CPU (or exits). Later in lab 7 you will implement preemptive scheduling, which allows the kernel to re-take control of the CPU from an environment after a certain time has passed even if the environment does not cooperate.
We are going to make JOS support "symmetric multiprocessing" (SMP), a multiprocessor model in which all CPUs have equivalent access to system resources such as memory and I/O buses. While all CPUs are functionally identical in SMP, during the boot process they can be classified into two types: the bootstrap processor (BSP) is responsible for initializing the system and for booting the operating system; and the application processors (APs) are activated by the BSP only after the operating system is up and running. Which processor is the BSP is determined by the hardware and the BIOS. Up to this point, all your existing JOS code has been running on the BSP.
In an SMP system, each CPU has an accompanying local APIC (LAPIC) unit. The LAPIC units are responsible for delivering interrupts throughout the system. The LAPIC also provides its connected CPU with a unique identifier. In this lab, we make use of the following basic functionality of the LAPIC unit (in kern/lapic.c):
cpunum()). STARTUP interprocessor interrupt (IPI) from
the BSP to the APs to bring up other CPUs (see
lapic_startap()).apic_init()).A processor accesses its LAPIC using memory-mapped I/O (MMIO). In MMIO, a portion of physical memory is hardwired to the registers of some I/O devices, so the same load/store instructions typically used to access memory can be used to access device registers. You've already seen one IO hole at physical address 0xA0000 (we use this to write to the VGA display buffer). The LAPIC lives in a hole starting at physical address 0xFE000000 (32MB short of 4GB), so it's too high for us to access using our usual direct map at KERNBASE. The JOS virtual memory map leaves a 4MB gap at MMIOBASE so we have a place to map devices like this. Since later labs introduce more MMIO regions, you'll write a simple function to allocate space from this region and map device memory to it.
Exercise 1.
Implement mmio_map_region in kern/pmap.c. To
see how this is used, look at the beginning of
lapic_init in kern/lapic.c. You'll have to do
the next exercise, too, before the tests for
mmio_map_region will run.
Before booting up APs, the BSP should first collect information
about the multiprocessor system, such as the total number of
CPUs, their APIC IDs and the MMIO address of the LAPIC unit.
The mp_init() function in kern/mpconfig.c
retrieves this information by reading the MP configuration
table that resides in the BIOS's region of memory.
The boot_aps() function (in kern/init.c) drives
the AP bootstrap process. APs start in real mode, much like how the
bootloader started in boot/boot.S, so boot_aps()
copies the AP entry code (kern/mpentry.S) to a memory
location that is addressable in the real mode. Unlike with the
bootloader, we have some control over where the AP will start
executing code; we copy the entry code to 0x7000
(MPENTRY_PADDR), but any unused, page-aligned
physical address below 640KB would work.
After that, boot_aps() activates APs one after another, by
sending STARTUP IPIs to the LAPIC unit of the corresponding
AP, along with an initial CS:IP address at which the AP
should start running its entry code (MPENTRY_PADDR in our
case). The entry code in kern/mpentry.S is quite similar to
that of boot/boot.S. After some brief setup, it puts the AP
into protected mode with paging enabled, and then calls the C setup
routine mp_main() (also in kern/init.c).
boot_aps() waits for the AP to signal a
CPU_STARTED flag in cpu_status field of
its struct CpuInfo before going on to wake up the next one.
Exercise 2.
Read boot_aps() and mp_main() in
kern/init.c, and the assembly code in
kern/mpentry.S. Make sure you understand the control flow
transfer during the bootstrap of APs. Then modify your implementation
of page_init() in kern/pmap.c to avoid adding
the page at MPENTRY_PADDR to the free list, so that we
can safely copy and run AP bootstrap code at that physical address.
Your code should pass the updated check_page_free_list()
test (but might fail the updated check_kern_pgdir()
test, which we will fix soon).
Question
KERNBASE just like
everything else in the kernel, what is the purpose of macro
MPBOOTPHYS? Why is it
necessary in kern/mpentry.S but not in
boot/boot.S? In other words, what could go wrong if it
were omitted in kern/mpentry.S?
When writing a multiprocessor OS, it is important to distinguish
between per-CPU state that is private to each processor, and global
state that the whole system shares. kern/cpu.h defines most
of the per-CPU state, including struct CpuInfo, which stores
per-CPU variables. cpunum() always returns the ID of the
CPU that calls it, which can be used as an index into arrays like
cpus. Alternatively, the macro thiscpu is
shorthand for the current CPU's struct CpuInfo.
Here is the per-CPU state you should be aware of:
Per-CPU kernel stack.
Because multiple CPUs can trap into the kernel simultaneously,
we need a separate kernel stack for each processor to prevent them from
interfering with each other's execution. The array
percpu_kstacks[NCPU][KSTKSIZE] reserves space for NCPU's
worth of kernel stacks.
In Lab 2, you mapped the physical memory that bootstack
refers to as the BSP's kernel stack just below
KSTACKTOP.
Similarly, in this lab, you will map each CPU's kernel stack into this
region with guard pages acting as a buffer between them. CPU 0's
stack will still grow down from KSTACKTOP; CPU 1's stack
will start KSTKGAP bytes below the bottom of CPU 0's
stack, and so on. inc/memlayout.h shows the mapping layout.
Per-CPU TSS and TSS descriptor.
A per-CPU task state segment (TSS) is also needed in order to specify
where each CPU's kernel stack lives. The TSS for CPU i is stored
in cpus[i].cpu_ts, and the corresponding TSS descriptor is
defined in the GDT entry gdt[(GD_TSS0 >> 3) + i]. The
global ts variable defined in kern/trap.c will
no longer be useful.
Per-CPU current environment pointer.
Since each CPU can run different user process simultaneously, we
redefined the symbol curenv to refer to
cpus[cpunum()].cpu_env (or thiscpu->cpu_env), which
points to the environment currently executing on the
current CPU (the CPU on which the code is running).
Per-CPU system registers.
All registers, including system registers, are private to a
CPU. Therefore, instructions that
initialize these registers, such as lcr3(),
ltr(), lgdt(), lidt(), etc., must
be executed once on each CPU. Functions env_init_percpu()
and trap_init_percpu() are defined for this purpose.
In addition to this, if you have added any extra per-CPU state or performed any additional CPU-specific initialization (by say, setting new bits in the CPU registers) in your solutions to challenge problems in earlier labs, be sure to replicate them on each CPU here!
Exercise 3.
Modify mem_init_mp() (in kern/pmap.c) to map
per-CPU stacks starting
at KSTACKTOP, as shown in
inc/memlayout.h. The size of each stack is
KSTKSIZE bytes plus KSTKGAP bytes of
unmapped guard pages. Your code should pass the new check in
check_kern_pgdir().
Exercise 4.
The code in trap_init_percpu() (kern/trap.c)
initializes the TSS and
TSS descriptor for the BSP. It worked in Lab 3, but is incorrect
when running on other CPUs. Change the code so that it can work
on all CPUs. (Note: your new code should not use the global
ts variable any more.)
When you finish the above exercises, run JOS in QEMU with 4 CPUs using make qemu CPUS=4 (or make qemu-nox CPUS=4), you should see output like this:
... Physical memory: 66556K available, base = 640K, extended = 65532K check_page_alloc() succeeded! check_page() succeeded! check_kern_pgdir() succeeded! check_page_installed_pgdir() succeeded! SMP: CPU 0 found 4 CPU(s) enabled interrupts: 1 2 SMP: CPU 1 starting SMP: CPU 2 starting SMP: CPU 3 starting ... [00000000] new env 00001000 kernel panic on CPU 0 at kern/trap.c:323: Page fault in kernel mode
Our current code spins after initializing the AP in
mp_main(). Before letting the AP get any further, we need
to first address race conditions when multiple CPUs run kernel code
simultaneously. The simplest way to achieve this is to use a big
kernel lock.
The big kernel lock is a single global lock that is held whenever an
environment enters kernel mode, and is released when the environment
returns to user mode. In this model, environments in user mode can run
concurrently on any available CPUs, but no more than one environment can
run in kernel mode; any other environments that try to enter kernel mode
are forced to wait.
kern/spinlock.h declares the big kernel lock, namely
kernel_lock. It also provides lock_kernel()
and unlock_kernel(), shortcuts to acquire and
release the lock. You should apply the big kernel lock at four locations:
i386_init(), acquire the lock before the BSP wakes up the
other CPUs.
mp_main(), acquire the lock after initializing the AP,
and then call sched_yield() to start running environments
on this AP.
trap(), acquire the lock when trapped from user mode.
To determine whether a trap happened in user mode or in kernel mode,
check the low bits of the tf_cs.
env_run(), release the lock right before
switching to user mode. Do not do that too early or too late, otherwise
you will experience races or deadlocks.
Exercise 5.
Apply the big kernel lock as described above, by calling
lock_kernel() and unlock_kernel() at
the proper locations.
How to test if your locking is correct? You can't at this moment! But you will be able to after you implement the scheduler in the next exercise.
Question
Challenge! The big kernel lock is simple and easy to use. Nevertheless, it eliminates all concurrency in kernel mode. Most modern operating systems use different locks to protect different parts of their shared state, an approach called fine-grained locking. Fine-grained locking can increase performance significantly, but is more difficult to implement and error-prone. If you are brave enough, drop the big kernel lock and embrace concurrency in JOS!
It is up to you to decide the locking granularity (the amount of data that a lock protects). As a hint, you may consider using spin locks to ensure exclusive access to these shared components in the JOS kernel:
Your next task in this lab is to change the JOS kernel so that it can alternate between multiple environments in "round-robin" fashion. Round-robin scheduling in JOS works as follows:
sched_yield() in the new kern/sched.c
is responsible for selecting a new environment to run.
It searches sequentially through the envs[] array
in circular fashion,
starting just after the previously running environment
(or at the beginning of the array
if there was no previously running environment),
picks the first environment it finds
with a status of ENV_RUNNABLE
(see inc/env.h),
and calls env_run() to jump into that environment. sched_yield() must never run the same environment
on two CPUs at the same time. It can tell that an environment
is currently running on some CPU (possibly the current CPU)
because that environment's status will be ENV_RUNNING.sys_yield(),
which user environments can call
to invoke the kernel's sched_yield() function
and thereby voluntarily give up the CPU to a different environment. Exercise 6.
Implement round-robin scheduling in sched_yield()
as described above. Don't forget to modify
syscall() to dispatch sys_yield().
Make sure to invoke sched_yield() in mp_main.
Modify kern/init.c to create three (or more!) environments that all run the program user/yield.c.
Run make qemu. You should see the environments switch back and forth between each other five times before terminating, like below.
Test also with several CPUS: make qemu CPUS=2.
... Hello, I am environment 00001000. Hello, I am environment 00001001. Hello, I am environment 00001002. Back in environment 00001000, iteration 0. Back in environment 00001001, iteration 0. Back in environment 00001002, iteration 0. Back in environment 00001000, iteration 1. Back in environment 00001001, iteration 1. Back in environment 00001002, iteration 1. ...
After the yield programs exit, there will be no runnable environment in the system, the scheduler should invoke the JOS kernel monitor. If any of this does not happen, then fix your code before proceeding.
Question
env_run() you should have
called lcr3(). Before and after the call to
lcr3(), your code makes references (at least it should)
to the variable e, the argument to env_run.
Upon loading the %cr3 register, the addressing context
used by the MMU is instantly changed. But a virtual
address (namely e) has meaning relative to a given
address context--the address context specifies the physical address to
which the virtual address maps. Why can the pointer e be
dereferenced both before and after the addressing switch?
Challenge! Add a less trivial scheduling policy to the kernel, such as a fixed-priority scheduler that allows each environment to be assigned a priority and ensures that higher-priority environments are always chosen in preference to lower-priority environments. If you're feeling really adventurous, try implementing a Unix-style adjustable-priority scheduler or even a lottery or stride scheduler. (Look up "lottery scheduling" and "stride scheduling" in Google.)
Write a test program or two
that verifies that your scheduling algorithm is working correctly
(i.e., the right environments get run in the right order).
It may be easier to write these test programs
once you have implemented fork() and IPC
in labs 6 and 7.
Challenge!
The JOS kernel currently does not allow applications
to use the x86 processor's x87 floating-point unit (FPU),
MMX instructions, or Streaming SIMD Extensions (SSE).
Extend the Env structure
to provide a save area for the processor's floating point state,
and extend the context switching code
to save and restore this state properly
when switching from one environment to another.
The FXSAVE and FXRSTOR instructions may be useful,
but note that these are not in the old i386 user's manual
because they were introduced in more recent processors.
Write a user-level test program
that does something cool with floating-point.
Although your kernel is now capable of running and switching between multiple user-level environments, it is still limited to running environments that the kernel initially set up. You will now implement the necessary JOS system calls to allow user environments to create and start other new user environments.
Unix provides the fork() system call
as its process creation primitive.
Unix fork() copies
the entire address space of calling process (the parent)
to create a new process (the child).
The only differences between the two observable from user space
are their process IDs and parent process IDs
(as returned by getpid and getppid).
In the parent,
fork() returns the child's process ID,
while in the child, fork() returns 0.
By default, each process gets its own private address space, and
neither process's modifications to memory are visible to the other.
You will provide a different, more primitive
set of JOS system calls
for creating new user-mode environments.
With these system calls you will be able to implement
a Unix-like fork() entirely in user space,
in addition to other styles of environment creation.
The new system calls you will write for JOS are as follows:
sys_exofork:sys_exofork call.
In the parent, sys_exofork
will return the envid_t of the newly created
environment
(or a negative error code if the environment allocation failed).
In the child, however, it will return 0.
(Since the child starts out marked as not runnable,
sys_exofork will not actually return in the child
until the parent has explicitly allowed this
by marking the child runnable using....)sys_env_set_status:ENV_RUNNABLE or ENV_NOT_RUNNABLE.
This system call is typically used
to mark a new environment ready to run,
once its address space and register state
has been fully initialized.sys_page_alloc:sys_page_map:sys_page_unmap:
For all of the system calls above that accept environment IDs,
the JOS kernel supports the convention
that a value of 0 means "the current environment."
This convention is implemented by envid2env()
in kern/env.c.
We have provided a very primitive implementation
of a Unix-like fork()
in the test program user/dumbfork.c.
This test program uses the above system calls
to create and run a child environment
with a copy of its own address space.
The two environments
then switch back and forth using sys_yield
as in the previous exercise.
The parent exits after 10 iterations,
whereas the child exits after 20.
Exercise 7.
Implement the system calls described above
in kern/syscall.c and make sure syscall() calls
them.
You will need to use various functions
in kern/pmap.c and kern/env.c,
particularly envid2env().
For now, whenever you call envid2env(),
pass 1 in the checkperm parameter.
Be sure you check for any invalid system call arguments,
returning -E_INVAL in that case.
Test your JOS kernel with user/dumbfork
and make sure it works before proceeding.
Challenge!
Add the additional system calls necessary
to read all of the vital state of an existing environment
as well as set it up.
Then implement a user mode program that forks off a child environment,
runs it for a while (e.g., a few iterations of sys_yield()),
then takes a complete snapshot or checkpoint
of the child environment,
runs the child for a while longer,
and finally restores the child environment to the state it was in
at the checkpoint
and continues it from there.
Thus, you are effectively "replaying"
the execution of the child environment from an intermediate state.
Make the child environment perform some interaction with the user
using sys_cgetc() or readline()
so that the user can view and mutate its internal state,
and verify that with your checkpoint/restart
you can give the child environment a case of selective amnesia,
making it "forget" everything that happened beyond a certain point.
If make grade is failing, and you are trying to figure out why a particular test case is failing, run ./grade-lab5 -v, which will show you the output of the kernel builds and QEMU runs for each test, until a test fails. When a test fails, the script will stop, and then you can inspect jos.out to see what the kernel actually printed.
This completes the lab. In the lab directory, commit your changes with git commit and type make handin to get instructions for submitting your code.
See our page on GitHub and Pull Requests for detailed information on pull requests and submitting your code.