Advanced Operating Systems
Debugging in L4
When developing an operating system on top of L4 you do not have the luxury of using a source level debugger such as gdb. There are still a number of techniques at your disposal to assist debugging, however.
The Pistachio Kernel debugger
If your operating system causes certain types of faults you are likely to be dropped into the L4 kernel debugger. You should also be able to break into the kernel debugger at any time with the escape (ESC) key (provided it is compiled into your kernel).
The kernel debugger has many options that can be displayed by hitting the ? key.
--- KD# breakin --- --- KD# breakin --- > help BS - back up to previous menu ? - this help message ESC - back to previous menu a - architecture specifics c - KDB configuration B - generic bootinfo SPC - show current user exception frame F - show exception frame K - dump kernel interface page p - dump page table s - search for an exception frame g - continue execution S - list all address spaces d - dump memory P - dump physical memory D - dump memory in other space 6 - Reset system q - show scheduling queue t - show thread control block T - shows thread control block (extended) # - statistics r - enable/disable/list tracepoints >
The most useful functions you will use are (in no particular order) most likely SPC, F, K, g, d (and D), q, t (and T), r and 6. Take the time to learn to use them, they are a very handy and will save you many hours (or even days) of debugging.
L4 also allows you to assign a debug name to threads. This can be very handy when creating new threads to aid in debugging.
Debugging a stop
Often when using L4 you will cause the kernel to enter the debugger for one reason or another. This example takes you through debugging an unhandled exception, eg. the output below:
:Exception occured: 0x07, EPC=0x0000000000a013f0 VA=0x000000001c800030 STATUS=0x040180e0 --- KD# Unhandled Exception --- -- current ASID is 2, CPU 0 --
The first thing you want to do is show the thread control block
(TCB) of the thread that cause the exception. The command
NB: Even though
The first thing you need to do is work out which thread is
Once you have determined which thread is executing you want to
work out the code it is executing. The
Now that we know the faulting address we now want to find out which line of code contains the fault. The first step is to find which executable file the fault is in. Dite helps us here:
Dite shows us in this case the the fault is in the
You can now use the searching facility in
In this case I am lucky and it is a small piece of function so tracking down the bug shouldn't be too hard. In large functions your skill at reading assembler code, (that was one of the recommended skills for this course remember), comes into play. Of course this should encourage to keep you functions small.
More on objdump
Objdump is a very handy utility for working out exactly what is where in an executable so you can work out what exactly is going wrong.
The two standard incantations for objdump are:% armv5b-softfloat-linux-objdump -dl my_elf.file | less
and% armv5b-softfloat-linux-objdump -lx my_elf.file | less
The first command (-dl) disassesmbles the text segment and shows you all the instructions and at what address. Using this information you can find out things such as:
NB: In case it's not yet obvious, you will need to get up to speed on your ARM assembly and not be afraid to get your hands dirty if you want to minimise the time you spend debugging.
The second objdump command (-lx) is useful for when addresses appear inside an object file but outside of the text segment. This is especially useful when debugging ELF loading. The -lx option displays section and symbol information. Further options can be added to dump data segments etc. man objdump is your friend.
Kernel Debugger Tracing
Kernel debug tracing is a handy tool that can be used for many things:
Kernel debugger configuration is achieved by editing your top
level SConstruct file, specifically you can add debug options to
the line that instantiates the "pistachio" application. For
instance to enable tracing you add
The simplest way to enable tracing from the kernel debugger is to use r to enter the tracing submenu, then E to enable all tracepoints. You can also ? from the tracing submenu to list all the options. It can be very handy to only trace certain kernel events. NOTE: You will not see any trace events for IPC messages delievered on the fastpath. There is no tracing for this case (otherwise it wouldn't be so fast!).
The run queues
q in the kernel debugger shows all the current L4 threads, in order of priority, and whether or not they are runnable.
> showqueue : (roottask) (IRQ 03) (IRQ 00) (IRQ 01) (IRQ 02) : (0010c001) : 00104001 (00108001) (00110001) idle : idle
This dump shows 10 threads. 5 real threads, 4 interrupt IRQ threads and the
idle thread. Four of the real threads are spawned by the roottask, one is the
syscall and init loops, another is used by the timer.c service and the last
thread is spawned by the IXP400's Operating System Access Layer (OSAL),
Threads with brackets around their thread ID, eg. (00108001) are currently blocked. Thread IDs without brackets are runnable. In the above example the only runnable thread is the idle thread. Other threads may become runnable due to a timeout, IPC or interrupt.
Entering the kernel debugger
The kernel debugger is a life-line to debug your code. Being able to enter the kernel debugger to examine the state at the right time can make hours of difference to debugging.
Examining Thread State
There is a lot of information about a thread that you may want or need to find out. There are a number of ways to find thread state from within the kernel debugger.
The t command be used to dump a thread's TCB. This can be either the current thread (eg. during a debug enter, exception or whatever), or you can specify a thread ID. A sample TCB is shown here. Interesting fields are marked in bold and explained below. The explanation is only a rough guide and may be incomplete. The exact semantics can be checked in the L4 source.
showtcb tcb/tid/name [current]: roottask === roottask == TCB: e0020000 == ID: 00100001 = d1f00000/d1f00000 == PRIO: 0xff === UIP: 005302e8 queues: rswl wait : NIL_THRD:NIL_THRD space: f0028000 USP: 0056d3e4 tstate: WAIT_FE ready: roottask:roottask pdir : 00000000 KSP: e00206e4 sndhd : NIL_THRD send : NIL_THRD:NIL_THRD pager: NIL_THRD total quant: 0x0 us, ts length : 0x2710 us, curr ts: 0x2710 us resources: 00000000 [ek], ARM [PID: 0, vspace: 0, domain: 1] scheduler: roottask send redirector: ANY_THRD recv redirector: ANY_THRD partner: ANY_THRD saved partner: NIL_THRD saved state: ABORTED
The T command be used to dump a thread's TCB and extended information as shown below. The first part is basically the same, interesting bits are highlighted and described below.
showtcbext tcb/tid/name [current]: roottask === roottask == TCB: e0020000 == ID: 00100001 = d1f00000/d1f00000 == PRIO: 0xff === UIP: 005302e8 queues: rswl wait : NIL_THRD:NIL_THRD space: f0028000 USP: 0056d3e4 tstate: WAIT_FE ready: roottask:roottask pdir : 00000000 KSP: e00206e4 sndhd : NIL_THRD send : NIL_THRD:NIL_THRD pager: NIL_THRD total quant: 0x0 us, ts length : 0x2710 us, curr ts: 0x2710 us resources: 00000000 [ek], ARM [PID: 0, vspace: 0, domain: 1] scheduler: roottask send redirector: ANY_THRD recv redirector: ANY_THRD partner: ANY_THRD saved partner: NIL_THRD saved state: ABORTED user handle: 00000000 cop flags: 00 preempt flags: 00 [~] exception handler: NIL_THRD virtual sender: NIL_THRD intended receiver: NIL_THRD incomming notify bits: 00000000 notify mask: 00000000 last preempted_ip: 00000000 preempt_callback_ip: 00000000 mr( 0): 000741c0 00701cc7 00700210 00000000 00000000 00000000 00000000 00000000 mr( 8): 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 mr(16): 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 mr(24): 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 Message Tag: 0 untyped, label = 7, flags = -X-- Acceptor: 00000000 (a) Error code: 0
The extended information also lists message and buffer register contents. This can be useful to debug the exact contents of IPC messages, especially when used in conjunction with kernel break-in on IPC send events.
The IPC message tag is shown in a nice, easy to read format.
The SPC (space bar) command be used to dump a thread's register set (eg. exception frame). Below is an example exception frame.
frame == Stack frame: e0020eb4 == cpsr = 0, pc = e0020800, sp = 0, lr = f00159c0 r0 = f001b318, r1 = f0002d5c, r2 = 0, r3 = 0, r4 = 0 r5 = 0, r6 = 0, r7 = 0, r8 = 0, r9 = 4 r10 = f0002be0, r11 = e0020800, r12 = ac
This dump shows the address of the stack frame, status and cause registers, EPC, and the general register file. Other stack frames can be inspected with the F command. You can find stack frames via the TCB and the s command.
Often when debugging your code you will want to reboot the machine, however
you may be working remotely, or otherwise not be able to physically reboot
the machine. The kernel debugger can do this for you. First drop into the
kernel debugger by hitting
If for some reason this doesn't work L4 may have gotten itself into an unexpected state and you will have to manually reboot the machine.
Last modified: Thu Jul 27 15:16:11 EST 2006