How breakpoints are set

I am fascinated by the debuggers. I love them so much that I wrote a small and very basic debugger as one of my projects recently. In this post I am going to write down what I’ve learned about how can a debugger set a breakpoint.. This post can be divided into these following sections.

What’s a breakpoint?
What’s a debugger?
What does the debugger need to do to set a breakpoint?
How can the debugger make the debuggee process halt?

What’s a breakpoint?

A breakpoint makes your program stop whenever a certain point in the program is reached.

What’s a debugger?

You can consider your debugger to be a program which forks() to create a child process and then calls execl() to load the process we want to debug. I used execl() in my code, but any of the system calls from the exec family of functions can be used.

snip20161006_10
The debugger process

And here’s the run_child() function which calls the execl() with the debuggee process’s executable name and path.
snip20161009_17

We see a call to ptrace() in run_child() function before calling execl(). Let’s, for the moment, not go into what ptrace() is, even though it’s very important to understand how does a debugger work. We will eventually come to it.

Now we have two processes running –

  1. The debugger as the parent process.
  2. And the debugee as the child process.

Let’s now try to think abstractly in a sort of hand-wavy manner what does a debugger need to do to set a breakpoint in the child process. The debugger needs the child process to stop where the breakpoint has been set. But how?

What does the debugger need to do to set a breakpoint?

Let’s examine the phrase “setting a breakpoint” a little closely. We say a process is running when the instructions of the process are being executed by the processor. Where are those instructions located? In the text/code section of the process’s virtual memory.

By setting a breakpoint we expect the debuggee process to halt at a certain point. Which means that we expect the debuggee process to stop just before executing some instruction. What can that instruction be? Well, if the user has set a breakpoint at the beginning of a function, it’s the first instruction at the start of the function. If the user has set a breakpoint at some line number in some file, the instruction is the first instruction in the series of instructions that correspond to that line.

Therefore the debugger needs to make the process halt right before executing that instruction.

In my project, I chose the instruction underlined in the following screenshot.
screen-shot-2016-10-17-at-10-51-31-am

How can the debugger make the debuggee process halt right before executing a particular instruction?

The debugger replaces the instruction (at least part of the instruction) at the start of the debuggee process with an instruction that generates a software interrupt. Hence, when this modified instruction is executed by the processor, the SIGTRAP signal is delivered and that is all that is needed to make the process stop. I have skipped a lot of details here, but we will discover them as we go.

But let’s first discover how does the debugger modify an instruction?
The instructions reside at the process’s text section which is mapped to the virtual memory when a process is loaded. Hence to modify the instruction the debugger needs to know the address of that instruction.

How does the debugger find out the address of an instruction?

If you are compiling a C/C++ program, you pass “-g” option to the compiler to generate some extra information. Those extra information contain these mappings and they are stored in a format known as DWARF in the object file. On Linux, the DWARF format is used to store the debug information inside the ELF file. Isn’t that cool! ELF stands for Executable Linkable Format. It’s the format to represent object files, executable files and shared libraries.

What is the instruction that the debugger puts in place of the original instruction?

The debugger overrides the first byte of the location where the original instruction was with the instruction “int 3”. “int 3” has 1 byte opcode “0xcc”, hence the debugger touches only the first byte of the memory address.

What’s an “int 3” instruction? “int n” instructions generate a call to the exception handler specified by the destination operand. “int 3” generates a call to the debug exception handler. The exception handler is a part of Kernel code and it delivers the signal SIGTRAP to the process, in our example to the process we are debugging.

How and when does the debugger change an instruction of the debugee process?

And now the most exciting part of this post has arrived. Using a very powerful system call ptrace().

Let’s understand ptrace.
What does ptrace() do? ptrace is a system call by which our debugger controls the execution of the process we are debugging. The debugger uses ptrace() to examine and change the memory and registers of the process that we are debugging.

If you look at the code here, before execl()’ing the debuggee process we called ptrace() with PTRACE_TRACEME which indicates that this process is to be traced by its parent process i.e the debugger.

The man page for ptrace mentions this –

If the PTRACE_O_TRACEEXEC option is not in effect, all successful calls to execl(2) by the traced process will cause it to be sent a SIGTRAP signal, giving the parent a chance to gain control before the new program begins execution.

Which simply means that the debugger gets notified via the wait()/waitpid() system call just before the debuggee process starts. And there the debugger has its golden chance to modify the text/code section of the process being debugged.

It also means that while being traced the debuggee process will stop each time a signal is being delivered and the tracer (the debugger) will be notified at its next call of waitpid(). Hence, when the signal SIGTRAP is delivered to the debugee process, it will stop and the debugger will be notified, which is exactly what we want. We want the debuggee process to stop and the debugger get notified when the debuggee process executes the “int 3” instruction. The debugger gets the information about the cause of the stopping of the debuggee process from the status returned by waitpid().

The default behaviour of SIGTRAP is to dump core and abort the process but we can not debug a process if it’s killed. Can we? Hence the debugger ignores the SIGTRAP signal and causes the debuggee process to continue.

Here’s the code to set “int 3” instruction at the first byte of the address that contained the original instruction. As you can see, the same old ptrace() function is used to first get the original instruction at the address so that we save the original instruction to restore it later. And then the same old ptrace() function is used with a different flag PTRACE_POKETEXT to set the new instruction which has “int 3” at its first byte.
screen-shot-2016-10-16-at-10-26-13-pm

What does the debugger do now that the “breakpoint has been hit”?

  • First, the debugger needs to restore the original instruction at the address where the breakpoint has been set.
  • Second, once it has been restored that original instruction should be executed once the debugger lets the debuggee process restart its execution.

How does the debugger restore the original instruction? Just the way the debugger sets “int 3” to set a breakpoint. Here’s the code. While setting the breakpoint we saved the original instruction, now all we need to do is to set it back at the given address.
screen-shot-2016-10-16-at-10-32-42-pm

How is the original instruction in the debuggee process then executed?
The program counter of the debuggee now points to the next instruction now that it has already executed the instruction at the address where we had out “int 3”.

To make the processor execute the original instruction in the debuggee process, we need to set the value of the program counter – %eip (in case of x86 machines) or %rip (in case of x86 64 machines) of the debuggee process to the address again.

And how can we set the instruction pointer of the debuggee process?
Using ptrace()! ptrace() has this super awesome capability of letting us “change the tracee’s memory and registers.” PTRACE_GETREGS makes ptrace copy the general purpose registers of the debuggee process into a struct. And PTRACE_SETREGS modifies the debuggee process’s general purpose registers. Here’s the code that does that
screen-shot-2016-10-17-at-10-39-46-am

Once the debugger restored the debuggee process’s program counter it let’s the process continue and the way to do that is following –
screen-shot-2016-10-17-at-10-41-12-am

And that’s how a debugger can set breakpoints.

The full code of the debugger can be found here.

I presented on this topic at the Thursday evening presentation at RC. Here’s the link to the slides.

References:
Eli Bendersky’s articles on debuggers
Call to interrupt procedures
Interrupts and interrupt handlers

How breakpoints are set

10 thoughts on “How breakpoints are set

  1. Tony Wooster says:

    How does this work with repeating breakpoints? We want to have the breakpoint reinstalled by the time the process comes back around to the instruction (what if, for example, the instruction jumps right back to itself?). After we repoke in the original instruction, we’re shot, right?

    I was thinking we could just restore registers but with eip/rip pointing to our own code area that looks like: [original instruction], jmp next user instruction address. But that fails if the original instruction is inspecting eip/rip. So then we’d have to take special care to rewrite those somehow… It’s doable, but messy. Any ideas how that’s done in production debuggers?

    1. satabdidas says:

      The manual for GDB internals says the following.

      “…. The basic theory is that GDB will replace a program instruction with a trap, illegal divide, or some other instruction that will cause an exception, and then when it’s encountered, GDB will take the exception and stop the program. When the user says to continue, GDB will restore the original instruction, single-step, re-insert the trap, and continue on. Note that this is only true in all-stop mode, where all threads are explicitely stopped by GDB whenever any of them hits a breakpoint. In non-stop mode, removing the trap even for a short time would open a window of time for other threads to miss the breakpoint. This is covered in the chapter about single-stepping. ”

      I haven’t implemented that yet. But I think it’ll be a worthwhile experiment to understand the behaviour.

      1. Yes, the usual way to do this is just replace the instruction, single step, then set the breakpoint again. The single step will be done by either a hardware single step through ptrace (if the target supports it), or the debugger will emulate the instruction, work out the possible branch targets, and set software breakpoints to do the step.

        Another possible way to continue from a breakpoint is to have a buffer allocated in the target where you place the original instruction you replaced with the breakpoint, followed by a jump to the instruction following the breakpoint. That way you don’t need to bother single-stepping; you just set the PC to the start of the buffer and continue.

        1. This paper describes some of GDB’s non-stop functionality, along with the displaced stepping method outlined above: http://www.bmrtech.com/uploadfile/image/whitepaper/mentorpaper_multcore_db.pdf

          LLDB’s code base is generally pretty readable, so you could explore that if you desired. This is the file which contains the low-level communication with the ptrace functionality: http://llvm.org/svn/llvm-project/lldb/trunk/source/Plugins/Process/Linux/NativeProcessLinux.cpp

  2. Nicolás says:

    Very educational!

    By the way, the code for run_child is missing from the blog post, I get the set_breakpoint code instead.

  3. Alan says:

    Okay, so you’ve set the breakpoint back to the original and told it to continue, but how do we stop at the breakpoint again next time around the loop?

  4. Johnf763 says:

    Thanks so much for sharing this excellent info! I’m seeking forward to see much more posts! dcfdfadkgggd

Comments are closed.