Book review: The Art of Debugging with GDB, DDD and Eclipse (Chapter 6).

Special Topics

What if it doesn't even compile or load?

Phantom line numbers in syntax error messages

Sometimes, the line number from the compiler error message is not very informative. Typically, the problem is a missing closing brace or semicolon. To locate it, you can start by commenting out the function where you suspect the problem to be. Don't be surprised by linker errors if you do this, but stay cool and use binary search principles to locate the location of the problem.

Missing Libraries

If you have a program that uses functions from another .o file, you can get 'undefined reference to f' errors from the linker.

A library can be static (functions become part of the resulting executable file) or dynamic (the functions are not physically attached to the calling code until the program is actually executed).

To create a static library and compile your main program with it:

  gcc -g -c b.c
  ar rc lib88.a b.o
  gcc -g a.c -l88 -Lz

With the -L option, you tell the linker in what directories other than the current one it should look for the functions.

To create a dynamic library (to avoid space-wasting separate copies of the same library) and then link to it, use:

gcc -fPIC -c b.c
gcc -shared -o lib88.so b.o
gcc -g a.c -l88 -Lz

The resulting program executable will merely contain a notation that this program makes use of the library lib88.so. The linking itself will now occur at runtime. With the ldd command, you can check which libraries a program needs, and where the OS finds them:

  ldd a.out

To add a library to the OS's normal library search path, in bash you can do:

  LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/Debug/z
  export LD_LIBRARY_PATH

For missing library problems, ofteh the root of the problem lies in a program called pkgconfig. The default directory that pkgconfig searches for .pc files depends on the location of pkgconfig itself, and must sometimes be adapted by setting the environment variable PKG_CONFIG_PATH.

Debugging GUI programs

The example will be for the curses library, but the applied principles are valid for other GUI libraries as well.

Debugging Curses Programs

For debugging curses programs, you must tell GDB to have the program execute in a different terminal window than the one that GDB is running in with the tty command. First, go to another window and run the tty command there to determine the ID for that window. Then, in the GDB window you type:

  tty /dev/pts/8

You must also type something like

  sleep 10000

in the execution window, so that your keyboard input to that window wil go to that program rather than to the shell.

In DDD, you can arrange a separate window for execution of the program by choosing 'View | Execution Window'. Don't type the sleep command into that window, as DDD does that for you.

In Eclipse, you can use the option 'C/C++ Attach to Local Application'. First start your program in a separate shel window, then select Run | Open Debug Dialog and from the list, choose the process you wish GDB to attach.

Book review: The Art of Debugging with GDB, DDD and Eclipse (Chapter 5).

Debugging in a multiple-activities context

Debugging Client/Server Network Programs

This section presents only a short example. The most important tricks to remember are:

GDB had a jump command, but in general you should use this command with caution.

Including errno.h can be useful for your debugging, as is the use of the strace program for checking the result of system calls.

In really complex network debugging cases, you might want to use a program to track individual TCP/IP packets.

Debugging threaded code

The POSIX standard Pthreads is used for the example in this section.

Modern operating systems use timesharing to manage multiple running programs in such a way that they appear to the user to execute simultaneously. After a certain timeslice, processes are preempted and a context switch occurs. Because you do not know the order in which threads will be scheduled, the debugging is more difficult.

Threads vs. processes: a thread is designed to occupy less memory and its context switch time is less than that for processes. Threads are therefore also sometimes called leightweight processes.

The global variables of the parent program in a threaded environment are shared by all threads and serve as the main method of communication between the threads.

Preemptive thread management policy: a thread in a program can be interrupted at any time by another thread. This makes bugs not always reproducible.

Each time a new thread is created, GDB announces it. To know what each thread is doing, you can use the GDB info threads command. The asterisk in the output shows you what thread you're in. You can inspect the stack of any thread by switching to that thread and then issuing the bt command:

  thread 3
  bt

Some other examples which should speak for themselves:

  break 88 thread 3
  break 88 thread 3 if x==y

In DDD, select Status | Threads, and a window will pop up, displaying all threads in the manner of the GDB info threads. You can click a thread to switch the debugger's focus to it. In DDD, you cannot make a breakpoint thread-specific, so you have to use the DDD Console and issue GDB commands.

Eclipse constantly displays your thread list, as opposed to having to request it, as in DDD. The call stack is shown in the thread list. In Eclipse, you can set a breakpoint for a specific thread: right-click and select 'Filtering'.

Debugging parallel applications

• Shared memory: Multiple CPUs all have access to some common physical memory.
• Message passing: code running on each CPU can only access that CPU’s local memory, and it communicates with the others by sending messages.

Message-Passing Systems

The book's examples use MPI.

GDB allows you to dynamically attach the debugger to an already-running process, using the process number: first find process number, then

  gdb programname 2776

Shared-memory systems

• True Shared Memory: OpenMP, internally makes use of threads.
• Software Distributed Shared-Memory Systems

Quite some difficulties arise when debugging this kind of programs. Pay attention!

Extended Example

Most important message from this example: pay attention with line numbers! They are not always what you think they are! For the rest, nothing too interesting here...

Book review: The Art of Debugging with GDB, DDD and Eclipse (Chapter 4).

When a program crashes

Bugs are often associated with the mishandling or misuse of pointers. This is what this chapter is about.

Background Material: Memory Management

By far the most common cause of a crash is for a program to attempt to access a memory location without having the permission to do so. In such a case, the OS will announce that the program has caused a segmentation fault. On Windows, the corresponding term is a general protection fault. The role played by virtual memory and how virtual memory issues relate to segfaults is the focus of this chapter.

On Unix platforms, a program's set of allocated virtual addresses is typically laid out something like:

• Text section (.text): the machine instructions produced by the compiler from your program's source code.
• Data section: all the program variables that are allocated at compile time (global variables). It consists of two subsections: .data with initialized variables, and .bss for uninitialized data
• Heap: area for additional memory requests from the operating system via calls to for example malloc() or new.
• Stack: space for dynamically allocated data like function calls. The stack grows and shrinks each time a function is called and returns.

If w is the word size of your machine in bits, then the virtual address space has a range from 0 to 2^w-1. Your program typically uses only a tiny fraction of this, and the OS may also reserve part for its own work.

The virtual address space is viewed as organized into chunks called pages. Physical memory (RAM, ROM) is also viewed as divided into pages. Some of the pages of a program are stored in pages of physical memory by the OS (these pages are resident, the rest is stored on disk). During program execution, program pages must be brought into and out of memory. To manage this, the OS maintains a page table for each process. Each of the process' virtual pages has an entry in the table with the following information:

• The current physical location of this page in memory or on disk
• Permissions: read, write, execute, for this page

Note that the OS will not allocate partial pages to a program.

When we run a program, the OS creates a page table that it uses to manage the virtual memory of the process that executes the program code. As the program executes, it will continually access its various sections and the page table will be consulted. Permissions from the page table are important here. The addresses a program generates are virtual and will be converted to a virtual page number v, which is checked in the page table against its permissions.

Important remark for debugging: from the absence of a seg fault, we can't conclude that a memory operation is correct. Seg faults do not always occur in situations where you might expect them to. This is somehow linked to the concept of page-size and how your variables are aligned in these pages.

Signals indicate exceptional conditions and are reported during program execution to allow the OS (or your own code) to react to a variety of events. Each signal has its own default signal handler, which is a function that is called when that particular signal is raised on a process. You can also write your own handlers, but these may cause complications when using GDB/DDD/Eclipse.

When program commits memory-access violation, a SIGSEGV signal is raised on the process and a core file is written to disk.

To tell GDB not to stop when certain signals occur, use the handle command.

Other sources of crashes besides segmentation faults:

• Floating-Point Exceptions (SIGFPE)
• bus error (SIGBUS): accessing a physical address that does not exist or pointer errors

Core Files

If a core file is created during a run of your program, you can open your debugger on it and then proceed with your usual GDB operations.

Core files contain a detailed description of the program's state when it died. The Unix command file helpfully tells you the name of the executable that dumped a particular core file.

The writing of core files may be suppressed by your shell. In bash, you can control the creation of core files with the ulimit command:

  ulimit -c n

where n is the maximum size for a core file, in kilobytes.

Extended example

An extended debugging example was given in this subsection and I refer to the book for the details. Most important is that this example emcompasses many aspects of debugging:

• The Principle of Confirmation
• Using core files for post-mortem analysis of a process that crashed
• Correcting, compiling, and re-running a program without ever leaving GDB
• The inadequacy of printf()-style debugging
• Using good old fashioned brain power

Book review: The Art of Debugging with GDB, DDD and Eclipse (Chapter 3).

Chapter 3: Inspecting and setting variables.

Our Main Example Code

The main example is an implementation of a binary tree.

Advanced Inspection and Setting of Variables

Instead of using three print commands as in

  p tmp->val
  p tmp->left
  p tmp->right

you can print a struct in its entirety using

  p *tmp

Even better is using GDB's display or disp command, which tells GDB to automatically print the specified item each time there is a pause in execution (due to a breakpoint, the next or step commands,...)

You can also use GDB's commands command to execute several commands (like print) when a breakpoint is hit. Next to the print command, the printf() command can be helpful for this. Its formatting is similar to that of its C-language namesake. The command set for a given breakpoint can be modified dynamically, or simply canceled by redefining an empty set:

  commands 1
  end

If your code contains functions to print certain data-structures, you can call these from GDB using the call command:

  call printtree(root)

In DDD, the Right-Click / Display option is very useful for displaying linked data-structures! To follow a link, right-click on the link and select the `Display *()' choice. To remove something, right-click and choose the Undisplay option. To avoid windows cluttered with data, you can run DDD with the --separate option, which will create three separate windows for Source Code, Console and Data.

Eclipse has its Variables view where you can inspect the value of variables. By default, this view does not show global variables. You can add those by right-clicking within the Variables view and selecting 'Add Global Variables'.

For printing values of dynamically created arrays (e.g. int *x;), you can use for example:

  p x
  p *x
  p x[5]
  p *x@25
  p (int [25]) *x

In DDD it's probably best to also use these commands in the Console. In Eclipse, you select the variabe in the Variables view by right-clicking and selecting `Display As Array'.

For C++ code, the same GDB commands work, but with somewhat different output. Also, GDB needs you to specify variables according to the same scope rules that C++ uses. For example, don't use

  p *root

but specify root via its full name as

  p *node::root

GDB's ptype command is handy to get a quick review of the structure of a class or struct. In DDD, you can right-click the class or variable name and choose `What Is' to get the same information. Eclipse has its Outline view for this.

With the info locals command in GDB, you can get a list of values of all the local variables in the current stack frame. In DDD, you can display these by clicking Data | Display Local Variables. Eclipse displays the local variables in the Variables view.

To examine memory at a given address directly, GDB provides the x (examine) command. In DDD one selects Data | Memory. Eclipse has a Memory view, in which you can create Memory Monitors. These things are mainly useful in assembly language contexts.

The print and display commands allow you to specify alternative formats for hex, character, string and floating point:

  p/x myvar
  p/c myvar
  p/s myvar
  p/f myvar

To temporarily disable, re-enable and delete a display item, do:

  info disp
  dis disp 1
  enable disp 1
  undisp 1

Setting Variables from Within GDB/DDD/Eclipse

In GDB, you can very easily set values with for example:

  set x = 12

There is no mouse-based way to do this in DDD, so do it from the Console. In Eclipse, right-click the variable in the Variables view and select Change Value. Note also that the changes you make to argv[1], argv[2] etc. with the set args command will not occur until the next time you issue the run command.

GDB has the info args command to check the arguments of the current function. DDD provides this when you click Data | Display Arguments.

GDB's Own Variables

Output values from GDB's print command are labeled $1, $2, and so on, and are collectively called the value history. They can be used to produce shortcuts in future print commands, for example:

  (gdb) p tmp->left
  $1 = (struct node *) 0x80496a8
  (gdb) p *$1

There is also a special value history variable $.

You can also store certain values in convenience variables. For example

  set $q = p
  p $*q

The variable $q here is called a convenience variables. These variables can change values according to C rules.

Book review: The Art of Debugging with GDB, DDD and Eclipse (Chapter 2).

Chapter 2: stopping to take a look around

Chapter 2 deals with breakpoints, watchpoints and catchpoints, which are collectively called breakpoints in the GDB documentation. Each breakpoint is assigned a unique identifier, starting at 1. Important to note is that if you type break 35 you will break right before line 35, so line 35 is not executed yet!

With the info breakpoints command, you get all the info you need about your breakpoints. In DDD and Eclipse, similar functionality is available using your mouse.

Setting breakpoints in DDD and Eclipse is easy with the mouse. In GDB, there are many ways to specify a breakpoint. Some examples:

  break myfunction
  break line_number
  break myfilename:line_number
  break myfilename:myfunction

With tbreak you can set a temporary breakpoint, which automatically gets deleted after the first time it is reached.

Another important remark is that internally, GDB works with machine language instructions, not lines of source code. This explains why the location of a breakpoint is sometimes different from where you intended it to be.

For C++ code, the catch command might be useful.

Note also that GDB has the concept of focus. The current focus is the source code you're currently seeing. It changes when you apply the list command to another source file, when you step into code from a different source file or when you hit a breakpoint while executing code in a different source file.

Type quit to leave GDB.

To avoid having to redefine your breakpoints, it is important not to exit GDB during a debug and recompile session. Note also that breakpoints at fixed line numbers can change after you've edited and recompiled the code! If you do want to quit GDB, you can use the .gdbinit file to save your breakpoints and other GDB commands.

There are many different ways to delete a breakpoint. Some examples are:

  delete
  delete 1 2 3
  clear
  clear myfunction
  clear myfile:myfunction
  clear linenumber
  clear myfile:linenumber

Similary for disabling breakpoints:

  disable
  disable 1 3
  enable 1 3
  enable once 1 3

In DDD and Eclipse, deleting and disabling breakpoints can easily be done with the mouse. Drag and drop of breakpoints also works in DDD, which by the way also has undo/redo functionality.

The info breakpoints command gives you the following information:

• Identifier (Num): unique identifier
• Type (Type): breakpoint, watchpoint or catchpoint
• Disposition (Disp): keep, del, dis
• Enable Status (Enb)
• Address (Address): location in memory
• Location (What)

and it also gives you how many times a breakpoint was hit already.

To resume execution at a breakpoint, you can use the step and continue commands.

Book review: The Art of Debugging with GDB, DDD and Eclipse (Chapter 1).

In order to become a better programmer, I am reading books.  However, sometimes, reading books thoroughly and remembering their content is not always easy.  I therefore decided to write summarizing reviews of the books that I read.  Below is the first of a set of summaries for the book `The Art of Debugging with GDB, DDD and Eclipse'.

Chapter 1

The book covers only GDB, DDD and Eclipse as debugging tools and C and C++ as programming languages.  The first chapter defines the Principle of Confirmation as `you as a programmer trying to confirm that what you believe is true, is actually true'.  Debuggers can help with this quest for confirmation.  Adding trace code (printf and friends) is discouraged: using a debugger is of course much easier and takes far less time.  Debuggers also allow you to find the location of a bug and set watchpoints.  The authors mention the following debugging principles:

• Start small: run your program on easy, simple cases.
• Use a top-down approach: do not step into functions when not necessary.
• Use a debugging tool to determine the location of a segmentation fault.
• Determine the location of an infinite loop by using your debugger's interrupt and backtrace features.
• Use binary search.

There are some differences between the text-based GDB debugger and GUI-based debuggers like DDD and Eclipse.  Setting and clearing breakpoints and stepping through code is much easier in DDD and Eclipse.  However, using GDB also has some advantages:

• It starts up quickly.
• It can be used in cases where a GUI-debugger cannot be used (ssh remote sessions etc).
• It needs less screen space.
• Debugging a GUI with a GUI can be problematic.

GDB also has a Terminal User Interface (TUI) mode which can be invoked with the -tui option or by typing CTRL-X-A when in GDB.  In TUI-mode, use the arrow keys to scroll up and down the source code.

The main debugger operations are:

• Stepping through code: breakpoints, single-stepping (next=next line, step=enter function), resume (continue), temporary breakpoints (tbreak, until, finish).
• Inspecting variables: in GDB this can be done with the print command, in DDD and Eclipse by hovering over a variable with the mouse.
• Setting watchpoints: pausing execution whenever a value of a specified variable changes.  A GDB example is watch (i>10).
• Moving up and down the call stack: in GDB you use the frame, up, down and backtrace commands for this.
  

Getting help about a GDB-command can be done using the straightforward help command (e.g. help breakpoints).

The chapter ends with an introductory debugging session.  Interesting new commands that appear in this session are break and condition.  Another interesting GDB-feature that is mentioned is its .gdbinit startup files.  Somewhere in the chapter, the authors also mention cgdb as an alternative to GDB.

Stay tuned for Chapter 2!