Remote Debug With gdb/gdbserve

Overview

Remote debugging is useful or necessary, for example, in following scenarios:

• There’s no full debugger on the target host, but a small stub (e.g. gdbserver) is available.
• There’s no full source on the target host(for various reasons, such as size, security, etc). In a large project, synchronizing source codes on different hosts is either convenient nor safe. This is probably the most common case in the real world.
• Debug input/output may pollute the target application input/output, for example, you are debugging a full-screen editor.

In practice, it’s better to have gdb and gdbserver with the same version. I had an experience that gdb (on centos5, v7.0.1) couldn’t connect to gdbserver (on debian 8, v7.7.1).

In the following example, the source codes locate on the host debian, the application is build and debug on debian, and the application is actually run on the remote host centos.

Prepare The Executable

No need to say, you need -g to keep symbols when you build your application. One additional tip: because the target application runs remotely, you don’t actually need to keep symbols in the target binary.

jason@debian$ gcc -g -o app app.c
jason@debian$ objcopy --only-keep-debug app app.debug
jason@debian$ strip -g -o app.remote app
jason@debian$ scp app.remote tony@centos:path/app

Explanation:

1. build app with symbol
2. extract symbols into a separate file. Now app.debug contains all debug information. This is common distribution practice.
3. generate a version of executable without symbols, which (app.remote) is the version you actually distribute. The binary app still contains full symbols for debugger (you can, however, using app.remote plus app.debug, but what’s the point of that?)
4. distribute the binary to the target host (host: centos, user: tony)

Launch At Target Host

On the target host (centos), start the application

tony@centos$ gdbserver localhost:4444 app
Process app created; pid = 10307
Listening on port 4444

The example uses tcp protocol. You can use serial com, in some special cases (such as in embedded system). The host part in host:port pair is not actually used in the current gdbserver version, so you can give it anything.

Start Debugging Session

Start debugging on the host debian. From the perspective of execution (on centos), gdb is running remotely (on debian); from the perspective of debugger, gdb (on debian) is debugging a program running on the remote host (centos).

jason@debian$ gdb app
... license info
Reading symbols from /home/jason/app...done.
(gdb) target remote centos:4444
Remote debugging using centos:4444
Reading symbos from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
(gdb) break main
(gdb) run

Once gdb connects to gdbserver, you can debug as usual as a normal gdb session, such as step, break, print variables, etc. One thing to remember is that the input and output of the program happen on the remote host.

On the remote host (centos), the gdbserver session looks like

...
Listening on port 4444
Remote debugging from host 192.168.205.96

... normal application input/output

Child exited with status 0
GDBserver exiting
tony@centos$