Debugging OPEN-R applications efficiently with openSDK

One of the biggest advantages of openSDK is that it allows you to run the usual debug tools on OPEN-R applications. In this document we will demonstrate how to use both GDB and Valgrind.

Unfortunately, at the time of writing, most debug tools don't handle dlopen'ed libraries after a chroot call very well (some even crash). This means that to get useful debug information you'll want to run openSDK without chroot most of the times (by specifying the --chroot=no argument).

 

General Debug Tips

To get good debug results you need to compile the source files with debug symbols (-g option). The standard Sony makefiles already do this, but they 'strip' the binaries after compiling the files, thus loosing the debug symbols. So if you are using similar makefiles you need to skip the 'strip' program execution. Our samples's makefiles have been patched so that you can make them skip 'strip' (by calling 'make STRIPOBJECTS=0').

Other source of good information is the compiler. To get the most of it, compile your programs with the -Wall argument and use a recent gcc version (newer versions generate more and better warnings). This way you'll be able to fix possible crashes before they happen.

 

Valgrind

Valgrind is a very powerful (yet simple) debugging tool that reports memory errors (like accesses to non-initialized memory or buffer overflows), memory leaks and crashes. Valgrind can also be used to profile applications (time, memory and cache usage). Running Valgrind with openSDK is pretty straitforward:

linux openSDK # valgrind ./OPENRloader --chroot=no samples/common/RobotDesign
==12521== Memcheck, a memory error detector.
==12521== Copyright (C) 2002-2007, and GNU GPL'd, by Julian Seward et al.
==12521== Using LibVEX rev 1732, a library for dynamic binary translation.
==12521== Copyright (C) 2004-2007, and GNU GPL'd, by OpenWorks LLP.
==12521== Using valgrind-3.2.3, a dynamic binary instrumentation framework.
==12521== Copyright (C) 2000-2007, and GNU GPL'd, by Julian Seward et al.
==12521== For more details, rerun with: -v
==12521==
not running with chroot()
Starting boot procedures for ERS-7...

Initializing thread pool ...
thread pool created with 16 threads.
not connected to the actuators server
Loaded module: MS/OPEN-R/MW/OBJS/POWERMON.BIN (thread id: 285694864::3)
RobotDesign::DoInit()
Loaded module: MS/OPEN-R/MW/OBJS/ERS-7.BIN (thread id: 294087568::4)
RobotDesign::DoStart()
==12521== Thread 26:
==12521== Conditional jump or move depends on uninitialised value(s)
==12521==    at 0x402BBD7: RobotDesign::DoStart(OSystemEvent const&) (RobotDesign.cc:40)
==12521==    by 0x402B878: __start_module (RobotDesignStub.cc:146)
==12521==    by 0x805A17B: module_executor(void*) (helper.cc:40)
==12521==    by 0x40414BA: start_thread (in /lib/libpthread-2.5.so)
==12521==    by 0x42349CD: clone (in /lib/libc-2.5.so)
==12521==
==12521== Invalid write of size 2
==12521==    at 0x402BBE2: RobotDesign::DoStart(OSystemEvent const&) (RobotDesign.cc:43)
==12521==    by 0x402B878: __start_module (RobotDesignStub.cc:146)
==12521==    by 0x805A17B: module_executor(void*) (helper.cc:40)
==12521==    by 0x40414BA: start_thread (in /lib/libpthread-2.5.so)
==12521==    by 0x42349CD: clone (in /lib/libc-2.5.so)
==12521==  Address 0x42F8FA4 is 4 bytes inside a block of size 5 alloc'd
==12521==    at 0x4021888: malloc (in /usr/lib/valgrind/x86-linux/vgpreload_memcheck.so)
==12521==    by 0x402BBD3: RobotDesign::DoStart(OSystemEvent const&) (RobotDesign.cc:39)
==12521==    by 0x402B878: __start_module (RobotDesignStub.cc:146)
==12521==    by 0x805A17B: module_executor(void*) (helper.cc:40)
==12521==    by 0x40414BA: start_thread (in /lib/libpthread-2.5.so)
==12521==    by 0x42349CD: clone (in /lib/libc-2.5.so)
OPENR::GetRobotDesign() : ERS-7
I am ERS-7.

In the example above we present the RobotDesign example from Sony, but modified to generate some errors. When reading the stack traces you can ignore the calls until the __start_module() function, as anything below that belongs to openSDK itself. Reports about memory leakages in pthread_create() can also be safely ignored.

 

GDB

Running GDB is easy as well as Valgrind

linux openSDK # gdb ./OPENRloader
GNU gdb 6.6
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
Using host libthread_db library "/lib/libthread_db.so.1".
(gdb) run --chroot=no samples/common/RobotDesign
Starting program: /cvs/openSDK/openSDK/OPENRloader --chroot=no samples/common/RobotDesign
[Thread debugging using libthread_db enabled]
[New Thread -1211237696 (LWP 13050)]
not running with chroot()
Starting boot procedures for ERS-7...

Initializing thread pool ...
[New Thread -1211241584 (LWP 13053)]
(...)
thread pool created with 16 threads.
[New Thread -1345524848 (LWP 13069)]
[New Thread -1353917552 (LWP 13070)]
not connected to the actuators server
[New Thread -1404273776 (LWP 13076)]
Loaded module: MS/OPEN-R/MW/OBJS/POWERMON.BIN (thread id: 2890693520::3)
[New Thread -1412822128 (LWP 13077)]
Loaded module: MS/OPEN-R/MW/OBJS/ERS-7.BIN (thread id: 2882145168::4)
RobotDesign::DoInit()
RobotDesign::DoStart()

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1412408432 (LWP 13342)]
0xb7daeb80 in strcpy () from /lib/libc.so.6
(gdb) bt
#0  0xb7daeb80 in strcpy () from /lib/libc.so.6
#1  0xb7fb8c1a in RobotDesign::DoStart (this=0xb7fba568, event=@0xabd05310) at RobotDesign.cc:43
#2  0xb7fb88a9 in __start_module () at RobotDesignStub.cc:146
#3  0x0805a2fc in module_executor (arg=0x8066cc8) at loader/helper.cc:40
#4  0xb7f974bb in start_thread () from /lib/libpthread.so.0
#5  0xb7e029ce in clone () from /lib/libc.so.6
(gdb) up
#1  0xb7fb8c1a in RobotDesign::DoStart (this=0xb7fba568, event=@0xabd05310) at RobotDesign.cc:43
43              strcpy(non_init_var, NULL);

In this example we have modified the RobotDesign example from Sony to crash with a simple strcpy() call (as can be seen in the GDB output). As with Valgrind, GDB stack traces also contain references to openSDK internal functions that you can ignore (contexts until the __start_module() function).

 

AIBO and OPEN-R are a trademark or a registered trademark of Sony Corporation.