Name

Thread Debugging — Overview of eCos Kernel thread-aware debugging

Description

Thread-aware debugging refers to the ability to interrogate the list of threads active within an application when the system is stopped (halted). This is normally when the code has either stopped at a breakpoint, or when execution is interrupted via a hosted debug session (e.g. from GDB).

Helper Symbols

For eCosPro to aid external host-based debug tools, a set of helper symbols are defined to provide information on the size (width) and offset of useful fields, or relevant constant values, instead of addresses. The majority of these symbols are named to avoid possible namespace clashes with applications, but for historical reasons some architecture specific symbols are valid in the C/C++ namespace. Similarly different architectures export their own symbols.

An external tool that wishes to interpret thread information can check for the presence of the specifically named symbol, and add support accordingly.

These symbols are held in the symbol table of the ELF file, but have no cost impact (code or data size) on the actual binary loaded into the target (either via a debugger, or if an application binary is stored on the target). Obviously a stripped executable will lose the helper symbols, but debugging using a stripped ELF file would always pose some restrictions.

Required

These symbols are required for accessing the list of threads and the currently active thread. If they are not present in the symbol table then it indicates an eCos build without a thread scheduler, and hence there is no need for thread-aware debug support.

Cyg_Thread::thread_list
Pointer to the first thread descriptor in the chain of created threads.
Cyg_Scheduler_Base::current_thread
Pointer to the thread context for the currently active thread.

Common

When an eCos scheduler is configured, information describing the thread context is provided to enable generic scanning code to be implemented in an external tool regardless whether some eCos features are enabled or disabled. Since individual eCos configurations can have features present that change the shape of the actual thread descriptor structure, we need the important fields for scanning a list of threads to be available in each ELF file.

__ecospro_syminfo.size.cyg_thread.list_next, __ecospro_syminfo.size.cyg_thread.state, __ecospro_syminfo.size.cyg_thread.sleep_reason, __ecospro_syminfo.size.cyg_thread.wake_reason, __ecospro_syminfo.size.cyg_thread.unique_id, __ecospro_syminfo.size.cyg_thread.name, __ecospro_syminfo.size.cyg_thread.priority, __ecospro_syminfo.size.cyg_thread.stack_ptr
The presence of a field in the thread descriptor structure can be determined by a non-zero size symbol being provided. These symbols give the size, in bytes, of the relevant field at the offset specified by the corresponding __ecospro_syminfo.off.* symbol. This allows any host tool to provide features based on the conditional presence of fields.
__ecospro_syminfo.off.cyg_thread.list_next
Offset to field that points to the next thread descriptor.
__ecospro_syminfo.off.cyg_thread.state, __ecospro_syminfo.off.cyg_thread.sleep_reason, __ecospro_syminfo.off.cyg_thread.wake_reason, __ecospro_syminfo.off.cyg_thread.unique_id, __ecospro_syminfo.off.cyg_thread.name, __ecospro_syminfo.off.cyg_thread.priority
Offsets for useful fields in a thread descriptor structure.
__ecospro_syminfo.off.cyg_thread.stack_ptr
This symbol is the offset for the field containing the address of the stacked register context for inactive threads.

Cortex-M

Cortex-M Base

For Cortex-M targets the following symbols describe the main CPU register state. The presence of the symbol __ecospro_syminfo.cortexm.thread.saved can be used as indicator of an eCos Cortex-M application.

__ecospro_syminfo.cortexm.thread.saved
This symbol provides the actual PC address of the point in the code where a switch actually occurs, It may be useful depending on how the external tool interprets the stacked context information.
__ecospro_syminfo.value.HAL_SAVEDREGISTERS.THREAD, __ecospro_syminfo.value.HAL_SAVEDREGISTERS.EXCEPTION, __ecospro_syminfo.value.HAL_SAVEDREGISTERS.INTERRUPT
These symbols provide the values in the type field used to identify the shape of the stacked context. Since only threads will be accessed by the Cyg_Thread::thread_list list the host tools should only ever encounter __ecospro_syminfo.value.HAL_SAVEDREGISTERS.THREAD type contexts (with the optional FPU register state indicator flag). The values for the other types of Cortex-M contexts are provided for completeness only.
__ecospro_syminfo.size.HAL_SavedRegisters.Thread
This symbol provides the total size, in bytes, of a stacked CPU context for contexts of type __ecospro_syminfo.value.HAL_SAVEDREGISTERS.THREAD.
__ecospro_syminfo.size.HAL_SavedRegisters.u.thread.r
Total size, in bytes, of all the core CPU registers present in a stacked context.
__ecospro_syminfo.size.HAL_SavedRegisters.u.thread.type
Size, in bytes, of the field that encodes the type of stacked context. Since for Cortex-M targets the actual context stored (and its size) depends on whether the individual thread has any hardware FPU context saved. This is needed to ensure only valid information is used when dealing with lazy per-thread hardware FPU support.
__ecospro_syminfo.size.HAL_SavedRegisters.u.thread.basepri, __ecospro_syminfo.size.HAL_SavedRegisters.u.thread.sp, __ecospro_syminfo.size.HAL_SavedRegisters.u.thread.pc
Size of individual context fields referenced by the corresponding offset symbol.
__ecospro_syminfo.off.HAL_SavedRegisters.u.thread.type, __ecospro_syminfo.off.HAL_SavedRegisters.u.thread.basepri, __ecospro_syminfo.off.HAL_SavedRegisters.u.thread.sp, __ecospro_syminfo.off.HAL_SavedRegisters.u.thread.r, __ecospro_syminfo.off.HAL_SavedRegisters.u.thread.pc
Offsets into the stacked context for the core register values.

Cortex-M FPU

When a Cortex-M configuration with hardware FPU support configured is used then the following optional symbols will be present with non-zero values where appropriate.

__ecospro_syminfo.value.HAL_SAVEDREGISTERS.WITH_FPU
This value provides the bitmask flag OR-ed into the type field provided at the __ecospro_syminfo.off.HAL_SavedRegisters.u.thread.type offset used to identify individual thread contexts that contain FPU state.
__ecospro_syminfo.size.HAL_SavedRegisters.u.thread.fpscr
Size, in bytes, of the FPSCR register stacked in the context.
__ecospro_syminfo.size.HAL_SavedRegisters.u.thread.s
Total size, in bytes, of the single-precision vector stacked in the context.
__ecospro_syminfo.off.HAL_SavedRegisters.u.thread.fpscr
Offset of FPSCR register in the stacked context.
__ecospro_syminfo.off.HAL_SavedRegisters.u.thread.s
Offset of the single-precision register vector in the stacked context.

ARM

ARM/Cortex-A Base

For ARM/Cortex-A targets the following symbols describe the main CPU register state. The presence of the symbol ARMREG_SIZE can be used as indicator of an eCos arm architecture application.

ARMREG_SIZE
This symbol, if non-zero, provides the total size of the stacked context for inactive threads.
armreg_r0, armreg_r1, armreg_r2, armreg_r3, armreg_r4, armreg_r5, armreg_r6, armreg_r7, armreg_r8, armreg_r9, armreg_r10, armreg_fp, armreg_ip, armreg_sp, armreg_lr, armreg_pc, armreg_cpsr
These symbols provide the offset of the corresponding register within the stacked context referenced from the thread object __ecospro_syminfo.off.cyg_thread.stack_ptr field.

ARM/Cortex-A FPU

Optional ARM FPU symbols.

ARMREG_FPUCONTEXT_SIZE
If non-zero then this symbol indicates that the eCos applicaton has been configured with hardware FPU support. The value is the total size of the stacked inactive thread context.
armreg_fpscr
Offset with the stacked context of the FPSCR register.

ARM/Cortex-A FPU Single-Precision

Symbols present when single-precision ARM FPU is configured.

armreg_s_vec
The offset to the start of the stacked single-precision register vector.
ARMREG_S_COUNT
Number of single-precision registers present from the armreg_s_vec offset.

ARM/Cortex-A FPU Double-Precision

Symbols present when doubled-precision ARM FPU is configured.

armreg_vfp_vec
The offset to the start of the stacked double-precision register vector.
ARMREG_VFP_COUNT
Number of single-precision registers present from the armreg_vfp_vec offset.

ColdFire

Symbols provided by eCos ColdFire targets. The presence of the symbol hal_context_pcsr_offset can be used as an identifier for ColdFire targets.

hal_context_size
The overall stacked context size.
hal_context_fpu_size
If hardware FPU support is configured this symbol provides the size of the stacked FPU context.
hal_context_pcsr_size, hal_context_integer_size
These symbols provide information on the size (width) of individual registers.
hal_context_pcsr_offset, hal_context_integer_d0_offset, hal_context_integer_d2_offset, hal_context_integer_a0_offset, hal_context_integer_a2_offset, hal_context_fpu_offset, hal_context_other_offset
The offsets within the stacked context for the processor state.
hal_context_rte_adjust
The value, in bytes, if a PCSR RTE adjustment is used.

Other Useful Symbols

Some other standard eCos symbols may be present that could also be useful for external debug tools.

idle_thread
This symbol can be used to identify the idle (background) thread descriptor object if useful to the thread-aware host debug tool.
cyg_libc_main_thread
This symbol will not be present in the application symbol table if the relevant object is not defined. When CYGSEM_LIBC_STARTUP_MAIN_THREAD is configured this symbol can be used to reference the thread descriptor object for the main() C thread created by the run-time.

GDB

The following documentation uses the GDB command-line interface for its examples, although the thread-aware debug support is applicable to applications that access GDB via its programmatic interface, e.g. Eclipse.

When a GDB debug session halts the CPU, either from the code hitting a previously set breakpoint or via the user requesting a halt, it will display the state of the currently active CPU state and select the currently executing thread. When displaying threads via the info threads command the currently selected thread is highlighted by an asterisk ( *) character. Therefore, immediately after a halt this will indicate the active, running, thread. Examining the CPU register state will report the state of this active thread.

With thread-aware debugging for the target application available, GDB will display a list of all known threads when given the command info threads. The command thread id can be used to switch context to other threads, providing the ability to examine their CPU register state, call stack, and local variables.

[Note]Notes:
  1. The documentation for all GDB features is beyond the scope of this reference. Please refer to the website GDB: The GNU Project Debugger for definitive documentation of the GDB thread debug support.
  2. All GDB execution operations such as single stepping and return from function call will always apply to the currently active, executing thread, NOT the thread currently selected by the developer or user within the debugger.

Depending on the target system being connected, the act of loading an eCos application into memory will not necessarily initialise all the memory and hardware state. The eCos application run-time startup code will normally initialise memory alongside other I/O requirements. Since it can be possible to execute thread interrogation commands before any target code has been executed, it can be useful having helper macros in your .gdbinit script to minimise misinformation being displayed. The following clear_ecos_thread_pointers GDB macro is an example which could be executed after loading an application and before any system initialisation code in the loaded application has been executed. It ensures that debug commands to interrogate thread state will not parse stale/undefined information from uninitialised memory.

define clear_ecos_thread_pointers
  set *((unsigned int *)&Cyg_Thread::thread_list) = 0
  set *((unsigned int *)&Cyg_Scheduler_Base::current_thread) = 0
end

document clear_ecos_thread_pointers
When starting a new debug session from application reset the run-time
code that clears BSS will not have been executed, so stale/unitialised
state may be present in memory. For RTOS aware thread debugging as
provided by external tools the GDB server may be confused and report
invalid state if the thread state is interrogated before the initial
eCos run-time initialisation has cleared the BSS area. This macro just
ensures that the relevant eCos pointers are NULL prior to debugging.
end

Of course if a hardware debugger is being used to connect to an existing application session (rather than loading and starting a new application session) then the macro should not be called.

It is useful to wrap the steps needed to connect to a target in a helper macro. e.g.:

define connocd
  target extended-remote localhost:3333
  load
  break cyg_test_exit
  break cyg_assert_fail
  display/i $pc
  clear_ecos_thread_pointers
end

So that all of the normal steps for loading and setting the debug environment for an application can be performed by a single command:

(gdb) connocd
0x080016dc in ?? ()

Loading section .rom_vectors, size 0x8 lma 0x90000000
Loading section .text, size 0x7b34 lma 0x90000008
Loading section .rodata, size 0x678 lma 0x90007b40
Loading section .data, size 0x180 lma 0x900081b8
Start address 0x90000008, load size 33588
Transfer rate: 94 KB/sec, 6717 bytes/write.
Breakpoint 1 at 0x900040f4: file ecospro-path/packages/infra/current/src/tcdiag.cxx, line 391.
Function "cyg_assert_fail" not defined.
Make breakpoint pending on future shared library load? (y or [n]) [answered N; input not from terminal]
(gdb)

The use of such macros from a GDB script file can make the task of debugging less cumbersome.

Ronetix PEEDI

[Note]Note

The Ronetix PEEDI firmware must be updated to at least version 21.2.0 to ensure the correct operation of the thread-aware debugging support.

The PEEDI [TARGET] section option COREn_OS can be used to introduce a thread/context description using the generic PEEDI support.

The example PEEDI configuration files supplied with eCosPro releases 4.5.8 and above should already have suitable RTOS support fragments. For example, the file packages/hal/arm/arm9/sam9g45ek/<version>/misc/peedi.sam9g45ek.cfg contains a [OS_ECOS_ARM] section, referenced from the head of the file via the CORE0_OS=OS_ECOS_ARM setting.

With a suitable COREn_OS the PEEDI will parse the eCos thread lists and stacked register contents when interrogating threads other than the current thread of executing on the CPU.

OpenOCD

OpenOCD provides the -rtos eCos option that can be used to configure thread-aware debug support in the configuration file used for the OpenOCD session.

[Note]Note

As of 2021-04-01 the generic eCos thread-aware debug support is specific to eCosPro releases of OpenOCD only. However, the intent is to contribute support for generic eCos thread-aware debug support to the OpenOCD project.

At its simplest the OpenOCD configuration file just needs to specify:

$_TARGETNAME configure -rtos eCos

NOTE: When OpenOCD -rtos support for eCos is configured the act of executing target remote or target extended-remote to connect to an OpenOCD instance will cause the configured OpenOCD RTOS support to perform an update_threads operation against the current memory state. This is to allow a debug session to be attached to an active system. However, it does mean that for an undefined memory state (power-on, CPU reset with undefined DRAM state, an application with a different thread context shape to the previous application new different-configuration application to be subsequently loaded after connecting to the OpenOCD GDB server) that the GDB server may report spurious thread information upon request.

The following is example output when the application is halted in a thread named "busy". The name, state and priority of the other available threads is also shown:

(gdb) info thr
 Id  Target Id         Frame
* 1  Thread 12 (Name: busy, State: Ready Pri: 20) 0x20009f50 in thread_busy (data=30000) at
     ecospro-path/packages/kernel/current/tests/fpint_thread_switch.cxx:566
  2  Thread 1 (Name: Idle Thread, State: Ready Pri: 31) Cyg_Scheduler::unlock_inner (new_lock=0) at
     ecospro-path/packages/kernel/current/src/sched/sched.cxx:233
  3  Thread 2 (Name: Test, State: Sleeping (WAIT) Pri: 3) Cyg_Scheduler::unlock_inner (new_lock=1) at
     ecospro-path/packages/kernel/current/src/sched/sched.cxx:233
  4  Thread 11 (Name: highpri, State: Sleeping (DELAY) Pri: 10) Cyg_Scheduler::unlock_inner (new_lock=0)
     at ecospro-path/packages/kernel/current/src/sched/sched.cxx:233

The OpenOCD GDB server can be left executing between GDB application debug sessions. It does not need to be re-started for every GDB session.

GDB stubs

Unlike the hardware debug approaches described above which benefit from bare-metal hardware support (SWD, JTAG, BDM, etc.), eCos also supports the use of GDBstubs which can be built into the application and accessed via an I/O channel (e.g. serial, Ethernet, etc.), or provided via a boot monitor/loader (e.g. RedBoot covering in Chapter 208, Getting Started with RedBoot) that provides an environment for executing applications. Such support has a run-time cost (code+data space as well as CPU cycles), and is only usable after some level of system initialisation has occurred. A hardware debug solution is therefore preferred.

However, if GDBstubs is the only available/possible solution, the eCos GDBstubs implementation supports thread aware debugging.