Kernel thread-aware debugging

Kernel thread-aware debugging
	Part I. The eCos Kernel

Description

Thread-aware debugging refers to the ability to interrogate the list of threads active within an application when the system is stopped (halted). This is normally when the code has either stopped at a breakpoint, or when execution is interrupted via a hosted debug session (e.g. from GDB).

Helper Symbols

For eCosPro to aid external host-based debug tools, a set of helper symbols are defined to provide information on the size (width) and offset of useful fields, or relevant constant values, instead of addresses. The majority of these symbols are named to avoid possible namespace clashes with applications, but for historical reasons some architecture specific symbols are valid in the C/C++ namespace. Similarly different architectures export their own symbols.

An external tool that wishes to interpret thread information can check for the presence of the specifically named symbol, and add support accordingly.

These symbols are held in the symbol table of the ELF file, but have no cost impact (code or data size) on the actual binary loaded into the target (either via a debugger, or if an application binary is stored on the target). Obviously a stripped executable will lose the helper symbols, but debugging using a stripped ELF file would always pose some restrictions.

Required

These symbols are required for accessing the list of threads and the currently active thread. If they are not present in the symbol table then it indicates an eCos build without a thread scheduler, and hence there is no need for thread-aware debug support.

Cyg_Thread::thread_list: Pointer to the first thread descriptor in the chain of created threads.
Cyg_Scheduler_Base::current_thread: Pointer to the thread context for the currently active thread.

Common

When an eCos scheduler is configured, information describing the thread context is provided to enable generic scanning code to be implemented in an external tool regardless whether some eCos features are enabled or disabled. Since individual eCos configurations can have features present that change the shape of the actual thread descriptor structure, we need the important fields for scanning a list of threads to be available in each ELF file.

__ecospro_syminfo.size.cyg_thread.list_next, __ecospro_syminfo.size.cyg_thread.state, __ecospro_syminfo.size.cyg_thread.sleep_reason, __ecospro_syminfo.size.cyg_thread.wake_reason, __ecospro_syminfo.size.cyg_thread.unique_id, __ecospro_syminfo.size.cyg_thread.name, __ecospro_syminfo.size.cyg_thread.priority, __ecospro_syminfo.size.cyg_thread.stack_ptr: The presence of a field in the thread descriptor structure can be determined by a non-zero size symbol being provided. These symbols give the size, in bytes, of the relevant field at the offset specified by the corresponding __ecospro_syminfo.off.* symbol. This allows any host tool to provide features based on the conditional presence of fields.
__ecospro_syminfo.off.cyg_thread.list_next: Offset to field that points to the next thread descriptor.
__ecospro_syminfo.off.cyg_thread.state, __ecospro_syminfo.off.cyg_thread.sleep_reason, __ecospro_syminfo.off.cyg_thread.wake_reason, __ecospro_syminfo.off.cyg_thread.unique_id, __ecospro_syminfo.off.cyg_thread.name, __ecospro_syminfo.off.cyg_thread.priority: Offsets for useful fields in a thread descriptor structure.
__ecospro_syminfo.off.cyg_thread.stack_ptr: This symbol is the offset for the field containing the address of the stacked register context for inactive threads.

Cortex-M

Cortex-M Base

For Cortex-M targets the following symbols describe the main CPU register state. The presence of the symbol __ecospro_syminfo.cortexm.thread.saved can be used as indicator of an eCos Cortex-M application.

__ecospro_syminfo.cortexm.thread.saved: This symbol provides the actual PC address of the point in the code where a switch actually occurs, It may be useful depending on how the external tool interprets the stacked context information.
__ecospro_syminfo.value.HAL_SAVEDREGISTERS.THREAD, __ecospro_syminfo.value.HAL_SAVEDREGISTERS.EXCEPTION, __ecospro_syminfo.value.HAL_SAVEDREGISTERS.INTERRUPT: These symbols provide the values in the type field used to identify the shape of the stacked context. Since only threads will be accessed by the Cyg_Thread::thread_list list the host tools should only ever encounter __ecospro_syminfo.value.HAL_SAVEDREGISTERS.THREAD type contexts (with the optional FPU register state indicator flag). The values for the other types of Cortex-M contexts are provided for completeness only.
__ecospro_syminfo.size.HAL_SavedRegisters.Thread: This symbol provides the total size, in bytes, of a stacked CPU context for contexts of type __ecospro_syminfo.value.HAL_SAVEDREGISTERS.THREAD.
__ecospro_syminfo.size.HAL_SavedRegisters.u.thread.r: Total size, in bytes, of all the core CPU registers present in a stacked context.
__ecospro_syminfo.size.HAL_SavedRegisters.u.thread.type: Size, in bytes, of the field that encodes the type of stacked context. Since for Cortex-M targets the actual context stored (and its size) depends on whether the individual thread has any hardware FPU context saved. This is needed to ensure only valid information is used when dealing with lazy per-thread hardware FPU support.
__ecospro_syminfo.size.HAL_SavedRegisters.u.thread.basepri, __ecospro_syminfo.size.HAL_SavedRegisters.u.thread.sp, __ecospro_syminfo.size.HAL_SavedRegisters.u.thread.pc: Size of individual context fields referenced by the corresponding offset symbol.
__ecospro_syminfo.off.HAL_SavedRegisters.u.thread.type, __ecospro_syminfo.off.HAL_SavedRegisters.u.thread.basepri, __ecospro_syminfo.off.HAL_SavedRegisters.u.thread.sp, __ecospro_syminfo.off.HAL_SavedRegisters.u.thread.r, __ecospro_syminfo.off.HAL_SavedRegisters.u.thread.pc: Offsets into the stacked context for the core register values.

Cortex-M FPU

When a Cortex-M configuration with hardware FPU support configured is used then the following optional symbols will be present with non-zero values where appropriate.

__ecospro_syminfo.value.HAL_SAVEDREGISTERS.WITH_FPU: This value provides the bitmask flag OR-ed into the type field provided at the __ecospro_syminfo.off.HAL_SavedRegisters.u.thread.type offset used to identify individual thread contexts that contain FPU state.
__ecospro_syminfo.size.HAL_SavedRegisters.u.thread.fpscr: Size, in bytes, of the FPSCR register stacked in the context.
__ecospro_syminfo.size.HAL_SavedRegisters.u.thread.s: Total size, in bytes, of the single-precision vector stacked in the context.
__ecospro_syminfo.off.HAL_SavedRegisters.u.thread.fpscr: Offset of FPSCR register in the stacked context.
__ecospro_syminfo.off.HAL_SavedRegisters.u.thread.s: Offset of the single-precision register vector in the stacked context.

ARM

ARM/Cortex-A Base

For ARM/Cortex-A targets the following symbols describe the main CPU register state. The presence of the symbol ARMREG_SIZE can be used as indicator of an eCos arm architecture application.

ARMREG_SIZE: This symbol, if non-zero, provides the total size of the stacked context for inactive threads.
armreg_r0, armreg_r1, armreg_r2, armreg_r3, armreg_r4, armreg_r5, armreg_r6, armreg_r7, armreg_r8, armreg_r9, armreg_r10, armreg_fp, armreg_ip, armreg_sp, armreg_lr, armreg_pc, armreg_cpsr: These symbols provide the offset of the corresponding register within the stacked context referenced from the thread object __ecospro_syminfo.off.cyg_thread.stack_ptr field.

ARM/Cortex-A FPU

Optional ARM FPU symbols.

ARMREG_FPUCONTEXT_SIZE: If non-zero then this symbol indicates that the eCos applicaton has been configured with hardware FPU support. The value is the total size of the stacked inactive thread context.
armreg_fpscr: Offset with the stacked context of the FPSCR register.

ARM/Cortex-A FPU Single-Precision

Symbols present when single-precision ARM FPU is configured.

armreg_s_vec: The offset to the start of the stacked single-precision register vector.
ARMREG_S_COUNT: Number of single-precision registers present from the armreg_s_vec offset.

ARM/Cortex-A FPU Double-Precision

Symbols present when doubled-precision ARM FPU is configured.

armreg_vfp_vec: The offset to the start of the stacked double-precision register vector.
ARMREG_VFP_COUNT: Number of single-precision registers present from the armreg_vfp_vec offset.

ColdFire

Symbols provided by eCos ColdFire targets. The presence of the symbol hal_context_pcsr_offset can be used as an identifier for ColdFire targets.

hal_context_size: The overall stacked context size.
hal_context_fpu_size: If hardware FPU support is configured this symbol provides the size of the stacked FPU context.
hal_context_pcsr_size, hal_context_integer_size: These symbols provide information on the size (width) of individual registers.
hal_context_pcsr_offset, hal_context_integer_d0_offset, hal_context_integer_d2_offset, hal_context_integer_a0_offset, hal_context_integer_a2_offset, hal_context_fpu_offset, hal_context_other_offset: The offsets within the stacked context for the processor state.
hal_context_rte_adjust: The value, in bytes, if a PCSR RTE adjustment is used.

Other Useful Symbols

Some other standard eCos symbols may be present that could also be useful for external debug tools.

idle_thread: This symbol can be used to identify the idle (background) thread descriptor object if useful to the thread-aware host debug tool.
cyg_libc_main_thread: This symbol will not be present in the application symbol table if the relevant object is not defined. When CYGSEM_LIBC_STARTUP_MAIN_THREAD is configured this symbol can be used to reference the thread descriptor object for the main() C thread created by the run-time.

GDB

The following documentation uses the GDB command-line interface for its examples, although the thread-aware debug support is applicable to applications that access GDB via its programmatic interface, e.g. Eclipse.

When a GDB debug session halts the CPU, either from the code hitting a previously set breakpoint or via the user requesting a halt, it will display the state of the currently active CPU state and select the currently executing thread. When displaying threads via the info threads command the currently selected thread is highlighted by an asterisk ( *) character. Therefore, immediately after a halt this will indicate the active, running, thread. Examining the CPU register state will report the state of this active thread.

With thread-aware debugging for the target application available, GDB will display a list of all known threads when given the command info threads. The command thread id can be used to switch context to other threads, providing the ability to examine their CPU register state, call stack, and local variables.

	Notes:
The documentation for all GDB features is beyond the scope of this reference. Please refer to the website GDB: The GNU Project Debugger for definitive documentation of the GDB thread debug support. All GDB execution operations such as single stepping and return from function call will always apply to the currently active, executing thread, NOT the thread currently selected by the developer or user within the debugger.

Notes:

The documentation for all GDB features is beyond the scope of this reference. Please refer to the website GDB: The GNU Project Debugger for definitive documentation of the GDB thread debug support.
All GDB execution operations such as single stepping and return from function call will always apply to the currently active, executing thread, NOT the thread currently selected by the developer or user within the debugger.

Depending on the target system being connected, the act of loading an eCos application into memory will not necessarily initialise all the memory and hardware state. The eCos application run-time startup code will normally initialise memory alongside other I/O requirements. Since it can be possible to execute thread interrogation commands before any target code has been executed, it can be useful having helper macros in your .gdbinit script to minimise misinformation being displayed. The following clear_ecos_thread_pointers GDB macro is an example which could be executed after loading an application and before any system initialisation code in the loaded application has been executed. It ensures that debug commands to interrogate thread state will not parse stale/undefined information from uninitialised memory.

define clear_ecos_thread_pointers
  set *((unsigned int *)&Cyg_Thread::thread_list) = 0
  set *((unsigned int *)&Cyg_Scheduler_Base::current_thread) = 0
end

document clear_ecos_thread_pointers
When starting a new debug session from application reset the run-time
code that clears BSS will not have been executed, so stale/unitialised
state may be present in memory. For RTOS aware thread debugging as
provided by external tools the GDB server may be confused and report
invalid state if the thread state is interrogated before the initial
eCos run-time initialisation has cleared the BSS area. This macro just
ensures that the relevant eCos pointers are NULL prior to debugging.
end

Of course if a hardware debugger is being used to connect to an existing application session (rather than loading and starting a new application session) then the macro should not be called.

It is useful to wrap the steps needed to connect to a target in a helper macro. e.g.:

define connocd
  target extended-remote localhost:3333
  load
  break cyg_test_exit
  break cyg_assert_fail
  display/i $pc
  clear_ecos_thread_pointers
end

So that all of the normal steps for loading and setting the debug environment for an application can be performed by a single command:

(gdb) connocd
0x080016dc in ?? ()

Loading section .rom_vectors, size 0x8 lma 0x90000000
Loading section .text, size 0x7b34 lma 0x90000008
Loading section .rodata, size 0x678 lma 0x90007b40
Loading section .data, size 0x180 lma 0x900081b8
Start address 0x90000008, load size 33588
Transfer rate: 94 KB/sec, 6717 bytes/write.
Breakpoint 1 at 0x900040f4: file ecospro-path/packages/infra/current/src/tcdiag.cxx, line 391.
Function "cyg_assert_fail" not defined.
Make breakpoint pending on future shared library load? (y or [n]) [answered N; input not from terminal]
(gdb)

The use of such macros from a GDB script file can make the task of debugging less cumbersome.

Ronetix PEEDI

	Note
	The Ronetix PEEDI firmware must be updated to at least version 21.2.0 to ensure the correct operation of the thread-aware debugging support.

The PEEDI [TARGET] section option COREn_OS can be used to introduce a thread/context description using the generic PEEDI support.

The example PEEDI configuration files supplied with eCosPro releases 4.5.8 and above should already have suitable RTOS support fragments. For example, the file packages/hal/arm/arm9/sam9g45ek/<version>/misc/peedi.sam9g45ek.cfg contains a [OS_ECOS_ARM] section, referenced from the head of the file via the CORE0_OS=OS_ECOS_ARM setting.

With a suitable COREn_OS the PEEDI will parse the eCos thread lists and stacked register contents when interrogating threads other than the current thread of executing on the CPU.

OpenOCD

OpenOCD provides the -rtos eCos option that can be used to configure thread-aware debug support in the configuration file used for the OpenOCD session.

	Note
	eCoscentric contributed the previously eCosPro specific eCos thread-aware debug support to the `OpenOCD` project. As of 2023-01-15 the support was merged into the `OpenOCD` mainline.

At its simplest the OpenOCD configuration file just needs to specify:

$_TARGETNAME configure -rtos eCos

NOTE: When OpenOCD -rtos support for eCos is configured the act of executing target remote or target extended-remote to connect to an OpenOCD instance will cause the configured OpenOCD RTOS support to perform an update_threads operation against the current memory state. This is to allow a debug session to be attached to an active system. However, it does mean that for an undefined memory state (power-on, CPU reset with undefined DRAM state, an application with a different thread context shape to the previous application new different-configuration application to be subsequently loaded after connecting to the OpenOCD GDB server) that the GDB server may report spurious thread information upon request.

The following is example output when the application is halted in a thread named "busy". The name, state and priority of the other available threads is also shown:

(gdb) info thr
 Id  Target Id         Frame
* 1  Thread 12 (Name: busy, State: Ready Pri: 20) 0x20009f50 in thread_busy (data=30000) at
     ecospro-path/packages/kernel/current/tests/fpint_thread_switch.cxx:566
  2  Thread 1 (Name: Idle Thread, State: Ready Pri: 31) Cyg_Scheduler::unlock_inner (new_lock=0) at
     ecospro-path/packages/kernel/current/src/sched/sched.cxx:233
  3  Thread 2 (Name: Test, State: Sleeping (WAIT) Pri: 3) Cyg_Scheduler::unlock_inner (new_lock=1) at
     ecospro-path/packages/kernel/current/src/sched/sched.cxx:233
  4  Thread 11 (Name: highpri, State: Sleeping (DELAY) Pri: 10) Cyg_Scheduler::unlock_inner (new_lock=0)
     at ecospro-path/packages/kernel/current/src/sched/sched.cxx:233

The OpenOCD GDB server can be left executing between GDB application debug sessions. It does not need to be re-started for every GDB session.

Segger JLink/JTrace

Segger do not allow source distributions of RTOS aware plugins based on their SDK. This policy unfortunately restricts eCos and eCos eCosPro support to pre-built shared library files only. There does not seem to be any obvious technical reason for this binary-only restriction, considering the simplicity of the exposed SDK API and limited feature set required to support RTOS aware debugging (as can be seen by the functionality required for the open-source OpenOCD "-rtos" support, and the generic config-file description approach built into the PEEDI (closed source) firmware.

Currently only 64-bit Linux x86_64 is available via the file libRTOSPlugin_eCosPro.so. Please contact eCosCentric to discuss the options available if other host platforms are required.

For example, the following is used to start a JLink GDB server session connected to a STM32F429I-DISCO board with the eCosPro RTOS aware plugin selected:

$ JLinkGDBServer -device stm32f429zi -if swd -rtos libRTOSPlugin_eCosPro.so

The JLinkGDBServer can be left executing between GDB application debug sessions. It does not need to be re-started for every GDB session.

GDB stubs

Unlike the hardware debug approaches described above which benefit from bare-metal hardware support (SWD, JTAG, BDM, etc.), eCos also supports the use of GDBstubs which can be built into the application and accessed via an I/O channel (e.g. serial, Ethernet, etc.), or provided via a boot monitor/loader (e.g. RedBoot covering in Chapter 228, Getting Started with RedBoot) that provides an environment for executing applications. Such support has a run-time cost (code+data space as well as CPU cycles), and is only usable after some level of system initialisation has occurred. A hardware debug solution is therefore preferred.

However, if GDBstubs is the only available/possible solution, the eCos GDBstubs implementation supports thread aware debugging.


Kernel Real-time Characterization		Kernel and infrastructure instrumentation