文章目录
Summary
Conduct Kernel debugging test on Embedded Linux S3C2440 by using KGDB. KGDB has been integrated into Linux kernel from version 2.6.15, it requires a serial connection to the development host. It is a cross GDB running on the development host workstation. Refer to earlier blog entry Embedded Linux S3C2440 – Kernel Module, all the tests are based on Foo.ko kernel module and test application Footest.
Startup the cross running GDB
Refer to Embedded Linux S3C2440 – Kernel Module, KGDB has been compiled into kernel and been enabled. We need to configure Embedded Linux S3C2440 to pass parameter to KGDB moudle to inform which port we use and send system request to trigger KGDB, as below steps shows,
Open Minicom and startup the Embedded Linux S3C2440 board, login as root, then do below actions,
[root@mini2440 /root]# echo ttySAC0 > /sys/module/kgdboc/parameters/kgdboc
kgdb: Registered I/O driver kgdboc.
[root@mini2440 /root]# echo g > /proc/sysrq-trigger
SysRq :
Above actions tell kernel which UART will be used for KGDB, Kgdboc means KGDB over console, so above commands shows Embedded Linux S3C2440 board kernel has entered KGDB and waiting for the connection from host GDB. KGDB is using serial link, we need to exit minicom. As below steps show,
Press Ctrl-A then Z to enter menu of minicom.
Press Q to quit without reset.
Select Yes when you have blow screen.
+----------------------+
| Leave without reset? |
| Yes No |
Start the GDB on host machine,
[root@localhost ~]# cd /home/iot/mini2440/linux-3.8.7/
[root@localhost linux-3.8.7]# arm-linux-gdb vmlinux
GNU gdb (GDB) 7.5.1
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu --target=arm-buildroot-linux-uclibcgnueabi".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/iot/mini2440/linux-3.8.7/vmlinux...done.
(gdb)
Set the baud rate,
(gdb) set remotebaud 115200
(gdb)
Connect GDB to the S3C2440 board KGDB,
(gdb) target remote /dev/ttyUSB0
Remote debugging using /dev/ttyUSB0
kgdb_breakpoint () at kernel/debug/debug_core.c:1012
1012 arch_kgdb_breakpoint();
(gdb) step
arch_kgdb_breakpoint () at kernel/debug/debug_core.c:1011
1011 wmb(); /* Sync point before breakpoint */
(gdb)
Above actions include, startup GDB and pass the kernel ELF file vmlinux to GDB, vmlinux is uncompressed version kernel image, connect to S3C2440 board using GDB ‘target remote’ command.
Set the break point at module.c module loading position
The driver foo.ko was compiled as a module, we need to set a break point at where the module is loaded. System call sys_init_module will copy the module information from user space to kernel space and it calls load_module(), set the break point at line 3068 and continue run kernel, as below shows.
(gdb) b kernel/module.c:3068
Breakpoint 1 at 0xc00525d4: file kernel/module.c, line 3068.
(gdb)
(gdb)
Connect Linux S3C2440 board from host through telnet.
And transfer the kernel module foo.ko and application footest from host to S3C2440 board.
[root@localhost ~]# telnet 192.168.0.11
Trying 192.168.0.11...
Connected to 192.168.0.11.
Escape character is '^]'.
mini2440 login: root
[root@mini2440 /root]#
[root@mini2440 /root]# tftp -g -r foo.ko 192.168.0.1
foo.ko 100% |********************************************************************************************************************************************| 69081 0:00:00 ETA
[root@mini2440 /root]# chmod 777 foo.ko
[root@mini2440 /root]# tftp -g -r footest 192.168.0.1
footest 100% |********************************************************************************************************************************************| 6122 0:00:00 ETA
[root@mini2440 /root]# chmod 777 footest
[root@mini2440 /root]# insmod foo.ko
After above insmod command, the foo.ko module is loaded, the GDB will stop at the break point, kernel/module.c:3068, as below gdb output shows,
[New Thread 485]
[Switching to Thread 485]
Breakpoint 1, do_init_module (mod=0xbf000344) at kernel/module.c:3068
3068 ret = do_one_initcall(mod->init);
(gdb) step
do_one_initcall (fn=0xbf002000) at init/main.c:682
682 int count = preempt_count();
(gdb)
ELF sections .text address from module_core and .init.text from module_init
As foo.ko is kernel module, there is no symbol information about the this module in vmlinux. We cannot set a break point at foo.ko module, and we do not know which address will the Linux kernel link to foo.ko module. Linux stores this module’s .text section address in the module information structure struct module in the module_core_element.
By using lsmod macro below, we can obtain the address of struct module allocated to foo.ko module, we will use the address retrieved to obtain the module_core address to be used as module’s .text address. And foo_init(void) function is defined with __init macro #define __init __section(.init.text) __cold notrace, this macro expands into a compiler attribute that directs the linker to place the marked portion of code into a specially named ELF section named .init.text. So the compiled foo_init() function will be placed into .init.text section of the foo.ko object module.
lsmod macro
The GDB lsmod macros in file /root/.gdbinit
#gdb List Modules Macro
define lsmod
printf "Address\t\tModule\n"
set $m=(struct list_head *)&modules
set $done=0
while ( !$done )
# list_head is 4-bytes into struct module
set $mp=(struct module *)((char *)$m->next - (char *)4)
printf "0x%08X\t%s\n", $mp, $mp->name
if ( $mp->list->next == &modules)
set $done=1
end
set $m=$m->next
end
end
Module loading procedure debugging
When the foo.ko module is loaded, the kernel allocates a chunk of memory for the main body of the module, which is pointed to by the struct module member named module_core. It then allocates a separate chunk of memory to hold the .init.text section. After the initialization function called, the kernel frees the memory that contained the initialization function. So the object module is split, and we need to inform gdb to be able to use symbolic data for debugging the initialization function.
Below gdb commands is to add symbol file and set break points.
(gdb) lsmod
Address Module
0xBF000344 foo
(gdb) set $m=(struct module*)0xBF000344
(gdb) p $m->module_core
$1 = (void *) 0xbf000000
(gdb) p $m->module_init
$2 = (void *) 0xbf002000
(gdb) add-symbol-file ./drivers/char/foo.ko 0xbf000000 -s .init.text 0xbf002000
add symbol table from file "./drivers/char/foo.ko" at
.text_addr = 0xbf000000
.init.text_addr = 0xbf002000
(y or n) y
Reading symbols from /home/iot/mini2440/linux-3.8.7/drivers/char/foo.ko...done.
(gdb) b foo_read
Breakpoint 2 at 0xbf0000a0: file drivers/char/foo.c, line 59.
(gdb) b foo_write
Breakpoint 3 at 0xbf000148: file drivers/char/foo.c, line 46.
(gdb) b do_write
Breakpoint 4 at 0xbf000100: file drivers/char/foo.c, line 28.
(gdb)
Step into kernel module foo.ko,
(gdb) step
current_thread_info () at /home/iot/mini2440/linux-3.8.7/arch/arm/include/asm/thread_info.h:99
99 return (struct thread_info *)(sp & ~(THREAD_SIZE - 1));
(gdb) step
do_one_initcall (fn=0xbf002000 <foo_init>) at init/main.c:685
685 if (initcall_debug)
(gdb) step
682 int count = preempt_count();
(gdb) step
current_thread_info () at /home/iot/mini2440/linux-3.8.7/arch/arm/include/asm/thread_info.h:99
99 return (struct thread_info *)(sp & ~(THREAD_SIZE - 1));
(gdb) step
do_one_initcall (fn=0xbf002000 <foo_init>) at init/main.c:685
685 if (initcall_debug)
(gdb) step
681 {
(gdb) step
685 if (initcall_debug)
(gdb) step
682 int count = preempt_count();
(gdb) step
685 if (initcall_debug)
(gdb) step
688 ret = fn();
(gdb) step
foo_init () at drivers/char/foo.c:119
119 result = register_chrdev(FOO_MAJOR, "scull", &foo_fops);
(gdb) step
register_chrdev (fops=0x36c <foo_fops>, name=0x150 <Address 0x150 out of bounds>, major=229) at drivers/char/foo.c:119
119 result = register_chrdev(FOO_MAJOR, "scull", &foo_fops);
(gdb) step
2125 return __register_chrdev(major, 0, 256, name, fops);
(gdb) step
__register_chrdev (major=major@entry=229, baseminor=baseminor@entry=0, count=count@entry=256, name=0xbf000299 "scull", name@entry=0x150 <Address 0x150 out of bounds>, fops=0xbf0002d8, fops@entry=0x36c <foo_fops>) at fs/char_dev.c:267
267 {
(gdb) step
272 cd = __register_chrdev_region(major, baseminor, count, name);
(gdb) step
__register_chrdev_region (major=major@entry=229, baseminor=baseminor@entry=0, minorct=minorct@entry=256, name=name@entry=0xbf000299 "scull") at fs/char_dev.c:97
97 {
(gdb) step
102 cd = kzalloc(sizeof(struct char_device_struct), GFP_KERNEL);
(gdb)
Now step into and debug the application,
[root@mini2440 /root]# mknod /dev/foo c 229 0
[root@mini2440 /root]# ./footest
[New Thread 488]
[Switching to Thread 488]
Breakpoint 3, foo_write (filp=0xc2dc6180, buffer=0xbeaa8c70 "Hello, there", count=12, f_pos=0xc2d8ff78) at drivers/char/foo.c:46
46 {
(gdb)
(gdb) step
50 copy_from_user(drv_buf, buffer, count);
(gdb) step
copy_from_user (n=12, from=0xbeab9c70, to=0x538 <drv_buf>) at drivers/char/foo.c:50
50 copy_from_user(drv_buf, buffer, count);
(gdb)
Output of the application
[root@mini2440 /root]# mknod /dev/foo c 229 0
[root@mini2440 /root]# ./footest
The length of string buf is: 12
write Hello, there.
read ereho, tlleHsuccess!.
.
[root@mini2440 /root]#
We can see the input string was Hello, there, the expected output should be reversed string ereht ,olleH, however the output was ereho, tlleH. I think below code logic was not correct,
for(i = 0; i<(len>>1); i++, len--)
{
temp = drv_buf[len-1];
drv_buf[len-1] = drv_buf[i];
drv_buf[i] = temp;
}
The input len was 12. The wrong logic is as below,
i
| len
| len>>1
| status
|
0
| 12
| 6
| i = 0; i < 6
|
1
| 11
| 5
| i = 1; i < 5
|
2
| 10
| 5
| i = 2; i < 5
|
3
| 9
| 4
| i = 3; i < 4
|
Above loop happened 4 times, it should be 6 times. half of 12, so the result was wrongly as ereho, tlleH. The correct logic should be below, and the result shall be ereht ,olleH.
i
| len (fixed value)
| len>>1
| status
|
0
| 12
| 6
| i = 0; i < 6
|
1
| 12
| 6
| i = 1; i < 6
|
2
| 12
| 6
| i = 2; i < 6
|
3
| 12
| 6
| i = 3; i < 6
|
4
| 12
| 6
| i = 4; i < 6
|
5
| 12
| 6
| i = 5 i < 6
|
Following below debugging process, the loop stops when i = 4.
Breakpoint 4, do_write () at drivers/char/foo.c:28
28 {
(gdb) step
32 for(i = 0; i<(len>>1); i++, len--)
(gdb) p i
$3 = 0
(gdb) i len
Undefined info command: "len". Try "help info".
(gdb) p $len
$4 = void
(gdb) p len
Cannot access memory at address 0x938
(gdb) step
35 drv_buf[len-1] = drv_buf[i];
(gdb) p i
$5 = 0
(gdb) p drv_buf
Cannot access memory at address 0x538
(gdb) p drv_buf[0]
Cannot access memory at address 0x538
(gdb) p *drv_buf@10
Cannot access memory at address 0x538
(gdb) x/xw 0x538
0x538 <drv_buf>: Cannot access memory at address 0x538
(gdb) list
30 int len = WRT_LEN;
31 char temp;
32 for(i = 0; i<(len>>1); i++, len--)
33 {
34 temp = drv_buf[len-1];
35 drv_buf[len-1] = drv_buf[i];
36 drv_buf[i] = temp;
37 }
38
39 }
(gdb) p len
$6 = 12
(gdb) p *dri_buf
No symbol "dri_buf" in current context.
(gdb) p *drv_buf
Cannot access memory at address 0x538
(gdb) x/xb &drv_buf
0x538 <drv_buf>: Cannot access memory at address 0x538
(gdb) x/xb drv_buf
0x538 <drv_buf>: Cannot access memory at address 0x538
(gdb) x/1xb drv_buf
0x538 <drv_buf>: Cannot access memory at address 0x538
(gdb) p drv_buf[0]
Cannot access memory at address 0x538
(gdb) p drv_buf[1]
Cannot access memory at address 0x539
(gdb) where
#0 do_write () at drivers/char/foo.c:35
#1 0xbf0001b8 in foo_write (filp=<optimized out>, buffer=<optimized out>, count=12, f_pos=<optimized out>) at drivers/char/foo.c:53
#2 0xc0096950 in vfs_write (file=file@entry=0xc2dc6180, buf=buf@entry=0xbeaa8c70 "Hello, there", count=<optimized out>, count@entry=12, pos=pos@entry=0xc2d8ff78) at fs/read_write.c:428
#3 0xc0096b24 in sys_write (fd=<optimized out>, buf=0xbeaa8c70 "Hello, there", count=12) at fs/read_write.c:475
#4 0xc000e4a0 in kuser_cmpxchg32_fixup ()
#5 0xc000e4a0 in kuser_cmpxchg32_fixup ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) whatis drv_buf
type = char [1024]
(gdb) backtrace
#0 do_write () at drivers/char/foo.c:35
#1 0xbf0001b8 in foo_write (filp=<optimized out>, buffer=<optimized out>, count=12, f_pos=<optimized out>) at drivers/char/foo.c:53
#2 0xc0096950 in vfs_write (file=file@entry=0xc2dc6180, buf=buf@entry=0xbeaa8c70 "Hello, there", count=<optimized out>, count@entry=12, pos=pos@entry=0xc2d8ff78) at fs/read_write.c:428
#3 0xc0096b24 in sys_write (fd=<optimized out>, buf=0xbeaa8c70 "Hello, there", count=12) at fs/read_write.c:475
#4 0xc000e4a0 in kuser_cmpxchg32_fixup ()
#5 0xc000e4a0 in kuser_cmpxchg32_fixup ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) p/x 0x538
$7 = 0x538
(gdb) x/i
0x539 <drv_buf+1>: Cannot access memory at address 0x538
(gdb) p tmp
No symbol "tmp" in current context.
(gdb) l
40
41
42
43
44
45 ssize_t foo_write(struct file *filp, const char *buffer, size_t count, loff_t *f_pos)
46 {
47 if(count > MAX_BUF_LEN)
48 count = MAX_BUF_LEN;
49
(gdb) step
34 temp = drv_buf[len-1];
(gdb) p temp
$8 = <optimized out>
(gdb) p len
$9 = 12
(gdb) p i
$10 = 0
(gdb) step
35 drv_buf[len-1] = drv_buf[i];
(gdb) step
36 drv_buf[i] = temp;
(gdb) step
32 for(i = 0; i<(len>>1); i++, len--)
(gdb) p i
$11 = 0
(gdb) step
35 drv_buf[len-1] = drv_buf[i];
(gdb) p i
$12 = 1
(gdb) step
34 temp = drv_buf[len-1];
(gdb) step
35 drv_buf[len-1] = drv_buf[i];
(gdb) step
36 drv_buf[i] = temp;
(gdb) p i
$13 = 1
(gdb) step
32 for(i = 0; i<(len>>1); i++, len--)
(gdb) step
35 drv_buf[len-1] = drv_buf[i];
(gdb) step
34 temp = drv_buf[len-1];
(gdb) step
35 drv_buf[len-1] = drv_buf[i];
(gdb) step
36 drv_buf[i] = temp;
(gdb) step
^[[A32 for(i = 0; i<(len>>1); i++, len--)
(gdb) step
35 drv_buf[len-1] = drv_buf[i];
(gdb) p
$14 = 1
(gdb) p i
$15 = 3
(gdb) p i
$16 = 3
(gdb) p
$17 = 3
(gdb) step
34 temp = drv_buf[len-1];
(gdb) step
35 drv_buf[len-1] = drv_buf[i];
(gdb) step
36 drv_buf[i] = temp;
(gdb) step
32 for(i = 0; i<(len>>1); i++, len--)
(gdb) step
39 }
(gdb) p i
$18 = 4
(gdb) step
foo_write (filp=<optimized out>, buffer=<optimized out>, count=12, f_pos=<optimized out>) at drivers/char/foo.c:55
55 }
(gdb) p len
No symbol "len" in current context.
(gdb) c
Continuing.
Breakpoint 2, foo_read (filp=0xc2dc6180, buffer=0xbeaa8c70 "Hello, there", count=100, ppos=0xc2d8ff78) at drivers/char/foo.c:59
59 {
(gdb)
The serial port output was like below,
foo initialized.
device open success!.
user write data to driver.
user read data from driver.
Device release.
device open success!.
user write data to driver.
user read
Cannot access memory at address
Above debugging process has one Cannot access memory at address error, refer to Linux Device Drivers, Second Edition by Alessandro Rubini, Jonathan Corbet, I guess it was the reason.
Note that you can’t disassemble a module function, because the debugger is acting on vmlinux, which doesn’t know about your module. If you try to disassemble a module by address, gdb is most likely to reply “Cannot access memory at xxxx.” For the same reason, you can’t look at data items belonging to a module. They can be read from /dev/mem if you know the address of your variables, but it’s hard to make sense out of raw data extracted from system RAM.
Reference
The 101 of ELF files on Linux: Understanding and AnalysisDebugging The Linux Kernel Using GdbEmbedded Linux S3C2440 – Kernel ModuleLinux Device Drivers, Second Edition by Alessandro Rubini, Jonathan Corbet
