[转帖]crash工具分析Kdump下vmcore文件常用命令总结(三)(实例易懂)

crash,工具,分析,kdump,vmcore,文件,常用命令,总结,实例,易懂 · 浏览次数 : 0

小编点评

**Kdump调试机理总结(一)** Kdump调试机理总结(一) **Kdump配置及使用详细总结(二)** Kdump配置及使用详细总结(二) **gdb调试常见命令总结** gdb调试常见命令总结 **Linux下objdump反编译文件命令总结** Linux下objdump反编译文件命令总结

正文

一、简介

本文主要介绍使用crash工具对kdump生成的vmcore文件进行分析,解析常见的crash命令,前面已讲述两章关于Kdump的内容,读者感兴趣可以点击下面的链接:

1、Kdump调试机理总结(一)

2、Kdump配置及使用详细总结(二)
系统产生异常时,触发Kdump机制,启动捕获内核,用以对生产内核下的内存进行收集和转存。
Kdump用于对内存镜像的转储,它不但可以转储内存镜像到本地硬盘,还可以将内存镜像通过NFS,SSH等协议转储到不同机器的设备上。
本文是将生产内核的内存数据转储在/var/crash目录下,下面开始对其进行分析。

分析vmcore的主要目的:
1、明白内核崩溃的大致原因;
2、对内核崩溃的原因做更细致的分析;

crash是一个被广泛应用的内核奔溃转储文件分析工具。crash工具跟gdb很类似,它可以交互的分析运行中的内核,也可以分析由kdump、netdump、diskdump、xendump产生的core dump文件

二、vmcore分析

2.1 环境及相关文件准备
使用crash工具分析vmcore,需要:
(1)crash工具(crash);
(2)崩溃转储文件(vmcore);
(3)发生崩溃的内核映像文件(vmlinux),包含调试内核所需调试信息;
(4)crash下不同命令分析vmcore文件。

具体操作:
(1)安装crash工具

[root@localhost /]# yum install crash

    (2)崩溃转储文件(vmcore)
    发生Kdump后在/var/crash下会生成vmcore文件,也可以手动进入Kdump生成vmcore,具体操作可参考本人博客kdump第二篇,具体链接在文章开头。

    (3)准备内核映像文件
    一般系统在安装后在/boot目录下,也有个内核映像文件,vmlinuxz开头的文件,但是它是压缩过后的,无法完成调试工作。
    我们需要一个带内核调试信息的vmlinux镜像,它有两种获取方式:
    1、手动编译vmlinux
    手动编译就是正常的编译一个Linux内核,不具体介绍了。
    需要注意的是,make menuconfig里配置CONFIG_DEBUG_INFO=y。
    2、官网下载vmlinux
    下载带有完整调试信息的内核映像文件,内核调试信息包kernel-debuginfo有两个:

    kernel-debuginfo
    kernel-debuginfo-common
    

      对于centos系统,可以在http://debuginfo.centos.org/上下载到各发行版本所需的调试包。
      可以通过直接解开这两个包找到vmlinux或者安装这两个包在/lib/debug/lib/modules/3.10.0.x86_64/目录下看到vmlinux内核映像文件:

      [root@localhost /]# rpm2cpio xxx.rpm|cpio -idvm //当前目录解开该安装包
      

        (4)crash下不同命令分析vmcore文件
        1、进入crash环境

        [root@localhost /]# crash /lib/debug/lib/modules/3.10.0.el7.x86_64/vmlinux /var/crash/127.0.0.1-2022-05-06-17\:25\:34/vmcore
        

        GNU gdb (GDB) 7.6
        Copyright (C) 2013 Free Software Foundation, Inc.
        License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
        This is free software: you are free to change and redistribute it.
        There is NO WARRANTY, to the extent permitted by law. Type "show copying"
        and "show warranty" for details.
        This GDB was configured as "x86_64-unknown-linux-gnu"...

          KERNEL<span class="token operator">:</span> <span class="token operator">/</span>boot<span class="token operator">/</span>vmlinux                     
        DUMPFILE<span class="token operator">:</span> <span class="token operator">/</span>var<span class="token operator">/</span>crash<span class="token operator">/</span><span class="token number">127.0</span><span class="token number">.0</span><span class="token number">.1</span><span class="token operator">-</span><span class="token number">2022</span><span class="token operator">-</span><span class="token number">05</span><span class="token operator">-</span><span class="token number">06</span><span class="token operator">-</span><span class="token number">17</span><span class="token operator">:</span><span class="token number">25</span><span class="token operator">:</span><span class="token number">41</span><span class="token operator">/</span>vmcore  <span class="token punctuation">[</span>PARTIAL DUMP<span class="token punctuation">]</span>
            CPUS<span class="token operator">:</span> <span class="token number">8</span>
            DATE<span class="token operator">:</span> Fri May <span class="token number">06</span> <span class="token number">17</span><span class="token operator">:</span><span class="token number">25</span><span class="token operator">:</span><span class="token number">34</span> <span class="token number">2022</span>
          UPTIME<span class="token operator">:</span> <span class="token number">00</span><span class="token operator">:</span><span class="token number">34</span><span class="token operator">:</span><span class="token number">45</span>
        

        LOAD AVERAGE: 3.37, 3.71, 3.38
        TASKS: 1316
        NODENAME: fe0fdb76-b761-11e9-b107-0014101e89e7
        RELEASE: 3.10.0+
        VERSION: #1 SMP Thu Dec 24 16:27:16 CST 2020
        MACHINE: x86_64 (3408 Mhz)
        MEMORY: 31.9 GB
        PANIC: "SysRq : Trigger a crash"
        PID: 1942
        COMMAND: "bash"
        TASK: ffff88068c957300 [THREAD_INFO: ffff88062b8f4000]
        CPU: 2
        STATE: TASK_RUNNING (SYSRQ)

          2、bt命令
          backtrace打印内核栈回溯信息,bt pid 打印指定进程栈信息。

          crash> bt 1942
          PID: 1942   TASK: ffff88068c957300  CPU: 2   COMMAND: "bash"
           #0 [ffff88062b8f7b48] machine_kexec at ffffffff81051e9b
           #1 [ffff88062b8f7ba8] crash_kexec at ffffffff810f27e2
           #2 [ffff88062b8f7c78] oops_end at ffffffff81689948
           #3 [ffff88062b8f7ca0] no_context at ffffffff816793f1
           #4 [ffff88062b8f7cf0] __bad_area_nosemaphore at ffffffff81679487
           #5 [ffff88062b8f7d38] bad_area_nosemaphore at ffffffff816795f1
           #6 [ffff88062b8f7d48] __do_page_fault at ffffffff8168c6ce
           #7 [ffff88062b8f7da8] do_page_fault at ffffffff8168c863
           #8 [ffff88062b8f7dd0] page_fault at ffffffff81688b48
              [exception RIP: sysrq_handle_crash+22]
              RIP: ffffffff813baf16  RSP: ffff88062b8f7e80  RFLAGS: 00010046
              RAX: 000000000000000f  RBX: ffffffff81a7b180  RCX: 0000000000000000
              RDX: 0000000000000000  RSI: ffff88086ec8f6c8  RDI: 0000000000000063
              RBP: ffff88062b8f7e80   R8: 0000000000000092   R9: 0000000000000e37
              R10: 0000000000000e36  R11: 0000000000000003  R12: 0000000000000063
              R13: 0000000000000246  R14: 0000000000000004  R15: 0000000000000000
              ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
           #9 [ffff88062b8f7e88] __handle_sysrq at ffffffff813bb6d2
          #10 [ffff88062b8f7ec0] write_sysrq_trigger at ffffffff813bbbaf
          #11 [ffff88062b8f7ed8] proc_reg_write at ffffffff812494bd
          #12 [ffff88062b8f7ef8] vfs_write at ffffffff811dee9d
          #13 [ffff88062b8f7f38] sys_write at ffffffff811df93f
          #14 [ffff88062b8f7f80] system_call_fastpath at ffffffff81691049
              RIP: 00007fb320bcb500  RSP: 00007ffde533c198  RFLAGS: 00000246
              RAX: 0000000000000001  RBX: ffffffff81691049  RCX: ffffffffffffffff
              RDX: 0000000000000002  RSI: 00007fb3214eb000  RDI: 0000000000000001
              RBP: 00007fb3214eb000   R8: 000000000000000a   R9: 00007fb3214d5740
              R10: 0000000000000001  R11: 0000000000000246  R12: 0000000000000001
              R13: 0000000000000002  R14: 00007fb320e9f400  R15: 0000000000000002
              ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b
          

          解析:
          可以看到最后几步触发了缺页异常,进入crash_kexec的流程,最后调用 machine_kexec()。这通常是一个硬件相关的函数。它会引导启动捕获内核,从而完成 kdump 的过程。
          代码就是走到了sysrq_handle_crash函数首地址+0x22这段命令的时候,触发的缺页异常。

          注意:
          这里,对应x86-64汇编,应用层下来的系统调用对应的6个参数存放的寄存器依次对应:rdi、rsi、rdx、rcx、r8、r9。对于多于6个参数的,仍存储在栈上。

            3、log命令
            打印vmcore所在的系统内核dmesg日志信息

            crash> log
            [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd070]
            [    0.000000] Linux version 4.20.0-rc4-00007-gef78e5e (root@localhost.localdomain) (gcc version 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a424ecfbc167ef90065c0eeb7f91977701] (Linaro GCC 7.3-2018.05)) #3 SMP PREEMPT Wed Jan 15 07:52:10 PST 2020
            [    0.000000] Machine model: linux,dummy-virt
            [    0.000000] efi: Getting EFI parameters from FDT:
            [    0.000000] efi: UEFI not found.
            [    0.000000] cma: Reserved 32 MiB at 0x00000000be000000
            [    0.000000] NUMA: No NUMA configuration found
            [    0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x00000000bfffffff]
            [    0.000000] NUMA: NODE_DATA [mem 0xbdfea840-0xbdfebfff]
            [    0.000000] Zone ranges:
            [    0.000000]   DMA32    [mem 0x0000000040000000-0x00000000bfffffff]
            [    0.000000]   Normal   empty
            [    0.000000] Movable zone start for each node
            [    0.000000] Early memory node ranges
            [    0.000000]   node   0: [mem 0x0000000040000000-0x00000000bfffffff]
            [    0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x00000000bfffffff]
            [    0.000000] On node 0 totalpages: 524288
            [    0.000000]   DMA32 zone: 8192 pages used for memmap
            [    0.000000]   DMA32 zone: 0 pages reserved
            [    0.000000]   DMA32 zone: 524288 pages, LIFO batch:63
            [    0.000000] psci: probing for conduit method from DT.
            [    0.000000] psci: PSCIv0.2 detected in firmware.
            [    0.000000] psci: Using standard PSCI v0.2 function IDs
            [    0.000000] psci: Trusted OS migration not required
            [    0.000000] random: get_random_bytes called from start_kernel+0xa8/0x418 with crng_init=0
            [    0.000000] percpu: Embedded 23 pages/cpu @(____ptrval____) s55704 r8192 d30312 u94208
            [    0.000000] pcpu-alloc: s55704 r8192 d30312 u94208 alloc=23*4096
            [    0.000000] pcpu-alloc: [0] 0 [0] 1 
            [    0.000000] Detected PIPT I-cache on CPU0
            [    0.000000] CPU features: enabling workaround for ARM erratum 832075
            [    0.000000] CPU features: enabling workaround for ARM erratum 834220
            [    0.000000] CPU features: enabling workaround for EL2 vector hardening
            [    0.000000] CPU features: detected: Kernel page table isolation (KPTI)
            [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 516096
            [    0.000000] Policy zone: DMA32
            [    0.000000] Kernel command line: rdinit=/linuxrc console=ttyAMA0
            [    0.000000] Memory: 2009884K/2097152K available (10876K kernel code, 1414K rwdata, 5100K rodata, 1344K init, 380K bss, 54500K reserved, 32768K cma-reserved)
            [    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
            [    0.000000] rcu: Preemptible hierarchical RCU implementation.
            [    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=2.
            [    0.000000]  Tasks RCU enabled.
            [    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
            [    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
            [    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
            [    0.000000] GICv2m: range[mem 0x08020000-0x08020fff], SPI[80:143]
            [    0.000000] arch_timer: cp15 timer(s) running at 62.50MHz (virt).
            [    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x1cd42e208c, max_idle_ns: 881590405314 ns
            

              4、dis命令
              dis -l (function+offset) 10 反汇编出指令所在代码,10代表打印该指定位置开始的10行信息。

              crash> dis -l sysrq_handle_crash+22 10
              /home/zhugeyifan/source/jd4000/jd4000_x86/packages/linux_lsp/kernel/linux-3.10.0/drivers/tty/sysrq.c: 138
              0xffffffff813baf16 <sysrq_handle_crash+22>:     movb   $0x1,0x0
              /home/zhugeyifan/source/jd4000/jd4000_x86/packages/linux_lsp/kernel/linux-3.10.0/drivers/tty/sysrq.c: 139
              0xffffffff813baf1e <sysrq_handle_crash+30>:     pop    %rbp
              0xffffffff813baf1f <sysrq_handle_crash+31>:     retq   
              /home/zhugeyifan/source/jd4000/jd4000_x86/packages/linux_lsp/kernel/linux-3.10.0/drivers/tty/sysrq.c: 85
              0xffffffff813baf20 <sysrq_handle_loglevel>:     nopl   0x0(%rax,%rax,1) [FTRACE NOP]
              0xffffffff813baf25 <sysrq_handle_loglevel+5>:   push   %rbp
              /home/zhugeyifan/source/jd4000/jd4000_x86/packages/linux_lsp/kernel/linux-3.10.0/drivers/tty/sysrq.c: 90
              0xffffffff813baf26 <sysrq_handle_loglevel+6>:   xor    %eax,%eax
              /home/zhugeyifan/source/jd4000/jd4000_x86/packages/linux_lsp/kernel/linux-3.10.0/drivers/tty/sysrq.c: 89
              0xffffffff813baf28 <sysrq_handle_loglevel+8>:   movl   $0x7,0x6322be(%rip)        # 0xffffffff819ed1f0 <console_printk>
              /home/zhugeyifan/source/jd4000/jd4000_x86/packages/linux_lsp/kernel/linux-3.10.0/drivers/tty/sysrq.c: 85
              0xffffffff813baf32 <sysrq_handle_loglevel+18>:  mov    %rsp,%rbp
              0xffffffff813baf35 <sysrq_handle_loglevel+21>:  push   %rbx
              /home/zhugeyifan/source/jd4000/jd4000_x86/packages/linux_lsp/kernel/linux-3.10.0 -- MORE --  forward: <SPACE>, <ENTER> or j  backward: b or k
              /drivers/tty/sysrq.c: 88
              0xffffffff813baf36 <sysrq_handle_loglevel+22>:  lea    -0x30(%rdi),%ebx
              

              解析:
              代码位置+函数+函数偏移位置+该位置的汇编指令...
              很容易定义到具体的哪一行代码,更方便我们去分析问题的原因!

                5、mod命令
                mod 查看当时内核加载的所有内核模块信息
                重装加载进改内核模块:

                crash> mod -s my_test_lkm /mnt/shared/my_test.ko
                     MODULE       NAME                            SIZE  OBJECT FILE
                ffffffffc07c7000  my_test                    12740  /mnt/shared/my_test.ko
                
                • 1
                • 2
                • 3

                6、sym命令

                “sym 内存地址”转换指定符号为其虚拟地址,显示系统中对应的符号表信息,并且具体到源代码的那一行
                如上面bt打印的RIP: ffffffffc07c5024,使用sym转换查看系统符号信息
                sym addr 通过虚拟地址,显示symbol和源码位置

                crash> sym ffffffffc07c5024
                ffffffffc07c5024 (t) my_openat+36 [my_test_lkm] /mnt/hgfs/test_ko/lkm-test05/my_lkm.c: 25
                

                sym symbol 通过symbol,显示虚拟地址和源码位置:
                crash> sym vfs_fstat
                ffffffff81196f30 (T) vfs_fstat ../debug/kernel-3.8.13/linux-3.8.13-44.1.1.el6uek/fs/stat.c: 59

                  7、ps命令
                  ps 打印内核崩溃时,正常的进程信息
                  带 > 标识代表是活跃的进程,ps pid打印某指定进程的状态信息:

                  crash> ps 27005
                     PID    PPID  CPU       TASK        ST  %MEM     VSZ    RSS  COMM
                  > 27005   7783   1  ffff997b388ae180  RU   0.2   91732   4124  pickup
                  > 
                  查看指定进程的进程树,显示进程父子关系(ps -p pid) 
                  crash> ps -p 85151
                  PID: 0      TASK: ffffffff818b6420  CPU: 0   COMMAND: "swapper/0"
                   PID: 1      TASK: ffff881f91dae040  CPU: 28  COMMAND: "init"
                    PID: 14544  TASK: ffff881f8d7b05c0  CPU: 11  COMMAND: "init.tfa"
                     PID: 85138  TASK: ffff880bab01a400  CPU: 8   COMMAND: "tfactl"
                      PID: 85151  TASK: ffff880b7a728380  CPU: 17  COMMAND: "perl"
                  ps -t [pid]: 显示进程运行时间
                  

                    8、files命令
                    files pid 打印指定进程所打开的文件信息
                    在这里插入图片描述

                    9、vm命令
                    vm pid 打印某指定进程当时虚拟内存基本信息
                    在这里插入图片描述

                    10、task命令
                    task 查看当前进程或指定进程task_struct和thread_info的信息
                    在这里插入图片描述

                    11、kmem命令
                    查看当时系统内存使用信息

                    kmem [-f|-F|-c|-C|-i|-v|-V|-n|-z|-o|-h] [-p | -m member[,member]]
                           [[-s|-S] [slab] [-I slab[,slab]]] [-g [flags]] [[-P] address]]
                    kmem -i         //查看内存整体使用情况
                    kmem -s        //查看slab使用情况
                    kmem [addr]   //搜索地址所属的内存结构
                    crash> kmem -i
                                     PAGES        TOTAL      PERCENTAGE
                        TOTAL MEM   511276         2 GB         ----
                             FREE   506631       1.9 GB   99% of TOTAL MEM
                             USED     4645      18.1 MB    0% of TOTAL MEM
                           SHARED      353       1.4 MB    0% of TOTAL MEM
                          BUFFERS        0            0    0% of TOTAL MEM
                           CACHED      480       1.9 MB    0% of TOTAL MEM
                             SLAB     1930       7.5 MB    0% of TOTAL MEM
                    

                    TOTAL HUGE 0 0 ----
                    HUGE FREE 0 0 0% of TOTAL HUGE

                    TOTAL SWAP 0 0 ----
                    SWAP USED 0 0 0% of TOTAL SWAP
                    SWAP FREE 0 0 0% of TOTAL SWAP

                    COMMIT LIMIT 255638 998.6 MB ----
                    COMMITTED 479 1.9 MB 0% of TOTAL LIMIT
                    crash>

                      12、struct命令

                      struct -o [struct]            //显示结构体中成员的偏移
                      struct [struct] [address]    //显示对应地址结构体的值
                      [struct] [address]          //简化形式显示对应地址结构体的值
                      [struct] [address] -xo     //打印结构体定义和大小
                      [struct].member[address]  //显示某个成员的值
                      
                      • 1
                      • 2
                      • 3
                      • 4
                      • 5

                      1、查看结构体成员变量,直接struct后面跟结构体名,比如我要看下内核中dentry结构体的定义,可以这样:

                      crash> struct dentry
                      struct dentry {
                          unsigned int d_flags;
                          seqcount_t d_seq;
                          struct hlist_bl_node d_hash;
                          struct dentry *d_parent;
                          struct qstr d_name;
                          struct inode *d_inode;
                          unsigned char d_iname[32];
                          unsigned int d_count;
                          spinlock_t d_lock;
                          const struct dentry_operations *d_op;
                          struct super_block *d_sb;
                          long unsigned int d_time;
                          void *d_fsdata;
                          struct list_head d_lru;
                          union {
                              struct list_head d_child;
                              struct callback_head d_rcu;
                          } d_u;
                          struct list_head d_subdirs;
                          struct hlist_node d_alias;
                      }
                      SIZE: 192
                      

                        2、查看成员变量在结构体中的偏移量,struct加-o选项:

                        crash> struct dentry -o
                        struct dentry {
                            [0] unsigned int d_flags;
                            [4] seqcount_t d_seq;
                            [8] struct hlist_bl_node d_hash;
                           [24] struct dentry *d_parent;
                           [32] struct qstr d_name;
                           [48] struct inode *d_inode;
                           [56] unsigned char d_iname[32];
                           [88] unsigned int d_count;
                           [92] spinlock_t d_lock;
                           [96] const struct dentry_operations *d_op;
                          [104] struct super_block *d_sb;
                          [112] long unsigned int d_time;
                          [120] void *d_fsdata;
                          [128] struct list_head d_lru;
                                union {
                                    struct list_head d_child;
                                    struct callback_head d_rcu;
                          [144] } d_u;
                          [160] struct list_head d_subdirs;
                          [176] struct hlist_node d_alias;
                        }
                        SIZE: 192
                        

                        crash> struct dentry.d_inode
                        struct dentry {
                        [48] struct inode *d_inode;
                        }

                          3、查看内存中指定结构体的值,struct后跟指针地址

                          crash> struct dentry ffff881fbdc02c80
                          struct dentry {
                            d_flags = 136, 
                            d_seq = {
                              sequence = 4
                            }, 
                            d_hash = {
                              next = 0x0, 
                              pprev = 0xffffc900031bd8a0
                            }, 
                            d_parent = 0xffff881fbdc02ec0, 
                            d_name = {
                              {
                                {
                                  hash = 1819047278, 
                                  len = 4
                                }, 
                                hash_len = 18998916462
                              }, 
                              name = 0xffff881fbdc02cb8 "null"
                            }, 
                          

                            注:如果要查看二阶指针的值,可以通过rd命令需要先获取一级指针的值,然后再用struct 结构体名 + addr获取具体的值
                            13、rd命令
                            读取内存内容

                            rd [addr] [len]                  //查看指定地址,长度为len的内存
                            rd -S [addr][len]               //尝试将地址转换为对应的符号
                            rd [addr] -e [addr]            //查看指定内存区域内容
                            crash> rd ffff0000085dc0d0 32
                            ffff0000085dc0d0:  a8c17bfd39000020 d503201fd65f03c0    ..9.{...._.. ..
                            ffff0000085dc0e0:  910003fda9bf7bfd 97ec827fd50342ff   .{.......B......
                            ffff0000085dc0f0:  d65f03c0a8c17bfd 910003fda9bd7bfd   .{...._..{......
                            ffff0000085dc100:  b0005d73a90153f3 aa0103f4911b2262   .S..s]..b"......
                            ffff0000085dc110:  f90017a1f9400041 910083a2d2800001   A.@.............
                            ffff0000085dc120:  37f8018097f909a2 f10bfc7ff94013a3   .......7..@.....
                            ffff0000085dc130:  b000656154000228 f940128491018021   (..Tae..!.....@.
                            ffff0000085dc140:  b940402252800000 1100044279000083   ...R"@@....yB...
                            ffff0000085dc150:  911b2273b9004022 f9400261f94017a2   "@..s"....@.a.@.
                            ffff0000085dc160:  b50000c1ca010041 a8c37bfda94153f3   A........SA..{..
                            ffff0000085dc170:  128002a0d65f03c0 97ebee0b17fffff7   .._.............
                            ffff0000085dc180:  910003fda9be7bfd aa0003f4a90153f3   .{.......S......
                            ffff0000085dc190:  940b05adf9400013 97ec5fb991012260   ..@.....`"..._..
                            ffff0000085dc1a0:  97edbddd91052260 940b05c5aa1403e0   `"..............
                            ffff0000085dc1b0:  97f0ec85aa1303e0 a8c27bfda94153f3   .........SA..{..
                            ffff0000085dc1c0:  d503201fd65f03c0 52800021a9bf7bfd   .._.. ...{..!..R
                            

                              下面将通过struct file**fd的地址,获取fd指针数组中各个file结构体成员地址(二阶指针存放的是file指针的地址,所以可以通过rd获取一级指针的地址;二阶指针+8,则指向下一个file结构体指针):

                              14、p命令
                              p命令可以用来打印出表达式或者变量的值

                              crash> p jiffies
                              jiffies = $7 = 4314305182
                              crash>
                              crash> p old_close_func
                              old_close_func = $5 = (close_t) 0xffffffff811ffd60 <SyS_close>
                              crash> p SyS_close
                              SyS_close = $6 = 
                               {long (long)} 0xffffffff811ffd60 <SyS_close>
                              crash> sym SyS_close
                              ffffffff811ffd60 (T) SyS_close /usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0.el7.x86_64/fs/open.c: 1113
                              
                              • 1
                              • 2
                              • 3
                              • 4
                              • 5
                              • 6
                              • 7
                              • 8
                              • 9
                              • 10
                              • 11

                              15、查看某个命令使用方法及使用示例
                              使用help+某个命令,可以查看这个命令的使用方法及使用示例:
                              比如我们想看struct命令如何使用,可以执行 help struct命令:

                              crash> help struct
                              

                              NAME
                              struct - structure contents

                              SYNOPSIS
                              struct struct_name[.member[,member]][-o][-l offset][-rfuxdp][address | symbol]
                              [count | -c count]

                              DESCRIPTION
                              This command displays either a structure definition, or a formatted display
                              of the contents of a structure at a specified address. When no address is
                              specified, the structure definition is shown along with the structure size.
                              A structure member may be appended to the structure name in order to limit
                              the scope of the data displayed to that particular member; when no address
                              is specified, the member's offset and definition are shown.

                              struct_name  name of a C<span class="token operator">-</span>code structure used by the kernel<span class="token punctuation">.</span>
                                  <span class="token punctuation">.</span>member  name of a structure member<span class="token punctuation">;</span> to display multiple members of a
                                           structure<span class="token punctuation">,</span> use a comma<span class="token operator">-</span>separated list of members<span class="token punctuation">.</span>
                                       <span class="token operator">-</span>o  show member offsets when displaying structure definitions<span class="token punctuation">;</span> 
                                           <span class="token keyword">if</span> used with an address or symbol argument<span class="token punctuation">,</span> each member will
                                           be preceded by its virtual address<span class="token punctuation">.</span>
                                <span class="token operator">-</span>l offset  <span class="token keyword">if</span> the address argument is a pointer to a structure member that
                                           is contained by the target data structure<span class="token punctuation">,</span> typically a pointer
                                           to an embedded list_head<span class="token punctuation">,</span> the offset to the embedded member may
                                           be entered in either of the following manners<span class="token operator">:</span>
                                             <span class="token number">1.</span> in <span class="token string">"structure.member"</span> format<span class="token punctuation">.</span>
                                             <span class="token number">2.</span> a number of bytes<span class="token punctuation">.</span> 
                                       <span class="token operator">-</span>r  raw dump of structure data<span class="token punctuation">.</span>
                                       <span class="token operator">-</span>f  address argument is a dumpfile offset<span class="token punctuation">.</span>
                                       <span class="token operator">-</span>u  address argument is a user virtual address in the current
                                           context<span class="token punctuation">.</span>
                                       <span class="token operator">-</span>x  override <span class="token keyword">default</span> output format with hexadecimal format<span class="token punctuation">.</span>
                                       <span class="token operator">-</span>d  override <span class="token keyword">default</span> output format with decimal format<span class="token punctuation">.</span>
                                       <span class="token operator">-</span>p  <span class="token keyword">if</span> a structure member is a pointer value<span class="token punctuation">,</span> show the member's
                                           data type on the output line<span class="token punctuation">;</span> and on the subsequent <span class="token function">line</span><span class="token punctuation">(</span>s<span class="token punctuation">)</span><span class="token punctuation">,</span>
                                           dereference the pointer<span class="token punctuation">,</span> display the pointer target's symbol
                                           value in brackets <span class="token keyword">if</span> appropriate<span class="token punctuation">,</span> and <span class="token keyword">if</span> possible<span class="token punctuation">,</span> display the
                                           target data<span class="token punctuation">;</span> requires an address argument<span class="token punctuation">.</span>
                                  address  hexadecimal address of a structure<span class="token punctuation">;</span> <span class="token keyword">if</span> the address points
                                           to an embedded list_head structure contained within the
                                           target data structure<span class="token punctuation">,</span> then the <span class="token string">"-l"</span> option must be used<span class="token punctuation">.</span>
                                   symbol  symbolic reference to the address of a structure<span class="token punctuation">.</span>
                                    count  count of structures to dump from an array of structures<span class="token punctuation">;</span>
                                           <span class="token keyword">if</span> used<span class="token punctuation">,</span> this must be the last argument entered<span class="token punctuation">.</span>
                                 <span class="token operator">-</span>c count  <span class="token string">"-c"</span> is only required <span class="token keyword">if</span> <span class="token string">"count"</span> is not the last argument
                                           entered or <span class="token keyword">if</span> a negative number is entered<span class="token punctuation">;</span> <span class="token keyword">if</span> a negative
                                           value is entered<span class="token punctuation">,</span> <span class="token function">the</span> <span class="token punctuation">(</span>positive<span class="token punctuation">)</span> <span class="token string">"count"</span> structures that
                                           lead up to and include the target structure will be displayed<span class="token punctuation">.</span>
                              

                              Structure data, sizes, and member offsets are shown in the current output
                              radix unless the -x or -d option is specified.

                                三、其他相关链接

                                1、Kdump调试机理总结(一)

                                2、Kdump配置及使用详细总结(二)

                                3、gdb调试常见命令总结

                                4、Linux下objdump反编译文件命令总结

                                与[转帖]crash工具分析Kdump下vmcore文件常用命令总结(三)(实例易懂)相似的内容:

                                [转帖]crash工具分析Kdump下vmcore文件常用命令总结(三)(实例易懂)

                                一、简介 本文主要介绍使用crash工具对kdump生成的vmcore文件进行分析,解析常见的crash命令,前面已讲述两章关于Kdump的内容,读者感兴趣可以点击下面的链接: 1、Kdump调试机理总结(一) 2、Kdump配置及使用详细总结(二) 系统产生异常时,触发Kdump机制,启动捕获内核

                                [转帖]Kdump调试机理详细总结(一)

                                https://blog.csdn.net/luckiers/category_11796393.html 一、简介 本文主要讲解Kdump的运行机理,后续两个章节会详细介绍kdump的使用和如何分析coredump文件信息,具体链接如下: Kdump配置及使用详细总结(二) crash工具分析vm

                                [转帖]Kdump配置及使用(详细)总结(二)

                                一、简介 本文主要介绍如何打开Kdump并对其相关文件进行配置。前面章节已经对Kdump调试机理进行总结总结,具体可以点击下面链接: Kdump调试机理总结(一) crash工具分析vmcore文件常用命令总结(三) 二、Kdump整体介绍 1、kdump kdump 是一种先进的基于 kexec

                                [转帖]使用 Crash 工具分析 Linux dump 文件

                                前言 Linux 内核(以下简称内核)是一个不与特定进程相关的功能集合,内核的代码很难轻易的在调试器中执行和跟踪。开发者认为,内核如果发生了错误,就不应该继续运行。因此内核发生错误时,它的行为通常被设定为系统崩溃,机器重启。基于动态存储器的电气特性,机器重启后,上次错误发生时的现场会遭到破坏,这使得

                                【转帖】linux 内核分析工具 Dtrace、SystemTap、火焰图、crash等

                                << System语言详解 >> 关于 SystemTap 的书。 我们在分析各种系统异常和故障的时候,通常会用到 pstack(jstack) /pldd/ lsof/ tcpdump/ gdb(jdb)/ netstat/vmstat/ mpstat/truss(strace)/iostat/s

                                [转帖]jvm crash when overwritten running jar

                                https://www.jianshu.com/p/bf0a051e4c63 现象 策划热更完配置表后 jvm直接就crash了(开发机linux) crash日志 日志分析 从crash日志看是reload配置表 使用Reflections扫描配置类 最终读取jar包 java.util.zip.

                                [转帖]实战案例分享:根据 JVM crash 日志定位和分析问题

                                https://cloud.tencent.com/developer/article/1744442 1. JVM crash了 下面是一份crash report, 下面是截取了crash report的部分,用于分析: # Problematic frame: # V [libjvm.so+0

                                [转帖]Core dump实战分析之Java版

                                https://www.jianshu.com/p/2cdf71f99209 Core dump实战分析(Java版) 背景 项目中的battleserver进程在某一段时间总是crash,无法找到具体Crash原因 Java通过JNI调用Luajit 那么进程Crash如何找到JNI的堆栈(C层)

                                [转帖]Linux kill & Java shutdownhook

                                https://www.jianshu.com/p/8001a66d37c9 shutdown-hook 建议加上shutdown的钩子 如果程序出现了内存溢出crash 则现在代码是没有任何保护措施的 或者说运维不小心关闭了服务器等 或者运维不小心kill了游戏服务器进程等 无法避免kill -9

                                [转帖]JVM metaspace outofmemory

                                https://www.jianshu.com/p/1ca44f94e42f 解决服务器进程退出问题(metaspace溢出) 现象 策划反应服务器进不去,远程看了一下进程消失了(crash) 有时候也会出现能登录,但是无法执行操作(进程还在),无法被正常shutdown 进程根目录下出现了java