Sometimes it is useful to configure kernel crash dumps in order to troubleshoot panics on hardware servers with Parallels Cloud Server, Parallels Server Bare Metal, or Parallels Virtuozzo Containers for Linux.
kexec-toolson the Hardware Node:
[root@pvcfl46x64 ~]# yum install kexec-tools
NOTE: In RHEL 6.x-based distributions,
kexec-toolsis installed automatically and the rest of the configuration is applied as well. These steps can be used to check the configuration.
Append the GRUB configuration with the kernel option
crashkernel=128M@16Mfor i386 or x86_64 and with
crashkernel=256M@256Mfor Itanium for PVC versions prior to 4.7. For versions more recent than 4.7, consider using
crashkernel=129M@0Mfor i386 and x86_64 architectures.
[root@pvcfl46x64 ~]# grep crash /etc/grub.conf kernel /boot/vmlinuz-2.6.18-028stab070.14 ro root=LABEL=/1 debug console=ttyS0,115200 console=tty crashkernel=128M@16M
NOTE: For 2.6.32-042stabXXX kernels and newer releases, be sure to use the "crashkernel=129M@0M" option in p.2.
- Reboot HW.
Enable Kdump on HW:
[root@pvcfl46x64 ~]# chkconfig kdump on [root@pvcfl46x64 ~]# chkconfig --list kdump kdump 0:off 1:off 2:on 3:on 4:on 5:on 6:off
Start Kdump on HW:
[root@pvcfl46x64 ~]# service kdump start No kdump initial ramdisk found. [WARNING] Rebuilding /boot/initrd-2.6.18-028stab070.14kdump.img Starting kdump: [ OK ]
NOTE: If you are configuring crash dumps on a Parallels Server Bare Metal machine, then add Parallels modules to "blacklist" before starting/restarting
[root@psbm5~]# tail -n1 /etc/kdump.conf blacklist extra/parallels/
Or you may stop Parallels Hypervisor before creating the
initrdimage, so that modules are not included in the image:
~# service parallels-server stop ~# touch /etc/kdump.conf ~# service kdump restart
If you want to store dumps on the external server, then additional steps are required. Edit "/etc/kdump.conf" to use the appropriate server:
[root@pvcfl46x64 ~]# cat /etc/kdump.conf net firstname.lastname@example.org path /var/crash/pvcfl46x64/
Note: Make sure that the partition is specified and that the "path" is big enough to store kernel dumps. The amount of available free space should be big enough to store several dumps of a size equal to the RAM.
[root@pvcfl46x64 ~]# service kdump propagate Generating new ssh keys... done. email@example.com's password: /root/.ssh/kdump_id_rsa.pub has been added to ~root/.ssh/authorized_keys2 on 10.55.63.11 [root@pvcfl46x64 ~]#
[root@pvcfl46x64 ~]# service kdump restart Stopping kdump: [ OK ] Detected change(s) the following file(s): /etc/kdump.conf Rebuilding /boot/initrd-2.6.18-028stab070.14kdump.img Starting kdump: [ OK ] [root@pvcfl46x64 ~]#
Now, when the kernel crashes, the dump can be found under "/var/crash/pvcfl46x64/."
On a non-production server, you can test the configuration as follows. For a production server, ensure that critical services are stopped to avoid possible data loss:
# echo 1 > /proc/sys/kernel/sysrq # echo c > /proc/sysrq-trigger
If everything is OK, you should see an output on the server console that looks similar to that displayed below:
The crash dump should appear as follows:
[root@pvcfl46x64 ~]# ls -la /var/crash/2011-03-03-05\:55/ total 649940 drwxr-xr-x 2 root root 4096 Mar 3 05:56 . drwxr-xr-x 3 root root 4096 Mar 3 05:55 .. -r-------- 1 root root 2018588492 Mar 3 05:56 vmcore
Parallels Containers user guide:
Dedoimedo.com also has a lot of useful information about kernel crash troubleshooting:
- Analyzing Linux kernel crash dumps with crash - The one tutorial that has it all
- Collecting and analyzing Linux kernel crashes - Kdump
In case of using Ksplice Uptrack solution for rebootless security updates, the generated crash dumps are not appropriate for analysis by Parallels Support. Such dumps should be forwarded to Ksplice Team.
10055 How to configure memory dump generation on Windows