Linux

Rapidly detecting Linux kernel source features for driver deployment

A while back I wrote about genconfig.sh, a technique for auto-detecting Linux kernel features, in order to improve portability of an open source, out-of-tree filesystem driver I developed as part of ElectricAccelerator. genconfig.sh has enabled me to maintain compatibility across a wide range of Linux kernels with relative ease, but recently I noticed that the script was unacceptably slow. On the virtual machines in our test lab, genconfig.sh required nearly 65 seconds to execute. For my 11-person team, a conservative estimate of time wasted waiting for genconfig.sh is nearly an entire person-month per year. With a little effort, I was able to reduce execution time to about 7 seconds, nearly 10x faster! Here’s how I did it.

A brief review of genconfig.sh

genconfig.sh is a technique for detecting the source code features of a Linux kernel. Like autoconf configure scripts, genconfig.sh uses a series of trivial C source files to probe for various kernel source features, like the presence or absence of the big kernel lock. For example, here’s the code used to determine whether the kernel has the set_nlink() helper function:

1
2
3
4
5
#include <linux/fs.h>
void dummy(struct inode *i)
{
set_nlink(i, 0);
}

If a particular test file compiles successfully, the feature is present; if the compile fails, the feature is absent. The test results are used to set a series of C preprocessor #define directives, which in turn are used to conditionally compile driver code suitable for the host kernel.

Reaching the breaking point

When I first implemented genconfig.sh in 2009 we only supported a few versions of the Linux kernel. Since then our support matrix has swollen to include every variant from 2.6.9 through 3.5.0, including quirky “enterprise” distributions that habitually backport advanced features without changing the reported kernel version. But platform support tends to be a mostly one-way street: once something is in the matrix, it’s very hard to pull it out. As a consequence, the number of feature tests in genconfig.sh has grown too, from about a dozen in the original implementation to over 50 in the latest version. Here’s a real-time recording of a recent version of genconfig.sh on one of the virtual machines in our test lab:

genconfig.sh executing on a test system; click for full size

Accelerator actually has two instances of genconfig.sh, one for each of the two kernel drivers we bundle, which means every time we install Accelerator we burn about 2 minutes waiting for genconfig.sh — 25% of the 8 minutes total it takes to run the install. All told I think a conservative estimate is that this costs my team nearly one full person-month of time per year, between time waiting for CI builds (which do automated installs), time waiting for manual installs (for testing and verification) and my own time spent waiting when I’m making updates to support new kernel versions.

genconfig.sh: The Next Generation

I had a hunch about the source of the performance problem: the Linux kernel build system. Remember, the pattern repeated for each feature test in the original genconfig.sh is as follows:

  1. Emit a simple C source file, called test.c.
  2. Invoke the kernel build system, using make and a trivial kernel module makefile:
    1
    2
    3
    conftest-objs := test.o
    obj-m := conftest.o
    EXTRA_CFLAGS += -Werror
  3. Check the exit status from make to decide whether the test succeeded or failed.

The C sources used to probe for features are trivial, so it seemed unlikely that the compilation itself would take so long. But we don’t have to speculate — if we use Electric Make instead of GNU make to run the test, we can use the annotated build log and ElectricInsight to see exactly what’s going on:

ElectricInsight visualization of Linux kernel module build, click for full size.

Overall, using the kernel build system to compile this single file takes nearly 2 seconds — not a big deal with only a few tests, but it adds up quickly. To be clear, the only part we actually care about is the box labeled /root/__conftest__/test.o, which is about 1/4 second. The remaining 1 1/2 seconds? Pure overhead. Perhaps most surprising is the amount of time burned just parsing the kernel makefiles — the huge bright cyan box on the left edge, as well as the smaller bright cyan boxes in the middle. Nearly 50% of the total time is just spent parsing!

At this point an idea struck me: there’s no particular reason genconfig.sh must invoke the kernel build system separately for each probe. Why not write out all the probe files upfront and invoke the kernel build system just once to compile them all in a single pass? In fact, with this strategy you can even use parallel make (eg, make -j 4) to eke out a little bit more speed.

Of course, you can’t use the exit status from make with this approach, since there’s only one invocation for many tests. Instead, genconfig.sh can give each test file a descriptive name, and then check for the existence of the corresponding .o file after make finishes. If the file is present, the feature is present; otherwise the feature is absent. Here’s the revised algorithm:

  1. Emit a distinct C source file for each kernel feature test. For example, the sample shown above might be created as set_nlink.c. Another might be write_begin.c.
  2. Invoke the kernel build system, using make -j 4 and a slightly less trivial kernel module makefile:
    1
    2
    3
    conftest-objs := set_nlink.o write_begin.o ...
    obj-m := conftest.o
    EXTRA_CFLAGS += -Werror
  3. Check for the existence of each .o file, using something like if [ -f set_nlink.o ] ; then … ; fi to decide whether the test succeeded or failed.

The net result? After an afternoon of refactoring, genconfig.sh now completes in about 7 seconds, nearly 10x faster than the original:

Updated genconfig.sh executing on a test system, click for full size.

The only drawback I can see is that the script no longer has that satisfying step-by-step output, instead dumping everything out at once after a brief pause. I’m perfectly happy to trade that for the improved performance!

Future Work and Availability

This new strategy has significantly improved the usability of genconfig.sh. After I finished the conversion, I wondered if the same technique could be applied to autoconf configure scripts. Unfortunately I find autoconf nearly impossible to work with, so I didn’t make much progress exploring that idea. Perhaps one of my more daring (or stubborn) readers will take the ball and run with it there. If you do, please comment below to let us know the results!

The new version of genconfig.sh is used in ElectricAccelerator 6.2, and can also be seen in the open source Loopback File System (lofs) on my Github repo.

HOWTO: ship a custom kernel driver for Linux

Pop quiz, hotshot: your company has developed a Linux kernel driver as part of its product offering. How do you deliver this driver such that your product is compatible with a significant majority of the Linux variants you are likely to encounter in the field? Consider the following:

  • RedHat Enterprise Linux 4 is based on kernel version 2.6.9
  • RHEL 5 is based on kernel version 2.6.18
  • RHEL 6 is based on 2.6.32
  • openSUSE 11.0 is based on 2.6.25
  • openSUSE 11.1 is based on 2.6.27
  • Ubuntu 9.04 is based on 2.6.28
  • Ubuntu 9.10 is based on 2.6.31
  • Ubuntu 10.04 is based on 2.6.32
  • Ubuntu 10.10 is based on 2.6.35

I could go on, but hopefully you get the point — “Linux” is not a single, identifiable entity, but rather a collection of related operating systems. And thus the question: how do you ship your driver such that you can install and use it on a broad spectrum of Linux variants? This is a problem that I’ve had to solve in my work.

Fundamentally, the solution is simple: ship the driver in source form. But that answer isn’t much help unless you can make your driver source-compatible with a wide range of kernel versions, spanning several years of Linux development. The solution to that problem is simple too, in hindsight, and yet I haven’t seen it used or described elsewhere: test for specific kernel features using something like a configure script; set preprocessor macros based on the results of the tests; and use the macros in the driver source to conditionally include code as needed. But before I get into the details of this solution, let’s look briefly at a few alternative solutions and why each was rejected.

Rejected alternatives: how NOT to ship a custom driver for Linux

Based on my informal survey of the state-of-the-art in this field, it seems there are three common approaches to solving this problem:

  1. Arrange for your driver to be bundled with the Linux kernel. If you can pull this off, fantastic! You’ve just outsourced the effort of porting your driver to the people who build and distribute the kernel. Unfortunately, kernel developers are not keen on bundling drivers that are not generally useful — that is, your driver has to have some utility outside of your specific application, or you can forget getting it bundled into the official kernel. Also, if you have any interesting IP in your driver, open-sourcing it is probably not an option.
  2. Prebuild your driver for every conceivable Linux variant. If you know which Linux variants your product will support, you could build the driver for each, then choose one of the prebuilt modules at installation time based on the information in /etc/issue and uname -r. VMWare uses this strategy — after installing VMWare Workstation, take a look in /usr/lib/vmware/modules/binary: you’ll find about a hundred different builds of their kernel modules, for various combinations of kernel versions, distributions and SMP-status. The trouble with this strategy is that it adds significant complexity to your build and release process: you need a build environment for every one of those variants. And all those modules bloat your install bundle. Finally, no matter how many distro’s you prebuild for, it will never be enough: somebody will come along and insist that your code install on their favorite variant.
  3. Ship source that uses the LINUX_VERSION_CODE and KERNEL_VERSION macros. These macros, defined by the Linux kernel build system, allow you to conditionally include code based on the version of the kernel being built. In theory this is all you need, if you know which version introduced a particular feature. But there are two big problems. First, you probably don’t know exactly which version introduced each feature. You could figure it out with some detective work, but who’s got the time to do that? Second, and far more troublesome, most enterprise Linux distributions (RHEL, SUSE, etc.) backport features and fixes from later kernels to their base kernel — without changing the value of LINUX_VERSION_CODE. Of course that renders this mechanism useless.

genconfig.sh: a configure script for kernel modules

Conceptually, genconfig.sh works the same way as an autoconf configure script: it uses a series of trivial test programs to check for different kernel features or constructs. The success or failure of each test to compile determines whether the corresponding feature is present, and by extension whether or not a particular bit of code ought to be included in the driver.

For example, in some versions of the Linux kernel (2.6.9, eg), struct inode includes a member called i_blksize. If present, this field should be set to the blocksize of the filesystem that owns the inode. It’s used in the implementation of the stat(2) system call. It’s a minor detail, but if you’re implementing a filesystem driver, it’s important to get it right.

We can determine whether or not to include code for this field by trying to compile a trivial kernel module containing just this code:

#include <linux/fs.h>
void dummy(void)
{
    struct inode i;
    i.i_blksize = 0;
    return;
}

If this code compiles, then we know to include code for managing the i_blksize field. We can create a header file containing a #define corresponding to this knowledge:

#define HAVE_INODE_I_BLKSIZE

Finally, the driver code uses that definition:

#ifdef HAVE_INODE_I_BLKSIZE
  inode->i_blksize = FS_BLOCKSIZE;
#endif

We can construct an equally trivial test case for each feature that is relevant to our driver. In the end we get a header with a series of defines, something like this:

#define HAVE_INODE_I_BLKSIZE
#define HAVE_3_ARG_INT_POSIX_TEST_LOCK
#define HAVE_KMEM_CACHE_T
#define HAVE_MODE_IN_VFS_SYMLINK
#define HAVE_3_ARG_PERMISSION
#define HAVE_2_ARG_UMOUNT_BEGIN
#define HAVE_PUT_INODE
#define HAVE_CLEANUP_IN_KMEM_CACHE_CREATE
#define HAVE_WRITE_BEGIN
#define HAVE_ADDRESS_SPACE_OPS_EXT
#define HAVE_SENDFILE
#define HAVE_DENTRY_IN_FSYNC

By referencing these definitions in the driver source code, we can make it source-compatible with a wide range of Linux kernel versions. To add support for a new kernel, we just have to determine which changes affect our module, write tests to check for those features, and update only the affected parts of our driver source.

This is more nimble, and far more manageable, than shipping prebuilt binaries for an endless litany of kernel variants. And it’s much more robust than relying on LINUX_VERSION_CODE: rather than implicitly trusting that a feature is present or absent based on an unreliable version string, we know for certain whether that feature is present, because we explicitly tried to use it.

Belt and suspenders: ensuring the driver works correctly

Now we have a strategy for shipping a driver that will build and load on a broad array of Linux variants. But this approach has introduced a new problem: how can we be sure that this driver that was just auto-configured and compiled on-the-fly will actually work as expected?

The solution to this problem has two components. First, we identified about a dozen specific Linux variants that are critical to our customers. The driver is exhaustively tested on each of these “tier 1” variants in every continuous integration build — over 3,000 automated unit tests are run against the driver on each. Of course, 12 variants is only a tiny fraction of the thousands of permutations that are possible, but by definition these variants represent the most important permutations to get right. We will know immediately if something has broken the driver on one of these variants.

Next, we ship a stripped down version of that unit test suite, and execute that automatically when the driver is built. This suite has only about 25 tests, but those tests cover every major piece of functionality — a reasonable compromise between coverage and simplicity. With this install-time test suite, we’ll know if there’s a problem with the driver on a particular platform as soon as somebody tries to install it.

Demonstration code

For demonstration purposes I have placed a trivial filesystem driver on my github repo. This driver, base0fs, was generated using the FiST filesystem generator, patched to make use of the genconfig.sh concept.

HOWTO: hide address-book status icons in Thunderbird 3.x

I just updated to the latest version of Thunderbird, 3.1.7. There’s lots to like in this release compared to the archaic 2.0.0.24 I was using before. Unfortunately, it also includes one feature that I really dislike: the address-book status icon. This is where Thunderbird puts a star next to each email address in the message header:

Address-book status icons in Thunderbird 3.x on Linux

I have three complaints about this “improvement”:

  1. I don’t really care if the email is in my address book. Don’t force me to deal with information that I’m not interested in.
  2. The icons are very distracting — one by itself might be alright, but a cluster of them, in bright yellow, just outside the area of visual focus (the message body) continually pulls my eye away from what I’m trying to do.
  3. It looks like my 7 year old daughter ran amok with a sheet of foil star stickers. There’s a time and a place for that kind of thing, and my mail reader is not it.

After far more research and experimentation than I care to admit (including downloading and searching the Thunderbird source code!), I figured out how to disable the address-book status icon in Thunderbird 3.x. Here’s the result:

Hidden address-book status icons

How to hide address-book status icons in Thunderbird 3.x

To hide address-book status icons in Thunderbird, you need to add the following code to your userChrome.css:

.emailStar {
  width: 0em;
  visibility:hidden;
}

userChrome.css lives in a chrome subdirectory of your Thunderbird profile directory. If you don’t already have a userChrome.css, just create one with the above contents.

After updating userChrome.css you’ll have to restart Thunderbird to see the change. Enjoy your newly uncluttered Thunderbird UI!

Bash completion for ElectricAccelerator

Maybe you haven’t noticed, but Electric Make (emake) has a lot of command-line options. Besides the options it inherits from emulating GNU make (or NMAKE), it has about fifty of its own options, from –emake-annodetail to –emake-tmpdir. Remembering them all, and their exact spelling, and their allowed values is a nuisance, even for me — and I created half of those options myself. So, I spent the last few evenings hacking together Bash TAB completion support for emake (download from github here), with pretty good results:

$ emake --emake-h<TAB><TAB>
--emake-history=        --emake-historyfile=  
--emake-history-force=

In addition to helping with emake options, it can help me remember the valid values for those options:

$ emake --emake-history=<TAB><TAB>
create  merge  read

It handles options with compounds values too, like –emake-annodetail:

$ emake --emake-annodetail=<TAB><TAB>
basic  env  file  history  lookup  registry  waiting
$ emake --emake-annodetail=file,<TAB><TAB>
file,basic     file,history   file,registry  
file,env       file,lookup    file,waiting   
$ emake --emake-annodetail=file,history,<TAB><TAB>
file,history,basic     file,history,registry
file,history,env       file,history,waiting
file,history,lookup

It can even do TAB completion on targets in makefiles, thanks to some clever code inherited from the gmake completion module that I used as the basis for my emake completion module:

$ emake <TAB><TAB>
all        check      distclean  Makefile   
buildtest  clean      install

And since I was already tinkering with TAB completion for emake, it wasn’t much work to do TAB completion for ElectricInsight (einsight) as well. In that case, TAB completion doesn’t really do a whole lot — einsight doesn’t have many command-line options. But intelligent TAB completion is still pretty handy for one specific reason: I can make it only match files with the correct extension — .xml and .anno:

$ ls
build-272.dlog  build-272.xml
build-273.dlog  build-273.xml
$ einsight <TAB>
$ einsight build-27<TAB><TAB>
build-272.xml  build-273.xml

Rather than suggesting all of the files in the directory, Bash now knows to suggest only the .xml files when I invoke einsight.

I was surprised to find that setting up custom TAB completion for my applications is pretty easy: just create a shell function that generates a list of possible completions based on a partial command-line, then instruct the shell to use that function to handle completions for whatever command you like. As far as I can tell, the mechanism is pretty flexible — you’re limited only by your own Bash scripting skill. If you’re interested in doing something like this yourself, I suggest you check out these online tutorials, as well as the Bash Completion project, which includes completion modules for nearly 200 commands.

Availability and installation

You can download the TAB completion module for emake and einsight from my github repository. As far as installation goes, you have a few options:

  1. Hook into the bash-completion package. Many modern Linux distributions, including Ubuntu 9.x/10.x and SUSE 11 install the bash-completion package and set up the default bashrc file to use it. On those systems, you can just copy accelerator.sh to /etc/bash_completion.d.
  2. Modify your personal .bashrc. If your system doesn’t have the bash-completion package, or if you can’t add files to etc, you can modify your own .bashrc to source accelerator.sh on startup. In that case I would rename it to $(HOME)/.accelerator.sh, so that it is normally hidden from directory listings, and add source $(HOME)/.accelerator.sh to your .bashrc file.

The image at the top of this post is free; you can redestribute it and/or modify it according to the terms of the Free Art License; it is based on this image by Aurelio Heckert.

HOWTO: install kernel debuginfo packages on SUSE Linux Enterprise Server 11

I needed to debug a kernel crash on SUSE Linux Enterprise Server 11 today. If you’re not familiar with debugging Linux kernel crashes, you need the kernel debug symbols in order to analyze the crash dump. These are typically not part of the kernel image itself, but instead are bundled into a kernel debuginfo package corresponding to the kernel that produced the crash dump.

Although I’ve done this on RedHat Enterprise Linux many times, I had never debugged a kernel crash on SUSE before, so I was not familiar with the process for acquiring the debuginfo packages with that distro. I couldn’t find any single set of instructions explaining how to get the packages, and although it wasn’t hard, I figured I’d try to save somebody else a little time by writing down the steps I followed.

SUSE uses a package manager called ZYpp. I used zypper, the command-line interface to ZYpp, to install the packages.

Step 1: enable the debuginfo repositories

Before zypper can install the debuginfo packages, it must be able to find them. The packages reside in specialized debuginfo repositories, which are normally not enabled, although the system is aware of them. Use zypper repos to get a list of the repositories:

lin4-ea6:~ # zypper repos
# | Alias                                                    | Name                                                   | Enabled | Refresh
--+----------------------------------------------------------+--------------------------------------------------------+---------+--------
1 | SUSE-Linux-Enterprise-Server-11 11-0                     | SUSE-Linux-Enterprise-Server-11 11-0                   | Yes     | No     
2 | SUSE-Linux-Enterprise-Software-Development-Kit-11_11-0   | SUSE-Linux-Enterprise-Software-Development-Kit-11 11-0 | Yes     | No     
3 | SUSE-Linux-Enterprise-Software-Development-Kit-11_11-0_1 | SUSE-Linux-Enterprise-Software-Development-Kit-11 11-0 | Yes     | No     
4 | nu_novell_com:SLE11-Debuginfo-Pool                       | SLE11-Debuginfo-Pool                                   | No      | Yes    
5 | nu_novell_com:SLE11-Debuginfo-Updates                    | SLE11-Debuginfo-Updates                                | No      | Yes    
6 | nu_novell_com:SLES11-Extras                              | SLES11-Extras                                          | No      | Yes    
7 | nu_novell_com:SLES11-Pool                                | SLES11-Pool                                            | No      | Yes    
8 | nu_novell_com:SLES11-Updates                             | SLES11-Updates                                         | Yes     | Yes

You want the two Debuginfo repos. To enable them, use zypper modifyrepo with the alias of the repo:

lin4-ea6:~ # zypper modifyrepo --enable nu_novell_com:SLE11-Debuginfo-Pool
Repository 'nu_novell_com:SLE11-Debuginfo-Pool' has been sucessfully enabled.
lin4-ea6:~ # zypper modifyrepo --enable nu_novell_com:SLE11-Debuginfo-Updates
Repository 'nu_novell_com:SLE11-Debuginfo-Updates' has been sucessfully enabled.

Step 2: find the debuginfo package for your crash

It’s critical to get the debuginfo package that matches the kernel that created your crash dump. It’s easy to determine the version you need: check the README.txt alongside the vmcore in the crash directory:

lin4-ea6:~ # cat /var/crash/2010-11-30-00:43/README.txt
Kernel crashdump
----------------

Crash time     : 2010-11-30 00:43 (+0000)
Kernel version : 2.6.27.45-0.1-pae
Host           : SLES-11-32
Dump level     : 0
Dump format    : compressed

In this case, I need the debuginfo for the pae variant of kernel version 2.6.27.45-0.1. Now, search the package repository for kernel debuginfo packages with zypper search:

lin4-ea6:~ # zypper search -s kernel-*-debuginfo*

Loading repository data...
Reading installed packages...

S | Name                     | Type    | Version          | Arch | Repository             
--+--------------------------+---------+------------------+------+------------------------
  | kernel-default-debuginfo | package | 2.6.27.54-0.2.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-default-debuginfo | package | 2.6.27.48-0.12.1 | i586 | SLE11-Debuginfo-Updates
  | kernel-default-debuginfo | package | 2.6.27.48-0.6.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-default-debuginfo | package | 2.6.27.48-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-default-debuginfo | package | 2.6.27.45-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-default-debuginfo | package | 2.6.27.42-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-default-debuginfo | package | 2.6.27.39-0.3.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-default-debuginfo | package | 2.6.27.37-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-default-debuginfo | package | 2.6.27.29-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-default-debuginfo | package | 2.6.27.25-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-default-debuginfo | package | 2.6.27.23-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-default-debuginfo | package | 2.6.27.21-0.1.2  | i586 | SLE11-Debuginfo-Updates
  | kernel-default-debuginfo | package | 2.6.27.19-5.1    | i586 | SLE11-Debuginfo-Pool   
  | kernel-pae-debuginfo     | package | 2.6.27.54-0.2.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-pae-debuginfo     | package | 2.6.27.48-0.12.1 | i586 | SLE11-Debuginfo-Updates
  | kernel-pae-debuginfo     | package | 2.6.27.48-0.6.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-pae-debuginfo     | package | 2.6.27.48-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-pae-debuginfo     | package | 2.6.27.45-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-pae-debuginfo     | package | 2.6.27.42-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-pae-debuginfo     | package | 2.6.27.39-0.3.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-pae-debuginfo     | package | 2.6.27.37-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-pae-debuginfo     | package | 2.6.27.29-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-pae-debuginfo     | package | 2.6.27.25-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-pae-debuginfo     | package | 2.6.27.23-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-pae-debuginfo     | package | 2.6.27.21-0.1.2  | i586 | SLE11-Debuginfo-Updates
  | kernel-pae-debuginfo     | package | 2.6.27.19-5.1    | i586 | SLE11-Debuginfo-Pool   
  | kernel-source-debuginfo  | package | 2.6.27.54-0.2.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-source-debuginfo  | package | 2.6.27.48-0.12.1 | i586 | SLE11-Debuginfo-Updates
  | kernel-source-debuginfo  | package | 2.6.27.48-0.6.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-source-debuginfo  | package | 2.6.27.48-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-source-debuginfo  | package | 2.6.27.45-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-source-debuginfo  | package | 2.6.27.42-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-source-debuginfo  | package | 2.6.27.39-0.3.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-source-debuginfo  | package | 2.6.27.37-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-source-debuginfo  | package | 2.6.27.29-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-source-debuginfo  | package | 2.6.27.25-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-source-debuginfo  | package | 2.6.27.23-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-source-debuginfo  | package | 2.6.27.21-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-source-debuginfo  | package | 2.6.27.19-5.1    | i586 | SLE11-Debuginfo-Pool   
  | kernel-vmi-debuginfo     | package | 2.6.27.54-0.2.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-vmi-debuginfo     | package | 2.6.27.48-0.12.1 | i586 | SLE11-Debuginfo-Updates
  | kernel-vmi-debuginfo     | package | 2.6.27.48-0.6.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-vmi-debuginfo     | package | 2.6.27.48-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-vmi-debuginfo     | package | 2.6.27.45-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-vmi-debuginfo     | package | 2.6.27.42-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-vmi-debuginfo     | package | 2.6.27.39-0.3.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-vmi-debuginfo     | package | 2.6.27.37-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-vmi-debuginfo     | package | 2.6.27.29-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-vmi-debuginfo     | package | 2.6.27.25-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-vmi-debuginfo     | package | 2.6.27.23-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-vmi-debuginfo     | package | 2.6.27.21-0.1.2  | i586 | SLE11-Debuginfo-Updates
  | kernel-vmi-debuginfo     | package | 2.6.27.19-5.1    | i586 | SLE11-Debuginfo-Pool   
  | kernel-xen-debuginfo     | package | 2.6.27.54-0.2.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-xen-debuginfo     | package | 2.6.27.48-0.12.1 | i586 | SLE11-Debuginfo-Updates
  | kernel-xen-debuginfo     | package | 2.6.27.48-0.6.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-xen-debuginfo     | package | 2.6.27.48-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-xen-debuginfo     | package | 2.6.27.45-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-xen-debuginfo     | package | 2.6.27.42-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-xen-debuginfo     | package | 2.6.27.39-0.3.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-xen-debuginfo     | package | 2.6.27.37-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-xen-debuginfo     | package | 2.6.27.29-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-xen-debuginfo     | package | 2.6.27.25-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-xen-debuginfo     | package | 2.6.27.23-0.1.1  | i586 | SLE11-Debuginfo-Updates
  | kernel-xen-debuginfo     | package | 2.6.27.21-0.1.2  | i586 | SLE11-Debuginfo-Updates
  | kernel-xen-debuginfo     | package | 2.6.27.19-5.1    | i586 | SLE11-Debuginfo-Pool

You can see there are several versions of each variant available. One tricky thing is that there isn’t an exact match for the kernel version I need. I’m looking for 2.6.27.45-0.1; the closest thing to it is 2.6.27.45-0.1.1. This seems to be nothing more than a minor inconsistency in labeling: the 2.6.27.45-0.1.1 is the correct package.

Step 3: install the kernel debuginfo package

Having identified the package, you are ready to install it with zypper install:

lin4-ea6:~ # zypper install kernel-pae-debuginfo=2.6.27.45-0.1.1
Loading repository data...
Reading installed packages...
Resolving package dependencies...

The following NEW package is going to be installed:
  kernel-pae-debuginfo 


The following package is not supported by its vendor:
  kernel-pae-debuginfo 


Overall download size: 153.1 M. After the operation, additional 673.8 M will be used.
Continue? [YES/no]:

At this prompt, you should type YES and hit Enter. zypper will download the package (which may take a while depending on the speed of your internet connection), showing a progress bar as it does:

Retrieving package kernel-pae-debuginfo-2.6.27.45-0.1.1.i586 (1/1), 153.1 M (673.8 M unpacked)
Retrieving: kernel-pae-debuginfo-2.6.27.45-0.1.1.i586.rpm [90% (1.1 M/s)]

Then zypper will install the package, again with a progress bar:

Retrieving package kernel-pae-debuginfo-2.6.27.45-0.1.1.i586 (1/1), 153.1 M (673.8 M unpacked)
Retrieving: kernel-pae-debuginfo-2.6.27.45-0.1.1.i586.rpm [done (244.7 K/s)]
Installing: kernel-pae-debuginfo-2.6.27.45-0.1.1 [84%]

When installation is complete, there is no more notification than that the progress bar reads “done”:

Retrieving package kernel-pae-debuginfo-2.6.27.45-0.1.1.i586 (1/1), 153.1 M (673.8 M unpacked)
Retrieving: kernel-pae-debuginfo-2.6.27.45-0.1.1.i586.rpm [done (244.7 K/s)]
Installing: kernel-pae-debuginfo-2.6.27.45-0.1.1 [done]

Step 4: symlink the debuginfo into the crash directory

The debuginfo package, for some reason, does not install the debuginfo files to a location known to the crash utility, so the final step is to make the debuginfo file available to crash before we invoke it. We do so by creating a symlink to the debuginfo file alongside the kernel image in the crash directory:

lin4-ea6:~ # ln -s /usr/lib/debug/boot/vmlinux-2.6.27.45-0.1-pae.debug /var/crash/2010-11-30-00:43

Decompress the kernel image and you’re in business:

lin4-ea6:~ # gzip -d /var/crash/2010-11-30-00:43/vmlinux-2.6.27.45-0.1-pae.gz
lin4-ea6:~ # cd /var/crash/2010-11-30-00:43
lin4-ea6:/var/crash/2010-11-30-00:43 # crash vmlinux-2.6.27.45-0.1-pae vmcore