Hacking loadable kernel modules with perl

I was reading Alan Coopersmith's blog entry on how he hacked the Xserver with the aid of perl, and that reminded me of a hack I did myself a while back, using perl.

Here's the scenario: I work mainly from home, using a Solaris 10 machine running a debug kernel and VPN to connect to work. The VPN connection requires the use of a loadable STREAMS driver supplied by Cisco. I also run Gnome, but one shortcoming of gnome-term is it doesn't support /dev/console. To get around this I have a single dtterm started with the command-line dtterm -C -map -title Console, which means that any console output is captured by the dtterm. Normally it's iconified, but the -map flag means that if anything is output to /dev/console it de-iconifies so I can see the message.

Here's the rub: If you run a debug kernel you get a kernel thread that periodically tries to unload any loadable kernel modules. This is to flush out any problems with modules that don't load and unload properly, for example by leaking memory. The Cisco VPN module locks itself into memory (which is sort of OK), but every time it gets an unload request it logs the fact with the following cmn_err() message which ends up on /dev/console, which definitely isn't OK.

Oct  6 18:18:06 myhost vpnmod: VPN Module 5.2.4 Unload BUSY (Wed Mar 13 00:31:06 MST 2002)

Every time this happens my /dev/console dtterm de-iconifies and I end up playing whack-a-mole.

OK, let's nail that sucker. It's complicated by the fact that we don't have source for the Cisco VPN module, but who's Operating System it it anyway, right? ;-) First, let's find the culprit:

# modinfo | grep vpn
100 7b600000  1f650   -   1  vpnmod (VPN module)
# ifconfig bge0 modlist
0 arp
1 ip
2 vpnmod
3 bge
# find /kernel -name vpnmod
/kernel/strmod/sparcv9/vpnmod
/kernel/strmod/vpnmod
# cp /kernel/strmod/sparcv9/vpnmod vpnmod.sparc.64
# elfdump -s vpnmod.sparc.64 | egrep 'cmn_err|print'
     [178]  0x000000000000 0x000000000000  NOTY GLOB  D    0 UNDEF       vsprintf
     [193]  0x000000000000 0x000000000000  NOTY GLOB  D    0 UNDEF       vcmn_err
     [255]  0x000000000000 0x000000000000  NOTY GLOB  D    0 UNDEF       cmn_err

So there's the potential culprits. We know vpnmod is a STREAMs driver, and that it feels the need to tell us whenever it gets an unload request, so let's try to narrow down where that might be. If we look at the Device Driver Entry Points section of the 'Writing Device Drivers' manual, we can see that all loadable modules (including STREAMS drivers) have to implement the _init, _fini and _info entry points, and that the _fini entry point is called when a module is being unloaded, so that looks like the obvious place to start.

#  dis -F _fini vpnmod.sparc.64 | less
                ****   DISASSEMBLER  ****


disassembly for vpnmod.sparc.64

section .text
_fini()
    _init+0x120:            9d e3 bf 30  save         %sp, -0xd0, %sp
    _init+0x124:            11 00 00 00  sethi        %hi(0x0), %o0
    _init+0x128:            90 12 20 00  or           %o0, vpn_open, %o0        ! vpn_open
    _init+0x12c:            91 2a 30 20  sllx         %o0, 32, %o0
    _init+0x130:            13 00 00 00  sethi        %hi(0x0), %o1
    _init+0x134:            90 02 00 09  add          %o0, %o1, %o0
    _init+0x138:            90 12 20 00  or           %o0, 0x0, %o0
    _init+0x13c:            40 00 00 00  call         _init+0x13c
    _init+0x140:            01 00 00 00  nop          
    _init+0x144:            d0 27 a7 eb  st           %o0, [%fp + 0x7eb]
    _init+0x148:            d0 07 a7 eb  ld           [%fp + 0x7eb], %o0
    _init+0x14c:            80 a2 20 00  cmp          %o0, 0x0
    _init+0x150:            12 48 00 13  bne,pt       %icc, _init+0x19c
    _init+0x154:            01 00 00 00  nop          
    _init+0x158:            11 00 00 00  sethi        %hi(0x0), %o0
    _init+0x15c:            d0 77 a7 df  stx          %o0, [%fp + 0x7df]
    _init+0x160:            d2 5f a7 df  ldx          [%fp + 0x7df], %o1
    _init+0x164:            92 12 60 00  or           %o1, 0x0, %o1
    _init+0x168:            d2 77 a7 df  stx          %o1, [%fp + 0x7df]
    _init+0x16c:            d0 5f a7 df  ldx          [%fp + 0x7df], %o0
    _init+0x170:            91 2a 30 20  sllx         %o0, 32, %o0
    _init+0x174:            d0 77 a7 df  stx          %o0, [%fp + 0x7df]
    _init+0x178:            11 00 00 00  sethi        %hi(0x0), %o0
    _init+0x17c:            d2 5f a7 df  ldx          [%fp + 0x7df], %o1
    _init+0x180:            92 02 40 08  add          %o1, %o0, %o1
    _init+0x184:            d2 77 a7 df  stx          %o1, [%fp + 0x7df]
    _init+0x188:            d0 5f a7 df  ldx          [%fp + 0x7df], %o0
    _init+0x18c:            90 12 20 00  or           %o0, 0x0, %o0
    _init+0x190:            d0 77 a7 df  stx          %o0, [%fp + 0x7df]
    _init+0x194:            10 68 00 11  ba,pt        %xcc, _init+0x1d8
    _init+0x198:            01 00 00 00  nop          
    _init+0x19c:            13 00 00 00  sethi        %hi(0x0), %o1
    _init+0x1a0:            d2 77 a7 df  stx          %o1, [%fp + 0x7df]

    _init+0x1a4:            d0 5f a7 df  ldx          [%fp + 0x7df], %o0
    _init+0x1a8:            90 12 20 00  or           %o0, 0x0, %o0
    _init+0x1ac:            d0 77 a7 df  stx          %o0, [%fp + 0x7df]
    _init+0x1b0:            d2 5f a7 df  ldx          [%fp + 0x7df], %o1
    _init+0x1b4:            93 2a 70 20  sllx         %o1, 32, %o1
    _init+0x1b8:            d2 77 a7 df  stx          %o1, [%fp + 0x7df]
    _init+0x1bc:            11 00 00 00  sethi        %hi(0x0), %o0
    _init+0x1c0:            d2 5f a7 df  ldx          [%fp + 0x7df], %o1
    _init+0x1c4:            92 02 40 08  add          %o1, %o0, %o1
    _init+0x1c8:            d2 77 a7 df  stx          %o1, [%fp + 0x7df]
    _init+0x1cc:            d0 5f a7 df  ldx          [%fp + 0x7df], %o0
    _init+0x1d0:            90 12 20 00  or           %o0, 0x0, %o0
    _init+0x1d4:            d0 77 a7 df  stx          %o0, [%fp + 0x7df]
    _init+0x1d8:            90 10 20 00  clr          %o0
    _init+0x1dc:            13 00 00 00  sethi        %hi(0x0), %o1
    _init+0x1e0:            92 12 60 00  or           %o1, vpn_open, %o1        ! vpn_open
    _init+0x1e4:            93 2a 70 20  sllx         %o1, 32, %o1
    _init+0x1e8:            15 00 00 00  sethi        %hi(0x0), %o2
    _init+0x1ec:            92 02 40 0a  add          %o1, %o2, %o1
    _init+0x1f0:            92 12 60 00  or           %o1, 0x0, %o1
    _init+0x1f4:            15 00 00 00  sethi        %hi(0x0), %o2
    _init+0x1f8:            94 12 a0 00  or           %o2, vpn_open, %o2        ! vpn_open
    _init+0x1fc:            95 2a b0 20  sllx         %o2, 32, %o2
    _init+0x200:            17 00 00 00  sethi        %hi(0x0), %o3
    _init+0x204:            94 02 80 0b  add          %o2, %o3, %o2
    _init+0x208:            94 12 a0 00  or           %o2, 0x0, %o2
    _init+0x20c:            d6 5f a7 df  ldx          [%fp + 0x7df], %o3
    _init+0x210:            19 00 00 00  sethi        %hi(0x0), %o4
    _init+0x214:            98 13 20 00  or           %o4, vpn_open, %o4        ! vpn_open
    _init+0x218:            99 2b 30 20  sllx         %o4, 32, %o4
    _init+0x21c:            1b 00 00 00  sethi        %hi(0x0), %o5
    _init+0x220:            98 03 00 0d  add          %o4, %o5, %o4
    _init+0x224:            98 13 20 00  or           %o4, 0x0, %o4
    _init+0x228:            40 00 00 00  call         _init+0x228
    _init+0x22c:            01 00 00 00  nop          
    _init+0x230:            d0 47 a7 eb  ldsw         [%fp + 0x7eb], %o0
    _init+0x234:            b0 10 00 08  mov          %o0, %i0
    _init+0x238:            81 cf e0 08  return       %i7 + 0x8
    _init+0x23c:            01 00 00 00  nop          

Hmm, wait a minute, there doesn't apppear to be any calls to cmn_err and friends in there, and if we look carefully at the call instructions we notice that the calls all call the location of the call instruction itself - what on earth is happening? Well, loadable modules are more-or-less standard ELF files, and when they are loaded into the kernel they undergo the same kind of link editing as userland ELF files get from ld.so.1. What we actually need to do is to look at the function after relocation, and the easiest way to do that is to look on a running system using mdb:

# mdb -k
Loading modules: [ unix krtld genunix specfs ufs sd ip sctp usba s1394 nca
crypto random nfs audiosup ptm ipc ]
> vpnmod`_fini ::dis
vpnmod`_fini:                   save      %sp, -0xd0, %sp
vpnmod`_fini+4:                 sethi     %hi(0), %o0
vpnmod`_fini+8:                 or        %o0, 0, %o0
vpnmod`_fini+0xc:               sllx      %o0, 0x20, %o0
vpnmod`_fini+0x10:              sethi     %hi(0x70150000), %o1
vpnmod`_fini+0x14:              add       %o0, %o1, %o0
vpnmod`_fini+0x18:              or        %o0, 0x130, %o0
vpnmod`_fini+0x1c:              call      -0x7a3d10b8   <mod_remove>
vpnmod`_fini+0x20:              nop
vpnmod`_fini+0x24:              st        %o0, [%fp + 0x7eb]
vpnmod`_fini+0x28:              ld        [%fp + 0x7eb], %o0
vpnmod`_fini+0x2c:              cmp       %o0, 0
vpnmod`_fini+0x30:              bne,pt    %icc, +0x4c   <vpnmod`_fini+0x7c>
vpnmod`_fini+0x34:              nop
vpnmod`_fini+0x38:              sethi     %hi(0), %o0
vpnmod`_fini+0x3c:              stx       %o0, [%fp + 0x7df]
vpnmod`_fini+0x40:              ldx       [%fp + 0x7df], %o1
vpnmod`_fini+0x44:              or        %o1, 0, %o1
vpnmod`_fini+0x48:              stx       %o1, [%fp + 0x7df]
vpnmod`_fini+0x4c:              ldx       [%fp + 0x7df], %o0
vpnmod`_fini+0x50:              sllx      %o0, 0x20, %o0
vpnmod`_fini+0x54:              stx       %o0, [%fp + 0x7df]
vpnmod`_fini+0x58:              sethi     %hi(0x7b61c800), %o0
vpnmod`_fini+0x5c:              ldx       [%fp + 0x7df], %o1
vpnmod`_fini+0x60:              add       %o1, %o0, %o1
vpnmod`_fini+0x64:              stx       %o1, [%fp + 0x7df]
vpnmod`_fini+0x68:              ldx       [%fp + 0x7df], %o0
vpnmod`_fini+0x6c:              or        %o0, 0x60, %o0
vpnmod`_fini+0x70:              stx       %o0, [%fp + 0x7df]
vpnmod`_fini+0x74:              ba,pt     %xcc, +0x44   
vpnmod`_fini+0x78:              nop
vpnmod`_fini+0x7c:              sethi     %hi(0), %o1
vpnmod`_fini+0x80:              stx       %o1, [%fp + 0x7df]
vpnmod`_fini+0x84:              ldx       [%fp + 0x7df], %o0
vpnmod`_fini+0x88:              or        %o0, 0, %o0
vpnmod`_fini+0x8c:              stx       %o0, [%fp + 0x7df]
vpnmod`_fini+0x90:              ldx       [%fp + 0x7df], %o1
vpnmod`_fini+0x94:              sllx      %o1, 0x20, %o1
vpnmod`_fini+0x98:              stx       %o1, [%fp + 0x7df]
vpnmod`_fini+0x9c:              sethi     %hi(0x7b61c800), %o0
vpnmod`_fini+0xa0:              ldx       [%fp + 0x7df], %o1
vpnmod`_fini+0xa4:              add       %o1, %o0, %o1
vpnmod`_fini+0xa8:              stx       %o1, [%fp + 0x7df]
vpnmod`_fini+0xac:              ldx       [%fp + 0x7df], %o0
vpnmod`_fini+0xb0:              or        %o0, 0x98, %o0
vpnmod`_fini+0xb4:              stx       %o0, [%fp + 0x7df]
vpnmod`_fini+0xb8:              clr       %o0
vpnmod`_fini+0xbc:              sethi     %hi(0), %o1
vpnmod`_fini+0xc0:              or        %o1, 0, %o1
vpnmod`_fini+0xc4:              sllx      %o1, 0x20, %o1
vpnmod`_fini+0xc8:              sethi     %hi(0x7b61c800), %o2
vpnmod`_fini+0xcc:              add       %o1, %o2, %o1
vpnmod`_fini+0xd0:              or        %o1, 0xa0, %o1
vpnmod`_fini+0xd4:              sethi     %hi(0), %o2
vpnmod`_fini+0xd8:              or        %o2, 0, %o2
vpnmod`_fini+0xdc:              sllx      %o2, 0x20, %o2
vpnmod`_fini+0xe0:              sethi     %hi(0x7b61c800), %o3
vpnmod`_fini+0xe4:              add       %o2, %o3, %o2
vpnmod`_fini+0xe8:              or        %o2, 0x90, %o2
vpnmod`_fini+0xec:              ldx       [%fp + 0x7df], %o3
vpnmod`_fini+0xf0:              sethi     %hi(0), %o4
vpnmod`_fini+0xf4:              or        %o4, 0, %o4
vpnmod`_fini+0xf8:              sllx      %o4, 0x20, %o4
vpnmod`_fini+0xfc:              sethi     %hi(0x7b61d400), %o5
vpnmod`_fini+0x100:             add       %o4, %o5, %o4
vpnmod`_fini+0x104:             or        %o4, 0x3d0, %o4
vpnmod`_fini+0x108:             call      -0x7a4a2388   <cmn_err>
vpnmod`_fini+0x10c:             nop
vpnmod`_fini+0x110:             ldsw      [%fp + 0x7eb], %o0
vpnmod`_fini+0x114:             mov       %o0, %i0
vpnmod`_fini+0x118:             return    %i7 + 8
vpnmod`_fini+0x11c:             nop

Ahah! At vpnmod`_fini+0x108 there's a call to cmn_err. If we look a the cmn_err manpage, we see the prototype is void cmn_err(int level, char *format...);, so the format string will be passed in the %o2 register, this being sparc. Looking back through the disassembly, we can work out that %o2 will have the value 0x7b61c8a0 at the time of the call to cmn_err. Back to mdb:

> 0x7b61c8a0/S
0x7b61c8a0:     VPN Module %s Unload %s (%s)\n

Bingo. That matches the string we are seeing printed to /dev/console, so we have identified the offending call, all we now have to do is to get rid of it. The obvious thing is to replace the call with a nop instruction. We can calculate the offset of the offending instruction fairly easily: we know the offset fom the beginning of the _fini routine, it's 0x108, and we can use dump to tell us the start address of the routine:

# dump -t -n _fini vpnmod.sparc.64

vpnmod.sparc.64:

              ***** SYMBOL TABLE INFORMATION *****

[Index]  Value           Size         Type	Bind	Other	Shndx	Name
.symtab:
[252]	 0x1ca4          115724       2		1	0	0x1	_fini

So that puts the offending call instruction at 0x1ca4 + 0x108 or 0x1dac from the start of the vpnmod code segment. Let's just double-check we are right with mdb (output shortened for clarity):

> ::modinfo
 ID         LOADADDR     SIZE REV MODULE NAME
:
100         7b600000    1f650   1 vpnmod (VPN module)
:
> 7b600000+0x1dac ::dis
:
vpnmod`_fini+0x108:             call      -0x7a4a2388   <cmn_err>
:
>

Yep, that works. However, an ELF file such as vpnmod isn't a straight memory image, so we can't just seek that number of bytes into the file and write a nop in there, we need to make sure it's the correct offset from the start of the segment that holds the code - the .text segment. We can find that bit of information out as follows:

# dump -hvn .text vpnmod.sparc.64  

vpnmod.sparc.64:

	   **** SECTION HEADER TABLE ****
[No]	Type	Flags	Addr           Offset         Size            Name
	Link	Info	Adralgn        Entsize

[1]	PBIT    -AI	0              0x40           0x1c40c         .text
	0	0	0x8            0              

So that's telling us that the .text (code) segment of the ELF file is 0x40 bytes from the start of the file, which means that we can finally calculate the position within the ELF file of the instruction we want to nop out - <offset of .text section in file> + <offset of _fini routine from start of .text segment> + <offset of call instruction from start of _fini> = 0x40 + 0x1ca4 + 0x108 = 0x1dec. Yay! A we have to do is splat a nop over the 4 bytes starting at that position and we are done, right?

No, unfortunately not. Remember I said that the kernel link-edits the module as it loads it in? Well, if we just nop the offending instruction krtld (the kernel linker) will still attempt to relocate the call to cmn_err, specifically it assumes the first byte is already a call opcode (0x40) and it will write the offset to cmn_err into the 2nd, 3rd and 4th bytes of the call instruction - only it won't be a valid call instruction any more as we've just written a nop over the top of it. Damn.

What we need to do is to prevent the kernel performing the relocation for the call that we wish to nop out. How on earth do we do that? Well, let's poke around inside the ELF file some more. We can use elfdump to dump out the relocation information in the file, and we already know that the relocation we are looking for is for a call to cmn_err that is 0x1dac bytes from the start of the .text segment.

# elfdump -r vpnmod.sparc.64 | less           

Relocation Section:  .rela.text
        type                          offset     addend  section        with respect to
:
        R_SPARC_WDISP30               0x1c8c          0  .rela.text     cmn_err  
        R_SPARC_WDISP30               0x1dac          0  .rela.text     cmn_err  
        R_SPARC_WDISP30               0x1e18          0  .rela.text     cmn_err  
:

There it is. A quick read of the libgelf manpage reveals the following information:

     gelf_getrela()
           Retrieves the  Elf32_Rela  or  Elf64_Rela  information
           from  the  relocation  table  at  the given index. dst
           points to the location where the GElf_Rela  relocation
           entry will be stored.

So it's off to /usr/include to find the definition of Elf64_Rela:

typedef struct {
	Elf64_Addr	r_offset;
	Elf64_Xword	r_info;		/* sym, type: ELF64_R_... */
	Elf64_Sxword	r_addend;
} Elf64_Rela;

OK, let's see if we can persuade the linker to change what it does with the relocation that we want to nobble. Searching for the existing relocation type, R_SPARC_WDISP30, in the /usr/include header files reveals this:

:
#define R_SPARC_NONE		0		/* relocation type */
#define R_SPARC_8		1
#define R_SPARC_16		2
#define R_SPARC_32		3
#define R_SPARC_DISP8		4
#define R_SPARC_DISP16		5
#define R_SPARC_DISP32		6
#define R_SPARC_WDISP30		7
:

Hmm, that R_SPARC_NONE sure looks interesting. At this point we suspect that the mighty Linker and Libraries Guide may have some words of wisdom - although it documents the workings of ld and ld.so.1, much of the information on how ELF files work is also applicable to the kernel. Sure enough, Chapter 7 on Object File Format has a strong hint that R_SPARC_NONE relocations are ignored by the linker, so if we set the r_info field of the appropriate Elf64_Rela entry in the .rela.text section to R_SPARC_NONE we should be able to get the kernel linker to ignore the relocation for the call we wish to nop out. At this point, I went off and looked at the source of the krtld, the kernel linker to confirm that this was in fact what happened - an option that once Solaris is released as Open Source will be open to you as well :-)

Our story is almost told - the process of overwriting the appropriate relocation table section is much the same as that used for nop-ing out the function call - find the start of the .rela.text table in the ELF file, find the index in the .rela.text table of the entry we want to null out, write zeros over it, and we're done.

Those of you who have lasted this far are probably asking "What on earth has this got to do with perl?". Well, there are some slight differences between the way this has to be done on 32-bit sparc, 64-bit sparc and i386, and I needed fixed versions of the VPN module for all those platforms. The vpnmod module is also unnecessarily chatty when it's loaded up during boot, and I wanted to shut it up then as well. I therefore wrote a perl script to do all this automatically. Unfortunately we aren't permitted to post source code on our blogs so I can't share it with you, but I can let you see it in action:

# patchmod _fini+0x108 vpnmod.sparc.64 vpnmod.sparc.64.patched
.text segment base is 0x40, length is 0x1c40c
.rela.text segment base is 0x21808, length is 0x11880,
    each entry is 0x18 bytes long
function _fini base is 0x1ca4, length is 0x1c40c
call opcode is at offset 0x1dac in the .text segment
call opcode is at offset 0x1dec in the file
relocation for address 0x1dac is at index 363 in .rela.text,
    and is a call to cmn_err
relocation table entry is at offset 0x23a10 in the file
copying and patching file
done

And if we dig around with dis and dump we can see that the script has had the desired effect:

# dis -F _fini vpnmod.sparc.64.patched | less
:
    _init+0x218:            99 2b 30 20  sllx         %o4, 32, %o4
    _init+0x21c:            1b 00 00 00  sethi        %hi(0x0), %o5
    _init+0x220:            98 03 00 0d  add          %o4, %o5, %o4
    _init+0x224:            98 13 20 00  or           %o4, 0x0, %o4
    _init+0x228:            01 00 00 00  nop          
    _init+0x22c:            01 00 00 00  nop          
    _init+0x230:            d0 47 a7 eb  ldsw         [%fp + 0x7eb], %o0
    _init+0x234:            b0 10 00 08  mov          %o0, %i0
    _init+0x238:            81 cf e0 08  return       %i7 + 0x8
:
# dump -rvp -n .rela.text vpnmod.sparc.64.patched | less
:
0x1c8c		cmn_err			R_SPARC_WDISP30	  0
0		0			R_SPARC_NONE	  0
0x1e18		cmn_err			R_SPARC_WDISP30	  0
:

It's worth noting that I could have done this all in C - Solaris has libraries for directly manipulating ELF files, see the gelf(3ELF) and elf(3ELF) manpages, but hey, I was in a hurry ;-)

Add a comment

Posted by Alan Burlison on 07 October 2004 07:06:40 BST#