Hacking loadable kernel modules with perl
I was reading Alan Coopersmith's blog entry on how he hacked the Xserver with the aid of perl, and that reminded me of a hack I did myself a while back, using perl.
Here's the scenario: I work mainly from home, using a Solaris 10 machine
running a debug kernel and VPN to connect to work. The VPN connection requires
the use of a loadable STREAMS driver supplied by Cisco. I also run Gnome, but
one shortcoming of gnome-term is it doesn't support /dev/console
.
To get around this I have a single dtterm started with the command-line
dtterm -C -map -title Console
, which means that any console output
is captured by the dtterm. Normally it's iconified, but the -map
flag means that if anything is output to /dev/console
it
de-iconifies so I can see the message.
Here's the rub: If you run a debug kernel you get a kernel thread that
periodically tries to unload any loadable kernel modules. This is to flush out
any problems with modules that don't load and unload properly, for example by
leaking memory. The Cisco VPN module locks itself into memory (which is sort
of OK), but every time it gets an unload request it logs the fact with the
following cmn_err()
message which ends up on
/dev/console
, which definitely isn't OK.
Oct 6 18:18:06 myhost vpnmod: VPN Module 5.2.4 Unload BUSY (Wed Mar 13 00:31:06 MST 2002)
Every time this happens my /dev/console
dtterm de-iconifies and I
end up playing whack-a-mole.
OK, let's nail that sucker. It's complicated by the fact that we don't have source for the Cisco VPN module, but who's Operating System it it anyway, right? ;-) First, let's find the culprit:
# modinfo | grep vpn 100 7b600000 1f650 - 1 vpnmod (VPN module) # ifconfig bge0 modlist 0 arp 1 ip 2 vpnmod 3 bge # find /kernel -name vpnmod /kernel/strmod/sparcv9/vpnmod /kernel/strmod/vpnmod # cp /kernel/strmod/sparcv9/vpnmod vpnmod.sparc.64 # elfdump -s vpnmod.sparc.64 | egrep 'cmn_err|print' [178] 0x000000000000 0x000000000000 NOTY GLOB D 0 UNDEF vsprintf [193] 0x000000000000 0x000000000000 NOTY GLOB D 0 UNDEF vcmn_err [255] 0x000000000000 0x000000000000 NOTY GLOB D 0 UNDEF cmn_err
So there's the potential culprits. We know vpnmod is a STREAMs driver, and
that it feels the need to tell us whenever it gets an unload request, so let's
try to narrow down where that might be. If we look at the
Device Driver Entry Points section of the 'Writing Device Drivers' manual,
we can see that all loadable modules (including STREAMS drivers) have to
implement the _init
, _fini
and _info
entry points, and that the _fini
entry point is called when a
module is being unloaded, so that looks like the obvious place to start.
# dis -F _fini vpnmod.sparc.64 | less **** DISASSEMBLER **** disassembly for vpnmod.sparc.64 section .text _fini() _init+0x120: 9d e3 bf 30 save %sp, -0xd0, %sp _init+0x124: 11 00 00 00 sethi %hi(0x0), %o0 _init+0x128: 90 12 20 00 or %o0, vpn_open, %o0 ! vpn_open _init+0x12c: 91 2a 30 20 sllx %o0, 32, %o0 _init+0x130: 13 00 00 00 sethi %hi(0x0), %o1 _init+0x134: 90 02 00 09 add %o0, %o1, %o0 _init+0x138: 90 12 20 00 or %o0, 0x0, %o0 _init+0x13c: 40 00 00 00 call _init+0x13c _init+0x140: 01 00 00 00 nop _init+0x144: d0 27 a7 eb st %o0, [%fp + 0x7eb] _init+0x148: d0 07 a7 eb ld [%fp + 0x7eb], %o0 _init+0x14c: 80 a2 20 00 cmp %o0, 0x0 _init+0x150: 12 48 00 13 bne,pt %icc, _init+0x19c _init+0x154: 01 00 00 00 nop _init+0x158: 11 00 00 00 sethi %hi(0x0), %o0 _init+0x15c: d0 77 a7 df stx %o0, [%fp + 0x7df] _init+0x160: d2 5f a7 df ldx [%fp + 0x7df], %o1 _init+0x164: 92 12 60 00 or %o1, 0x0, %o1 _init+0x168: d2 77 a7 df stx %o1, [%fp + 0x7df] _init+0x16c: d0 5f a7 df ldx [%fp + 0x7df], %o0 _init+0x170: 91 2a 30 20 sllx %o0, 32, %o0 _init+0x174: d0 77 a7 df stx %o0, [%fp + 0x7df] _init+0x178: 11 00 00 00 sethi %hi(0x0), %o0 _init+0x17c: d2 5f a7 df ldx [%fp + 0x7df], %o1 _init+0x180: 92 02 40 08 add %o1, %o0, %o1 _init+0x184: d2 77 a7 df stx %o1, [%fp + 0x7df] _init+0x188: d0 5f a7 df ldx [%fp + 0x7df], %o0 _init+0x18c: 90 12 20 00 or %o0, 0x0, %o0 _init+0x190: d0 77 a7 df stx %o0, [%fp + 0x7df] _init+0x194: 10 68 00 11 ba,pt %xcc, _init+0x1d8 _init+0x198: 01 00 00 00 nop _init+0x19c: 13 00 00 00 sethi %hi(0x0), %o1 _init+0x1a0: d2 77 a7 df stx %o1, [%fp + 0x7df] _init+0x1a4: d0 5f a7 df ldx [%fp + 0x7df], %o0 _init+0x1a8: 90 12 20 00 or %o0, 0x0, %o0 _init+0x1ac: d0 77 a7 df stx %o0, [%fp + 0x7df] _init+0x1b0: d2 5f a7 df ldx [%fp + 0x7df], %o1 _init+0x1b4: 93 2a 70 20 sllx %o1, 32, %o1 _init+0x1b8: d2 77 a7 df stx %o1, [%fp + 0x7df] _init+0x1bc: 11 00 00 00 sethi %hi(0x0), %o0 _init+0x1c0: d2 5f a7 df ldx [%fp + 0x7df], %o1 _init+0x1c4: 92 02 40 08 add %o1, %o0, %o1 _init+0x1c8: d2 77 a7 df stx %o1, [%fp + 0x7df] _init+0x1cc: d0 5f a7 df ldx [%fp + 0x7df], %o0 _init+0x1d0: 90 12 20 00 or %o0, 0x0, %o0 _init+0x1d4: d0 77 a7 df stx %o0, [%fp + 0x7df] _init+0x1d8: 90 10 20 00 clr %o0 _init+0x1dc: 13 00 00 00 sethi %hi(0x0), %o1 _init+0x1e0: 92 12 60 00 or %o1, vpn_open, %o1 ! vpn_open _init+0x1e4: 93 2a 70 20 sllx %o1, 32, %o1 _init+0x1e8: 15 00 00 00 sethi %hi(0x0), %o2 _init+0x1ec: 92 02 40 0a add %o1, %o2, %o1 _init+0x1f0: 92 12 60 00 or %o1, 0x0, %o1 _init+0x1f4: 15 00 00 00 sethi %hi(0x0), %o2 _init+0x1f8: 94 12 a0 00 or %o2, vpn_open, %o2 ! vpn_open _init+0x1fc: 95 2a b0 20 sllx %o2, 32, %o2 _init+0x200: 17 00 00 00 sethi %hi(0x0), %o3 _init+0x204: 94 02 80 0b add %o2, %o3, %o2 _init+0x208: 94 12 a0 00 or %o2, 0x0, %o2 _init+0x20c: d6 5f a7 df ldx [%fp + 0x7df], %o3 _init+0x210: 19 00 00 00 sethi %hi(0x0), %o4 _init+0x214: 98 13 20 00 or %o4, vpn_open, %o4 ! vpn_open _init+0x218: 99 2b 30 20 sllx %o4, 32, %o4 _init+0x21c: 1b 00 00 00 sethi %hi(0x0), %o5 _init+0x220: 98 03 00 0d add %o4, %o5, %o4 _init+0x224: 98 13 20 00 or %o4, 0x0, %o4 _init+0x228: 40 00 00 00 call _init+0x228 _init+0x22c: 01 00 00 00 nop _init+0x230: d0 47 a7 eb ldsw [%fp + 0x7eb], %o0 _init+0x234: b0 10 00 08 mov %o0, %i0 _init+0x238: 81 cf e0 08 return %i7 + 0x8 _init+0x23c: 01 00 00 00 nop
Hmm, wait a minute, there doesn't apppear to be any calls to
cmn_err
and friends in there, and if we look carefully at the call
instructions we notice that the calls all call the location of the call
instruction itself - what on earth is happening? Well, loadable modules are
more-or-less standard ELF files, and when they are loaded into the kernel they
undergo the same kind of link editing as userland ELF files get from
ld.so.1.
What we actually need to do is to look at the function after
relocation, and the easiest way to do that is to look on a running system using mdb:
# mdb -k Loading modules: [ unix krtld genunix specfs ufs sd ip sctp usba s1394 nca crypto random nfs audiosup ptm ipc ] > vpnmod`_fini ::dis vpnmod`_fini: save %sp, -0xd0, %sp vpnmod`_fini+4: sethi %hi(0), %o0 vpnmod`_fini+8: or %o0, 0, %o0 vpnmod`_fini+0xc: sllx %o0, 0x20, %o0 vpnmod`_fini+0x10: sethi %hi(0x70150000), %o1 vpnmod`_fini+0x14: add %o0, %o1, %o0 vpnmod`_fini+0x18: or %o0, 0x130, %o0 vpnmod`_fini+0x1c: call -0x7a3d10b8 <mod_remove> vpnmod`_fini+0x20: nop vpnmod`_fini+0x24: st %o0, [%fp + 0x7eb] vpnmod`_fini+0x28: ld [%fp + 0x7eb], %o0 vpnmod`_fini+0x2c: cmp %o0, 0 vpnmod`_fini+0x30: bne,pt %icc, +0x4c <vpnmod`_fini+0x7c> vpnmod`_fini+0x34: nop vpnmod`_fini+0x38: sethi %hi(0), %o0 vpnmod`_fini+0x3c: stx %o0, [%fp + 0x7df] vpnmod`_fini+0x40: ldx [%fp + 0x7df], %o1 vpnmod`_fini+0x44: or %o1, 0, %o1 vpnmod`_fini+0x48: stx %o1, [%fp + 0x7df] vpnmod`_fini+0x4c: ldx [%fp + 0x7df], %o0 vpnmod`_fini+0x50: sllx %o0, 0x20, %o0 vpnmod`_fini+0x54: stx %o0, [%fp + 0x7df] vpnmod`_fini+0x58: sethi %hi(0x7b61c800), %o0 vpnmod`_fini+0x5c: ldx [%fp + 0x7df], %o1 vpnmod`_fini+0x60: add %o1, %o0, %o1 vpnmod`_fini+0x64: stx %o1, [%fp + 0x7df] vpnmod`_fini+0x68: ldx [%fp + 0x7df], %o0 vpnmod`_fini+0x6c: or %o0, 0x60, %o0 vpnmod`_fini+0x70: stx %o0, [%fp + 0x7df] vpnmod`_fini+0x74: ba,pt %xcc, +0x44vpnmod`_fini+0x78: nop vpnmod`_fini+0x7c: sethi %hi(0), %o1 vpnmod`_fini+0x80: stx %o1, [%fp + 0x7df] vpnmod`_fini+0x84: ldx [%fp + 0x7df], %o0 vpnmod`_fini+0x88: or %o0, 0, %o0 vpnmod`_fini+0x8c: stx %o0, [%fp + 0x7df] vpnmod`_fini+0x90: ldx [%fp + 0x7df], %o1 vpnmod`_fini+0x94: sllx %o1, 0x20, %o1 vpnmod`_fini+0x98: stx %o1, [%fp + 0x7df] vpnmod`_fini+0x9c: sethi %hi(0x7b61c800), %o0 vpnmod`_fini+0xa0: ldx [%fp + 0x7df], %o1 vpnmod`_fini+0xa4: add %o1, %o0, %o1 vpnmod`_fini+0xa8: stx %o1, [%fp + 0x7df] vpnmod`_fini+0xac: ldx [%fp + 0x7df], %o0 vpnmod`_fini+0xb0: or %o0, 0x98, %o0 vpnmod`_fini+0xb4: stx %o0, [%fp + 0x7df] vpnmod`_fini+0xb8: clr %o0 vpnmod`_fini+0xbc: sethi %hi(0), %o1 vpnmod`_fini+0xc0: or %o1, 0, %o1 vpnmod`_fini+0xc4: sllx %o1, 0x20, %o1 vpnmod`_fini+0xc8: sethi %hi(0x7b61c800), %o2 vpnmod`_fini+0xcc: add %o1, %o2, %o1 vpnmod`_fini+0xd0: or %o1, 0xa0, %o1 vpnmod`_fini+0xd4: sethi %hi(0), %o2 vpnmod`_fini+0xd8: or %o2, 0, %o2 vpnmod`_fini+0xdc: sllx %o2, 0x20, %o2 vpnmod`_fini+0xe0: sethi %hi(0x7b61c800), %o3 vpnmod`_fini+0xe4: add %o2, %o3, %o2 vpnmod`_fini+0xe8: or %o2, 0x90, %o2 vpnmod`_fini+0xec: ldx [%fp + 0x7df], %o3 vpnmod`_fini+0xf0: sethi %hi(0), %o4 vpnmod`_fini+0xf4: or %o4, 0, %o4 vpnmod`_fini+0xf8: sllx %o4, 0x20, %o4 vpnmod`_fini+0xfc: sethi %hi(0x7b61d400), %o5 vpnmod`_fini+0x100: add %o4, %o5, %o4 vpnmod`_fini+0x104: or %o4, 0x3d0, %o4 vpnmod`_fini+0x108: call -0x7a4a2388 <cmn_err> vpnmod`_fini+0x10c: nop vpnmod`_fini+0x110: ldsw [%fp + 0x7eb], %o0 vpnmod`_fini+0x114: mov %o0, %i0 vpnmod`_fini+0x118: return %i7 + 8 vpnmod`_fini+0x11c: nop
Ahah! At vpnmod`_fini+0x108
there's a call to
cmn_err
. If we look a the cmn_err
manpage, we see the
prototype is void cmn_err(int level, char *format...);
, so the
format string will be passed in the %o2
register, this being sparc.
Looking back through the disassembly, we can work out that %o2
will have the value 0x7b61c8a0
at the time of the call to
cmn_err
. Back to mdb:
> 0x7b61c8a0/S 0x7b61c8a0: VPN Module %s Unload %s (%s)\n
Bingo. That matches the string we are seeing printed to
/dev/console
, so we have identified the offending call, all we now
have to do is to get rid of it. The obvious thing is to replace the call with
a nop instruction. We can calculate the offset of the offending instruction
fairly easily: we know the offset fom the beginning of the _fini
routine, it's 0x108, and we can use dump
to tell us the start
address of the routine:
# dump -t -n _fini vpnmod.sparc.64 vpnmod.sparc.64: ***** SYMBOL TABLE INFORMATION ***** [Index] Value Size Type Bind Other Shndx Name .symtab: [252] 0x1ca4 115724 2 1 0 0x1 _fini
So that puts the offending call instruction at 0x1ca4 + 0x108 or 0x1dac from the start of the vpnmod code segment. Let's just double-check we are right with mdb (output shortened for clarity):
> ::modinfo ID LOADADDR SIZE REV MODULE NAME : 100 7b600000 1f650 1 vpnmod (VPN module) : > 7b600000+0x1dac ::dis : vpnmod`_fini+0x108: call -0x7a4a2388 <cmn_err> : >
Yep, that works. However, an ELF file such as vpnmod isn't a straight memory
image, so we can't just seek that number of bytes into the file and write
a nop in there, we need to make sure it's the correct offset from the start of
the segment that holds the code - the .text
segment. We can find
that bit of information out as follows:
# dump -hvn .text vpnmod.sparc.64 vpnmod.sparc.64: **** SECTION HEADER TABLE **** [No] Type Flags Addr Offset Size Name Link Info Adralgn Entsize [1] PBIT -AI 0 0x40 0x1c40c .text 0 0 0x8 0
So that's telling us that the .text (code) segment of the ELF file is 0x40 bytes from the start of the file, which means that we can finally calculate the position within the ELF file of the instruction we want to nop out - <offset of .text section in file> + <offset of _fini routine from start of .text segment> + <offset of call instruction from start of _fini> = 0x40 + 0x1ca4 + 0x108 = 0x1dec. Yay! A we have to do is splat a nop over the 4 bytes starting at that position and we are done, right?
No, unfortunately not. Remember I said that the kernel link-edits the module
as it loads it in? Well, if we just nop the offending instruction krtld (the
kernel linker) will still attempt to relocate the call to
cmn_err
, specifically it assumes the first byte is already
a call opcode (0x40) and it will write the offset to cmn_err
into
the 2nd, 3rd and 4th bytes of the call instruction - only it won't
be a valid call instruction any more as we've just written a
nop over the top of it. Damn.
What we need to do is to prevent the kernel performing the relocation for the
call that we wish to nop out. How on earth do we do that? Well, let's poke
around inside the ELF file some more. We can use elfdump
to dump
out the relocation information in the file, and we already know that the
relocation we are looking for is for a call to cmn_err
that is
0x1dac bytes from the start of the .text
segment.
# elfdump -r vpnmod.sparc.64 | less Relocation Section: .rela.text type offset addend section with respect to : R_SPARC_WDISP30 0x1c8c 0 .rela.text cmn_err R_SPARC_WDISP30 0x1dac 0 .rela.text cmn_err R_SPARC_WDISP30 0x1e18 0 .rela.text cmn_err :
There it is. A quick read of the libgelf
manpage reveals the
following information:
gelf_getrela() Retrieves the Elf32_Rela or Elf64_Rela information from the relocation table at the given index. dst points to the location where the GElf_Rela relocation entry will be stored.
So it's off to /usr/include
to find the definition of Elf64_Rela
:
typedef struct { Elf64_Addr r_offset; Elf64_Xword r_info; /* sym, type: ELF64_R_... */ Elf64_Sxword r_addend; } Elf64_Rela;
OK, let's see if we can persuade the linker to change what it does with the
relocation that we want to nobble. Searching for the existing relocation type,
R_SPARC_WDISP30
, in the /usr/include
header files
reveals this:
: #define R_SPARC_NONE 0 /* relocation type */ #define R_SPARC_8 1 #define R_SPARC_16 2 #define R_SPARC_32 3 #define R_SPARC_DISP8 4 #define R_SPARC_DISP16 5 #define R_SPARC_DISP32 6 #define R_SPARC_WDISP30 7 :
Hmm, that R_SPARC_NONE
sure looks interesting. At this point we
suspect that the mighty Linker
and Libraries Guide may have some words of wisdom - although it documents
the workings of ld
and ld.so.1
, much of the
information on how ELF files work is also applicable to the kernel. Sure
enough, Chapter 7 on Object File Format has a strong hint that
R_SPARC_NONE
relocations are ignored by the linker, so if we set
the r_info
field of the appropriate Elf64_Rela
entry
in the .rela.text
section to R_SPARC_NONE
we should
be able to get the kernel linker to ignore the relocation for the call we wish
to nop out. At this point, I went off and looked at the source of the krtld,
the kernel linker to confirm that this was in fact what happened - an option
that once Solaris is released as Open Source will be open to you as well :-)
Our story is almost told - the process of overwriting the appropriate relocation
table section is much the same as that used for nop-ing out the function call -
find the start of the .rela.text
table in the ELF file, find the
index in the .rela.text
table of the entry we want to null out,
write zeros over it, and we're done.
Those of you who have lasted this far are probably asking "What on earth has this got to do with perl?". Well, there are some slight differences between the way this has to be done on 32-bit sparc, 64-bit sparc and i386, and I needed fixed versions of the VPN module for all those platforms. The vpnmod module is also unnecessarily chatty when it's loaded up during boot, and I wanted to shut it up then as well. I therefore wrote a perl script to do all this automatically. Unfortunately we aren't permitted to post source code on our blogs so I can't share it with you, but I can let you see it in action:
# patchmod _fini+0x108 vpnmod.sparc.64 vpnmod.sparc.64.patched .text segment base is 0x40, length is 0x1c40c .rela.text segment base is 0x21808, length is 0x11880, each entry is 0x18 bytes long function _fini base is 0x1ca4, length is 0x1c40c call opcode is at offset 0x1dac in the .text segment call opcode is at offset 0x1dec in the file relocation for address 0x1dac is at index 363 in .rela.text, and is a call to cmn_err relocation table entry is at offset 0x23a10 in the file copying and patching file done
And if we dig around with dis
and dump
we can see that
the script has had the desired effect:
# dis -F _fini vpnmod.sparc.64.patched | less : _init+0x218: 99 2b 30 20 sllx %o4, 32, %o4 _init+0x21c: 1b 00 00 00 sethi %hi(0x0), %o5 _init+0x220: 98 03 00 0d add %o4, %o5, %o4 _init+0x224: 98 13 20 00 or %o4, 0x0, %o4 _init+0x228: 01 00 00 00 nop _init+0x22c: 01 00 00 00 nop _init+0x230: d0 47 a7 eb ldsw [%fp + 0x7eb], %o0 _init+0x234: b0 10 00 08 mov %o0, %i0 _init+0x238: 81 cf e0 08 return %i7 + 0x8 : # dump -rvp -n .rela.text vpnmod.sparc.64.patched | less : 0x1c8c cmn_err R_SPARC_WDISP30 0 0 0 R_SPARC_NONE 0 0x1e18 cmn_err R_SPARC_WDISP30 0 :
It's worth noting that I could have done this all in C - Solaris has libraries for directly manipulating ELF files, see the gelf(3ELF) and elf(3ELF) manpages, but hey, I was in a hurry ;-)