Alan's Ramblings - mysql tag

Language wars are so boring

Fri, 24 Mar 2006 03:03:59 GMT

I've just read James Gosling's post on how he got flamed for daring to say that scripting languages aren't a panacea. Whereas the old language crusades used to be fought between the various "compiled languages" it seems the new jihad is between "compiled" and "dynamic" languages. The arguments are much the same, and are just as tedious as they rely on religious fervour rather than pragmatism and common sense.

As James points out there's a huge continuum of both problem spaces and languages that can be used to address them, the trick is to pick the right tool for the job at hand. The point that most of the participants in language wars miss is that in most cases the choice of language should be secondary, the most important deciding factor should be what tools and libraries are available for the language and how closely they address the problem domain. Syntax just isn't that important - I care far less about having to declare types for my variables than I do about having to write a thousands of lines of code that could more properly be replaced with a library function. I also quite like having the dangerous and tedious bits of programming taken care of, such as memory management - an area where dynamic languages and garbage collecting languages such as Java have a clear edge over more traditional choices such as C - however in some circumstances C is still the right choice, when other factors (e.g. predictability or outright speed) are more important.

The day job for the last 18 months has been nearly 100% Perl and heavily regexp based as well as requiring access to MySQL, LDAP and a load of other network-based stuff. At one point I rewrote some of the Perl regexp code in Java. It was easy to do, thanks to Java's good regexp library. Sure it was a bit more wordy, but it worked just fine. That's the point - the power of both Perl an Java isn't really the core language, it's the ecosystem in which they exist - in Perl's case the unspoken hero is CPAN, in Java's case it's the extensive class library that it comes with, along with the vast range of freely available external class libraries. Choosing a language based purely on whether it is "static" or "dynamic" makes about as much sense as choosing a car based on the paint colour - sure it's important, but it's not the most important factor.

Making a perl filehandle into an object

Wed, 22 Dec 2004 09:25:38 GMT

I'm working on a project that uses both perl and MySQL, and I needed an easy way of piping stuff from perl into mysqlimport so that I could do fast bulk uploads to the database. MySQL is a bit odd in how it does this - it insists that the data has to come from a file who's filename prefix is the same as the table into which you are loading. This precludes a vanilla perl piped open and requires the use of a named pipe - the easiest way being to create a named pipe under /tmp of the form <table name>.<pid>.

I needed several loaders open simultaneously, so the obvious thing was to abstract the functionality into a class. I still wanted to be able to use normal print statements to output the table rows to the loader, so I wanted the filehandle to simultaneously be an object as well. Creating objects in perl is done with the bless operator, which requires a reference to bless - you can't bless a normal scalar value into an object. Fortunately open() in later versions of perl actually gives you a reference to a filehandle:

$ perl -d -e 1

Loading DB routines from perl5db.pl version 1.25
Editor support available.

Enter h or `h h' for help, or `man perldebug' for more help.

main::(-e:1):	1
  DB<1> my $a;

  DB<2> open($a, '>', '/dev/null');

  DB<3> x $a
0  GLOB(0x3a40e0)
   -> *main::$a
         FileHandle({*main::$a}) => fileno(3)
  DB<4> bless($a, 'MyClass');

  DB<5> x $a
0  MyClass=GLOB(0x3a40e0)
   -> *main::$a
         FileHandle({*main::$a}) => fileno(3)
  DB<6>

Having blessed the filehandle into the appropriate class we can then invoke methods on it (e.g. $a->do_stuff()), as well as print to it directly (e.g. print($a "hello\n");). However, having an object without any associated properties isn't really much use - in my case I needed to be able to store the name of the named pipe I was using to communicate with mysqlimport so that I could remove it when the filehandle was closed. It's possible to use the perl tie and overloading mechanisms to do this, but as this is perl and tmtowtdi always applies, and there is in fact a simpler although less obvious way.

To follow this it's first necessary to understand a little about how perl actually stores values internally. Variables come in different types, the common ones being scalars, hashes and arrays, repectively denoted by the leading $, % and @ characters on variables. $a, %a and @a look like they are entirely different variables, but in fact they aren't - they are all slots in a single perl symbol table entry called "a". These symbol table entries are called typeglobs or globs for short and are accessible by using the "*" prefix on a variable, so *a refers to the typeglob where $a, %a and @a all live.

The "a" filehandle that we opened in the example above also has a hash slot, so if we want to store additional attributes on "a" we need some way of getting the associated hash slot in its typeglob. This is actually very easy, although the syntax is a little abstruse:

my $fh;
open($fh, '>', $fifo);
my $self = \%{*$fh};
$self->{fifo} = $fifo;
bless($fh, $class);

Let's pick apart the line that assigns to $self. $fh is actually a reference to a filehandle, so we dereference it with *$ to the entire "fh" glob. The %{...} says we want to access the hash slot of the "fh" glob, and the \ gets us a reference to that, so $self ends up being a reference to the hash slot associated with "fh". Phew. We can then assign to it as a normal hash reference. When we subsequently call a method on the blessed $fh filehandle, we can use exactly the same chant to get back the hash reference and access the data that we put in it. This trick is used by the standard perl IO::Socket class to squirrel away socket attributes, but it's often useful to be able to associate properties with a filehandle yourself so I think this particular technique deserves to be more widely known. If you want further information on how all this stuff hangs together, you should check out the perlref manpage.

MySQL - is there no end to my pain?

Wed, 18 Aug 2004 14:56:18 GMT

I've been doing all my MySQL development on an Opteron box running Solaris 10, but the production machine is an Ultra 60, and therefore sparc architecture. Having got everything ready to roll out, I set to to build MySQL on the U60 - a simple job, or so I thought - only to have it blow up during the compile:

"history.c", line 617: warning: implicit function declaration: strunvis
"history.c", line 655: warning: implicit function declaration: strvis
"history.c", line 655: undefined symbol: VIS_WHITE
cc: acomp failed for history.c

It turns out that on sparc the macro HAVE_VIS_H was being defined in config.h, but it wasn't defined on x86, which was kinda odd. Time to go spelunking through configure to figure out what was going on. It turns out that configure was probing for the file vis.h, using the usual 'write and compile a test program' method, which was succeeding. A 'find' on the immediately obvious directory trees didn't help, so the -H compiler flag came to the rescue:

$ printf '#include <vis.h>\nint main(){}\n' > try.c
$ cc -H -c try.c
/ws/on-tools-uk/SUNWspro/SOS8/prod/include/cc/vis.h
	/ws/on-tools-uk/SUNWspro/SOS8/prod/include/cc/sys/vis_proto.h
		/usr/include/sys/isa_defs.h
	/ws/on-tools-uk/SUNWspro/SOS8/prod/include/cc/sys/vis_define.h
	/ws/on-tools-uk/SUNWspro/SOS8/prod/include/cc/sys/vis_types.h
	/ws/on-tools-uk/SUNWspro/SOS8/prod/include/cc/sys/vis_asi.h
$

Ah. sparc has some additional instructions called the VIS instructions which are analagous to the MMX extensions on x86, and it was the header file for those that was being picked up, rather than the one that history.c was expecting. This leads rather nicely to Alan's First Rule of Configure Scripting:

Test for the symbol, not the header file.

I've had pain in this area before, early on in the development of Solaris 10 I had to fix perl's Configure. Perl's Configure is way smarter than the GNU one in that it does actually probe the symbol table of the libraries it it interested in, which is much faster and also more accurate. However the perl Configure was assuming that if it found '_foo', plain 'foo' also also existed. It even went as far as assuming that if '__foo' existed then 'foo' also existed, and that just isn't necessarily so. Fixing it proved to be an exercise in cross-platform development and remote debugging - at the same time as fixing it on Solaris I managed to break Configure on MPE/iX, and had to work with the maintainer for that platform to get it going again. It never ceases to amaze me how many platforms perl actually compiles and runs on - as I said, it's Configure is way smarter than the GNU one.

I can't really blame MySQL for this particular problem, as I guess they normally build with gcc. In the past I've spent considerable amounts of time making sure perl builds OK with our compiler, and I've also obtained some free Forte compiler licenses for the other perl developers. Not that I have any say at all in the matter, the usual disclaimers apply etc, but unbundling just Forte cc, CC, dbx and dmake and offering them for free would be really helpful to both our customers and us. Sure, we can still make money by charging for the full IDE, but I'd love if we could make those bits free. Heck, even Microsoft have realised that free compilers make good sense!

Anyway, back to my MySQL problem. I've now got to try to figure out the command-line magic I need to stop it picking up the wrong header. I've also logged a MySQL bug - the bug id is 5102.

MySQL bites my ass... again.

Wed, 11 Aug 2004 13:30:00 GMT

Having got all my data shovelled into MySQL, I foolishly decided that I'd like some of it back. I'd carefully normalised all my tables, so in order to retrieve it I needed to join across several tables plus I needed some subqueries in the select field list to do some lookups, so it ended up being a reasonably complex query - something along the lines of:

select
    a1.col1, c1.col2
    (select b1.col2 from b b1 where b1.col1 = a1.col2),
    (select b2.col2 from b b2 where b2.col1 = a1.col3)
from
    a a1 join b b3 on a1.col1 = b3.col1
    join c c1 on a1.col4 = c1.col1
order by
    b3.col4

Which worked just fine - right up until the rows where the value in a1.col3 was NULL, when instead of the second subselect returning the value NULL it returned the value looked up in the subselect on the previous line of the query!!!!! I've tried to create a smaller reproducible test case without success, and I really don't have the time to chase it down. Removing the third join on the b table makes the problem go away, but then I lose the required ordering - gah!

I can reformulate the query to remove the subselects by the crafty use of a left outer join, which will probably be more efficient anyway, but it should work as is. Bah.

I hates software!

Elbonian database technology, part II

Mon, 26 Jul 2004 06:20:47 GMT

cp kindly commented on my last post regarding Open Source databases, and suggested two others I should go look at, SapDB/MaxDB and Firebird. I looked at MaxDB, and it is a re-branded and enhanced version of SAP DB, SAP AG's open source database. It has been taken over by MySQL AG, and it appears that it will eventually be merged into MySQL. It therefore doesn't look like a particularly good choice, as it might not be around in the future.

I then went and looked at Firebird. This started out as Borland's InterBase product, which they released as open source in 2000, then changed their minds and tried to close it back up. The result was a fork - the original Borland code was taken and rewritten in C++ (no, I have no idea why) by the Firebird community (including the original Interbase author) that developed around the original Borland code release. It's an interesting story that anyone who is thinking about open sourcing a large piece of software (*cough*) should be aware of. The documentation is a bit thin - the Firebird website relies heavily on the Borland docs, but it seems to be a mature product, so I'm going to give it a whizz.