set -e and set -x

If you write shell scripts, you definitely need to know about two nice features that can be enabled through the set builtin:

• set -e: Enables checking of all commands. If a command exits with an error and the caller does not check such error, the script aborts immediately. Enabling this will make your scripts more robust. But don't wait until your script is "complete" to set the flag as an afterthought, because it will be a nightmare to fix the scrip to work with this feature enabled. Just write set -e as the very first line of your code; well... after the shell bang.
• set -x: If you are writing simple scripts that are meant to, well, script the execution of a few tasks (as opposed of being full-flown programs written in shell), set this flag to trace the execution of all commands. This will make the interpreter print each command right before it is executed, so it will aid you in knowing what is happening at any point in time.

Installing NetBSD/macppc on a Mac Mini G4

Yesterday, I spent a while installing NetBSD/macppc 5.0.1 on a Mac Mini G4. The process wasn't easy, as it involved the following steps. I'm omitting many details, as they are "common knowledge" to Mac users (or otherwise can be easily found on the net):

• After booting the installer from the CD image, drop into the shell.
• Use pdisk to create an Apple_HFS partition for the boot loader and two Apple_UNIX_SVR2 partitions, one for the root file system and another for swap.
• Run sysinst and install the system. When asked to repartition the disk, just say Use existing partition sizes.
• Once the system is installed, drop again into the shell before rebooting.
• Mount your hard disk into /mnt and chroot into it.
• Fetch a copy of pkgsrc.
• Install the sysutils/hfsutils package.
• Use hformat to create a new HFS file system in the Apple_HFS partition we created.
• Mount the installation CD.
• Copy, using hcopy, the ofwboot.xcf file from the CD to the boot partition.
• Reboot.
• Drop into the OpenFirmware setup (Command+Option+P+R).
• Set boot-device to hd:,\ofwboot.xcf.
• Set boot-file to netbsd.
• And here is the tricky thing to get the machine to auto-boot: Set boot-command to ." hello" cr " screen" output boot, not mac-boot.

I found the last command somewhere on the Internet (dunno where now), but, supposedly, a regular mac-boot should have worked. In fact, it works if you call this command from the prompt, but not during automatic boot. (It turns out to be a problem with the version of OpenFirmware I have.)

Just writing down the steps in case I need them later on. Installing Debian stable was much, much easier, but the installer for testing crashes every day with a different error, so I gave up.

(Oh, by the way, I did the same installation into an old PowerMac G3 and that was really painful. The machine refused to boot from any of the CDs I tried and the prebuilt kernels hang during initialization due to a bogus driver. In the end: netbooting and using custom kernels.)

Processing Makefile.am with M4

ATF's Makefile.am, which is a single Makefile for the whole tree, was already at the 1300 lines mark and growing. At this size, it is unmanageable, and a quick look at its contents reveals tons of repeated delicate code.

Why so much repeated code, you ask, if the whole point of Automake is to simplify Makefiles? Automake does in fact simplify Makefile code when you define targets known by Automake, such as binaries and/or libraries. However, as soon as you start doing fancy things with documentation, building tons of small programs or messing with shell scripts, things get out of control because you are left on your own to define their targets and their necessary build logic.

Up until now, I had just kept up with the boilerplate code... but now that I'm starting to add pretty complex rules to generate HTML and plain text documentation out of XML files, the complexity must go. And here comes my solution:

I've just committed an experiment to process Makefile.am with M4. I've been trying to look for prior art behind this idea and couldn't find any, so I'm not sure how well this will work. But, so far, this has cut down 350 lines of Makefile.am code.

How does this work? First of all, I've written a script to generate the Makefile.am from the Makefile.am.m4 and put it in admin/generate-makefile.sh. All this script does is call M4, but I want to keep this logic in a single place because it has to be used from two call sites as described below.

Then, I've added an autogen.sh script to the top-level directory that generates Makefile.am (using the previous script) and calls autoreconf -is. I'm against autogen.sh scripts that pretend to be smart instead of just calling autoreconf, but in this case I see no other way around it.

At last, I've modified Makefile.am to add an extra rule to generate itself based on the M4 version. This, of course, also uses generate-makefile.sh.

We'll see how this scales, but I'm so far happy with the results.

Extending sudo credentials

If you use sudo for, e.g. pkgsrc's just-in-time su, you may have often bitten by the problem that some compilations are slow and the build process stops right in the middle to ask you for a root password. If you go away while the system compiles, you'll be frustrated when you come back, as the process may still well be at the very beginning.

This happens because, unless disabled by the system administrator, your sudo credentials last for 5 minutes. If you hadn't used sudo for those 5 minutes, it will ask you for your password again. A simple workaround for the problem is to automatically renew your credentials, say, every 2 minutes. You can do this by running the following command (from the same console you are using later on!) right before starting a pkgsrc build:

$ ( while :; do sudo -v; sleep 120; done ) &

Best config setting ever

echo 'set editing-mode vi' >>~/.inputrc

This will enable vi-editing mode for all commands that use the GNU readline library (e.g. bash, python, bc, etc.), not only the shell. For the shell only (including non-bash shells), add 'set -o vi' to your shrc file.

I don't know why I didn't do this before given that I'm a pretty hard vi user. Still, for some reason, I kept using emacs-like key bindings for command-line editing. Not any more! However, be careful: if you are used to vim's visual editing mode, you'll keep hitting 'v' in the command line and getting super annoyed.

Enjoy!

1000 revisions for ATF

Mmm! Revision 7ca234b9aceabcfe9a8a1340baa07d6fdc9e3d33, committed about an hour ago, marks the 1000th revision in the ATF repository. Thanks for staying with me if you are following the project :)

Books by Joel Spolsky

I just finished reading the third book in a row from Joel Spolsky, titled Joel on Software. Before this one, I read More Joel on Software and The Best Software Writing 1, all in a bit over a month. Note: I hadn't read any book cover-to-cover for a loooong while. Very interesting and entertaining books; highly recommended.

Oh, and his writing style is really enjoyable. We, crappy blog writers, can learn a lot from him!

Rearchitecting ATF

During the last few weeks, I've been doing some ATF coding and, well... I'm not happy. At all. I keep implementing features but I feel, more and more, that ATF is growing out of control and that it is way too sluggish. It oughtn't be so slow. About 6 minutes to run the whole test suite in a Mac G3 I just got? HA! I bet I can do much, much, much better than that. Come on, we are targeting NetBSD, so we should support all those niche platforms rather well, and speed matters.

The thing is, the current code base grew out of a prototype that didn't have that much of a design. Well, it had a design but, in my opinion, it has turned to be a bad design. I couldn't imagine that we could hit the bottlenecks (speed) and user-interface issues (for example, the huge difficulties that involve debugging a failing test case) that we are hitting. So...

IT IS TIME FOR A CHANGE!!!

I'm currently working on a written specification of what ATF will look like, hopefully, in the not-so-distant future. It will take a while to get there, but with enough effort, we soon will. And life will be better. And no, I'm not talking about a from-scratch rewrite; that'd only hurt the project. I plan to take incremental and safe steps, keeping the code base running all the time, but I will do a major face-lift of everything. (I wish I could say "we" instead of "I" here. But not there yet.)

Why am I writing a specification, you ask? Well, because that forces me (or ANY developer) to think how I want the thing to look like and to decide, exactly, on what the design will be, which technologies will be used, which languages will be involved and in what components, etc. And no, I'm not talking of a class model design; I'm just talking about the main design of the whole picture, which is quite hard by itself. Plus having a spec will allow me to show it to you before I start coding and you will say "oh, wonderful, this new design sucks so much that I'm not going to bother with the new version". Or maybe hell will freeze and you will think, "mmm, this looks interesting, maybe it will solve these issues I'm having as regards speed, ease of debugging and ease of use".

Anyway, I hope to have a draft "soon" and to hear any of the two possible comments as a result!

Edit (July 29th): Alright, I have uploaded an extremely preliminary copy of the specification just so that you can see where my current ideas are headed. Expect many more changes to this document, so don't pay too much attention to the tiny details (most of which aren't there anyway yet).