NetBSD in Google Summer of Code 2010

For the 6th year in a row, NetBSD is a mentoring organization for Google Summer of Code 2010!

If you are a bright student willing to develop full-time for an open source project during this coming summer, consider applying with us! You will have a chance to work with very smart people and, most likely, in the area that you are most passionate about. NetBSD, being an operating system project, has offers for project ideas at all levels: from the kernel to the packaging system, passing by drivers, networking tools, user-space utilities, the system installer, automation tools and more!

I would like to point you at the 3 project proposals I'm willing to directly mentor:

• Optimize and speed-up ATF: Make the testing framework blazing fast so that running the NetBSD automated tests does not take ages on slow platforms.
  
• Reorganize ATF to improve modularity: Refactor pieces of the testing framework so that it is easier to redistribute, has cleaner interfaces and is easier to depend on from third-party projects.
  
• Rewrite pkg_comp with portability as a major goal: Use Python to create a tool to automatically build binary packages from within a sandbox.

If you find any of the above projects interesting, or if you have any other project proposal that you think I could mentor, do not hesitate to contact me. Feel free to send me a draft of your application, together with a bit of information about you, so that we can discuss your proposal and make sure it gets selected!

Or, if none of the projects above interests you, please do check out the full list of NetBSD project proposals. I'm sure you will find something that suits your interests :-)

New version of the monotone-server package in pkgsrc

Wow, it has been a long time... 5 years ago, I created the monotone-server package in pkgsrc, a package that provided an interactive script to set up a monotone server from scratch with, what I though, minimal hassle.

My package did the job just fine, but past year I was blown away by the simplicity of the same package in Fedora: their init.d script provides a set of extra commands to initialize the server before starting it up, and that is it. No need to mess with a separate interactive script; no need to create and memorize passphrases that you will never use; and, what's more, all integrated in the only single place that makes sense: in the init.d "service management" script.

It has been a while since I became jealous of their approach, but I've finally got to it: I've spent the last few days rewriting the monotone-server package in pkgsrc and came up with a similar scheme. And this new package just made its way to pkgsrc-HEAD! The new package comes with what I think is a detailed manual page that explains how to configure the server from scratch. Take a look and, if you find any mistakes, inconsistencies or improvements to be done, let me know!

In the meantime, I will log into my home server, rebuild the updated package and put it in production :-)

Introducing the ATF nofork branch

Despite my time for free software being virtually zero these days, I have managed to implement a prototype of what ATF would look like if it didn't implement forking and isolation in test programs. This feature has been often requested by users to simplify their life when debugging test cases.

I shouldn't repeat everything I posted on the atf-devel mailing list regarding this announcement, so please refer to that email for details. But I must say that the results look promising: the overall code of ATF is much simpler and also faster. (An execution I just tried cuts the run time of the ATF test suite from 1m 41s to 1m 16s.) Expect more simplifications and speed-ups!

set -e and set -x

If you write shell scripts, you definitely need to know about two nice features that can be enabled through the set builtin:

• set -e: Enables checking of all commands. If a command exits with an error and the caller does not check such error, the script aborts immediately. Enabling this will make your scripts more robust. But don't wait until your script is "complete" to set the flag as an afterthought, because it will be a nightmare to fix the scrip to work with this feature enabled. Just write set -e as the very first line of your code; well... after the shell bang.
• set -x: If you are writing simple scripts that are meant to, well, script the execution of a few tasks (as opposed of being full-flown programs written in shell), set this flag to trace the execution of all commands. This will make the interpreter print each command right before it is executed, so it will aid you in knowing what is happening at any point in time.

Installing NetBSD/macppc on a Mac Mini G4

Yesterday, I spent a while installing NetBSD/macppc 5.0.1 on a Mac Mini G4. The process wasn't easy, as it involved the following steps. I'm omitting many details, as they are "common knowledge" to Mac users (or otherwise can be easily found on the net):

• After booting the installer from the CD image, drop into the shell.
• Use pdisk to create an Apple_HFS partition for the boot loader and two Apple_UNIX_SVR2 partitions, one for the root file system and another for swap.
• Run sysinst and install the system. When asked to repartition the disk, just say Use existing partition sizes.
• Once the system is installed, drop again into the shell before rebooting.
• Mount your hard disk into /mnt and chroot into it.
• Fetch a copy of pkgsrc.
• Install the sysutils/hfsutils package.
• Use hformat to create a new HFS file system in the Apple_HFS partition we created.
• Mount the installation CD.
• Copy, using hcopy, the ofwboot.xcf file from the CD to the boot partition.
• Reboot.
• Drop into the OpenFirmware setup (Command+Option+P+R).
• Set boot-device to hd:,\ofwboot.xcf.
• Set boot-file to netbsd.
• And here is the tricky thing to get the machine to auto-boot: Set boot-command to ." hello" cr " screen" output boot, not mac-boot.

I found the last command somewhere on the Internet (dunno where now), but, supposedly, a regular mac-boot should have worked. In fact, it works if you call this command from the prompt, but not during automatic boot. (It turns out to be a problem with the version of OpenFirmware I have.)

Just writing down the steps in case I need them later on. Installing Debian stable was much, much easier, but the installer for testing crashes every day with a different error, so I gave up.

(Oh, by the way, I did the same installation into an old PowerMac G3 and that was really painful. The machine refused to boot from any of the CDs I tried and the prebuilt kernels hang during initialization due to a bogus driver. In the end: netbooting and using custom kernels.)

Processing Makefile.am with M4

ATF's Makefile.am, which is a single Makefile for the whole tree, was already at the 1300 lines mark and growing. At this size, it is unmanageable, and a quick look at its contents reveals tons of repeated delicate code.

Why so much repeated code, you ask, if the whole point of Automake is to simplify Makefiles? Automake does in fact simplify Makefile code when you define targets known by Automake, such as binaries and/or libraries. However, as soon as you start doing fancy things with documentation, building tons of small programs or messing with shell scripts, things get out of control because you are left on your own to define their targets and their necessary build logic.

Up until now, I had just kept up with the boilerplate code... but now that I'm starting to add pretty complex rules to generate HTML and plain text documentation out of XML files, the complexity must go. And here comes my solution:

I've just committed an experiment to process Makefile.am with M4. I've been trying to look for prior art behind this idea and couldn't find any, so I'm not sure how well this will work. But, so far, this has cut down 350 lines of Makefile.am code.

How does this work? First of all, I've written a script to generate the Makefile.am from the Makefile.am.m4 and put it in admin/generate-makefile.sh. All this script does is call M4, but I want to keep this logic in a single place because it has to be used from two call sites as described below.

Then, I've added an autogen.sh script to the top-level directory that generates Makefile.am (using the previous script) and calls autoreconf -is. I'm against autogen.sh scripts that pretend to be smart instead of just calling autoreconf, but in this case I see no other way around it.

At last, I've modified Makefile.am to add an extra rule to generate itself based on the M4 version. This, of course, also uses generate-makefile.sh.

We'll see how this scales, but I'm so far happy with the results.

Extending sudo credentials

If you use sudo for, e.g. pkgsrc's just-in-time su, you may have often bitten by the problem that some compilations are slow and the build process stops right in the middle to ask you for a root password. If you go away while the system compiles, you'll be frustrated when you come back, as the process may still well be at the very beginning.

This happens because, unless disabled by the system administrator, your sudo credentials last for 5 minutes. If you hadn't used sudo for those 5 minutes, it will ask you for your password again. A simple workaround for the problem is to automatically renew your credentials, say, every 2 minutes. You can do this by running the following command (from the same console you are using later on!) right before starting a pkgsrc build:

$ ( while :; do sudo -v; sleep 120; done ) &