The Julipedia: Blog

Lutok 0.1 available

2012-01-29T16:12:00.003-05:00

A few months ago, I introduced the Lutok project, a simple C++ API for Lua. To recap: the major goal of this API, which does not mimic the Lua C API bit by bit, is to enforce correct coding practices on the client side. This is done by (ab)using the RAII programming pattern to automatically free resources when not needed and to ensure that the Lua stack is correctly managed. The library also adheres to common C++ programming idioms and exposes exceptions for error management and uses the pimpl idiom to completely hide the Lua C API from clients of Lutok (unless you use the c_gate backdoor!).

Today, I am pleased to announce that the first formal release of Lutok, obviously named 0.1, is available for download! You can obtain this release by visiting the lutok-0.1.tar.gz download page.

Also, in preparation for this release, I have spent the weekend writing some little example programs to demonstrate the usage of Lutok, and also some installation instructions.

Hope you find this release useful and please do send me any comments you may have!

Hint: Yes, releasing Lutok 0.1 was a prerequisite for Kyua 0.3. So stay tuned ;-)

Kyua: Weekly status report

2012-01-23T10:28:00.002-05:00

Released ATF 0.15 and imported it into NetBSD.
Added support for integer/float printf-like modifiers to the utils::format module. These will be required to beautify size and time quantities in the reports and error messages.

I spent way more time than I wanted on this. At first, I attempted to use std::snprintf to parse and process the format modifiers for integers and floats so that I could avoid implementing a custom parser for them. While this sounds like a cool idea (yay, code reuse!), it resulted in a ugly, nasty and horrible mess. In the end, I just ended up implementing custom parsing of the formatters, which was way easier and "good enough" for Kyua's needs.

Started work on backporting ATF's new require.memory property into Kyua. This needs having a way to parse and format byte quantities in user-friendly forms (e.g. 1k, 2m, etc.)... hence the previous work on utils::format!
Set up a Google+ Page for Kyua. I have no idea what to use it for yet. Maybe the status reports should go in there. Ideas?

Kyua: Weekly status report

2012-01-15T22:21:00.003-05:00

Finally some progress!

Backported the require.memory changes in NetBSD to the ATF upstream code, and extended them to support OS X as well.
Backported local pkgsrc patches to ATF into the upstream code.
Started to prepare ATF 0.15 by doing test runs of NetBSD/i386 and NetBSD/amd64 and by building the code in various Linux distributions. Several build bugs fixed along the way.
Spent a long while trying to figure out how the Fedora package maintainer procedure has changed since 3 years ago to create packages for ATF, Lutok and Kyua. Not very successful yet unfortunately.

Nothing on the Kyua front, but getting a new release of ATF out of the door has higher priority now!

Kyua: Weekly status report

2012-01-10T22:44:00.001-05:00

It's that time of the week again (although delayed by over a day). Unfortunately, no progress either during past week. Being on semi-vacations doesn't leave much "free time"... However, I traveled back home yesterday and getting back to my daily routine should give some spare time!

Kyua: Weekly status report

2012-01-03T05:25:00.003-05:00

Happy new year!

No activity past week as I was completely away from the computer.

Kyua: Weekly status report

2011-12-26T06:38:00.000-05:00

The post title should mention ATF instead of Kyua... but I'm keeping the usual one for consistency:

Integrated timestamps into the XML and HTML reports generated by atf-report. These should soon show up in the continuous tests of NetBSD.
Work on integrating the use of POSIX timers into atf-run after Christos Zoulas performed these changes in the NetBSD tree. The result is quite awful because I need to keep compatibility with systems that do not provide the "new" interface...

Kyua: Weekly status report

2011-12-19T10:35:00.000-05:00

Unfortunately, not much activity this week due to travel reasons. Anyway, some work went in:

Preliminary code to generate HTML reports from kyua report. This is easy peasy but boring. The current code was written as a proof of concept and is awful, hence why it was not committed. I'm now working in cleaning it up.
Backported test program and test case timestamping into ATF based on a patch from Paul Goyette. This is a very useful feature to have, and it will have to be added to Kyua later. (It has always been planned to be added, but have not had the time yet.)

Kyua: Weekly status report

2011-12-11T22:16:00.001-05:00

Some significant improvements this week:

Finally submitted the code to store and load full test case definitions. This is quite tricky (and currently very, very ugly) but it works and it will allow the reports to include all kinds of information from the test cases.
Removed the Atffiles from the tree; yay! For a long time, I had been using atf-run to run broken tests because atf-run allowed me to watch the output of the test case being debugged. However, this has been unnecessary since the introduction of the debug command in late August. I now feel confident that these files can go. (And debug is much more powerful than atf-run because you can target a single test case instead of a whole test program.)
Some crazy work attempting to hide the name of SQLite types from the sqlite::statement interface. I've been only able to do so somewhat decently for bind, but all my attempts at doing the same with column result in horrible code so far. So no, such changes have not been submitted.
As of a few minutes ago, kyua test now records the output of the test cases (stdout and stderr) into the database. These will be invaluable for debugging of test cases, particularly when the reports are posted online.
Some preliminary work at implementing HTML reports. This, however, has not received much progress due to the previous item requiring completion.

I'm quite excited at this point. HTML reports are a few weeks away at most. Once that happens, it will be time to start considering replacing the atf-run / atf-report duo for good, particularly within NetBSD. This will certainly not be easy... but all the work that has gone into Kyua so far has this sole goal!

Kyua: Weekly status report

2011-12-05T19:20:00.001-05:00

Some more work towards allowing storing and loading of full test case definitions (which in turn will allow us to provide detailed HTML reports). I have local changes that do this, but they are gated by the lack of some additional tests and probably some optimizations, because they slow down kyua report significantly. Regarding commits, I have only submitted some related cleanup-changes.
I got distracted by invalid Doxygen comments and traced down why they were not being correctly validated (which is the whole point of running Doxygen during the build). It turns out I had to enable some extra EXTRACT_* settings in the Doxyfile. After doing so, I realized there are many, many documentation problems in the code... and I have been fixing them since. It's a tough operation, but I'm more than half-way through already. To give you an idea, the current diffstat shows about 650 new comment lines (!).

Kyua: Weekly status report

2011-11-28T10:06:00.001-05:00

My goal for this past week was to change the database layer to be able to store full definitions of the test cases, and to later be able to load these while scanning an action. This is to allow the report command to provide all kinds of information about the executed tests, not just their names and their results. However, adding this functionality has proven to be more complex than I wished because the current types to represent test programs and test cases are kinda broken: that the abstractions chosen a while ago do not seem to be appropriate, and this is complicating further changes.

Due to this, I ended up doing some cleanups. First, I reimplemented the way in which test programs that fail to list their test cases are represented. And second, I got rid of the useless test_case_id class, which exposed even further problems in the data types that represent test cases.

It's now time to sit and think if the current representations of test programs and test cases make sense and, if not, how to better redo them. Not going to be easy, but I hope to have some time for this cleanup during this upcoming week.

Kyua: Weekly status report

2011-11-20T22:19:00.001-05:00

I only have one thing to report this week, but oh boy it's big: the report command finally reports the results of a run of a test suite! Yes, you heard well: Kyua is, finally, able to keep a record of all the previous executions of test suites and allows you to extract reports of any of them a posteriori.

At the moment, the report is just some disorganized plain-text. For example:

$ kyua test

[... wait for the tests to run ...]

Committed action 82

$ kyua report
===> Skipped tests
[... too long to show ...]
===> Expected failures
integration/cmd_report_test:output__console__change_file  ->
    expected_failure: --output not implemented yet:
    atf-check failed; see the output of the test for details
===> Failed tests
store/transaction_test:put_test_case__ok  ->
    failed: Line 663: stmt.step() not met
===> Summary
Action: 82
Test cases: 934 total, 15 skipped, 1 expected failures,

0 broken, 1 failed

I'm now working on changing the database schema to be able to really store all the data about test cases, because at the moment I'm only storing their names. Once all the original data is stored, the report command will have lots more information to work with, and then will be the time to start improving the format of the reports. As mentioned earlier, having interactive HTML dashboards is high in the priority list, and a very important goal of Kyua altogether.

Stay tuned :-)

Kyua: Weekly status report

2011-11-14T11:21:00.001-05:00

Kyua has finally gained a report subcommand, aimed at processing the output data of an action (stored in the database) and generating a user-friendly report in a variety of formats. This is still extremely incomplete, so don't get your hopes too high yet ;-) The current version of the report command takes an action and all it does is dump its runtime context (run directory, environment variables, etc.). Consider it just a proof of concept.

I have now started work on loading the data of test case results for a particular action, and once that is done, the report command will start yielding really useful data: i.e. it will actually tell you what happened during a particular execution of a test suite. The way I'm approaching the work these days is by building the skeleton code to implement the basic functionality first (which actually involves writing a lot of nasty code), with the goal of adding missing pieces later bit by bit.

For example, at this moment I'm only targeting text-based outputs with a limited set of data. However, when that is done, adding extra data or different formats will be relatively easy. Generating HTML dashboards (without going through XML, as was the case of atf-report!) is definitely highly prioritized.

By the way: I just realized it has already been one year since Kyua saw life. Wow, time flies. And only now we are approaching a point where killing the atf-run / atf-report pair is doable. I'm excited.

Kyua: Weekly status report

2011-11-07T10:04:00.001-05:00

Many things have happened this week, but they can all be summarized in one single sentence: kyua test now records the results of the execution of a test suite into the SQLite database.

"Why is this important?", you ask. Well, that's a good question. Recording test results opens the gate to many long-awaited features that should be coming soon, such as the ability to inspect the history of a particular test, to query all available data of a test result and/or to generate a dashboard of test results. It's interesting to realize that most of these features are just one SQL query away. If you install Kyua, you can already run a few tests and then use kyua db-exec to issue arbitrary SQL queries against the database; the schema (see store/schema.sql) might look a bit convoluted, but a bunch of NATURAL JOINs will yield the desired output.

The feature requests that have the highest priority at this point are the ability to generate a report of the last tests run both as a text file and as an HTML dashboard, because having these features means we can finally kill the atf-run and atf-report pair. At this point I'm, once again, "stuck" while figuring out how to best organize the code to make all these things possible while still keeping a nice separation across the existing layers (cli, engine and store)... all without introducing much unnecessary complexity. But exciting times lie ahead!

Kyua: Weekly status report

2011-10-31T10:50:00.004-04:00

Submitted a placeholder implementation of the new persistence layer (store top-level directory). This only supports opening a database and ensuring its metadata is valid. See r253.
Added a db-exec CLI command to execute arbitrary SQL commands onto the database. This is handy during development and testing, but may also help users to extract information out of the database in those cases where the CLI does not cover their needs just yet. See r255.
Miscellaneous fixes and improvements to utils::env and utils::sqlite.
Preliminary code to support putting objects (like actions and contexts) into the database. I've been thinking about this for a while and finally came up with a design that completely decouples the persistence needs from the higher-level classes in the engine layer. I haven't submitted the code yet though, as it lacks tests. (Still thinking how to write the loading of objects though.)

Kyua: Weekly status report

2011-10-23T22:12:00.001-04:00

Moved all the logic code of the "debug", "list" and "test" commands from the CLI layer to the engine layer.

Up until now, the CLI modules implementing these commands contained all the logic to load Kyuafiles and iterating over them to find matching tests and applying the desired operation to them. This kind of code belongs in "driver" modules (aka controllers) of the engine layer, because there is nothing UI-related in them.
After this refactoring, the code left in the CLI modules is purely presentation-related, and the code in the engine implements all the logic.
The goal of these changes is to be able to hide the interactions with the database in these controllers. The CLI layer has no business in dealing with the database connection (other than allowing the user to specify which database to talk to, of course).

Implemented a very simple RAII model for SQLite transactions.
Some additions to the utils::sqlite bindings to simplify some common calling patterns (e.g. binding statement parameters by name).
Preliminary prototypes at database initialization. This involves creating new databases and populating them with the initial schema, plus dealing with database metadata to, e.g. detect if we are dealing with the correct schema version.

The code for this is still too crappy to be submitted, so don't look for it in the repository just yet!
The design document details many things that should be part of the schema (e.g. "sessions"0, but I've decided that I'll start easy with a simplified schema and later build on top of it. Otherwise there will be too many clunky moving parts to deal with while the fundamental ideas are not yet completely clear.

Fixes to let the code build and run again in NetBSD (macppc at least).

I've now been stuck for a few days trying to figure out what the best way to implement the conversion of (new) in-memory objects to database objects is, and how to later recover these objects. E.g. what the correct abstractions are to take test case results and put them in the database, and how to retrieve these results to generate reports later on. I now start to have a clear mental picture on how this should look like, but I have yet to see how it will scale.

Kyua: Weekly status report

2011-10-16T22:55:00.002-04:00

Added support to prepare and execute statements to utils::sqlite.
Added the UTILS_PURE definition to tag functions as pure. I should now sweep through old code to apply the attribute where possible.
Created a pkgsrc package for Vera++. Investigating if I can use this tool for coding style validation, as the current code of Kyua is a bit messy in this regard.
Made a quick attempt at getting kyua test record test results in a simple database; success! This was just a proof of concept, so it is not submitted yet.
Started to refactor the code to move many logic constructions from the cli to the engine. With the need to store test results in the database, it's clear that these semantics do not belong in the CLI, but the current code structure do not easily allow this. Need to add some "controller" classes in the engine to hide all the interaction among internal components.

Kyua: Weekly status report

2011-10-09T19:37:00.002-04:00

Some more reading on SQLite. The Using SQLite book is close to awesome; very easy to read and full of insightful tips.
Cleaned up my preliminary SQLite wrapper code. Still very incomplete, but I've bitten the bullet and decided to commit the new library as is; otherwise I'll just keep procrastinating. So, ladies and gentlemen, welcome the new utils::sqlite module in Kyua. At the moment, this just provides the barebones to open and close a database. I need to start playing around with preparing statements before I can model these in my wrapper classes. (Yes, the API structure is extremely similar to that in Lutok.)

Update (23:40): I sent this too early, thinking I would not have anything nice to report this week either. But as it turns out, I finally had some spare time this late evening and got a chance to submit the extremely-incomplete SQLite C++ bindings I've been toying around with. See the update above! :-)

Kyua: Weekly status report

2011-10-03T15:37:00.003-04:00

Unfortunately, not much time this week either :-(
I am currently working on some adjustments to the design document of the database to describe new ideas; the previous design was incomplete in some areas and/or not accurate enough to support the described use cases. However, I've not had the to time to finish these edits and publish them.

Kyua: Weekly status report

2011-09-26T18:03:00.004-04:00

Unfortunately, no activity this week other than some brainstorming on the database design.

Why? My shipment container from Dublin arrived and I spent most of the weekend organizing stuff, setting up my little NetBSD box and attending a friend's wedding!

VirtualBox and port forwarding (and Fusion)

2011-09-19T17:57:00.000-04:00

I have been a VMware Fusion user for a long time and I think I am about to be a VirtualBox convert. Let me explain you my story.

With the release of the Windows 8 Developer Preview, I felt like giving the system a try. Not because I particularly like or care about Windows, but just because I am genuinely curious about operating systems and wanted to check out their new Metro interface. So I went ahead, downloaded the ISO image, created a new virtual machine under Fusion 3, started the virtual machine, and saw Fusion crash while the Windows installer was booting.

Oh well, an emulation problem. I tried to fiddle with the virtual machine settings a bit, selecting different CPU virtualization settings and different hardware devices. No luck. After that, I resorted to Google to look for a solution to the problem and found that other people were experiencing the same issue. But, at the same time, I also found that VMware had just released Fusion 4 and that this new release works with Windows 8 just fine. Ah, cool! Or not... $50 for the upgrade. For my use cases (which amount to booting a few BSD and Linux virtual machines in console mode), it is hard to justify the upgrade — specially when it would just be to fulfill my curiosity and then delete the virtual machine 10 minutes later.

But wait! There is that thing called VirtualBox. I had used this product a while ago and did not enjoy doing so because its interface was utterly (really) confusing. It was nowhere near the polish of Fusion or Parallels. However, to my surprise, most of my major itches have been fixed and the new interface of VirtualBox is bearable. And it boots the Windows 8 preview just fine!

Right. So after installing Windows 8, playing with it for 10 minutes, and later deleting it, I decided to look around and see what else VirtualBox has to offer. It has many settings. Yes, I know, Fusion has many settings too, but most are hidden from the UI and can only be accessed by modifying the vmx text file. In the case of VirtualBox they are easily accessible.

I kept looking around... and there it was, the "Port Forwarding" option staring at me under the Network settings.

One of the features I've been wishing for a long, long, long time is the ability to easily configure a NATed (not bridged) virtual machine in my laptop, assign a host name to it from the graphical interface and then be able to SSH into the virtual machine from the laptop. As it turns out, this is doable in Fusion but it requires some nasty tinkering: one has to edit a dhcpd configuration file by hand to add a new stanza for the virtual machine (/Library/Application Support/VMware Fusion/vmnet8/dhcpd.conf). This involves manually copy/pasting the MAC address of the virtual machine into that file (which you must first extract from the vmx file), assigning it an IP in the corresponding subnet, and sticking the IP plus host name in the /etc/hosts file. This gets boring pretty quickly.

VirtualBox may or may not support this kind of setting, but it has the aforementioned "Port Forwarding" option which is interesting on its own. As you can imagine, this sets up forwarding of a port from the host system to a port of the virtual machine. Creating an entry for SSH is as easy as clicking a couple of buttons, choosing a local port on the host system (e.g. 2022) and forwarding it to port 22 of the guest system. Voila. You can now access your virtual machine locally and remotely without having to go through any of the host name nor DHCP madness I experienced before:

$ ssh -p 2022 localhost

You can later create an alias in your ~/.ssh/config file to make accessing the virtual machine a breeze:

Host myvm
Hostname localhost
Port 2022

And later:

$ ssh myvm

It couldn't be easier. This trivial feature is a killer for the kind of environment I am interested in. And I also know that VirtualBox provides some other interesting features, like headless mode, that are not (easily) available with Fusion.

So yes, I think I am a convert.

Kyua: Weekly status report (db designdoc edition!)

2011-09-18T19:10:00.000-04:00

Moved the code of utils::lua to a new project, Lutok.
Attempted to integrate a copy of Lutok into the Kyua source code to simplify installing Kyua. I have been playing with Subversion externals and Autoconf/Automake hacks to make this happen, but unfortunately haven't got a pleasant solution yet.
Modified Lutok to not expose the Lua C API at all from header files and cleaned up the Kyua code to cope with the changes.
Been chewing through the first chapters of the "Using SQLite" book to refresh my SQL skills.
And, most importantly, wrote a preliminary design document for the database store of Kyua and the reporting features. Comments certainly welcome! Be aware that this is how atf-report will be replaced, so once this is done we should be able to finally kill atf-run and atf-report altogether :-)

Name your C++ auto-cleaners

2011-09-17T23:26:00.000-04:00

As you may already know, RAII is a very powerful and popular pattern in the C++ language. With RAII, you can wrap non-stack-managed resources into a stack-managed object such that, when the stack-managed object goes out of scope, it releases the corresponding non-stack-managed object. Smart pointers are just one example of this technique, but so are IO streams too.

Before getting into the point of the article, bear with me for a second while I explain what the stack_cleaner object of Lutok is. The "stack cleaner" takes a reference to a Lua state and records the height of the Lua stack on creation. When the object is destroyed (which happens when the declaring function exits), the stack is returned to its previous height thus ensuring it is clean. It is always a good idea for a function to prevent side-effects by leaving its outside world as it was — and, like it or not, the Lua state is part of the outside world because it is an input/output parameter to many functions.

Let's consider a piece of code without using the stack cleaner:

void
my_function(lutok::state& state, const int foo)
{
    state.push_integer(foo);

    ... do something else in the state ...
    const int bar = state.to_integer();
    if (bar != 3) {
        state.pop(1); 
        throw std::runtime_error("Invalid data!");
    }
    state.pop(1);

}

Note that we have had to call state.pop(1) from "all" exit points of the function to ensure that the stack is left unmodified upon return of my_function. Also note that "all exit points" may not be accurate: in a language that supports exceptions, any statement may potentially raise an exception so to be really safe we should do:

void
my_function(lutok::state& state, const int foo)
{
    state.push_integer(foo);
    try { 
        ... do something else in the state ...
        const int bar = state.to_integer();
        if (bar != 3 
           throw std::runtime_error("Invalid data!");
    } catch (...) {
        state.pop(1);
        throw;
    }
    state.pop(1);
}

... which gets old very quickly. Writing this kind of code is error-prone and boring.

With an "auto-cleaner" object such as the stack_cleaner, we can simplify our code like this:

void
my_function(lutok::state& state, const int foo)
{
    lutok::stack_cleaner cleaner(state);

    state.push_integer(foo);
    ... do something else in the state ...
    const int bar = state.to_integer();
    if (bar != 3) 
        throw std::runtime_error("Invalid data!"); 
}

And we leave the boring task of determining when to actually call state.pop(1) to the compiler and the runtime environment. In this particular case, no matter how the my_function terminates, we ensure that the Lua stack will be left as the same size as it was before.

But, as I said earlier, all this was just an introduction to the idea that made me write this post.

When you declare an auto-cleaner object of any kind, be sure to give it a name. It has happened to me a few times already that I have written the following construct:

lutok::stack_cleaner(state);

... which is syntactically correct, harmless and "looks good" if you don't look closely. The compiler will chew along just fine because, even though we are declaring an anonymous object, its constructor and destructor may be doing who-knows-what, so their code must be called and thus the "unused variable" warning cannot really be raised.

However this does not give us the desired behavior. The cleaner object will be constructed and destructed in the same statement without having a chance to wrap any of the following code, because its scope is just the statement in which it was defined. In other words, the cleaner will have absolutely no effect on the rest of the function and thus will be useless.

So, moral of the story: always give a name to your auto-cleaner objects so that their scope is correctly defined and their destructor is run when you actually expect:

lutok::stack_cleaner ANY_NAME_HERE(state);

Introducing Lutok: A lightweight C++ API for Lua

2011-09-15T15:18:00.000-04:00

It has finally happened. Lutok is the result of what was promised in the "Splitting utils::lua from Kyua" web post.

Quoting the project web page:

Lutok provides thin C++ wrappers around the Lua C API to ease the interaction between C++ and Lua. These wrappers make intensive use of RAII to prevent resource leakage, expose C++-friendly data types, report errors by means of exceptions and ensure that the Lua stack is always left untouched in the face of errors. The library also provides a small subset of miscellaneous utility functions built on top of the wrappers.

Lutok focuses on providing a clean and safe C++ interface; the drawback is that it is not suitable for performance-critical environments. In order to implement error-safe C++ wrappers on top of a Lua C binary library, Lutok adds several layers or abstraction and error checking that go against the original spirit of the Lua C API and thus degrade performance.

Lutok was originally developed within Kyua but was later split into its own project to make it available to general developers.

Coming up with a name for this project was quite an odyssey, and is what has delayed is release more than I wanted. My original candidate was "luawrap" which, although not very original, was to-the-point and easy to understand. Unfortunately, that name did not clear with the legal department and I had to propose several other names, some of which were not acceptable either. Eventually, I settled with "Lutok", which comes from "LUa TOolKit".

At this point, the source tree of Lutok provides pretty much the same code as the utils::lua module of Kyua. While it may be enough to get you started, I'm pretty sure you will lack some functions in the state class. If that is the case, don't hesitate to file a bug report to let me know what is missing.

In case you missed the link above, the project page is here: Lutok in Google Code.

Using va_copy to safely pass ap arguments around

2011-09-12T22:34:00.000-04:00

A long time ago, while I was preparing an ATF release, I faced many failing tests and crashes in one of the platforms under test. My memory told me this was a problem in OpenSolaris, but the repository logs say that the problem really happened in Fedora 8 x86_64.
The problem manifested itself as segmentation faults pretty much everywhere, and I could trace such crashes down to pieces of code like the following, of which the C code of ATF is full of:

void
foo_fmt(const char *fmt, ...)
{
    va_list ap;

    va_start(ap, fmt);
    foo_ap(fmt, ap);
    va_end(ap);
}

void
foo_ap(const char *fmt, va_list ap)
{
    char buf[128];

    vsnprintf(buf, sizeof(buf), fmt, ap);

    ... now, do something with buf ...
}

The codebase of ATF provides _fmt and _ap variants for many functions to give more flexibility to the caller and, as shown above, the _fmt variant just relies on the _ap variant to do the real work.
Now, the crashes that appeared from the code above seemed to come from the call that consumes the ap argument, which in this case is vsnprintf. Interestingly, though, all the tests in other platforms but Linux x86_64 worked just fine, and this included OpenSolaris, other Linux distributions, some BSDs and even different hardware platforms.
As it turned out, you cannot blindly pass ap arguments around because they are not "normal" parameters (even though, unfortunately, they look like so!). In most platforms, the ap element will be just an "absolute" pointer to the stack, so passing the variable to an inner function calls is fine because the caller's stack has not been destroyed yet and, therefore, the pointer is still valid. But... the ap argument can have other representations. It'd be an offset to the stack instead of a pointer, or it'd be a data structure that holds all the variable parameters. If, for example, the ap argument held an offset, passing it to an inner function call would make such offset point to "garbage" because the stack would have been grown due to the new call frame. (I haven't investigated what specific representation is x86_64 using.)
The solution is to use the va_copy function to generate a new ap object that is valid for the current stack frame. This is easy, so as an example, we have to rewrite the foo_ap function above as follows:

void
foo_ap(const char *fmt, va_list ap)
{
    char buf[128];
    va_list ap2;

    va_copy(ap2, ap);
    vsnprintf(buf, sizeof(buf), fmt, ap2);
    va_end(ap2);

    ... now, do something with buf ...
}

This duplication of the ap argument pointing to the variable list of arguments ensures that ap2 can be safely used from the new stack frame.

Kyua: Weekly status report

2011-09-12T10:12:00.003-04:00

My plan for this week was to release utils::lua as a separate module. Unfortunately, this has not been possible because I've been fighting with legal to clear the name for the project. I don't have an approved name yet, so this will have to wait a bit more :-(
On another order of things, I have started writing a design document for the database that will collect test case results and other information. Will share it as soon as it is readable and more or less complete.

Diversions in Autoconf (actually, in M4sugar)

2011-09-06T18:05:00.002-04:00

Have you ever wondered how Autoconf reorganizes certain parts of your script regardless of the order in which you invoke the macros in your configure.ac script? For example, how come you can define --with-* and --enable-* flags anywhere in your script and these are all magically moved to the option-processing section of the final shell script? After all, Autoconf is just a collection of M4 macros, and a macro preprocessor's only work is to expand macros in the input with predefined output texts. Isn't it?

Enter M4sugar's diversions. Diversions are a mechanism that allows M4 macros to output code to different text blocks, which are later concatenated in a specific order to form the final script.

Let's consider an example (based on a few M4 macros to detect the ATF bindings from your own configure scripts). Suppose you want to define a macro FROB_ARG to provide a --with-frob argument whose argument must be either "yes" or "no". Also suppose you want to have another macro FROB_CHECK to detect whether libfrob exists. Lastly, you want the user to be able to use these two independently: when FROB_CHECK is used without invoking FROB_ARG first, you want it to unconditionally look for the library; otherwise, if FROB_ARG has been used, you want to honor its value.

We could define these macros as follows:

AC_DEFUN([FROB_ARG], [
    AC_ARG_WITH([frob],
                [AS_HELP_STRING([--with-frob=yes|no], [enable frob])],
                [with_frob=${withval}, [with_frob=yes]) 
])

AC_DEFUN([FROB_CHECK], [
    m4_divert_text([DEFAULTS], [with_frob=yes])

    if test "${with_frob}" = yes; then
        ... code to search for libfrob ... 
    elif test "${with_frob}" = no; then
        :  # Nothing to do. 
    else
        AC_MSG_ERROR([--with-frob must be yes or not]) 
    fi
])

Note the m4_divert_text call above: this macro invocation tells M4sugar to store the given text (with_frob=yes) in the DEFAULTS diversion. When the script is later generated, this text will appear at the beginning of the script before the command-line options are processed, completely separated from the shell logic that consumes this value later on.

With this we ensure that the with_frob shell variable is always defined regardless of the call to the FROB_ARG macro. If this macro is called, with_frob will be defined during the processing of the options and will override the value of the variable defined in the DEFAULTS section. However, if the macro has not been called, the variable will keep its default value for the duration of the script.

Of course, this example is fictitious and could be simplified in other ways. But, as you can see in the referred change and in the Autoconf code itself, diversions are extensively used for trickier purposes. In fact, Autoconf uses diversions to topologically sort macro dependencies in your script and output them in a specific order to satisfy cross-dependencies.

Isn't that cool? I can't cease to be amazed, but I also don't dare to look at how this works internally for my own sanity...

Introducing "Dirt and the City"

2011-09-04T20:36:00.002-04:00

A friend of mine recently started a blog called Dirt and the City and, because he has invited me to contribute to it, I feel like posting a reference here ;-)

Dirt and the City is a blog that intends to show you the "ugly" (literally) side of NYC. The blog is named after a famous TV show filmed in this same city; I haven't watched the show myself, but I bet that there is no dirt to be seen in it. If you are wondering what the city really looks like, this blog is your place.

But, before visiting the blog, be sure to take it as it is meant to be: just a fun collection of photos! If you ever want to visit NYC, don't let the contents of the blog change your plans. You will definitely enjoy your visit regardless.

Without further ado, and just in case you missed the link above, click here to visit Dirt and the City.

Kyua: Weekly status report

2011-09-04T20:28:00.001-04:00

Slow week. We had some team-related events at work and I have friends over from Dublin, so it has been hard to find some time to do work on Kyua. Regardless, here comes the weekly report:

Split utils::lua into its own package, per some user's request. I'm still setting up the separate project and have to do lots of cleanup on the code, so nothing is available yet.
Started experimenting on the long promised "tests results database". So far, I have started writing a small utils::sqlite3 C++, RAII-based wrapper for SQLite 3 and a SQL schema for the database. My rusty SQL skills don't help :-P

However, all this work is local so there have been no commits to the repository this week; sorry!

Splitting utils::lua from Kyua

2011-09-03T09:43:00.001-04:00

If you remember a post from January titled C++ interface to Lua for Kyua (wow, time flies), the Kyua codebase includes a small library to wrap the native Lua C library into a more natural C++ interface. You can take a look at the current code as of r129.

Quoting the previous post:

The utils::lua library provides thin C++ wrappers around the Lua C API to ease the interaction between C++ and Lua. These wrappers make intensive use of RAII to prevent resource leakage, expose C++-friendly data types, report errors by means of exceptions and ensure that the Lua stack is always left untouched in the face of errors. The library also provides a place (the operations module) to add miscellaneous utility functions built on top of the wrappers.

While the RAII wrappers and other C++-specific constructions are a very nice thing to have, this library has to jump through a lot of hoops to interact with binary Lua versions built for C. This makes utils::lua not usable for performance-critical environments. Things would be way easier if utils::lua linked to a Lua binary built for C++, but unfortunately that is not viable in most, if not all, systems with binary packaging systems (read: most Linux distributions, BSD systems, etc.).

That said, I've had requests from a bunch of people to provide utils::lua separately from Kyua regardless of any performance shortcomings it may have, and this is what I have started doing this weekend. So far, I already have a pretty clumsy standalone package (I'll keep the name to myself for now ;-) that provides this library on its own with the traditional Automake, Autoconf and Libtool support. Once this is a bit better quality, and once I modify Kyua to link against this external library and assess that things work fine, I'll make the decision on how to publish this (but most likely it should be a separate project in Google Code).

Splitting the code doesn't come with its own issues though: maintaining a separate package will involve more work and hopefully/supposedly, dealing with quite a few feature requests to add missing functionality! Also, it means that utils::lua cannot use any of the other Kyua libraries (utils::sanity for example), so I lose a bit of consistency across the Kyua codebase. I am also not sure about how to share non-library code (in particular, the m4 macros for Autoconf) across the two packages.

So, my question is: are you interested in utils::lua being shipped separately? :-) Do you have any cool use cases for it that you can share here?

Thanks!

Kyua: Weekly status report

2011-08-28T18:48:00.002-04:00

Not a very active week: I've been on-call four days and they have been quite intense. Plus I have had to go through a "hurricane" in NYC. That said, I had some time to do a bit of work on Kyua and the results have been nice :-)

Made calls to getopt_long(3) work with GNU Getopt by using the correct value of optind to reset option processing.
Improved the configure script to error out in a clearer way when missing dependencies (pkg.m4 and Lua) are not found.
Did some portability fixes.
And released Kyua 0.2! (along with a pkgsrc package)

At this point, I have to start thinking how to implement test suite reporting within Kyua (i.e. how to replace atf-report). This probably means learning SQLite and refreshing my incredibly rusty SQL skills.

Also, it's time to (probably) split the utils::lua library in a separate package because there is several people interested in this.

Kyua 0.2 released

2011-08-24T17:09:00.000-04:00

Dear readers,

I am very proud to announce the second formal release of Kyua, 0.2. This release comes with lots of internal changes and a few user-visible changes. Instead of listing all the changes and news here, I'll just recommend you to visit the 0.2 release page and read all the notes.

This release has been tested under NetBSD-current, Mac OS X Snow Leopard, Debian sid and Ubuntu 10.04.1 LTS.

Please report any problems that you encounter.

Have fun!

Kyua: Weekly status report

2011-08-21T23:47:00.000-04:00

Implemented the "debug" command. Still very rudimentary, this command allows the user to run a test case without capturing its stdout nor stderr to aid in debugging of failed test cases. In the future, this command will also allow things like keeping the work directory for manual inspection, or spawning a shell or a debugger in the work directory after a test case is executed.
Many build fixes under different platforms in preparation for a 0.2 release. In particular, Kyua now builds under Ubuntu 10.04.1 LTS but some tests fail.
Had to disable the execution of the bootstrap test suite within Kyua because it stalls in systems where the default shell is not bash. I presume this is a bug in GNU Autotest, so I filed a report.

Kyua: Weekly status report

2011-08-15T09:13:00.001-04:00

Changed the --config and --variable options to be program-wide instead of command-specific. The configuration file should be able to hold properties that tune the behavior of Kyua, not just the execution of tests, so this makes sense.
Added the config subcommand, which provides a way to inspect the configuration as read by Kyua.
Got rid of the test_suites_var function from configuration files and replaced it by simple assignments to variables in the test_suites global table.
Enabled detection of unused parameters by the compiler and fixed all warnings.
Changed developer mode to only control whether warnings are enforced or not (not to enable the warnings themselves) and made developer mode be disabled on formal releases.

Barring release testing, Kyua 0.2 should be ready soon :-)

Kyua: Weekly status report

2011-08-08T12:30:00.000-04:00

Added the ability to explicitly define timeouts for plain test programs. This just completes the work from past week that made running plain test programs at all but had an ugly TODO in it to implement this missing feature.
The bootstrap test suite now runs as a single test case within the whole Kyua test suite. Demonstrates the plain test programs interface functionality :-)
Started reshuffling code to make the <tt>--config</tt> and related flags program-wide. The goal is to allow the configuration file to tune the behavior of all of Kyua, so these flags must be made generic. They were previously specific to the <tt>test</tt> subcommand only.

Kyua: Weekly status report

2011-08-01T10:51:00.000-04:00

Implemented the "plain" test interface. This allows plugging "foreign test programs" into a Kyua-based test suite. (A foreign test program is a program that does not use any testing framework: it reports success and failure by means of an exit code.)
Generalized code between the atf and plain interfaces and did some cleanups.
Attempted to fix the ATF_REQUIRE_EQ macros in ATF to evaluate their arguments only once. This has proven to be tricky and therefore it is not done yet. At the moment, the macros reevaluate the arguments on a failure condition, which is not too big of a deal.

Kyua: Weekly status report

2011-07-25T09:51:00.000-04:00

Finished splitting the atf-specific code from the generic structures in the engine. The engine now supports the addition of extra test interfaces with minimal effort.
Started implementing a "plain" test interface for test programs that do not use any test framework. This is to allow muxing non-atf tests into atf-based test cases, which is required in the NetBSD tree.

Kyua: Weekly status report

2011-07-18T12:49:00.000-04:00

Slow week. I've been busy moving to NYC!

Kept working on the splitting of ATF-specific code from the core test abstractions. The work is now focused on refactoring the results-reporting pieces of the code, which are non-trivial.

Kyua: Weekly status report

2011-07-10T17:40:00.000-04:00

One of the major features I want in place for Kyua 0.2 is the ability to run "foreign" test programs as part of a test suite: i.e. to be able to plug non-ATF test programs into a Kyuafile. The rationale for this is to lower the entry barrier of newcomers to Kyua and, also, to allow running some foreign test suites that exist in the NetBSD source tree but that are currently not run.

The work this week has gone in the direction outlined above among other things:

Created an abstract base class to represent test programs and provided an implementation for ATF.
Did the same thing for test cases.
Moved the kyua-cli package from pkgsrc-wip into pkgsrc head. Installing Kyua is now a breeze under NetBSD (and possibly under other platforms supported by pkgsrc!)

The next steps are to generalize the test case results, clearly separate the ATF-specific code from the general abstractions, and add an implementation to run simple test programs.

Kyua: Weekly status report

2011-07-05T03:20:00.002-04:00

Belated update:

Created a pkgsrc package for kyua-cli. Still in pkgsrc-wip though because pkgsrc is in a feature freeze.
Wrote a little tutorial on how to run NetBSD tests using Kyua.
Started work on 0.2 by doing a minor UI fix in the about command.

I've now started to look at how to split the engine into different "runners" to add support for test programs written without ATF. Not that I plan to use this feature... but having it in place will ensure that the internal interfaces are clean and may help in adoption of Kyua.

Kyua: Weekly status report

2011-06-26T15:51:00.000-04:00

This has been the big week:

Wrote some user documentation for the kyua binary.
Fixed some distcheck problems.
Released Kyua 0.1!

The next immediate thing to do is to write a short tutorial on how to run the NetBSD tests with Kyua and get some people to actually try it. After that, there are many things to improve and features to add :-)

Kyua 0.1 released!

2011-06-23T12:44:00.000-04:00

Dear readers,

I'm very proud to announce that the very first release of Kyua, obviously numbered 0.1, has been published! The release page for kyua-cli-0.1 contains more details about what this release offers, and don't hesitate to join the kyua-discuss mailing list and ask if you have questions.

Kyua was started sometime in past October and it has taken over six months to get the first public version. (It is true that my free time has heavily fluctuated during this period though, so it does not mean that six months of intensive coding have gone into the release ;-) Kyua 0.1 hovers at almost 40K lines of code according to Ohloh and comes with 800 test cases, yet it only implements a replacement for the atf-run component of ATF.

I have a package ready for pkgsrc, but unfortunately can't submit it because pkgsrc is in a freeze in preparation for the next stable branch. No matter what, expect detailed instructions on how to install Kyua and how to run the NetBSD-current test suite with it very soon.

Enjoy, and keep the bug reports coming!

Kyua: Weekly status report

2011-06-19T16:10:00.000-04:00

A couple of things have happened:

Released ATF 0.14. This release was followed by an import into NetBSD and fixing of subsequent fallout.
Some performance improvements to atf-sh. After killing a bunch of complex shell constructions and removing lots of obsolete functions, the performance results are significant. There is still room for improvement of course, and I still need to quantify how these optimizations behave in single-core machines.

I certainly expected more progress this past week... but in case you don't know: I am moving countries very soon now, and as the move date approaches, there is more and more stuff to be done at home so less and less time for hacking.

Validating format strings in custom C functions

2011-06-17T11:40:00.000-04:00

In C, particularly due to the lack of dynamic strings, it's common to pass format strings around together with a variable set of arguments. A prototype like this is very common:

void my_printf(const char*, ...);

For the standard printf and similar functions, some compilers will ensure that the variable list of arguments matches the positional parameters in the format string and, if they don't match, raise a warning. This is, however, just a warning "hardcoded" to match these functions, as the compiler can't know how the variable arguments of our custom my_printf function relate to the first argument.

Or can it?

I was made aware of a nice GCC attribute that allows developers to tag printf-like functions in a manner that allows the compiler to perform the same validation of variable arguments and format strings. This is in the form of a GCC __attribute__ that also happens to work with CLang. Let's see an example to illustrate how this works:

#include <stdarg.h>
#include <stdio.h>

static void my_printf(const char*, ...)
    __attribute__((format(printf, 1, 2)));

static void
my_printf(const char* format, ...)
{
    va_list ap;

    printf("Custom printf: ");
    va_start(ap, format);
    vprintf(format, ap);
    va_end(ap);
}

int
main(void)
{
    my_printf("this is valid %d\n", 3);
    my_printf("but this is not %f\n", 3);
}

If we compile the code above:

$ clang example.c
example.c:22:33: warning: conversion specifies type 'double' but
the argument has type 'int' [-Wformat]
    my_printf("but this is not %f\n", 3);
                               ~^     ~
1 warning generated.

Very useful. This function attribute has been applied to many functions in the NetBSD tree and many bugs have been spotted thanks to it.

Instead of me explaining how the format attribute works, I'll refer you to the official documentation. The attribute recognizes several format styles and takes different arguments depending on them, so it is a bit tricky to explain. Plus, if you look at the extensive list of attributes, you may find some useful stuff ;-)

Happy debugging!

Kyua: Weekly status report

2011-06-13T08:03:00.000-04:00

Added support for recursion from the top-level Kyuafile. This Kyuafile should not reference any directories explicitly because the directories at the top level are supposed to be created by the installation of packages. Closed issue 9.
Improved error messages when the test programs are bogus. Closed issue 13.
Backported format-printf attribute improvements from NetBSD head to ATF.
Miscellaneous build and run fixes for both Kyua and ATF in NetBSD and OS X.
Cut a release candidate for atf-0.14 and started testing on NetBSD.

The kyua-cli codebase is now feature complete. Blocking the 0.1 release are the need to polish the release documents and the requirement of releasing atf-0.14 beforehand. Should happen soon :-)

Kyua: Weekly status report

2011-06-05T14:58:00.000-04:00

Some long-standing bug fixes / improvements have gone in this week:

Improvements to the cleanup routine, which is used to destroy the work directory of a test case after the test case has terminated:

Heavy refactoring to be tolerant to failures. These failures may arise when a child of the test case does not exit immediately and holds temporary files in the work directory open for longer than expected.
Any file systems that the test case leaves mounted within the work directory will now be unmounted, just as the ATF test interface mandates. I realize that this adds a lot of complexity to the runtime engine for very little gain. If/when we revise the tests interface, it will be worth to reconsider this and maybe leave the cleanup of mounted file systems to the test case altogether.
As a result, issue 17 has been fixed!

Kyua now captures common termination signals (such as SIGINT) and exits in a controlled manner. What this means is that Kyua will now kill any active test programs and clean up any existing work directories before exiting. What this also means is that issue 4 is fixed.

To increase amusements, a little FYI: the above points have never worked correctly in ATF, and the codebase of ATF makes it extremely hard to implement them right. I have to confess that it has been tricky to implement the above in Kyua as well, but I feel much more confident in that the implementation works well. Of course, there may be some corner cases left... but, all in all, it's more robust and easier to manage.

The list of pending tasks for 0.1 shortens!

Kyua: Weekly status report

2011-05-29T15:00:00.005-04:00

Some cool stuff this week, albeit not large-scale:

Implemented the --variable flag in the test command. This flag, which can be specified multiple times, allows a user to override any configuration variable (be it a built-in or a test-suite variable) from the command line. This is actually the same as atf-run's -v flag, but with a clear separation between built-in configuration settings and test-suite specific settings.
Added support for several environment variables to allow users (and tests) to override built-in paths. I can't imagine right now any legitimate use for these variables, but hardcoded values are bad in general, atf-run provided these same variables, and these variables are very handy for testing purposes.
Added support for the new require.files test-case metadata property to both ATF and Kyua. This new property allows tests to specify a set of files that they require in order to run, and is useful for those tests that can't run before make install is executed.

The functionality planned for the 0.1 release is now pretty much complete. There is still a few rough edges to clean, some documentation to write, and some little features to implement/fix. See the open bugs for 0.1 to get an idea of the remaining tasks.

Kyua: Weekly status report

2011-05-22T18:30:00.000-04:00

This week:

Cleaned up the internal code of the "list" command and added a few unit tests.
Added integration tests for the "test" command that focus mostly on the behavior of the "test" command itself. There is still a need for full integration tests that validate the execution of the test cases themselves and their cleanup, and these will be tricky to write.
Changed atf-c, atf-c++ and atf-sh to show a warning when a test program is run by hand. Users should really be using atf-run to execute the tests, or otherwise things like isolation or timeouts will not work (and they'll conclude that atf is broken!).

Joining the Board of Directors of The NetBSD Foundation

2011-05-22T07:01:00.000-04:00

Dear readers,

It is a great pleasure for me to announce that I have just joined the Board of Directors of The NetBSD Foundation.

If you are curious about how this all happened, here it goes: as described in the election procedure, someone who I don't know nominated me back in November of 2010 to become part of the new board composition. After the Nomination Committee made their way through the long list of nominees, interviews and deliberation, they proposed a slate for the new members of the board. This slate included two people who were renewing their previous term (tron@ and reed@) and two new members (spz@ and jmmv@) to replace the two members stepping down (agc@ and david@). The final approval of the proposed slate happened in April 2011 and, yesterday, it became official.

Looking back, I can't believe it has been already ~10 years since I first started using NetBSD. Ah, those were the times of 1.5. My responsibilities within the project have shifted a lot during this time, ranging from the maintenance of GNOME and several web site tasks, the development of the testing framework (which is still ongoing today), and, from today, to my new duties within the board.

I'm looking forward to working on this new area of the project and I hope to meet the requirements of such position. The two members being replaced will be missed, and keeping up to the high bar they left behind will be tough. That said, I hope spz@ and I will be able to meet the expectations.

If you have any ideas or concerns regarding the direction of The NetBSD Project, don't hesitate to send them to the board@!

Kyua: Weekly status report, BSDCan 2011 edition

2011-05-16T16:07:00.000-04:00

I spent past week in Ottawa, Canada, attending the BSDCan 2011 conference. The conference was composed of lots of interesting content and hosted many influential and insightful BSD developers. While the NetBSD presence was very reduced, I could have some valuable talks with both NetBSD and FreeBSD developers.

Anyway. As part of BSDCan 2011, I gave a talk titled "Automated testing in NetBSD: past, present and future". The talk focused on explaining what led to the development of ATF in the context of NetBSD, what related technologies exist in NetBSD (rump, anita and dashboards), what ATF's shortcomings are and how Kyua plans to resolve them. (Video coming soon, I hope.)

The talk was later followed by several questions and off-session conversations about testing in general in BSDs. Among these, I gathered a few random ideas / feelings:

The POSIX 1003.3 standard defines the particular results a test can emit (see the corresponding DejaGnu documentation). Both ATF and Kyua already implement all the results defined in the standard, but they use different names and extend the standard with many extra results. Given that the standard does not define useful concepts like "expected failures", an idea that came up is to provide a flag to force POSIX compliance at the cost of being less useful. Why? Just for the sake of saying that Kyua conforms to this standard.
The audience seemed to like the idea of a "tests results store" quite a bit, and the sound of SQLite for the implementation was not bullied. This is something I'm eager to work on, but not before I publish a 0.1 release.
I highlighted the possibility of allowing Kyua to run "foreign" test programs so that we could integrate the results into the database. This could be useful to run tests for which we (*BSD) have no control (e.g. gcc) in an integrated manner. The idea was not bullied by anyone either.
FreeBSD has already been looking at ATF / Kyua and they are open to collaboration.
OpenBSD won't import any new C++ code, and adding C-based tests to the tree while relegating the C++ runtime to the ports is not an option. Somehow I expected this.
Junos (the FreeBSD-based operating system from Jupiter Networks) recently imported ATF and they are happy with it so far. Yay!
Would be nice to have a feature to run tests remotely after, maybe, deploying a single particular test and its dependencies. This is gonna be tricky and not in my current immediate plans.

Other than that, I had little time to do some coding:

Fixed a problem in which both ATF and Kyua were not correctly resetting the timezone of the executed tests. I only found this because, after arriving in Canada, some Kyua tests would start to fail. (Yes, the fix is in both code bases!)
Added some support to capture deadly signals that terminate Kyua so that Kyua can print an informational message stating that something went wrong and which log file contains more information. See r121.

That's it folks! Thanks to those attending the conference and, in particular, to those that came to my talk :-)

Kyua: Weekly status report

2011-05-08T17:22:00.000-04:00

Unfortunately, no progress whatsoever this week :-( Too busy at work and preparing my upcoming trips. Time to fly to BSDCan 2011 tomorrow.

Kyua: Weekly status report

2011-05-01T11:59:00.000-04:00

Unfortunately, I have had no time for coding this week. The only things I could do were:

Fixed a few build problems on NetBSD introduced during past week's changes.
Built Kyua and ran a few tests on NetBSD/macppc (just for the joy of it).

Coding has been eclipsed by the preparation of my presentation for BSDCan 2011; at this point, this has priority over any code changes. I'd argue that preparing the presentation is also part of the project, so some time has been invested ;-)

More next week, hopefully, but I don't expect being able to do any big code improvements until after BSDCan.

Use explicit conditionals

2011-04-30T05:29:00.000-04:00

In C — or, for that matter, several other languages such as Python or C++ — most native types can be coerced to a boolean type: expressions that deliver integers, pointers or characters are automatically treated as boolean values whenever needed. For example: non-zero integer expression and non-NULL pointers evaluate to true whereas zero or NULL evaluate to false.

Many programmers take advantage of this fact by stating their conditionals like this:

void func(const int in) {
    if (in) {
        ... do something when in != 0 ...
    } else {
        ... do something else when in == 0 ...
    }
}

... or even do something like:

bool func(const struct mystruct *ptr) {
    int out = calculate_out(in);

    ... do something more with out ...

    return out;  // The return type is bool though; is this ok?
}

... but such idioms are sloppy and can introduce confusion or bugs. Yes, things work in many cases, but "work" is not a synonym for "good" ;-)

Taking advantage of coercions (automatic type conversions) typically makes the code harder to follow. In statically-typed languages, whenever you define a variable to be of a particular type, you should adhere to said type by all means so that no confusion is possible. Conceptually, an integer is not a boolean and therefore it should not be treated as such. But this is just a matter of style.

On the other hand, the automatic conversions can lead to hidden bugs; for example, in the second function above, did we really want to convert the "out" value to a boolean or was that a mistake? This is not so much a matter of style but a matter of careful programming, and having as much state as possible "in your face" can help prevent these kind of trivial errors.

In contrast, one would argue that having to provide explicit checks or casts on every expression is a waste of time. But keep in mind that code is supposed to be written once and read many times. Anything you can do to communicate your intent to the reader will surely be appreciated.

Disclaimer: the above is my personal opinion only. Not following the style above should have no implications on the resulting binary code. (Wow, this post had been sitting as a draft around here for a too long time.)

Kyua: Weekly status report

2011-04-24T14:15:00.000-04:00

Ouch; I'm exhausted. I just finished a multi-hour hacking session to get the implementation of the list subcommand in control. It is now in a very nice user-facing shape, although its code deserves a little bit of house cleaning (coming soon).

Anyway, this week's progress:

Added the kyuaify.sh script. This little tool takes a test suite (say, NetBSD's /usr/tests directory) and converts all its Atffiles into Kyuafiles. The tool is no sophisticated at all; in fact, it is a pretty simple script that I haven't tested with any other test suites so far. See the announcement of kyuaify for some extra details.
Added logging support for the Lua code and changed the Lua modules to spit some logging information while processing Kyuafiles and configuration files.
Added a mechanism in the user interface module to consistently print informational, warning and error messages.
Implemented proper test filtering (after several iterations). What does proper mean? Well, for starters, test filters are always relative to the test suite's root (although we already saw this in last week's report). But the most important thing is that the filters are now validated: nice, user-friendly errors will be reported when the collection of tests is non-disjoint, when it includes duplicate names or when any of the provided filters does not match any test case. I really need to document the rationale of these in the manual, but for now the r118 commit message includes a few details.
Drafted some notes for the BSDCan 2011 conference. I am quite tempted to reuse parts of the presentation from NYCBSDCon 2010, but I really want to give more emphasis on Kyua this time. In case you don't know, Kyua was first announced in NYCBSDCon 2010 and it was still a very immature project. The project has changed a lot since then.

The wishful plan for next week is to clean up the internals of the list command (by refactoring and adding unit tests) and implement preliminary integration tests for the test subcommand. The latter scares me quite a bit. But... hmm... I guess preparing the presentation for BSDCan 2011 has priority.

Kyua: Weekly status report

2011-04-17T15:29:00.000-04:00

This week started easy:

Added integration tests for the about and help subcommands. These were pretty easy to do.
Added integration tests for the list subcommand. I initially added these tests as expected failures to reason about the appearance and behavior of this command from the point of view of the user before actually working on the code... and I am still writing such code to make these tests pass!

This is where things got a bit awry. Polishing the behavior of the list command so that its interface is consistent among all flag and argument combinations is tricky and non-trivial. I ended up having to change code deep down in the source tree to implement missing features and to change existing bits and pieces:

Implemented support to print all test case properties, not only the user-specific ones. The properties recognized by the runtime engine are stored as individual arguments of a structure, so these required some externalization code.
Implemented "test case filtering". This allows users to select what tests to run on the command line at the test case granularity. For example, foo/bar selects all tests in a subdirectory if bar is a directory, or all the tests in the bar test program if it is a binary. But what is new (read: not found in ATF) is that you can even do foo/bar:test-1 where test-1 is the name of a test case within the foo/bar test program. I've been silently wishing for this feature to be available in ATF because it shortens the build/test/edit cycle, but it was not easy to add.
Changed the internal representation of test suites to ensure all test program names are relative to the root of the test suite. The root of the test suite is considered to be the directory in which the top-level Kyuafile lives (/usr/tests/ in NetBSD). The whole point of doing this is to provide some consistency to the filters when the user decides to execute tests that are not in the current directory. For example, the following are now equivalent:
$ cd /usr/tests && kyua list fs/tmpfs
$ kyua list -k /usr/tests/Kyuafile fs/tmpfs

The plans for the upcoming week are to finish with the clean up of the list command (which involves adding proper error reporting, refactoring of the command module and addition of unit tests) and start cleaning up the test command.

Also, remember that BSDCan 2011 is now around the corner and that I will be talking about ATF, Kyua and NetBSD in it! My plan was to have a kyua-cli-0.1 release by the time of the conference, although this will be tricky to achieve...

Kyua: Weekly status report

2011-04-10T15:00:00.000-04:00

Few things worth mentioning this week as reviewing Summer of Code student applications has taken priority. The good thing is that there are several strong applications for NetBSD; the bad thing is that none relate directly to testing. Anyway, the work this week:

Added a pkg-config file for atf-sh as well as an Autoconf macro to detect its presence. This is needed by Kyua to easily find atf-sh. (Yes, I know: this is an abuse of pkg-config, but it works pretty well and is consistent with atf-c and atf-c++.)
Implemented basic integration tests for Kyua in r98 using atf-sh. These tests are still very simple but provide a placeholder into which new tests will be plugged. Having good integration test coverage is key in preparation for a 0.1 release. Oh, and by the way, this revision has bumped the number of tests to 601, crossing the 600 barrier :-)

That's pretty much it. Now, back to attempting to fix my home server as a fresh installation of NetBSD/macppc has decided to not boot any more. (Yes, this has blocked most of my weekend...)

Kyua: Weekly status report

2011-04-03T15:00:00.000-04:00

This week's work has been quite active on the ATF front but not so much in the Kyua one. I keep being incredibly busy on the weekends (read: traveling!) so it's hard to get any serious development work done.

What has happened?

Finally tracked down and fixed some random atf-run crashes that had been hunting the NetBSD test suite for months (see PR bin/44176). The fix is in reality an ugly workaround for the fact that a work directory cannot be considered "stable" even after the test case terminates. There may be dying processes around that touch the work directory contents, and the cleanup code in atf-run was not coping well with those. As it turns out, this problem also exists in Kyua (even though it's not as pronounced because arbitrary failures when running a test case do not crash the runtime engine) so I filed issue 17 to address it.
Released ATF 0.13 and imported it both to NetBSD-current and pkgsrc. As a side note, Kyua requires the new features in this release, so putting it out there is a requirement to release Kyua 0.1. This new release does not have a big effect on NetBSD though, because the copy of ATF in NetBSD has been constantly receiving cherry-picks of the upstream fixes.
Replaced several TODO items in the Kyua code with proper calls to the logging subsystem. These TODO items were referring to conditions in the code that should not happen, but for which we cannot do any proper recovery (like errors in a destructor). Sure, these could be better signaled as an assertion... but these code paths can be triggered in extremely-tricky conditions and having Kyua crash because of them is not nice (particularly when the side-effects of executing that code paths are non-critical).

So, in retrospect, I have fulfilled the goal set past week of releasing ATF 0.13, but I haven't got to the addition of integration tests. Oh well... let's see if this upcoming week provides more spare time.

Kyua: Weekly status report

2011-03-27T15:00:00.005-04:00

This has been a slow week. In the previous report, I set the goal of getting Kyua to run the NetBSD test suite accurately (i.e. to report the same results as atf-run), and this has been accomplished. Actually, the changes required in Kyua to make this happen were minimal, but I got side-tracked fixing issues in NetBSD itself (both in the test suite and in the kernel!). So, the things done:

Fixed Kyua to correctly kill any dangling subprocesses of a test case and thus match the behavior of atf-run. This was the only change required to make issue 16 happen: i.e. to get Kyua to report the same results as atf-run for the NetBSD test suite.

Based on a suggestion from Antti Kantee, an alternative way to handle this would be to not kill any processes and just report the test case as broken if it fails to clean itself up. The rationale being that the runtime engine can kill dangling subprocesses in 99% of the occasions, but not always. The exception are those subprocesses that change their process group. It'd be better to make all cleanups explicit instead of hiding this corner case, as it can lead to confusion. Addressing this will have to wait though, as it is a pretty invasive change.
Before closing issue 16, I want to implement some integration tests for Kyua to ensure that the whole system behaves as we expect (which is what the NetBSD test suite is currently doing implicitly).

Kyua is pickier than atf-run: if a cleanup routine of a test case fails or crashes, Kyua will (correctly) report the test case as broken while atf-run will silently ignore this situation. Some NetBSD tests had crashing cleanup parts, so I fixed them.
Some test programs in NetBSD were leaving unkilled subprocesses behind. These subprocesses are daemons and thus fall out of the scope of what Kyua can detect and kill during the cleanup phase. I mistakenly tracked down the problem to rump, but Antti Kantee kindly found the real problem in the kernel (not in rump!).
As a side effect of processes being left behind, I extended the functionality of pidfile(3) and implemented pid file support in bozohttpd. This is to make the tests that spawn a bozohttpd in the background more robust, by giving them a way to forcibly kill the server during cleanup.

These changes are still under review and not committed yet.

For the upcoming week, I plan to add some basic integration tests to Kyua and release ATF 0.13. I've been running a NetBSD system with the latest ATF code integrated for a while (because Kyua requires it) and things have been working well.

Kyua: Weekly status report

2011-03-20T11:58:00.007-04:00

These days, I find myself talking about Kyua to "many" people. In particular, whenever a new feature request for ATF comes in, I promise the requester that the feature will be addressed as part of Kyua. However, I can imagine that this behavior leaves the requester with mixed feelings: it is nice that the feature will be implemented but, at the same time, it is very hard to know when because the web site of Kyua does not provide many details about its current status.

In an attempt to give Kyua some more visibility, I will start posting weekly activity reports in this blog. These reports will also include any work done on the ATF front, as the two projects are highly related at this point. I write these reports regularly at work and I feel like it is a pretty good habit: every week, you have to spend some time thinking about what you did for the project and you feel guilty if the list of tasks is ~zero ;-) It also, as I said, gives more visibility to the work being done so that outsiders know that the project is not being ignored.

Before starting with what has happened this week, a bit of context. I have been traveling like crazy and hosting guests over for the last 2 months. This has given me virtually no time to work on Kyua but, finally, I have got a chance to do some work this past week.

So, what are the news?

Implemented the --loglevel command line flag, which closes issue 14. Kyua now generates run-time logs of its internal activity to aid in postmortem debugging and this flag allows the user to control the verbosity of such logs.
Antti Kantee hacked support for atf-run in the NetBSD source tree to dump a stack trace of any crashing test program. I have backported this code to the upstream ATF code and filed issue 15 to implement this same functionality in Kyua.
Fixed a hang in atf-run that made it get stuck when a test case spawned a child processes and atf-run failed to terminate it. A quick test seems to indicate that Kyua is affected by a similar problem: it does not get stuck but it does not correctly kill the subprocesses. The problem will be addressed as part of issue 16.
Oh, and by the way: Kyua will be presented at BSDCan 2011.

My plans for this week are to make Kyua run the full NetBSD test suite without regressions when compared to ATF. Basically, the results of a test run with Kyua should be exactly the same as those of a test run with ATF. No dangling processes should be left behind.

Lastly, if you are interested in these reports and other Kyua news, you can subscribe to the kyua label feed and, if you want to stay up to date with any changes performed to the code, subscribe to the kyua-log mailing list.

A teeny tiny review of Twitterville

2011-01-27T16:12:00.003-05:00

After about two months, I finally finished reading Twitterville by Shel Israel (@shelisrael). One of my followers (@drio) asked for a review of the book, so here is my attempt to do so.

But first, a quick summary: Twitterville is a book that focuses on the dynamics of Twitter. It starts by explaining how Twitter works, but that is only a tiny introductory part of the book. The majority of the contents explain how people and business interact with each other by means of Twitter, and it does so by providing lots of real-life stories. The stories range from topics as diverse as businesses offering deals, to individuals raising funds for specific causes.

The book is easy to read and is well structured, and as you read through it you will realize that the author had to do some major research efforts to collect all the stories that he presents. I personally enjoyed the first half of the book a lot, but at some point I ran out of time for reading and my interest dropped. It was hard to pick on reading again because the book becomes quite repetitive after a few chapters; just keep in mind that it is a collection of personal experiences organized by different major topics and you won't be disappointed.

Twitterville has changed my view on Twitter. I have discovered many "use cases" for Twitter that I could not imagine and, as many people do, I used to disregard Twitter as a useless "status updates" system. Today, however, I have set up several Twitter searches to monitor some topics of my interest and I engage with people that I did not know beforehand. It kinda feels like a world-wide unorganized chat room to me... but, as the author mentions many times, the way you see and use Twitter is up to you and you alone!

Injecting C++ functions into Lua

2011-01-17T11:09:00.002-05:00

The C++ interface to Lua implemented in Kyua exposes a lua::state class that wraps the lower-level lua_State* type. This class completely hides the internal C type of Lua to ensure that all calls that affect the state go through the lua::state class.

Things get a bit messy when we want to inject native functions into the Lua environment. These functions follow the prototype represented by the lua_CFunction type:

typedef int (*lua_CFunction)(lua_State*);

Now, let's consider this code:

int
awesome_native_function(lua_State* state)
{
    // Uh, we have access to s, so we bypass the lua::state!
    ... do something nasty ...

    // Oh, and we can throw an exception here...
    //with bad consequences.
}

void
setup(...)
{
    lua::state state;
    state.push_c_function(awesome_native_function);
    state.set_global("myfunc");
    ... run some script ...
}

The fact that we must pass a lua_CFunction prototype to the lua_pushcfunction object means that such function must have access to the raw lua_State* pointer... which we want to avoid.

What we really want is the caller code to define a function such as:

typedef int (*cxx_function)(lua::state&)

In an ideal world, the lua::state class would implement a push_cxx_function that took a cxx_function, generated a thin C wrapper and injected such generated wrapper into Lua. Unfortunately, we are not in an ideal world: C++ does not have high-order functions and thus the "generate a wrapper function" part of the previous proposal does not really work.

What we can do instead, though, is to make the creation of C wrappers for these C++ functions trivial. And this is what r42 did. The approach I took is similar to this overly-simplified (and broken) example:

template< cxx_function Function >
int
wrap_cxx_function(lua_State* state)
{
    try {
        lua::state state_wrapper(state);
        return Function(state_wrapper);
    } catch (...) {
        luaL_error(state, "Geez, don't go into C's land!");
    }
}

This template wrapper takes a cxx_function object and generates a corresponding C function at compile time. This wrapper function ensures that C++ state does not propagate into the C world, as that often has catastrophical consequences. (Due to language limitations, the input function must have external linkage. So no, it cannot be static.)

As a result, we can rewrite our original snippet as:

int
awesome_native_function(lua::state& state)
{
    // See, we cannot access lua_State* now.

    ... do something ...
    throw std::runtime_error("And we can even do this!");
}

void
setup(...)
{
    lua::state state;
    state.push_c_function(
        wrap_cxx_function< awesome_native_function >);
    state.set_global("myfunc");
    ... run some script ...
}

Neat? I think so, but maybe not so much. I'm pretty sure there are cooler ways of achieving the above purpose in a cleaner way, but this one works nicely and has few overhead.

Error handling in Lua: the Kyua approach

2011-01-14T03:04:00.004-05:00

About a week ago, I detailed the different approaches I encountered to deal with errors raised by the Lua C API. Later, I announced the new C++ interface for Lua implemented within Kyua. And today, I would like to talk about the specific mechanism I implemented in this library to deal with the Lua errors.

The first thing to keep in mind is that the whole purpose of Lua in the context of Kyua is to parse configuration files. This is an infrequent operation, so high performance does not matter: it is more valuable to me to be able to write robust algorithms fast than to have them run at optimal speed. The other key point to consider is that I want Kyua to be able to use prebuilt Lua libraries, which are built as C binaries.

The approach I took is to wrap every single unsafe Lua C API call in a "thin" (FSVO thin depending on the case) wrapper that gets called by lua_pcall. Anything that runs inside the wrapper is safe to Lua errors, as they are caught and safely reported to the caller.

Lets examine how this works by taking a look at an example: the wrapping of lua_getglobal. We have the following code (copy pasted from the utils/lua/wrap.cpp file but hand-edited for publishing here):

static int
protected_getglobal(lua_State* state)
{
    lua_getglobal(state, lua_tostring(state, -1));
    return 1;
}

void
lua::state::get_global(const std::string& name)
{
    lua_pushcfunction(_pimpl->lua_state, protected_getglobal);
    lua_pushstring(_pimpl->lua_state, name.c_str());
    if (lua_pcall(_pimpl->lua_state, 1, 1, 0) != 0)
        throw lua::api_error::from_stack(_pimpl->lua_state,
            "lua_getglobal");
}

The state::get_global method is my public wrapper for the lua_getglobal Lua C API call. This wrapper first prepares the Lua stack by pushing the address of the C function to call and its parameters and then issues a lua_pcall call that executes the C function in a Lua protected environment.

In this case, the argument preparation for protected_getglobal is trivial because the lua_getglobal call does not require access to any preexisting values on the Lua stack. Things get much trickier when that happens as in the case of the lua_getglobal wrapper. I'll leave understanding how to do this as an exercise to the reader (but you can cheat by looking at line 154).

Anyway. The above looks all very nice and safe and the tests for the state::get_global function, even the ones that intentionally cause a failure, all work fine. So we are good, right? Nope! Unfortunately, the code above is not fully safe to Lua errors.

In order to prepare the lua_pcall execution, the code must push values on the stack. As it turns out, both lua_pushcfunction and lua_pushstring can fail if they run out of memory (OOM). Such failure would of course be captured inside a protected environment... but we have a little chicken'n'egg problem here. That said, OOM failures are rare so I'm going to leverage this fact and not worry about it. (Note to self: install a lua_atpanic handler to complain loudly if that ever happens.)

Addendum: Bundling Lua within my program and building it as a C++ binary with exception reporting enabled in luaconf.h would magically solve all my issues. I know. But I don't fancy the idea of bundling the library into my source tree for a variety of reasons.

C++ interface to Lua for Kyua

2011-01-08T08:17:00.005-05:00

Finally! After two weeks of holidays work, I have finally been able to submit Kyua's r39: a generic library that implements a C++ interface to Lua. The code is hosted in the utils/lua/ subdirectory.

From the revision description:

The utils::lua library provides thin C++ wrappers around the Lua C API to ease the interaction between C++ and Lua. These wrappers make intensive use of RAII to prevent resource leakage, expose C++-friendly data types, report errors by means of exceptions and ensure that the Lua stack is always left untouched in the face of errors. The library also provides a place (the operations module) to add miscellaneous utility functions built on top of the wrappers.

In other words: this code aims to decouple all details of the interaction with the Lua C API from the main code of Kyua so that the high level algorithms do not have to worry about Lua C API idiosyncrasies.

Further changes to Kyua to implement the new configuration system will follow soon as all the basic code to talk to Lua has been ironed out. Also expect some extra posts regarding the design decisions that went on this helper code and, in particular, about error reporting as mentioned in the previous post.

(Yep, Lua and Kyua sound similar. But that was never intended; promise!)

Error handling in Lua

2011-01-07T08:01:00.004-05:00

Some of the methods of the Lua C API can raise errors. To get an initial idea on what these are, take a look at the Functions and Types section and pay attention to the third field of a function description (the one denoted by 'x' in the introduction).

Dealing with the errors raised by these functions is tricky, not to say a nightmare. Also, the ridiculously-short documentation on this topic does not help. This post is dedicated to explain how these errors may be handled along with the advantages and disadvantages of each case.

The Lua C API provides two modes of execution: protected and unprotected. When in protected mode, all errors caused by Lua are caught and reported to the caller in a controlled manner. When in unprotected mode, the errors just abort the execution of the calling process by default. So, one would think: just run the code in protected mode, right? Yeah, well... entering protected mode is nontrivial and it has its own particularities that make interaction with C++ problematic.

Let's analyze error reporting by considering a simple example: the lua_gettable function. The following Lua code would error out when executed:

my_array = nil
return my_array["test"]

... which is obvious because indexing a non-table object is a mistake. Now let's consider how this code would look like in C (modulo the my_array assignment):

lua_getglobal(state, "my_array");
lua_pushstring(state, "test");
lua_gettable(state, -2);

Simple, huh? Sure, but as it turns out, any of the API calls (not just lua_gettable) in this code can raise errors (I'll call them unsafe functions). What this means is that, unless you run the code with a lua_pcall wrapper, your program will simply exit in the face of a Lua error. Uh, your scripting language can "crash" your host program out of your control? Not nice.

What would be nice is if each of the Lua C API unsafe functions reported an error (as a return value or whatever) and allowed the caller to decide what to do. Ideally, no state would change in the face of an error. Unfortunately, that is not the case but it is exactly what I would like to do. I am writing a C++ wrapper for Lua in the context of Kyua and fine granularity in error reporting means that automatic cleanup of resources managed by RAII is trivial.

Let's analyze the options that we have to control errors caused within the Lua C API. I will explain in a later post the one I have chosen for the wrapper in Kyua (it has to be later because I'm not settled yet!).

Install a panic handler

Whenever Lua code runs in an unprotected environment, one can use lua_atpanic to install a handler for errors. The function provided by the user is executed when the error occurs and, if the panic function returns, the program exits. To prevent exiting prematurely, one could opt for two mechanisms:

Make the panic handler raise a C++ exception. Sounds nice, right? Well, it does not work. The Lua library is generally built as a C binary which means that our panic handler will be called from within a C environment. As a result, we cannot throw an exception from our C++ handler and expect things to work: the exception won't propagate correctly from a C++ context to a C context and then back to C++. Most likely, the program will abort as soon as we leave the C++ world and enter C to unwind the stack.
Use setjmp before the call to the unsafe Lua function and recover with longjmp from within the panic handler. It turns out that this does work but with one important caveat: the stack is completely cleared before the call to the panic handler. As a result, this prevents the requirement of "leave the stack unmodified on failure" as is desired of any function (report errors early before changing state).

Run every single call in a protected environment

This is doable but complex and not completely right: to do this, we need to write a C wrapper function for every unsafe API function and run it with lua_pcall. The overhead of this approach is significant: something as simple as a call to lua_gettable turns into several stack manipulation operations, a call to lua_pcall and then further stack modifications to adjust the results.

Additionally, in order to prepare the call to lua_pcall, one has to use the multiple lua_push* functions to prepare the stack for the call. And, guess what, most of these functions that push values onto the stack can themselves fail. So... in order to prepare the environment for a safe call, we are already executing unsafe calls. (Granted, the errors in these case are only due to memory exhaustion... but still, the solution is not fully robust.)

Lastly, note that we cannot use lua_cpcall because it does discard all return values of the executed function. Which means that we can't really wrap single Lua operations. (We could wrap a whole algorithm though.)

Run the whole algorithm in a protected environment

This defeats the whole purpose of the per-function wrapping. We would need to provide a separate C/C++ function that runs all unsafe code and then call it by means of lua_pcall (or lua_cpcall) so that errors are captured and reported in a controlled manner. This seems very efficient... albeit not transparent and will surely cause issues.

Why is this problematic? Errors that happen inside the protected environment are managed by means of a longjmp. If the code wrapped by lua_pcall is a C++ function, it can instantiate objects. These objects have destructors. A longjmp outside of the function means that no destructors will run... so objects will leak memory, file descriptors, and anything you can imagine. Doom's day.

Yes, I know Lua can be rebuilt to report internal errors by means of exceptions which would make this particular problem a non-issue... but this rules out any pre-packaged Lua binaries (the default is to use longjmp and henceforth what packaged binaries use). I do not want to embed Lua into my source tree. I want to use Lua binary packages shipped with pretty much any OS (hey, including NetBSD!), which means that my code needs to be able to cope with Lua binaries that use setjmp/longjmp internally.

Closing remarks

I hope the above description makes any sense because I had to omit many, many details in order to make the post reasonably short. It could also be that there are other alternatives I have not considered, in which case I'd love to know them. Trying to find a solution to the above problem has already sucked several days of my free time, which translates in Kyua not seeing any further development until a solution is found!

Understanding setjmp/longjmp

2011-01-02T03:22:00.003-05:00

For a long time, I have been aware of the existence of the standard C functions setjmp and longjmp and that they can be used to simulate exceptions in C code. However, it wasn't until yesterday that I had to use them... and it was not trivial. The documentation for these functions tends to be confusing, and understanding them required looking for additional documents and a bit of experimentation. Let's see if this post helps in clarifying how these functions work.

The first call to setjmp causes the process state (stack, CPU registers, etc.) to be saved in the provided jmp_buf structure and, then, a value of 0 to be returned. A subsequent call to longjmp with the same jmp_buf structure causes the process to go "back in time" to the state stored in said structure. The way this is useful is that, when going back in time, we tweak the return value of the setjmp call so we can actually run a second (or third or more) path as if nothing had happened.

Let's see an example:

#include <setjmp.h>
#include <stdio.h>
#include <stdlib.h>

static jmp_buf buf;

static void
myfunc(void)
{
   printf("In the function.\n");

   ... do some complex stuff ...

   /* Go back in time: restore the execution context of setjmp
    * but make the call return 1 instead of 0. */
   longjmp(buf, 1);

   printf("Not reached.\n");
}

int
main(void) {
   if (setjmp(buf) == 0) {
       /* Try block. */
       printf("Trying some function that may throw.\n");
       myfunc();
       printf("Not reached.\n");
   } else {
       /* Catch block. */
       printf("Exception caught.\n");
   }
   return EXIT_SUCCESS;
}

The example above shows the following when executed:

Trying some function that may throw.
In the function.
Exception caught.

So, what happened above? The code starts by calling setjmp to record the execution state and the call returns 0, which causes the first part of the conditional to run. You can think of this clause as the "try" part of an exception-based code. At some point during the execution of myfunc, an error is detected and is "thrown" by a call to longjmp and a value of 1. This causes the process to go back to the execution of setjmp but this time the call returns 1, which causes the second part of the conditional to run. You can think of this second clause as the "catch" part of an exception-based code.

It is still unclear to me what the "execution context" stored in jmp_buf is: the documentation does not explain what kind of resources are correctly unwinded when the call to longjmp is made... which makes me wary of using this technique for exception-like handling purposes. Oh, and this is even less clear in the context of C++ code and, e.g. calls to destructors. Would be nice to expand the description of these APIs in the manual pages.

Happy new year!

2011-01-01T14:37:00.003-05:00

Dear readers,

2011 is here so...

Happy new year!

I hope you all are having a nice holiday season and enjoyed the new year's eve celebration, should it be something special for you.

My tentative resolutions for this year related to non-work and non-personal areas would be:

First, to revive this blog. I have been lately posting more frequently than has been usual and it has been a pleasant task. I would like to recover the habit of blogging several times per week and, for that, I need topics! Keep'em coming! I already have some topics on the queue but they need a bit of research on my side first.
Second, to bring Kyua to reality (i.e. to deprecate ATF). Work is continuing intensively and a preliminary release should be ready during Q1. This release will bring the much-needed replacement for atf-run.
And third... well, haven't thought that much about resolutions ;-) We will see.

Using RAII to clean up temporary values from a stack

2010-12-27T04:06:00.007-05:00

For the last couple of days, I have been playing around with the Lua C API and have been writing a thin wrapper library for C++. The main purpose of this auxiliary library is to ensure that global interpreter resources such as the global state or the execution stack are kept consistent in the presence of exceptions — and, in particular, that none of these are leaked due to programming mistakes when handling error codes.

To illustrate this point, let's forget about Lua and consider a simpler case. Suppose we lost the ability to pass arguments and return values from functions in C++ and all we have is a stack that we pass around. With this in mind, we could implement a multiply function as follows:

void multiply(std::stack< int >& context) {
    const int arg1 = context.top();
    context.pop();
    const int arg2 = context.top();
    context.pop();
    context.push(arg1 * arg2);
}

And we could call our function as this:

std::stack< int > context;
context.push(5);
context.push(6);
multiply(context);
const int result = s.top();
s.pop();

In fact, my friends, this is more-or-less what your C/C++ compiler is internally doing when converting code to assembly language. The way the stack is organized to perform calls is known as the calling conventions of an ABI (language/platform combination).

Anyway, back to our point. One important property of such a stack-based system is that any function that deals with the stack must leave it in a consistent state: if the function pushes temporary values (read: local variables) into the stack, such temporary values must be gone upon return no matter how the function terminates. Otherwise, the caller will not find the stack as it expects, which will surely cause trouble at a later stage. The above example works just fine because our function is extremely simple and does not put anything on the stack.

But things get messier when our functions can fail halfway through, and, in particular, if such failures are signaled by exceptions. In these cases, the function will abort abruptly and the function must take care to clean up any values that may still be left on the stack. Let's consider another example:

void magic(std::stack< int >& context) {
    const int arg1 = context.top();
    context.pop();
    const int arg2 = context.top();
    context.pop();

    context.push(arg1 * arg2);
    context.push(arg1 / arg2);
    try {
        ... do something with the two values on top ...

        context.push(arg1 - arg2);
        try {
            ... do something with the three values on top ...
        } catch (...) {
            context.pop();  // arg1 - arg2
            throw;
        }
        context.pop();
    } catch (...) {
        context.pop();  // arg1 / arg2
        context.pop();  // arg1 * arg2
        throw;
    }
    context.pop();
    context.pop();
}

The above is a completely fictitious and useless function, but serves to illustrate the point. magic() starts by pushing two values on the stack and then performs some computation that reads these two values. It later pushes an additional value and does some more computations on the three temporary values that are on the top of the stack.

The "problem" is that the computation code can throw an exception. If it does, we must sanitize the stack to remove the two or three values we have already pushed. Otherwise, the caller will receive the exception, it will assume nothing has happened, and will leak values on the stack (bad thing). To prevent this, we have added a couple of try/catch clauses to capture these possible exceptions and to clean up the already-pushed values before exiting the function. Unfortunately, this gets old very quickly: having to add try/catch statements surrounding every call is boring, ugly, and hard to read (remember that, potentially, any statement can throw an exception). You can see this in the example above with the two nested try/catch blocks.

To mitigate this situation, we can apply a RAII-like technique to make popping elements on errors completely transparent and automated. If we can make it transparent, writing the code is easier and reading it is trivial; if we can make it automated, we can be certain that our error paths (rarely tested!) correctly clean up any global state. In C++, destructors are deterministically executed whenever a variable goes out of scope, so we can use this to our advantage to clean up temporary values. Let's consider this class:

class temp_stack {
    std::stack< int >& _stack;
    int _pop_count;

public:
    temp_stack(std::stack< int >& stack_) :
        _stack(stack_), _pop_count(0) {}

    ~temp_stack(void)
    {
        while (_pop_count-- > 0)
            _stack.pop();
    }

    void push(int i)
    {
        _stack.push(i);
        _pop_count++;
    }
};

With this, we can rewrite our function as:

void magic(std::stack< int >& context) {
    const int arg1 = context.top();
    context.pop();
    const int arg2 = context.top();
    context.pop();

    temp_stack temp(context);

    temp_stack.push(arg1 * arg2);
    temp_stack.push(arg1 / arg2);
    ... do something with the two values on top ...

    temp_stack.push(arg1 - arg2);
    ... do something with the three values on top ...

    // Yes, we can return now.  No need to do manual pop()s!
}

Simple, huh? Our temp_stack function keeps track of how many elements have been pushed on the stack. Whenever the function terminates, be it due to reaching the end of the body or due to an exception thrown anywhere, the temp_stack destructor will remove all elements previously registered from the stack. This ensures that the function leaves the global state (the stack) as it was on entry — modulo the function parameters consumed as part of the calling conventions.

So how does all this play together with Lua? Well, Lua maintains a stack to communicate parameters and return values between C and Lua. Such stack can be managed in a similar way with a RAII class, which makes it very easy to write native functions that deal with the stack and clean it up correctly in all cases. I would like to show you some non-fictitious code right now, but it's not ready yet ;-) But when it is, it will be part of Kyua. Stay tuned!

And, to conclude: to make C++ code robust, wrap objects that need manual clean up (pointers, file descriptors, etc.) with small wrapper classes that perform such clean up on destruction. These classes are typically fully inlined and contain a single member field, so they do not impose any performance penalty. But, on the contrary, your code can avoid the need of many try/catch blocks, which are tricky to get right and hard to validate. (Unfortunately, this technique cannot be applied in, e.g. Java or Python, because the execution of the class destructors is completely non-deterministic and not guaranteed to happen whatsoever!)

Dependency injection: simple class constructors

2010-12-26T05:41:00.005-05:00

Following my previous post on dependency injection (DI for short), I wanted to show you today another example of code in which DI helps in making the code clearer and easier to validate. In this case, the person to blame for the original piece of code being criticized is me.

The atffile module in ATF provides a class to represent the contents of Atffiles. An Atffile is, basically, a file containing a list of test programs to run and a list of properties associated to these test programs. Let's consider the original implementation of this module:

class atffile {
    strings_vector _test_programs;
    strings_vector _properties;

public:
    atffile(const path& file)
    {
        std::ifstream input(file.c_str());
        _test_programs = ... parse list from input ...;
        _properties = ... parse list from input ...;
    }

    ... getters and other read-only methods ...
};

According to the object-oriented programming (OOP) principles we are taught over and over again, this seems like a reasonable design. An atffile object is entirely self-contained: if the constructor finishes successfully, we know that the new object matches exactly the representation of an Atffile on disk. The other methods in the class provide read-only access to the internal attributes, which ensures that the in-memory representation remains consistent.

However, this design couples the initialization of an object with external dependencies, and that is bad for two main reasons: first, because it makes testing (very) difficult; and, second, because it makes an apparently simple action (constructing an object) a potentially expensive task (reading from an external resource).

To illustrate the first point, let's consider a helper free-function that deals with an atffile object:

std::string
get_property(const atffile& file, const std::string& name,
             const std::string& defvalue)
{
    const strings_vector& props = file.properties();
    const strings_vector::const_iterator iter =
        props.find(name);
    if (iter == props.end())
        return defvalue;
    else
        return *iter;
}

Now, how do we write unit-tests for this function? Note that, to execute this function, we need to pass in an atffile object. And to instantiate an atffile, we need to be able to read a Atffile from disk because the only constructor for the atffile class has this dependency on an external subsystem. So, summarizing, to test this innocent function, we need to create a file on disk with valid contents, we need to instantiate an atffile object pointing to this file, and only then we can pass it to the get_property function. At this point, our unit test smells like an integration test, and it actually is for no real reason. This will cause our test suite to be more fragile (the test for this auxiliary function depends on the parsing of a file) and slow.

How can we improve the situation? Easy: decoupling the dependencies on external systems from the object initialization. Take a look at this rewritten atffile class:

class atffile {
    strings_vector _test_programs;
    strings_vector _properties;

public:
    atffile(const strings_vector& test_programs_,
            const strings_vector& properties_) :
        _test_programs(test_programs_),
        _properties(properties_)
    {
        assert(!_test_programs.empty());
    }

    static atffile
    parse(const path& file)
    {
        std::ifstream input(file.c_str());
        strings_vector test_programs_ =
            ... parse list from input ...;
        strings_vector properties_ =
            ... parse list from input ...;
        return atffile(test_programs_, properties_);
    }

    ... getters and other read-only methods ...
};

Note that this new design does not necessarily violate OOP principles: yes, we can now construct an object with fake values in it by passing them to the constructor, but that does not mean that such values can be inconsistent once the object is created. In this particular example, I have added an assertion in the constructor to reenforce a check performed by parse (that an atffile must list at least one test program).

With this new design in mind, it is now trivial to test the get_property function shown above: constructing an auxiliary atffile object is easy, because we can inject values into the object to later pass it to get_property: no need to create a temporary file that has to be valid and later parsed by the atffile code. Our test now follows the true sense of a unit test, which is much faster, less fragile and "to-the-point". We can later write integration tests if we so desire. Additionally, we can also write tests for atffile member functions, and we can very easily reproduce corner cases for them by, for example, injecting bad data. The only place where we need to create temporary Atffiles is when we need to test the parse class method.

So, to conclude: make your class constructors as simple as possible and, in particular, do not make your class constructors depend on external systems. If you find yourself opening resources or constructing other objects from within your constructor, you are doing it wrong (with very few exceptions).

I have been using the above principle for the last ~2 years and the results are neat: I am much, much more confident on my code because I write lots of more accurate test cases and I can focalize dependencies on external resources on a small subset of functions. (Yes, Kyua uses this design pattern intensively!)

Dependency injection and testing: an example

2010-12-23T11:11:00.004-05:00

A coworker just sent me some Python code for review and, among such code, there was the addition of a function similar to:

def PathWithCurrentDate(prefix, now=None):
    """Extend a path with a year/month/day subdirectory layout.

    Args:
        prefix: string, The path to extend with the date subcomponents.
        now: datetime.date, The date to use for the path; if None, use
            the current date.

    Returns:
        string, The new computed path with the date appended.
    """
    path = os.path.join(prefix, '%Y', '%m', '%d')
    if now:
        return now.strftime(path)
    else:
        return datetime.datetime.now().strftime(path)

The purpose of this function, as the docstring says, is to simplify the construction of a path that lays out files on disk depending on a given date.

This function works just fine... but it has a serious design problem (in my opinion) that you only see when you try to write unit tests for such function (guess what, the code to review did not include any unit tests for this). If I ask you to write tests for PathWithCurrentDate, how would you do that? You would need to consider these cases (at the very very least):

Passing now=None correctly fetches the current date. To write such a test, we must stub out the call to datetime.datetime.now() so that our test is deterministic. This is easy to do with helper libraries but does not count as trivial to me.
Could datetime.datetime.now() raise an exception? If so, test that the exception is correctly propagated to match the function contract.
Passing an actual date to now works. We know this is a different code path that does not call datetime.datetime.now(), but still we must stub it out to ensure that the test is not going through that past in case the current date actually matches the date hardcoded in the test as an argument to now.

My point is: why is such a trivial function so complex to validate? Why such a trivial function needs to depend on external state? Things become more obvious if we take a look at a caller of this function:

def BackupTree(source, destination):
    path = PathWithCurrentDate(destination)
    CreateArchive(source, os.path.join(path, 'archive.tar.gz'))

Now, question again: how do we test this? Our tests would look like:

def testOk(self):
    # Why do we even have to do this?
    ... create stub for datetime.datetime.now() to return a fake date ...

   CreateArchive('/foo', '/backups/prefix')
   ... validate that the archive was generated in the fake date directory ...

Having to stub out the call to datetime.datetime.now() before calling CreateArchive is a really, really weird thing at first glance. To be able to write this test, you must have deep insight of how the auxiliary functions called within the function work to know what dependencies on external state they have. Lots of black magic involved.

All this said, the above may not seem like a big issue because, well, a call to datetime.datetime.now() is cheap. But imagine that the call being performed deep inside the dependency tree was more expensive and dealt with some external state that is hard to mock out.

The trick to make this simpler and clearer is to apply a form of Dependency injection (or, rather, "value injection"). We want the PathWithCurrentDate function to be a simple data manipulation routine that has no dependencies on external state (i.e. make it purely functional). The easiest way to do so is to remove the now=None code path and pass the date in right from the most external caller (aka, the main() program). For example (skipping docstrings for brevity):

def PathWithCurrentDate(prefix, now):
    path = os.path.join(prefix, '%Y', '%m', '%d')
    return now.strftime(path)

def BackupTree(source, destination, backup_date):
    path = PathWithCurrentDate(destination, backup_date)
    CreateArchive(source, os.path.join(path, 'archive.tar.gz'))

With this approach, the dependency on datetime.datetime.now() (aka, a dependency on global state) completely vanishes from the code. The code paths to validate are less, and they are much simpler to test. There is no need to stub out a function call seemingly unused by BackupTree.

Another advantage of this approach can be seen if we were to have multiple functions accessing the same path. In this case, we would need to ensure that all calls receive the exact same date... what if the program kept running past 12AM and the "now" value changed? It is trivial to reason about this feature if the code does not have hidden queries to "now" (aka global state) within the code... but it becomes tricky to ensure our code is right if we can't easily audit where the "now" value is queried from!

The "drawback", as some will think, is that the caller of any of these functions must do more work on its own to provide the correct arguments to the called functions. "And if I always want the backup to be created on the current directory, why can't the backup function decide on itself?", they may argue. But, to me, the former is definitely not a drawback and the latter... is troublesome as explained in this post.

Another "drawback", as some others would say, is that testing is not a goal. Indeed it is not: testing is only a means to "correct" code, but it is also true that having testable code often improves (internal) APIs and overall design.

To conclude: the above is an over-simplistic case study and my explanations will surely not convince anyone to stop doing black evil "clever" magic from within functions (and, worse, from within constructors). You will only realize that the above makes any sense when you start unit-testing your code. Start today! :-)

Kyua: Design of the configuration system

2010-12-22T09:48:00.004-05:00

Over a week ago, I mostly finished the implementation of the runtime engine for test cases of Kyua and, along the way, realized that it is imperative to write a configuration system right now before the configuration code becomes messier than it already is.

To that end, I spent the last week working on a design document for the configuration system. Summarizing, the document describes what the requirements for the configuration files of Kyua are, what the possible alternatives to implement them are, and advocates the use of Lua — a tiny embedded programming language — to bring these configuration files to life.

It is your chance to get involved in the early stages of the development of Kyua! :-) Take a look at the email to kyua-discuss asking for comments and feel free to join the mailing list ;-)

Sticky bit trivia

2010-12-21T03:50:00.004-05:00

Did you ever wonder where the "sticky" part of the "sticky bit" name comes from? I actually didn't, but I just came across the Sticky bit page on Wikipedia through a tweet from @AaronToponce and discovered why.

If you have used any "recent" (the quotes are important) Unix-like system, you probably know what the sticky bit is used for: to restrict the deletion of files in a directory to, basically, the owner of such files. The sticky bit is used on /tmp (among other directories) so that anyone can create temporary files in it but only those that created the temporary files can delete them.

But that's not all that is to know.

The original purpose of the sticky bit was to mark frequently-used executable files so that the operating system kept their text segment on swap space. This speeded up subsequent executions of such programs because the system would not need to access the file system to load the binary: it could just use the image already kept in the swap space. This behavior became obsolete with the advent of memory-mapped executables.

Now it's clear why the sticky bit has the name it has, isn't it? But still, that's not all that is to know.

SunOS 4 introduced a new behavior for the sticky bit on regular, non-executable files: reads and writes to such files would bypass the buffer cache, thus basically telling the system to perform raw I/O on those files. This was particularly useful on swap files when NFS was involved.

With the above, I have just tried to summarize you the information that is in NetBSD's chmod(2) and sticky(7) manual pages; they contain much more detailed information. (And yep, that's right: contrary to what the Wikipedia article says, NetBSD does not support this behavior any more.) Hope you found this insightful if you did not know this little piece of history!

Getting the hang of Twitter searches

2010-12-17T10:43:00.003-05:00

I have had a Twitter account (@jmmv) for several years already but I have never leveraged its power. Why? Well... basically because I have never known how.

The Twitterville book by Shel Israel (@shelisrael), which I have been reading lately, has opened my mind quite a bit. Twitter is not so much about posting status updates, but more about sharing content and starting/joining conversations with other people. Today I like to think of Twitter as a world-wide chat-room.

Twitterville mentions many times that the key in Twitter usage is to search for content. But finding content among all the junk that floats in Twitter is hard. I had seen that it is possible to actually do searches in the Twitter web page, but that is not really usable. Yes, you can search once for something you are interested in... but you know what, you can do the same in any search engine and get more relevant results.

To me, what has made a difference is to switch to a client that does actually support live searches (TweetDeck in my case). With such a client, all you have to do is create a search for any given topic you may be remotely interested in and status updates will just pop up in your client as soon as someone posts about that particular topic. Easy, huh? See, it's like joining your favorite #topic chat-room.

With this in mind, you can, for example, create a search such as "#netbsd OR #freebsd OR #openbsd" to get real-time tweets about these BSD operating systems. It is a fact that you will see loads of junk (disable popup notifications recommended), but you will catch some interesting content. And the best of it, you can reply to that content. This is particularly useful because you can (try to) fix misconceptions that people have before they spread out too much.

That said, the value you see in Twitter and how you use it, is fully up to you. Different people will find different use cases, all of them interesting on their own.

And to conclude, I have to confess that while the above may seem obvious to many, it is something that has escaped my mind until last week.

Introducing Kyua

2010-12-16T14:11:00.005-05:00

Wow. I have just realized that I have not blogged at all about the project that has kept me busy for the past two months! Not good, not good. "What is this project?", I hear. Well, this project is Kyua.

A bit of background first: the Automated Testing Framework, or ATF for short, is a project that I started during the Summer of Code of 2007. The major goal of ATF was, and still is, to provide a testing framework for the NetBSD operating system. The ATF framework is composed of a set of libraries to aid in the implementation of test cases in C, C++ and shell, and a set of tools to ease the execution of such test cases (atf-run) and to generate reports of the execution (atf-report).

At that point in time, I would say that the original design of ATF was nice. It made test programs intelligent enough to execute their test cases in a sandboxed environment. Such test programs could be executed on their own (without atf-run) and they exposed the same behavior as when they were run within the runtime monitor, atf-run. On paper this was nice, but in practice it has become a hassle. Additionally, some of these design decisions mean that particular features (in particular, parallel execution of tests) cannot be implemented at all. At the end of 2009 and beginning of 2010, I did some major refactorings to the code to make the test programs dumber and to move much of the common logic into atf-run, which helped a lot in fixing the major shortcomings encountered by the users... but the result is that, today, we have a huge mess.

Additionally, while ATF is composed of different modules conceptually separate from each other, there is some hard implementation couplings among them that impose severe restrictions during development. Tangentially, during the past 2 years of working at Google (and coding mainly in Python), I have been learning new neat programming techniques to make code more testable... and these are not followed at all by ATF. In fact, while the test suite of ATF seems very extensive, it definitely is not: there are many corner cases that are not tested and for which implementing tests would be very hard (which means that nasty bugs have easily sneaked in into releases).

Lastly, a very important point that affects directly the success of the project. Outsiders that want to contribute to ATF have a huge entry barrier: the source repository is managed by Monotone, the bug tracker is provided by Gnats (a truly user-unfriendly system), and the mailing lists are offered by majordomo. None of these tools is "standard" by today's common practices, and some of them are tied to NetBSD's hosting which puts some outsiders off.

For all the reasons above and as this year has been moving along, I have gotten fed up with the ATF code base. (OK, things are not that bad... but in my mind they do ;-) And here is where Kyua comes into the game.

Kyua is a project to address all the shortcomings listed above. First of all, the project uses off-the-shelf development tools that should make it much, much easier for external people to contribute. Secondly, the project intends to be much more modular, providing a clear separation between the different components and providing code that is easily testable. Lastly, Kyua intends to remain compatible with ATF so that there are no major disruptions for users. You can (and should) think of Kyua as ATF 2.0, not as a vastly different framework.

As of today, Kyua implements a runtime engine that is on par, feature-wise, to the one provided by atf-run. It is able to run test cases implemented with the ATF libraries and it is able to test itself. It currently contains 355 test cases that run in less than 20 seconds. (Compare that to the 536 test cases of ATF, which take over a minute to run, and Kyua is still really far from catching up with all the functionality of ATF.) Next actions involve implementing reports generation and configuration files.

Anyway. For more details on the project, I recommend you to read the original posting to atf-devel or the project's main page and wiki. And of course, you can also download the preliminary source code to take a look!

Enjoy :-)

Creating atf-based tests for NetBSD src

2010-09-03T12:14:00.003-04:00

Thanks to Antti Kantee's efforts, atf is seeing increasing visibility in the NetBSD community during the past few months. But one of the major concerns that we keep hearing from our developers is "Where is the documentation?". Certainly I have been doing a pretty bad job at that, and the current in-tree documents are a bit disorganized.

To fix the short-term problem, I have written a little tutorial that covers pretty much every aspect that you need to know to write atf tests and, in particular, how to write such tests for the NetBSD source tree. Please refer to the official announcement for more details.

Comments are, of course, welcome! And if you can use this tutorial to write your first tests for NetBSD, let me know :-)

ATF 0.10 released

2010-07-02T11:18:00.003-04:00

Ladies and gentlemen: I have just released ATF 0.10! This release with such a magic number includes lots of new exciting features and provides a much simplified source tree.

Dive into the 0.10 release page for details!

I'm now working in getting this release into the NetBSD tree to remove some of the custom patches that have been superseded by the official release. Will be there soon.

And all this while I am a meetBSD in Kraków :-)

Testing NetBSD: Easy Does It

2010-06-24T04:20:00.002-04:00

Antti Kantee has been, for a while, writing unit/integration tests for the puffs and rump systems (for which he is the author) shipped with NetBSD. Recently, he has been working on fixing the NetBSD test suite to report 0 failures in the i386 platform so as to encourage developers to keep it that way while doing changes to the tree. The goal is to require developers to run the tests themselves before submitting code.

Antti has just published an introductory article, titled Testing NetBSD: Easy Does It, that describes what ATF and Anita are, how to use them and how they can help in NetBSD development and deployment. Nice work!

ATF 0.9 released (late announcement)

2010-06-18T13:48:00.003-04:00

Oops! Looks like I forgot to announce the release of ATF 0.9 here a couple of weeks ago. Just a short notice that the formal release has been available since June 3rd and that 0.9 has been in NetBSD since June 4th!

You can also enjoy a shiny-new web site! It even includes a FAQ!

And, as a side note: I have added a test target to the NetBSD Makefiles, so now it's possible to just do make test within any subdirectory of src/tests/ and get what you expect.

Trac installation for ATF

2010-05-10T14:33:00.004-04:00

During the past few months, I've got into the habit of using a bug tracker to organize my tasks at the work place. People assign tickets to me to get things done and I also create and self-assign tickets to myself to keep them as a reminder of the mini-projects to be accomplished. Sincerely, this approach works very well for me and keeps me focused.

Since then, I've been wishing to have a similar system set up for ATF. Yeah, we could use the Gnats installation provided by NetBSD... but I hate this issue tracking system. It's ancient, ugly, and I do really want a web interface to manage my tickets through.

So, this weekend, I finally took some time and set up a Trac installation for ATF to provide a decent bug/task tracking system. The whole Apache plus Trac setup was more complex than I imagined, but I do hope that the results will pay off :-)

Take a look at the official announcement for more details!

Ads gone

2010-05-10T05:22:00.003-04:00

Almost a year ago, I decided to give a try to AdSense. And, so far, the "earnings" have been ~30 EUR which I cannot even cash. Given this and how ugly and disturbing the ads look on the front page, I have disabled them. (I think the ads have gotten much worse over time... but as I do not pay attention to the front page, I didn't see them.) Thanks to Roman Valls for pointing this out!

ATF 0.8 imported into NetBSD

2010-05-08T04:21:00.002-04:00

Finished importing ATF 0.8 into the NetBSD source tree. Wow, the CVS import plus merge was much easier than I expected.

Note that, while the NetBSD test suite should continue to work as usual, there are some backwards incompatible changes in the command line interface of test programs. If you are used to run them by hand, expect different results. Please read the release news for details.

Now let's wait for complaints about broken builds! And enjoy this new release in your NetBSD-current system!

Announcing ATF 0.8

2010-05-07T10:41:00.003-04:00

Looks like today is a release day. I've just pushed ATF 0.8 out in the wild and will proceed to import it into pkgsrc and NetBSD later. Refer to the release announcement for details. This is an exciting release! You have been warned ;-)

Announcing etcutils 0.1

2010-05-07T05:03:00.003-04:00

During past week, I worked on a new package called etcutils. It provides a (reduced) tool-set to programmatically manage files in /etc and is specially designed to allow pkgsrc to update /etc/shells and /etc/services in a more consistent way.

I'm happy to say that the 0.1 release is now ready! Go to the etcutils web page for details. (I know that if you are a Linux user, you probably don't care about this because your distribution most likely already provides something similar... albeit more complex.)

I'll now proceed to import this new package into pkgsrc as sysutils/etcutils. Later on (most likely not today), the following should happen: rework pkginstall to use the new shells(8) utility to update /etc/shells, add a new feature to pkgsrc to abstract the updates to /etc/services, and swipe through pkgsrc to make all packages touching this file use the new frameworky option. (Hey FAM, that includes you!)

Forget about test(1)'s == operator

2010-04-24T17:24:00.004-04:00

Some implementations of test(1), in an attempt to be smart, provide non-standard operators such as ==. Please forget about those: they make your scripts non-portable and a pain to use in other systems. Why? Because, due to the way the shell works, failures in calls to test(1) will often just result in an error message (which may not be seen due to other output) and the script will happily continue running even if it missed to perform some important operation.

So... just use the standard equality operators:

= for string equality comparison.
-eq for numeric equality comparison.

Note that whenever I refer to test(1), I'm also talking about the [ ... ] construction in conditionals.

Also, please note that this also affects configure scripts, and the problem in these appears much more commonly than in other scripts!

Always define an else clause for portability #ifdefs

2010-04-22T09:51:00.005-04:00

If you use #ifdef conditionals in your code to check for portability features, be sure to always define a catch-all else clause that actually does something, even if this something is to error out.

Consider the following code snippet, quoted from gamin's tests/testing.c file:

if (arg != NULL) {
#ifdef HAVE_SETENV
  setenv("GAM_CLIENT_ID", arg, 1);
#elif HAVE_PUTENV
  char *client_id = malloc (strlen (arg) + sizeof "GAM_CLIENT_ID=");
  if (client_id)
  {
      strcpy (client_id, "GAM_CLIENT_ID=");
      strcat (client_id, arg);
      putenv (client_id);
  }
#endif /* HAVE_SETENV */
}
ret = FAMOpen(&(testState.fc));

The FAMOpen method queries the GAM_CLIENT_ID environment variable to set up the connections parameters to the FAM server. If the variable is not defined, the connection will still work, even though it will use some default internal value. In the test code above, the variable is explicitly set to let the tests use a separate server instance.

Now, did you notice that we have to conditional branches? One for setenv and one for putenv? It seems reasonable to assume that one or the other must be present on any Unix system. Unfortunately, this is flawed:

What happens if the code forgets to include config.h?
What happens if the configure script fails to detect both setenv and putenv? This is not that uncommon, given how some configure scripts are written.
What happens if neither setenv nor putenv are available?

The answer to the three questions is: in the above code snippet, the code builds just fine but will misbehave at run time: neither HAVE_SETENV nor HAVE_PUTENV are defined, so the code will not be able to define the required environment variable. However, FAMOpen will later be called and it will not behave as expected because the variable has not been set.

Note that this code snippet is just an example. I have seen many more instances of this exact same problem with worse consequences than the above. Read: they were not part of the test code, but just part of the regular code path.

So how do you implement the above in a saner way? You have two alternatives:

Add an #else clause that contains a fallback implementation. In the case above, we could, for example, prefer to use setenv if present because it has a nicer interface, and fall back to putenv if not found.
This has a disadvantage though: if you forget to include config.h or the configure script cannot correctly detect one of the possible implementations (even when present), you will always use the fallback implementation.
Keep each possible implementation correctly protected by a conditional, but add a #else clause that raises an error at build time. This will make sure that you never forget to define at least one of the portability macros for any reason. This is the preferred approach.

Following the second suggestion above, the code would get the following structure:

#if defined(HAVE_SETENV)
setenv(...);
#elif defined(HAVE_PUTENV)
putenv(...);
#else
#   error "Don't know how to set environment variables."
#endif

With this code, we can be sure that the code will not build if none of the possible implementations are selected. We can later proceed to investigate why that happened.

Where does Gnome use file monitoring?

2010-04-21T04:08:00.003-04:00

One of my readers of my post yesterday, wonders where Gnome uses the file monitoring APIs. Well, the answer is: everywhere.

Here are some examples:

Nautilus monitors all open folders so that it can update their contents whenever the underlying file store changes. Say you are viewing the Documents folder and you save a new Document from within OpenOffice into that folder. You definitely want Nautilus to show it immediately, without having to manually hit Refresh from the menu.
The trash applet monitors the trash folders and updates its icon from empty to full whenever one of these folders ceases to be non-empty.
The panel monitors the applications directory to notice when new applications get installed. This allows it to update the Applications menu immediately as soon as a new program gets installed into the system.
The GTK file open/save dialogs monitor the directory they are viewing for the same reason as Nautilus. (Actually, I'm unsure about this point. My Linux installation is an old Ubuntu 8.04 LTS that does not have GIO, so I can't verify in recent ones. However, this makes perfect sense and if not implemented, it should be!)
The background switcher control panel monitors the folders containing images to be able to show new installed backgrounds. (I'm not sure about this either. It doesn't happen in my Linux installation, but it also makes sense.)
Media players such as Rhythmbox and Banshee allow you to point them to a folder containing music and have an option to automatically add music to the library as soon as it pops up in such folder.
Potentially, any document editor, picture viewer, etc. monitors the documents opened in them so that these applications can notice external notifications to those files. This is useful to prevent overwriting a file with an out-dated in-memory copy. For example: you are taking some notes with GEdit. On a separate terminal window, you quickly edit the notes file with Vim to add a new note. When you go back to GEdit, you want the editor to tell you that the file has changed out of its control and offer you a choice: e.g. reload or ignore?

And many, many more other situations that I'm surely missing...

As you can see, it is fairly important to get the file monitoring subsystem working flawlessly. Otherwise, all these tiny details don't work and the end user experience is undermined.

New gio-fam package

2010-04-20T13:48:00.003-04:00

As briefly outlined in the previous post, new versions of Glib provide GIO, a library that intends to be a low-level file system API on top of the POSIX interface. This library provides an interface to asynchronously wait for file system change notifications including the creation, deletion and modification of files.

The monitoring functionality in GIO is modular: it is backed by different loadable plugins that implement OS-specific functionality. In particular, GIO uses an inotify module in Linux and a FAM module everywhere else.

Up until now, the devel/glib2 package in pkgsrc provided a build-time option to specify whether to build the GIO FAM plugin or not. Given that this plugin is built as a shared object that is loaded dynamically at run-time, having a build-time option for this is clearly wrong: it gives no choice to those relying on binary packages (e.g. end/new users). Furthermore, it adds a dependency on the ugly-FAM at the very bottom of the huge Gnome dependency chain. (As already stated, FAM is outdated and hard to set up.)

So, based on this, I've just removed all FAM support from devel/glib2 altogether and packaged its loadable module as sysutils/gio-fam.

Now waiting for a clean rebuild of the Gnome packages to see if the desktop now works on my machine by avoiding FAM/Gamin.

File system monitoring, Gnome and NetBSD

2010-04-20T09:21:00.006-04:00

A few days ago, I decided to start using NetBSD, as well as Gnome on NetBSD once again, mostly because the lack of their use makes my skills feel rusty in many different areas. While NetBSD has surprised me in a good way (I am running it on a Macbook Pro and things like wireless and DRI work), Gnome has not. There are tons of broken things that prevent a smooth user experience.

One of these broken things is the monitoring of changes in the file system. Actually, this has never worked 100%. But what is this and why does it matter, you ask? Well, file system monitoring is an internal component of the Gnome infrastructure that allows the desktop to receive notifications when files or directories change. This way, if, say, you are viewing the Downloads folder in Nautilus and you start downloading a file from Epiphany into that folder, Nautilus will realize the new file and show it immediately without requiring a manual refresh.

How to monitor the file system depends on the operating system. There are basically two approaches: polling and asynchronous notifications. Polling is suboptimal because the notifications are usually delayed. Asynchronous notifications are tied to the operating system: Linux provides inotify, NetBSD provides kqueue and other systems provide their own APIs.
In the past, Gnome monitored the file system by a combination of FAM, a system-level service that provides an API to file system monitoring, and GNOME VFS, a high-level layer that hides the interaction with FAM. This approach was good in spirit (client/server separation) but didn't work well:

FAM is abandoned.
Does not support kqueue out of the box.
FAM runs as root.
FAM is too hard to set up: it requires rpcbind, an addition to /etc/services, a sysctl tweak, and the configuration of a system-level daemon.

To solve some of these problems, a drop-in replacement for FAM was started. Gamin, as it is known, still does not fix everything:

Gamin is abandoned.
Supports kqueue out of the box, but does not work very well.
Actually, Gamin itself does not work. Running the tests provided by the distfile in a modern Linux system results in several test failures.

Anyway. Did you notice the abandoned pattern above? This is important: in the new world order, Gnome does not use FAM any more.

The new structure to monitor files is: the low-level glib library provides the gio module, which has some file system monitoring APIs. The GVFS module provides higher level abstractions to file system management, and relies on gio for file system monitoring. There is no more GNOME VFS any more.

The key point is: gio uses inotify directly; no abstraction layers in between. FAM support is still there for platforms without inotify, but as it is not used in Linux any more, it rots. Linux developers will never experience what it is to have a system that needs to use FAM to get this functionality to work.

At last, let's look at the status of all this in NetBSD:

The FAM package was patched to support kqueue. Although this kinda works, it is not perfect. Also, as mentioned above, FAM is, I'd say, the package with the hardest installation procedure of the whole Gnome platform.
The Gamin packages are nicer than the FAM package regarding their configuration. However, when using Gamin instead of FAM, all sorts of bugs appear in Gnome (it actually gets stuck during startup for me). The breakage of the unit tests does not provide any confidence, and the fact that Gamin is abandoned, the idea of fixing it doesn't make me thrive.
The glib2 package depends on FAM. This is ugly; really ugly. I had to shout WTF when I saw this, seriously.
Seeing the direction gio/gvfs take, it is obvious that things can only get worse in the future.

If time permits, I'm planning to work on improving this whole situation. Ideas include:

Splitting the FAM gio module out of the glib2 package. Ideally, this would happen upstream.
Implement a gio backend for kqueue.
Check if the core packages still using gnome-vfs have a more recent version that uses gvfs instead and, if so, update them.

Can't promise you anything other than, if I get to work on it, I will keep you posted!

NetBSD in Google Summer of Code 2010

2010-03-19T04:36:00.004-04:00

For the 6th year in a row, NetBSD is a mentoring organization for Google Summer of Code 2010!

If you are a bright student willing to develop full-time for an open source project during this coming summer, consider applying with us! You will have a chance to work with very smart people and, most likely, in the area that you are most passionate about. NetBSD, being an operating system project, has offers for project ideas at all levels: from the kernel to the packaging system, passing by drivers, networking tools, user-space utilities, the system installer, automation tools and more!

I would like to point you at the 3 project proposals I'm willing to directly mentor:

Optimize and speed-up ATF: Make the testing framework blazing fast so that running the NetBSD automated tests does not take ages on slow platforms.
Reorganize ATF to improve modularity: Refactor pieces of the testing framework so that it is easier to redistribute, has cleaner interfaces and is easier to depend on from third-party projects.
Rewrite pkg_comp with portability as a major goal: Use Python to create a tool to automatically build binary packages from within a sandbox.

If you find any of the above projects interesting, or if you have any other project proposal that you think I could mentor, do not hesitate to contact me. Feel free to send me a draft of your application, together with a bit of information about you, so that we can discuss your proposal and make sure it gets selected!

Or, if none of the projects above interests you, please do check out the full list of NetBSD project proposals. I'm sure you will find something that suits your interests :-)

New version of the monotone-server package in pkgsrc

2010-03-12T04:30:00.002-05:00

Wow, it has been a long time... 5 years ago, I created the monotone-server package in pkgsrc, a package that provided an interactive script to set up a monotone server from scratch with, what I though, minimal hassle.

My package did the job just fine, but past year I was blown away by the simplicity of the same package in Fedora: their init.d script provides a set of extra commands to initialize the server before starting it up, and that is it. No need to mess with a separate interactive script; no need to create and memorize passphrases that you will never use; and, what's more, all integrated in the only single place that makes sense: in the init.d "service management" script.

It has been a while since I became jealous of their approach, but I've finally got to it: I've spent the last few days rewriting the monotone-server package in pkgsrc and came up with a similar scheme. And this new package just made its way to pkgsrc-HEAD! The new package comes with what I think is a detailed manual page that explains how to configure the server from scratch. Take a look and, if you find any mistakes, inconsistencies or improvements to be done, let me know!

In the meantime, I will log into my home server, rebuild the updated package and put it in production :-)

Introducing the ATF nofork branch

2010-03-06T15:00:00.003-05:00

Despite my time for free software being virtually zero these days, I have managed to implement a prototype of what ATF would look like if it didn't implement forking and isolation in test programs. This feature has been often requested by users to simplify their life when debugging test cases.

I shouldn't repeat everything I posted on the atf-devel mailing list regarding this announcement, so please refer to that email for details. But I must say that the results look promising: the overall code of ATF is much simpler and also faster. (An execution I just tried cuts the run time of the ATF test suite from 1m 41s to 1m 16s.) Expect more simplifications and speed-ups!

set -e and set -x

2010-01-24T16:52:00.003-05:00

If you write shell scripts, you definitely need to know about two nice features that can be enabled through the set builtin:

set -e: Enables checking of all commands. If a command exits with an error and the caller does not check such error, the script aborts immediately. Enabling this will make your scripts more robust. But don't wait until your script is "complete" to set the flag as an afterthought, because it will be a nightmare to fix the scrip to work with this feature enabled. Just write set -e as the very first line of your code; well... after the shell bang.
set -x: If you are writing simple scripts that are meant to, well, script the execution of a few tasks (as opposed of being full-flown programs written in shell), set this flag to trace the execution of all commands. This will make the interpreter print each command right before it is executed, so it will aid you in knowing what is happening at any point in time.

Installing NetBSD/macppc on a Mac Mini G4

2010-01-11T18:22:00.003-05:00

Yesterday, I spent a while installing NetBSD/macppc 5.0.1 on a Mac Mini G4. The process wasn't easy, as it involved the following steps. I'm omitting many details, as they are "common knowledge" to Mac users (or otherwise can be easily found on the net):

After booting the installer from the CD image, drop into the shell.
Use pdisk to create an Apple_HFS partition for the boot loader and two Apple_UNIX_SVR2 partitions, one for the root file system and another for swap.
Run sysinst and install the system. When asked to repartition the disk, just say Use existing partition sizes.
Once the system is installed, drop again into the shell before rebooting.
Mount your hard disk into /mnt and chroot into it.
Fetch a copy of pkgsrc.
Install the sysutils/hfsutils package.
Use hformat to create a new HFS file system in the Apple_HFS partition we created.
Mount the installation CD.
Copy, using hcopy, the ofwboot.xcf file from the CD to the boot partition.
Reboot.
Drop into the OpenFirmware setup (Command+Option+P+R).
Set boot-device to hd:,\ofwboot.xcf.
Set boot-file to netbsd.
And here is the tricky thing to get the machine to auto-boot: Set boot-command to ." hello" cr " screen" output boot, not mac-boot.

I found the last command somewhere on the Internet (dunno where now), but, supposedly, a regular mac-boot should have worked. In fact, it works if you call this command from the prompt, but not during automatic boot. (It turns out to be a problem with the version of OpenFirmware I have.)

Just writing down the steps in case I need them later on. Installing Debian stable was much, much easier, but the installer for testing crashes every day with a different error, so I gave up.

(Oh, by the way, I did the same installation into an old PowerMac G3 and that was really painful. The machine refused to boot from any of the CDs I tried and the prebuilt kernels hang during initialization due to a bogus driver. In the end: netbooting and using custom kernels.)

Processing Makefile.am with M4

2009-10-25T08:33:00.002-04:00

ATF's Makefile.am, which is a single Makefile for the whole tree, was already at the 1300 lines mark and growing. At this size, it is unmanageable, and a quick look at its contents reveals tons of repeated delicate code.

Why so much repeated code, you ask, if the whole point of Automake is to simplify Makefiles? Automake does in fact simplify Makefile code when you define targets known by Automake, such as binaries and/or libraries. However, as soon as you start doing fancy things with documentation, building tons of small programs or messing with shell scripts, things get out of control because you are left on your own to define their targets and their necessary build logic.

Up until now, I had just kept up with the boilerplate code... but now that I'm starting to add pretty complex rules to generate HTML and plain text documentation out of XML files, the complexity must go. And here comes my solution:

I've just committed an experiment to process Makefile.am with M4. I've been trying to look for prior art behind this idea and couldn't find any, so I'm not sure how well this will work. But, so far, this has cut down 350 lines of Makefile.am code.

How does this work? First of all, I've written a script to generate the Makefile.am from the Makefile.am.m4 and put it in admin/generate-makefile.sh. All this script does is call M4, but I want to keep this logic in a single place because it has to be used from two call sites as described below.

Then, I've added an autogen.sh script to the top-level directory that generates Makefile.am (using the previous script) and calls autoreconf -is. I'm against autogen.sh scripts that pretend to be smart instead of just calling autoreconf, but in this case I see no other way around it.

At last, I've modified Makefile.am to add an extra rule to generate itself based on the M4 version. This, of course, also uses generate-makefile.sh.

We'll see how this scales, but I'm so far happy with the results.

Extending sudo credentials

2009-09-23T05:10:00.002-04:00

If you use sudo for, e.g. pkgsrc's just-in-time su, you may have often bitten by the problem that some compilations are slow and the build process stops right in the middle to ask you for a root password. If you go away while the system compiles, you'll be frustrated when you come back, as the process may still well be at the very beginning.

This happens because, unless disabled by the system administrator, your sudo credentials last for 5 minutes. If you hadn't used sudo for those 5 minutes, it will ask you for your password again. A simple workaround for the problem is to automatically renew your credentials, say, every 2 minutes. You can do this by running the following command (from the same console you are using later on!) right before starting a pkgsrc build:

$ ( while :; do sudo -v; sleep 120; done ) &

Best config setting ever

2009-08-31T07:36:00.004-04:00

echo 'set editing-mode vi' >>~/.inputrc

This will enable vi-editing mode for all commands that use the GNU readline library (e.g. bash, python, bc, etc.), not only the shell. For the shell only (including non-bash shells), add 'set -o vi' to your shrc file.

I don't know why I didn't do this before given that I'm a pretty hard vi user. Still, for some reason, I kept using emacs-like key bindings for command-line editing. Not any more! However, be careful: if you are used to vim's visual editing mode, you'll keep hitting 'v' in the command line and getting super annoyed.

Enjoy!

1000 revisions for ATF

2009-08-03T15:59:00.002-04:00

Mmm! Revision 7ca234b9aceabcfe9a8a1340baa07d6fdc9e3d33, committed about an hour ago, marks the 1000th revision in the ATF repository. Thanks for staying with me if you are following the project :)

Books by Joel Spolsky

2009-08-03T13:42:00.002-04:00

I just finished reading the third book in a row from Joel Spolsky, titled Joel on Software. Before this one, I read More Joel on Software and The Best Software Writing 1, all in a bit over a month. Note: I hadn't read any book cover-to-cover for a loooong while. Very interesting and entertaining books; highly recommended.

Oh, and his writing style is really enjoyable. We, crappy blog writers, can learn a lot from him!

Rearchitecting ATF

2009-07-27T17:19:00.004-04:00

During the last few weeks, I've been doing some ATF coding and, well... I'm not happy. At all. I keep implementing features but I feel, more and more, that ATF is growing out of control and that it is way too sluggish. It oughtn't be so slow. About 6 minutes to run the whole test suite in a Mac G3 I just got? HA! I bet I can do much, much, much better than that. Come on, we are targeting NetBSD, so we should support all those niche platforms rather well, and speed matters.

The thing is, the current code base grew out of a prototype that didn't have that much of a design. Well, it had a design but, in my opinion, it has turned to be a bad design. I couldn't imagine that we could hit the bottlenecks (speed) and user-interface issues (for example, the huge difficulties that involve debugging a failing test case) that we are hitting. So...

IT IS TIME FOR A CHANGE!!!

I'm currently working on a written specification of what ATF will look like, hopefully, in the not-so-distant future. It will take a while to get there, but with enough effort, we soon will. And life will be better. And no, I'm not talking about a from-scratch rewrite; that'd only hurt the project. I plan to take incremental and safe steps, keeping the code base running all the time, but I will do a major face-lift of everything. (I wish I could say "we" instead of "I" here. But not there yet.)

Why am I writing a specification, you ask? Well, because that forces me (or ANY developer) to think how I want the thing to look like and to decide, exactly, on what the design will be, which technologies will be used, which languages will be involved and in what components, etc. And no, I'm not talking of a class model design; I'm just talking about the main design of the whole picture, which is quite hard by itself. Plus having a spec will allow me to show it to you before I start coding and you will say "oh, wonderful, this new design sucks so much that I'm not going to bother with the new version". Or maybe hell will freeze and you will think, "mmm, this looks interesting, maybe it will solve these issues I'm having as regards speed, ease of debugging and ease of use".

Anyway, I hope to have a draft "soon" and to hear any of the two possible comments as a result!

Edit (July 29th): Alright, I have uploaded an extremely preliminary copy of the specification just so that you can see where my current ideas are headed. Expect many more changes to this document, so don't pay too much attention to the tiny details (most of which aren't there anyway yet).

The mess of ATF's code

2009-07-13T20:37:00.003-04:00

Yes. ATF's code is a "bit" messy, to put it bluntly. I'm quite happy with some of the newest bits but there are some huge parts in it that stink. The main reason for this is that the "ugly" parts were the ones that were written first, and they were basically a prototype; we didn't know all the requirements for the code at that point... and we still don't know them, but we know we can do much better. Even though I'm writing in plural... I'm afraid we = I at the moment :-P

So, is it time for the big-rewrite-from-scratch? NO! Joel Spolsky wrote about why this is a bad idea and I have to agree with him. Yeah, I'm basically the only developer of the code so everything is in my head, and I'd do a rewrite with a fresh mind, but... I'd lose tons of work and, specially, I'd lose tons of code that deals with tricky corner-cases that are hard to remember.

Sure, I want to clean things up but they'll happen incrementally. And preferably concurrently with feature additions. These two things could definitely happen at the same time if only I had infinite spare time...

Anyway, the major point of this post is to describe what I don't like about the current code base and how I'd like to see it changing:

A completely revamped C++ API for test cases. The current one sucks. It is not consistent with the C API. It lacks important functionality. It uses exceptions for test-case status reporting (yuck!). And it's ugly.
Clear separation of "internal/helper" APIs from the test APIs. You'll agree that the "fs" module, which provides path abstraction and other file system management routines, is something that cannot be part of ATF's API. ATF is about testing. Period. Either that fs module should be in a separate library or should be completely hidden from the public. Otherwise, it'll suffer from abuse and, what scares me, will have to become part of ATF's API. And likewise, most — really — most of the modules in the current code are internal.
Less dependencies from the C++ API to the C API. Most of the current C++ modules are wrappers of their corresponding C counterparts. This is nice for code reuse but makes the code extremely fragile. In C++, things like RAII can provide really robust code with minimum effort, but intermixing such C++ code with C makes things ugly really quickly. I'd like to find a way to keep the two libraries separate from each other (and thus keep the C++ binding "pure"), but at the same time I don't want to duplicate code... an interesting problem.
Split the tarball into smaller pieces. People writing test cases for C applications don't want to pull in a huge package that depends on C++ and whatnot. And ATF is huge. It takes forever to compile. And this is a serious issue for broad adoption. Note: whether the tools are written in C++ or not is a separate issue, because these are not a dependency for anything!
The shell binding is slow. Really slow compared to the other ones. Optimizations would be nice, but those do not address the root of the problem: it's costly to query information from shell-based tests at run time. I.e. it takes a long time to get the full list of test cases available in a test suite because you have to run every single test program with the -l flag. Keeping a separate file with test-case metadata alongside the binary could resolve this and allow more flexibility at run time.
And some other things.

Those are the major things I'd like to see addressed soon, but they involve tons of work. Of course, I'd like to be able to work on some features expected by other developers: easier debugging, DOCUMENTATION!...

So, helpers welcome :-)

Technicians and schedules

2009-06-23T08:19:00.002-04:00

Here I am, on the afternoon of a work day, sitting at home waiting for an eircom technician to come set it up my phone line. How nice. The story goes like this:

Two weeks ago, I placed an online order to request a phone line, explicitly specifying that the physical installation is already done (even though I don't know if it works or not, but that should be fairly easy for them to check). A few days later, the technician called me saying that he'd come today (two weeks after), anytime from 12.00 to 15.00, but that I'd call the company the same day to get a more accurate schedule.

Fine, I'll wait until the 23rd to do that call. But you know what happened, right? I called them this morning and they said that, effectively, the technician was coming today, from 12.30 onwards but they were unable to provide me any more specific information because the technicians have multiple appointments. What? Again, WHAT? At this age of technology, can't we implement a system to track technicians and their schedules? Can't we make some approximations of how long each visit will take? I bet it's trivial if you put in just some common sense.

People have jobs, and they can't leave anytime for unknown periods of time; granted, I have some more freedom at Google, but that is absolutely not the case for most other companies. If you have to be at home at 12.30 sharp, and the appointment will last 30 minutes approximately, that is one thing, but having to be at home from 12.30 for an unexpected period of time, that is a very different thing.

Just wondering... couldn't they just make the technician call you about 20-30 minutes before arrival so that you could make the same arrangements as him and be there at the same time? It doesn't seem such an insane request.

Child-process management in C for ATF

2009-06-21T13:40:00.006-04:00

Let's face it: spawning child processes in Unix is a "mess". Yes, the interfaces involved (fork, wait, pipe) are really elegant and easy to understand, but every single time you need to spawn a new child process to, later on, execute a random command, you have to write quite a bunch of error-prone code to cope with it. If you have ever used any other programming language with higher-level abstraction layers — just check Python's subprocess.Popen — you surely understand what I mean.

The current code in ATF has many places were child processes have to be spawned. I recently had to add yet another case of this, and... enough was enough. Since then, I've been working on a C API to spawn child processes from within ATF's internals and just pushed it to the repository. It's still fairly incomplete, but with minor tweaks, it'll keep all the dirty details of process management contained in a single, one-day-to-be-portable module.

The interface tries to mimic the one that was designed on my Boost.Process Summer of Code project, but in C, which is quite painful. The main idea is to have a fork function to which you pass the subroutine you want to run on the child, the behavior you want for the stdout stream and the behavior you want for the stderr steam. These behaviors can be any of capture (aka create pipes for IPC communcations), silence (aka redirect to /dev/null), redirect to file descriptor and redirect to file. For simplicity, I've omitted stdin. With all this information, the fork function returns you an opaque structure representing the child, from which you can obtain the IPC channels if you requested them and on which you can wait for finalization.

Here is a little example, with tons of details such as error handling or resource finalization removed for simplicity. The code below would spawn "/bin/ls" and store its output in two files named ls.out and ls.err:

static
atf_error_t
run_ls(const void *v)
{
 system("/bin/ls");
 return atf_no_error();
}

static
void
some_function(...)
{
 atf_process_stream_t outsb, errsb;
 atf_process_child_t child;
 atf_process_status_t status;

 atf_process_status_init_redirect_path(&outsb, "ls.out");
 atf_process_status_init_redirect_path(&errsb, "ls.err");

 atf_process_fork(&child, run_ls, &outsb, &errsb, NULL);
 ... yeah, here comes the concurrency! ...
 atf_process_child_wait(&child, &status);

 if (atf_process_status_exited(&status))
     printf("Exit: %d\n", atf_process_status_exitstatus(&status));
 else
     printf("Error!");
}

Yeah, quite verbose, huh? Well, it's the price to pay to simulate namespaces and similar other things in C. I'm not too happy with the interface yet, though, because I've already encountered a few gotchas when trying to convert some of the existing old fork calls to the new module. But, should you want to check the whole mess, check out the corresponding revision.