pkgsrc: Yesterday's changes

It has been a while since I've done any "big" changes in pkgsrc, basically because I've been quite busy with VCS Made Easy during the past month. To make things worse, I've got back to university and there is a lot of work to do.

But anyway, yesterday I was able to finish two major changes I had going on. The first one was the update of GNOME to 2.8.3, which was released past Wednesday. This version features several bug fixes and minor improvements, as you can expect from any minor release. Hopefully, 2.10 will be released in less than three weeks; but that will require a lot more of work.

The second one was a complete reorganization of the Boost packages, as I announced some days ago in another post. Previously, there was a single and big boost package that provided almost everything (OK, there were two more, providing threading and Python support, but they were somewhat ugly). After the changes, you can find this:

• devel/boost-build: Provides the bjam utility, the Boost.Build's core program. This is just a build dependency for all the other packages.
• devel/boost-docs: Installs all documentation files; made optional because they are quite big and you can usually access them online.
• devel/boost-headers: Provides all the header files required at build time. This is also a build dependency and, due to the nature of Boost, many libraries are only required at this stage (i.e., the final program won't need any extra libraries to run).
• devel/boost-libs: Builds and installs all binary libraries in threaded form, except Boost.Python (to avoid a dependency on Python when not wanted).
• devel/boost-python: The Boost Python binary library. This package is very similar to the one we had before, but now works.
• meta-pkgs/boost: A meta package that depends on all of the above, to ease the installation of a complete Boost suite.

Furthermore, there are also some internal changes in the way the libraries are built. On the one hand, threading is always enabled; not doing so adds too much complexity and breaks Boost.Python. On the other hand, all stuff is installed unversioned; this eases a lot the build of programs outside pkgsrc. You can also find some other changes, but these are, maybe, the major ones.

Enjoy!

Tabulators vs. spaces

Code indentation is a very personal thing; people use 2, 4, 8, or any other amount of whitespace to align their constructions. However, few of them pay attention to the usage of spaces or tabulators to accomplish what they want. Some others do, but problems remain. Let's analyze what these are:

Many text editors try to be smart and replace groups of eight spaces by a tabulator character. This may seem like a good idea to reduce on-disk space, because the resulting document will usually look the same. But IMNSHO, this is a very big mistake. The code ends up with such a mixture of spaces and tabulators that it's almost impossible to edit them easily, specially if you have to change the indentation of a block of code (either manually or automatically). Consider the following chunk of code:

void function(void) {
       if (...) { // 1 tab
               statement1; // 2 tabs
               statement2; // 16 spaces
               statement3; // 1 tab 8 spaces
       } // 8 spaces
}

Ridiculous, isn't it? Of course it is, but it happens a lot. Now, suppose Joe the user sets up his editor to show tabulators as two spaces. He will see the following:

void function(void) {
 if (...) {
   statement1;
               statement2;
         statement3;
       }
}

What a neat result. It gets even worse if the original writer does not use 8-spaces for tabulators. Just try to do a less over the file afterward; you'll be scared.

Some other people think that, by using tabulators, people will be able to read the code following their indentation preferences; i.e., adjusting the tabulator size (what we did in the previous example). Hahaha, what an utopia!

The first problem is that lines will be too long (or too short, but this is not a problem). Suppose you configure your editor to treat tabulators as two spaces. If you care about readability, you will write text until you hit the right margin of the screen, and break the line at that point. OK, good. But now, another reader looks at the code with a size of eight, and all lines wrap into multiple ones.

The second problem is that, to achieve what you want, you will have to pay a lot (and I really mean that) of attention to indentation, basically because your editor will keep doing nasty things. Consider this chunk of code:

void function(void) {
       if (...) { // 1 tab
               (void)printf("very long string", // 2 tabs
                            arg1, arg2); // 3 tabs, 5 spaces
       } // 1 tab
}

Note that we originally wanted to align the line containing arg1 to end up just after the parenthesis of the printf call. That line has three tabulators, but its indentation level is two. Therefore, it should only have two tabulators and 13 spaces. Otherwise, Joe looks at that code again, with a setting of two spaces per tabulator, and he sees:

void function(void) {
 if (...) { // 1 tab
   (void)printf("very long string", // 2 tabs
          arg1, arg2); // 3 tabs, 5 spaces
 } // 1 tab
}

Again, an ugly result.

Conclusion: handling code with mixed tabulators is difficult; too difficult to even worry about it. I've seen many projects (Boost, for example) banning their usage, and I do understand them. Personally, I use an indentation level of four spaces and completely disable tabulators from my editor, usually Vim. If you wonder how to do it, add the following lines to your ~/.vimrc file:

set expandtab
set shiftwidth=4
set softtabstop=4

And here are some minor goodies for GNU Emacs, which can be placed in your ~/.emacs file:

; Set the default fill column to something wider.
(setq default-fill-column 75)

; Highlight trailing whitespace in lines.
(setq-default show-trailing-whitespace t)

; Do not use TABs to indent, unless a buffer overrides it explicitly.
(setq-default indent-tabs-mode nil)

Edit (Feb 27th): Some minor changes in the code examples' comments.

Boost: Shared pointers

The Boost Smart Pointers library provides a set of classes to manage pointers. I've started using boost::shared_ptr and am very happy with the results. Let's see what it is.

shared_ptr is a simple templated class that holds a pointer to a dynamically allocated object, which is reference counted. The object it references is automatically destroyed when the last instance of the smart pointer is removed; this usually happens when it gets out of its code scope.

So the first and most useful application of this class is to remove (almost) all possible memory leaks from your code. You do not need to issue explicit calls to the delete operator any more, since the smart class destructor will do it for you. In other words: you get a cheap garbage collector in your code which requires no external functionality and which has a very small performance penalty.

Another use of it (and this is the main reason why I started using this class) is to make some interfaces (function prototypes) easier to understand. Consider the following class structure (coming from VCS Made Easy:

struct entity { ... }
struct tree : public entity { ... }
struct group : public entity {
   void add_entity(entity* e);
   ...
}

group is an entity container, as you can guess from its name and from its add_entity method. Note that this method takes a pointer to an entity object. Now the doubt arises: does this pointer refer to a dynamically allocated object or to something else (i.e., on the stack)? That is: will the class take care to delete all objects it holds or will leave that task to the caller?

In my first implementation, the function was meant to take a dynamically allocated instance, because group's destroyer manually took care to delete all of its contents. This destructor could produce errors if the user inserted an stack object in the container, because delete could not be applicable. Thus I had to stick a big comment near the function definition warning the user.

Using smart pointers, the interface becomes clearer, and there is no possibility to make the mistakes outlined above. The prototype becomes:

void add_entity(boost::shared_ptr e);

It's now clear that the function takes a pointer to a dynamically allocated object (due to the nature of the smart pointers). Anyway, if it referred to a locally allocated object (I'm not sure if shared_ptr can do that), this could cause no problems because only the last instance deleted could cause the real object to be freed (no matter in which order the objects were destroyed). Plus we've also got ridden of group's destroyer!

Therefore, this is an easy way to make your code safer and clearer. Note that there are several other interesting smart pointers that you may want to look at, but I'll leave those to the documentation.

Boost: First impressions

Despite my previous post, I decided to give a try to Boost and switched VCS Made Easy to use some of its libraries; the modified source code is not yet on the public server (hmm, the joys of Monotone), but I think it'll be soon.

I must say that the results are very satisfactory. At first, I dropped my custom file and directory classes in favor of Boost.Filesystem. During the conversion, I had to rework some of the existing code to comply with Boost's guidelines WRT global (environment) variables and portability. The code is now much cleaner, easier to read and probably less error prone (I still have to add unit tests).

Then I substituted several usages of std::ostringstream with printf-like formatters from Boost.Format. Using these, it's very easy to construct text messages, which also results in easier to read code. I think I'll convert many other strings to use this.

Another thing I've used is the boost::noncopyable class (from Boost.Utility) to easily mark several classes as non-copyable. This restriction was not in the code before, but it has to be because these classes are not safe to copy operations (pointers get duplicated, for example).

There are many other goodies I've discovered and which are worth using, but haven't got the time to implement them yet. Some examples include the smart pointers, the checked deletes or the whole testing framework (from Boost.Test).

I must also say that the documentation is excellent and everything is very well thought due to the peer-review. All libraries come with an explanation of their methods, the design choices made during their development, explanations about the rationale behind things that may look strange at first look, portability suggestions, FAQs...

Let's see what else I discover... but I think the doubts I had yesterday have been simply resolved :-)

Boost: To use or not to use it

For at least two weeks, I've been rewriting VCS Made Easy from scratch (and it's already working fairly well), a project I announced a long time ago. It's being written in C++ and the code is organized in three (independent) layers:

• The helper library: This provides classes to access operating system specific details. Among others, it has classes to manage processes, users, files, directories, etc. Everything is very simple and, depending on how you look at it, incomplete. I will probably need to add much more functionality to the existing classes (specially, the ones to manage the file system are very poor) and some new ones.
• The vcsme library: This includes all the logic of the program in a frontend-independent way and uses the helper library.
• The console frontend: Uses the vcsme library to service user requests. It does almost nothing on its own, aside parsing the command line.

So far, so good. But now I realize that many things I'm doing in the lowest layer, the helper library, are already implemented in Boost. More specially, what I'm interested in is the Boost Filesystem library. However, if I decide to use this one, I may end up using other functionality, such as the Boost Test library (due to its execution monitor and the unit tests framework) and the Boost Format library (to remove several ugly usages of std::ostringstream). These classes are more tested, more portable and more complete.

Unfortunately, there are some disadvantages. First of all, the Boost source package is huge (around 10Mb). And the binary ones are too, depending on which packaging system you use. This is solvable, though, if package maintainers split the Boost package into multiple small parts (like Debian does. (And I'll probably do it in pkgsrc.)

But the biggest inconvenient I find is that Boost does not cover all my needs, so I might be using it for a few stuff, but still need a lot of custom classes. AFAICT, there is no library to manage processes nor users, a thing I need for this specific project. However, I also think that if I keep the code clearly organized, these custom classes may, some day, end up being part of Boost (and this could be really, really nice).

So now I'm faced the choice of whether to use Boost or not... any comments?

Passing information to configure scripts

GNU Autoconf provides three ways to pass information to the generated configure scripts; these are enable-style flags, with-style flags and command line variables. Despite each one is provided with a very concrete goal in mind, many people overloads their meaning. Their purpose is clearly described in the manual, which these people "forget" to read. (I know it's impossible to read everything one would like to... not an excuse in this case, I'd say.)

So let's see which is the real purpose of the above methods:

• Enable-style flags: these are provided for packages that have optional compile-time features. They are defined using the AC_ARG_ENABLE macro. An example could be a --enable-debug argument. Note how this could change the behavior of the current package only, without adding additional dependencies nor requirements.
• With-style flags: these are provided for packages that require, or can optionally use, other software packages that are already installed. They are defined using the AC_ARG_WITH macro. An example that you can see in many packages is --with-x, used to manually enable or disable X11 support. Note how X11 is always an external package, hence the use of a with-style argument.
• Command line variables: anything else you can't do with the previous arguments. They are defined using the AC_ARG_VAR variable and are passed to the configure script as normal arguments. Most configure scripts provide some default variables defined this way. E.g., CC, which could be used as: ./configure CC=gcc, not CC=gcc ./configure. Be aware that both will work, but the semantics are quite different; I encourage you to read the manual.

Now let's analyze a problem I fixed two months ago, in which I required to pass the configuration script some information. For some "strange" reason, the Boost package creates libraries with different names (in fact, different suffixes) depending on the system where they are built on. Due to this, Monotone failed to build under NetBSD (or even SuSE Linux because it didn't know it had to use a different suffix.

I tried to fix it in an automated way, but AFAICT, it's almost impossible (or, at the very least, very error prone). So I decided to do it manually by passing the required information (the suffix name) to the configure script. Some people may have used a with-style argument, say --with-boost-suffix=<string>, but that'd have been incorrect. Remember that these arguments are to enable or disable external dependencies, but in this case we are not doing that. An enable-style argument would have been equally incorrect.

So I had to add a command line (environment) variable, called BOOST_SUFFIX, which takes the appropriate string for the current build system. The result is that, when needed, you can call the configure script with an extra argument that specifies how the library is named. I.e., what I have to do ATM is: ./configure BOOST_SUFFIX=-gcc.

How to get the user's home directory

Many programs need to access the user's home directory to look for files, specially configuration ones. To do that, they usually get the HOME environment variable's value and use it to construct the full path. However, if the application is security-sensible, this is probably inappropriate.

What happens if such application is run through sudo or with the setuid bit set? Let's see it:

[dawn jmmv] $ sudo /bin/sh -c 'id -un ; echo ${HOME}'
root
/home/jmmv

Oops! The application with extra privileges is getting the home directory of the unprivileged user! This is a serious problem, because the user could simply create a malicious configuration file (that exec's a third program, for example) and use it with more privileges than he has.

Fortunately, the user's home directory can be guessed using a safer method: that is, by reading its value from the /etc/passwd file. We only need to be careful to get the entry that matches the actual effective user (not the real one). How'd we do it?

struct passwd *p = getpwuid(geteuid());
(void)printf("Home directory is: \n", p->pw_dir);

If we now put the above in a little test program and run it, we get:

[dawn tmp] $ ./a.out
Home directory is: /home/jmmv
[dawn tmp] $ sudo ./a.out
Home directory is: /root

Well, I'm not completely sure this is the best way to go (i.e., if there is any way to still get a value decided by the user), but so far I think this is safer than just reading HOME's value. If anybody knows of any drawbacks, please share!

C++: Verifying program sanity

The C language provides a macro, called assert, that is used to verify conditions that must be true at any point of the program. These include preconditions, postconditions and invariants, all of which are explained in introductory programming courses. Whenever an assertion is violated, the program is abruptly stopped because there is most likely a bug in its code.

C++, like C, also provides the assert macro in the cassert standard header. However, there is another way to verify program sanity without using it. How? Using exceptions.

The std::logic_error exception is provided with this situation in mind: to verify that a certain condition holds true at a specific point of the program. If it does not, the program logic (hence the name) is incorrect and must be fixed. Using this method has some advantages, but also some disadvantages. Let's see them.

Using an exception, you can provide a descriptive error message of what failed. The assert macro only prints out the assertion triggered, which causes difficulties to decipher what really made the assertion false. Using an exception, you can add more information, such as the value of an specific parameter (in the case of a precondition) to let the developer know what's wrong.

Furthermore, when you catch one of these exceptions (you'd usually do that from your program's main function with a global try/catch block), you can tell the user that something went wrong, where to report the bug, what information to include, etc. In other words: you can decide how the program will end, instead of simply getting an abort trap.

However, there are some drawbacks in this approach too. The first one is that you'll have to manually create some macros to verify an assertion and throw the correct exception. This is to allow you to quickly disable all of them when building production code; otherwise you'd slow down your program quite a bit (it may not matter at all depending on the nature of your program).

But, maybe, the most important problem you can encounter is that you cannot use this method from within destroyers, as explained here. It may sound strange, but a destroyer can rightfully expect some preconditions to be true.

The method you use is up to you: it can be one of these two or a custom one you write (such as global functions used as logic error handlers). But, whichever it is, add assertions to your code.

C++: Containers of pointers

One of the things C++ provides to programmers is that pointers can be avoided in much situations, and IMHO, they should be (specially in public class interfaces). However, there are some times in which pointers must be used.

Consider an abstract base class and multiple specializations of it; if you want to define a container of all these objects, no matter which derived class they belong to, the container must hold the parent's type. But you can't use an object (nor a reference) of this type, basically for two reasons: you can't instantiate one and, even if you could (no abstract members), the derived object could loose all of its specific properties when converted to the parent one.

In the above scenario, you are required to use pointers. And, when doing so, you hit a problem: the operations that are applied to containers will work at the pointer level, not at the object level, so all comparisons will be wrong. Let's clarify this with an example.

Suppose we have a set of pointers to strings and we add several objects to it:

std::set<std::string*> s;
s.insert(new std::string("String 1"));
s.insert(new std::string("String 2"));
s.insert(new std::string("String 3"));

We are now asked to check whether "String 2" is part of the set (in this example we are checking for equality). You can't use the s.find operation, because it will compare pointers rather than strings; so unless you know the address of the object you are looking for, you won't be able to find it. A solution could be to manually iterate all over the set and check each element against the string we want. Ugly, isn't it?

Fortunately, we can use the standard algorithms, but we will have to construct a custom predicate that, given two pointers, compares their contents. Our predicate could look like the following (requires the functional header):

template <class T>
struct unref_equal_to : public std::binary_function<T, T, bool>
{
   bool operator()(const T& x, const T& y) const { return *x == *y; }
};

Despite the operator taking two references, it must take pointers (because T will have to be a pointer type); otherwise, nasty things will happen. Then, all it does is apply the == (equality) operator on the unreferenced objects and return the result, which will do what we want (in this case, compare the string contents).

Now, how do we apply this predicate to look for a specific string in our previous set? Easy; we need the std::find_if standard algorithm, which takes a predicate as an argument. Let's see what we'd do:

std::string what("String 2");
std::set<std::string*>::const_iterator i =
   std::find_if(s.begin(), s.end(),
                std::bind2nd(unref_equal_to<std::string*>(),
                &what));
if (i != s.end())
   std::cout << "String 2 is part of s!" << std::endl;

Don't let this scare you, specially the std::bind2nd part. This function is used to set the second argument of the binary function we created above (which has some special properties). OTOH, the std::find_if algorithm will take care to fill in the first argument on each iteration it does over the set, so we will get the expected comparisons.

Guessing Tcl/Tk configuration

When a program uses Tcl and/or Tk, its configuration script (if any) has to check for the presence of these libraries and retrieve some information about them, such as their version, the required link flags, etc. Unfortunately, the process to achieve this is rather obscure (AFAIK), thus it is unknown by many people. This results in unportable (and broken) configuration scripts.

So how is this done? Both libraries install a shell script which carries all the required information, so all you have to do is find the script, read it, and use the variables you need. These scripts are named tclConfig.sh and tkConfig.sh respectively, and are placed in the lib directory relative to their installation prefix. I.e., most of the times, they'll be placed in /usr/lib, /usr/local/lib or other locations such as /usr/pkg/lib in NetBSD.

Let's see a bit of code that shows how to get Tcl's link flags using GNU Autoconf:

AC_ARG_WITH(tcl,
           AS_HELP_STRING(--with-tcl=PREFIX, [Enable Tcl support]),,
           with_tcl=yes)
if test ${with_tcl} != no; then
   if test ${with_tcl} = yes; then
       tcl_dirs="/usr/local /usr"
   else
       tcl_dirs="${with_tcl}"
   fi

   AC_MSG_CHECKING(for tclConfig.sh script)
   found=no
   for d in ${tcl_dirs}; do
       if test -f ${d}/lib/tclConfig.sh; then
           found=yes
           break
       fi
   done
   AC_MSG_RESULT(${found})

   if test ${found} = no; then
       AC_MSG_ERROR(Tcl support requested but the library cannot
                    be found)
   fi
   . ${d}/lib/tclConfig.sh
   AC_MSG_NOTICE(using "${TCL_LIBS}" to link to Tcl)
   AC_SUBST(TCL_LIBS)
fi

The above code snippet does the following:

1. It defines a command line argument (--with-tcl) to enable or disable Tcl support. This flag accepts a directory name as its argument (aside yes and no) which is treated as Tcl's installation prefix. Note that, if Tcl support was not optional, you'd use AC_ARG_VAR to specify the prefix rather than a flag (because neither of "enable" nor "with" semantics match what you'd want to do).
2. If Tcl was enabled, it checks whether the user gave a path or simply let the argument take its default value. In the later case, the script falls back to a sane list of directories; otherwise, only the directory given by the user is used.
3. It walks the directory list looking for a tclConfig.sh script and, when found, stops the process.
4. If the script wasn't found, the configuration is aborted with a descriptive error message.
5. If the script was found, it's read and the TCL_LIBS variable is added to the substitution list so that it's available from the Makefiles. We'd do this for any other variable provided by the script. At this point, you'd also check for the detected version (TCL_VERSION and other variables) and compare it to your requisites.

Of course, all of the above also applies to Tk, just changing the name of one library for the other.

C++: Exceptions and destroyers

If you use exceptions to signal errors in C++, you should be really careful when throwing an exception from inside a class' destroyer function, because you can easily cause an abort trap. Consider the following code:

struct test {
  ~test(void) {
      throw int(0);
  }
};

int main(void) {
  try {
      test t;
      throw int(1);
  } catch (int e) {
  } catch (...) {
  }

  return 0;
}

Trivial, isn't it? Well, now compile it and run it: you will get a "nice" core dump. However, if you disable the throw statement inside ~test, the problem will go away. The problem will also disappear if you move test's instantiation outside the try block.

But... why does this happen? When the int(1) exception is thrown, the t object has to be destroyed before it can be handled, because its scope has to be cleaned (see how it's placed inside the try block). When this happens, the destroyer function throws a new exception (while we were handling another one), which could produce an infinite loop. Voila, you've got the abort trap.

So be aware when throwing an exception from a destroyer. Simply put, you must make sure that they will not generate exceptions; you have to handle any possible failure from within them.