Unprivileged pkgsrc builds

As you already know, pkgsrc can be used to build a large variety of software. What you may not be aware of is that it can be used without super-user privileges; i.e., as a regular user, you can take advantage of pkgsrc to build software in systems you do not own! (Provided that you are allowed to do that, of course.)

The "problem" is that it has never been easy to do so (at least not for novice users): you had to override many variables in mk.conf file to avoid setting forbidden ownerships, strict permissions, etc., with the risk of forgetting something important.

With the changes I just commited, it should be easier than ever. If you already have a running installation, setting UNPRIVILEGED=YES in mk.conf should DTRT, and you may be able to remove many of your configuration. (Check the mk/unprivileged.mk file for details about what it does.)

In case you are bootstrapping a new installation, just do ./bootstrap --ignore-user-check. This will tell the script that you want to do a non-root build, so it will enable the unprivileged mode and choose several default directories for you. For instance, it will use ~/pkg as the default prefix (changeable through --prefix), and all other pkgsrc-related directories will be made relative to it.

For more information, check out this FAQ (may take an hour to reflect the changes).

Distributed streaming

Data streaming, as known today, lets a single machine send real time data to a number of clients. This number is limited due to bandwidth constraints, as it is expensive (specially on the upload channel). The problem that arises is that individuals or little associations can't set up reliable streaming servers, as they won't be able to pay the costs for it (thus only being able to serve a very limited number of clients).

Where do I want to get? Well, two friends and I have been working on a little project (as a university task) to improve this situation. The idea is to apply the peer-to-peer networking scheme to data (multimedia) streaming, so that each client also acts as a server. This way, the number of clients is theorically unlimited; however, the reception latency increases and you may suffer data cuts if a node drops (the time needed to search for and connect to another peer).

While testing our application, we set up a server on a 56kbits line and we were able to connect more than 5 clients to it (yeah, it's a non-impressive number, but we didn't have more machines to test), receiving each one a constant data flow of (almost) 4Kbytes/s. That is an upload bandwidth of 20Kbytes/s, impossible to reach by our telephonic line (which barely serves the flow needed by the stream).

BTW, the application is network transparent, so it works with existing applications and protocols; ATM, it only handles SHOUTcast, as it's the protocol we focused on, but it should be extensible to others without much problems.

I don't know if we will make the code available (I have nothing against it, and I think my friends won't either). The thing is that I'm afraid it will suffer from stagnation, just like happened with our previous university-related project (Vigipac).

So, if you found the idea interesting, check out Peercast, a more mature application. Hmm... why did we reinvent the wheel? Well, we discovered it just a few days ago, when the project was already finished.

Styles in word-processors

When writing a long document in a word-processor... have you ever seen people creating a table of contents by hand? Numbering pages manually? Having to change the font face of their section headers one by one? Poor guys; I often feel annoyed because word processors impose no rules on how to write documents, and people don't care to discover all the marvelous features they provide. Yes, I'm talking about styles.

Styles are a way to organize your document logically (as in chapters, sections, paragraphs, etc.). When using styles, you don't have to — or ideally, you mustn't have to — care about how your document looks during its writing, only about its contents. All you have to do is tag special parts of it as titles, notes, outlines or any other kind of logical constructions that make sense for it.

But why? First of all because changing the whole look of your document is a matter of editing the specific style you used for each "logical construction". For example, if you wanted to leave 0.40mm between paragraphs, all you'd have to do is edit the "Paragraph" style to reflect your change and the entire document would benefit from it. On the other hand, the word processor understands how your document is internally organized, so it can do a lot of work for you, like generation of indexes, calculation of references to sections or figures, etc. You read it correctly: all these things can be automatically done by the program!

If you have ever written a page using XHTML Strict or Docbook/XML (to name a few), you know what I'm talking about. Styles are based on the same idea. For example, you are not allowed to use all those problematic tags present in HTML 4 (font being the main offender), because they are meant to change the look of your writing. In order to change how the document is rendered, you either use a CSS or an XSL, which define how each logical concept will be drawn.

To conclude: (almost?) all word processors support styles; it's a matter of looking for them and start using them. However, Openoffice is specially good at this (it could be improved in some areas, though); take a look at its Stylist, which is open by default. At last, a tip: remove the "formatting toolbar" from the program's window and forget about the "Format" menu; you will avoid many bad practices.

Shutting up /etc/security

For a long time, I've been receiving, almost daily, useless security reports from my NetBSD boxes. With "useless", I mean notifications of changes that I know are perfectly legitimate. That is, all they contained was:

======
/etc/resolv.conf diffs (OLD < > NEW)
======
--- /var/backups//etc/resolv.conf.current       2005-05-21 [...]
+++ /etc/resolv.conf    2005-05-22 [...]
@@ -1,3 +1,3 @@
-# Created by dhclient at: Sat May 21 15:08:21 CEST 2005
+# Created by dhclient at: Sun May 22 13:41:19 CEST 2005
 search home.network
 nameserver 127.0.0.1

As I said, I know that the resolv.conf file changes daily due to every run of the dhclient utility, so don't bother me telling that! What I wanted was to remove the date from resolv.conf's header, in order to avoid differences between runs unless something else really changed. After a bit of research I found a way to fix.

The thing is that dhclient runs a special script, called dhclient-script, which does some pre/post processing over network configuration. This script is not meant to be edited, but it allows to place functional hooks under /etc/dhclient-enter-hooks and /etc/dhclient-exit-hooks. And this is what I did: I created the /etc/dhclient-exit-hooks file on my machine to remove the date from the generated file:

if [ -f /etc/resolv.conf ]; then
   sed -e 's|^# Created by dhclient at:.*$|# Created by dhclient|'    </etc/resolv.conf >/etc/resolv.conf.save
   cp /etc/resolv.conf.save /etc/resolv.conf
   rm /etc/resolv.conf.save
fi

I'm not sure if this is the best way to solve this little problem, but at least it works.

pkg-config: Sanity-checking package dependencies

A while ago, I posted a series of pkg-config related messages: 1, 2 and 3. They were intended to give some background knowledge about that program, to let me easily explain something else: how pkg-config can be used to sanity-check packages — that is, to ensure that the libraries they depend on are the correct ones.

In pkgsrc, direct dependencies are specified by including special buildlink3.mk files. These files have a default version specification in them, so that a reasonable value is kept somewhere centralized. For example, in the current tree, when you include glib2's buildlink3.mk file, you get a dependency such as glib2>=2.6.1. So far so good.

Now suppose that you are packaging a program which requires, at least, version 2.6.4. You need to make sure that the new package specifies that, like in glib2>=2.6.4, to avoid problems if an older version is installed. The problem is that manually checking this is very difficult; if the developer has the required version (2.6.4) already installed, he won't notice that the configuration script is requiring a newer version than the one specified in the buildlink3.mk file.

Similar problems arise with indirect dependencies. Consider a package which requires libgnomeui and gtk2. If this package includes libgnomeui's buildlink3.mk file alone, it will build correctly, because this will automatically pull in gtk2's buildlink3.mk file. However, this is incorrect, because the resulting binary package will miss a direct dependency on gtk2. Discovering these issues is quite difficult.

The thing is that I have found a way to notice these problems in an automated fashion (rather than manually reviewing tons of code in configuration scripts). Given that the configure scripts (usually) call pkg-config for each direct dependency, it's a matter of capturing these calls, storing them in a log file to compare them later on with the package's specification files.

To achieve this, I've written a little patch for pkg-config which makes it write a log file with all the queries it receives and their results. Then, I've made pkgsrc use the patch to generate such file inside WRKDIR. And finally, I've written a script, called verifypc, that compares the log file with what the package's Makefile contains.

It saves me a lot of time when updating packages to newer versions, as I can quickly see if I need to modify the dependencies. However, the script still has multiple problems (such as thinking that 2.9 is newer than 2.10) and, frankly, is very ugly. I guess I'll have to clean up this stuff and include it in pkgsrc ;)

Recent pkgsrc changes

Eww, it has been a very long while since I wrote the previous post, and even more since I did any serious changes in pkgsrc. Real life tasks get in the middle and take too much time (fortunately, this stressful period is about to end). However, during the last few days, I've been able to work a bit on pkgsrc, so I'd like to comment the changes I've done.

First of all, GNOME has been brought up to 2.10.1, the latest stable release in the 2.10 branch. It should fix many bugs — specially one that has been bothering me a lot in Epiphany, as it failed to automatically open files after their download. Note this has been possible, mainly, due to all the updates Matthias Drochner has done; otherwise, I wouldn't have had the time to do everything at the moment.

Secondly and lastly, I've reworked the ao packages. The base libao package is now completely driver-agnostic; in other words, it only installs the core library, with no extra plug-ins. The reasons? On the one hand, simplification of the package's code; on the other hand, less chances for it to break on foreign systems (think about the old ugly hacks to handle the PLIST). Of course, to compensate the change, I've added three new packages: libao-macosx, libao-oss and libao-sun, which provide the correct platform-specific driver on the systems where they are allowed to be installed.