Brian Gough’s Notes

It’s one month till the next GNU Hackers Meeting in Paris (Thursday 25 August – Sunday 28 August). There are about 45 GNU maintainers and contributors registered so far (details of how to register are on the meeting web page).

Speakers include Jim Meyering on “The perils of relying on output streams in C”, Stefano Zacchiroli (Debian Project Leader) on “Distributions, upstreams and downstreams” and Jim Blandy (Mozilla) on “Extensibility in Firefox” among others.

The meeting is hosted by IRILL, the Initiative de Recherche et Innovation sur le Logiciel Libre at the INRIA building near the Place d’Italie. Looks like a great location:

Here’s a list of pre-release checks I’ve found useful over the years. Most of them should apply to any package, but they’ve been used for a C library (“libfoo” in the examples below, with exported header files foo/*.h and source files src/*.c).

• Make sure all internal functions are static, and all exported functions & variables prefixed with foo_ or FOO_. Here’s a command for checking this:

  nm -A -g -P *.a  | perl -a -n -e 'print if $F[1] !~ /foo_/ && $F[2] ne "U"'

• Make sure config.h is used consistently – it should be present in source files, but not in any exported headers.

  grep config.h foo/*.h    --- shouldn't match anything
  grep -L config.h src/*.c   --- gives files not using config.h

• Make a master header file of all exported headers and try compiling it to test for conflicts between different header files.

  cat foo/*.h > tmp.c
  gcc tmp.c

• Check that the library passes “make check” with various options. Some good ones are:

  -O3
  -mfpmath=sse -msse2
  -funsigned-char (to simulate Power/ARM)
  -std=c89 -ansi -pedantic (now that GCC uses c99 by default instead of c89)

• Test with valgrind. You can run all the tests under valgrind with the TESTS_ENVIRONMENT variable

   $ TESTS_ENVIRONMENT="valgrind --tool=memcheck" make check

  This only works if configure was run with –disable-shared, because otherwise the shared library libtool shell-scripts themselves are run under valgrind.

  Regarding memory leaks, the GNU coding standards say “Memory leak detectors such as valgrind can be useful, but don’t complicate a program merely to avoid their false alarms. For example, if memory is used until just before a process exits, don’t free it simply to silence a leak detector.”

  For libraries I’ve found it’s very helpful (actually, essential) to have a test suite that is leak-free. Then if any test program exits with memory still allocated it indicates a problem. This is how I have caught any memory leaks during development.

• Do make distcheck the local copy of install-sh provided by automake, in addition to /usr/bin/install. Depending on the end user’s environment, either of these could end up being used.

  make distcheck # will use /usr/bin/install
  make distcheck INSTALL=$HOME/foo/install-sh  # absolute location of install-sh

  When you do “make install” yourself, it will probably use /usr/bin/install since that is present on GNU/Linux systems. If so, you would not detect any problem with the install-sh shipped in your package – for example, if it had got corrupted for some reason (e.g. a stray edit or in my case, after doing a repository conversion I somehow ended up with my working directory containing the first ever version of install-sh that was checked in – which dated from 1994 – it didn’t work properly on modern systems, but I never noticed that because I was always using /usr/bin/install).

• Try running the build and make check with a memory limit to be sure that excessive memory is not required to do the compilation.

  ulimit -v 65536  #  some reasonable memory limit
  make
  make check

• Check that the texinfo manual works for postscript/pdf

  make dvi  # or make pdf

  It’s easy to miss a diagram or other texinput file that is needed by texi2dvi but not by makeinfo.

At FOSDEM, Alfred (ams@gnu.org) came up with the idea of “Autocache” – a system-wide precomputed cache of autoconf tests that makes every configure script run instantaneously. A crazy idea that cannot work completely in practice, of course, but there might be something to it.

He sent me an implementation which works like this: you make a configure script from all the autoconf tests in Gnulib (excluding some obvious ones that should not be cached). The idea is to pick tests which are unlikely to change on a given system. You run this huge configure script once and make a copy of the resulting config.cache file. You then use this as an initial config.cache for any package you’re compiling.

I tested it on GSRC and it worked surprisingly well – out of ~100 packages there were only a handful of failures where somehow the cache had bad results. Here are the timing results for the top ten longest-running GNU configure scripts with/without autocache.

In his presentation at FOSDEM on “Power, Freedom, Software. Why we need to divide and re-conquer our systems” Karsten Gerloff talked about the difference between decentralised systems, which are networks of client-server systems (such as Email), and distributed systems, where all nodes are equal (such as peer to peer filesharing networks like GNUnet).

If the client and the server of a decentralised system are both free software then anyone can start their own server and be free from control. In practice, decentralised systems have a tendency to become centralised, and in some cases highly centralised for most users.

The tendency to centralisation is a result of two factors: asymmetric costs and economies of scale.

1. Asymmetric costs: in client-server systems, the server generally requires more administration that the client. Users have an incentive to transfer administration to a third-party, either for a fee or trading away privacy, in the case of advertising-supported services.
2. Economies of scale: server administration is subject to significant economies of scale. Larger providers drive out smaller providers over time, leading to a high degree of centralisation.

These two factors form a positive feedback loop, as larger providers can offer better services at lower cost, thereby attracting more users.

As a specific example, consider email. In the early days, each group of users had their own mail server at their local site. Over time, email has been outsourced to ISPs, ISPs have merged and been taken over, and finally been overtaken by a small number of global mail providers with userbases in the hundreds of millions.

Although the factors driving this process are universal the degree of centralisation is not uniform, but depends on the relative cost of administration and value of privacy for different users. For example, email has become effectively centralised for individual non-technical users, where the relative cost of running a mail server is high and the perceived privacy problem is low. For large companies, the relative cost of running a mail server is lower, confidentiality of corporate information is valued, and most maintain their own mail systems in-house.

In his talk about Autotools at the FOSDEM GNU Dev Room, Ralf Wildenheus mentioned an autoconf option that I wasn’t aware of. It’s the “-C” option and it enables caching, which speeds up multiple runs of a ./configure script enormously.

$ ./configure --help
     --cache-file=FILE   cache test results in FILE [disabled]
 -C, --config-cache      alias for `--cache-file=config.cache'

$ ./configure -C
configure: loading cache config.cache
...

Caching used to be the default up to about 10 years ago, but was turned off as the cache gets out of sync with the system if new packages are installed. Also it really only benefits developers, as end-users typically only need to run ./configure once.

In the 1990’s I got used to caching being enabled by default and when it was turned off I never noticed the change — except that somehow running ./configure while developing seemed a lot slower in the past ten years!

Thanks to Ralf for pointing out this option as I am now using -C all the time.

The GNU dev room at FOSDEM 2011 was great, the room had 100 seats and every talk was packed out with people standing at the sides.

The GNU talks were:

Org-Mode: your life in plain text by Bastien Guerry on the amazing personal organizer mode for emacs

Dynamic hacking with Guile by Andy Wingo on the new release of Guile 2.0 and hacking a twitter clone (“Peeple”) live in 15 minutes using Guile’s new web framework – amazing,.

GNU Autotools by Ralf Wildenhues with lots of useful tips for autoconf and automake.

GNU Network Security Labyrinth by Simon Josefsson, a nice tutorial on how different security protocols have developed over time, in order to solve the problems with earlier attempts.

GNU recutils: your data in plain text by Jose E. Marchesi, tools to work with your data on the command-line.

There were also two talks from FSF Europe,

Power, Freedom, Software. Why we need to divide and re-conquer our systems by Karsten Gerloff.

Non-free software advertisement – presented by your government by Matthias Kirschner, about the successful PDFreaders.org campaign.

Thanks to José and Andy for organising the day, it was a great set of topics.

Karsten Gerloff talking about freedom and the cloud.

Simon Josefsson finishing his talk

The room

GNU Parallel is a nice recent addition to my unix toolkit. It can do a lot of things, my favorite is as a “map” operator for the command-line, automatically parallelising over the arguments of a command.

The command

parallel command {} ::: arg1 arg2 arg3 ...

is equivalent to

 command arg1 &
 command arg2 &
 command arg3 &
 ...

run in parallel. The {} is replaced by each argument in turn.

As part of the regular updates to GSRC I need to run “make check-update” for several hundred directories for all the gnu packages. With a for loop that takes a long time, as “check-update” examines remote urls to see if a new version of a package has been released. With parallel I can use

parallel make -C {} check-update ::: gnu/*/

to get the same effect but in parallel (by default 9 processes simultaneously – to change it use -j N).

The author of GNU Parallel, Ole Tange, is giving a talk at FOSDEM (Brussels) in the GNU Dev room on Saturday 5 Feb 2011, see http://www.gnu.org/ghm/2011/fosdem/ for details. Hope to see you there!

I’ve published a paperback edition of the Org Mode 7 Reference Manual (subtitled “Organize your life in GNU Emacs”). For each copy sold $1 is donated to the Org project.

I’ll be going to FOSDEM in February 2011 (Sat 5 – Sun 6) for the GNU Developers session on Saturday.

This year the GNU project has its own room for up to 100 people and the organisers are looking for GNU developers to give a talk about their package (see the GNU fosdem webpage for details) – the deadline for abstracts is 22 December. It is a good chance to tell others about your work and meet potential contributors, as the FOSDEM event attracts thousands of developers.

Whether you want to give a talk or not, you should email fosdem2011@gnu.org with your contact details if you plan to attend so the organisers can keep you updated.

The Software Freedom Law Centre has published Eben Moglen’s testimony to Congress on privacy and social networking.

. . . Facebook and similar centralized social networking services like to talk about their “privacy settings.” This is mere deception, a simple act of deliberate confusion. These “privacy settings” merely determine what one user can see of another user’s private data. The grave, indeed fatal, design error in social networking services like Facebook isn’t that Johnny can see Billy’s data. It’s that the service operator has uncontrolled access to everybody’s data, regardless of the so-called “privacy settings.” . . .

Testimony of Eben Moglen, December 2, 2010, US House of Representatives Subcommittee on Commerce, Trade & Consumer Protection