Brian Gough’s Notes

occasional GNU-related news

GNU Parallel – a map operator for the command line

GNU Parallel is a nice recent addition to my unix toolkit. It can do a lot of things, my favorite is as a “map” operator for the command-line, automatically parallelising over the arguments of a command.

The command

parallel command {} ::: arg1 arg2 arg3 ...

is equivalent to

 command arg1 &
 command arg2 &
 command arg3 &

run in parallel. The {} is replaced by each argument in turn.

As part of the regular updates to GSRC I need to run “make check-update” for several hundred directories for all the gnu packages. With a for loop that takes a long time, as “check-update” examines remote urls to see if a new version of a package has been released. With parallel I can use

parallel make -C {} check-update ::: gnu/*/

to get the same effect but in parallel (by default 9 processes simultaneously – to change it use -j N).

The author of GNU Parallel, Ole Tange, is giving a talk at FOSDEM (Brussels) in the GNU Dev room on Saturday 5 Feb 2011, see for details. Hope to see you there!