Features Download
From: Mark Seaborn <mrs <at> mythic-beasts.com>
Subject: Relocatability of autotools-based build process
Newsgroups: gmane.comp.file-systems.zero-install.devel
Date: Saturday 22nd March 2008 16:24:27 UTC (over 9 years ago)
Hi all,

I am investigating the possibility of building packages in bulk,
e.g. for all of GNOME's components, using a tool like JHBuild.

JHBuild currently knows how to "make install" components into a
--prefix directory.  I am investigating whether JHBuild could build
Debian and/or Zero-Install packages.  (There is a branch called
JHDebuild for building debs, but I have not found any description of
how it works.)

As an aside, the prevailing approach to packaging seems to be that:
 * Each component is packaged individually by hand.
 * The source package contains executable code, anywhere from 1 line to
   large shell scripts or makefiles.
 * This code tends to be boilerplate which is duplicated across all
   packages.  Libraries such as debhelper and cdbs can reduce the amount
   of boilerplate, but there is always a minimum of code that is
 * Debian packages are a good example of this approach.  Based on
   approach is simpler but similar.

I find JHBuild interesting because it avoids duplicating executable
code outside the upstream packages, and it lets the upstream code do
all the work.  JHBuild's XML files list SCM repositories and their
dependencies.  In the worst case, it lists special-case "configure"
options that are required.

The feasibility of building everything via Zero-Install comes back to
a question I asked last year:

I wrote:
> > Do you find there are many programs that can't be relocated by setting
> > environment variables?

Thomas Leonard wrote:
> Games mainly. They seem to like starting with 'cd /usr/games/...' for
> some reason.
(from http://lists.gnu.org/archive/html/plash/2007-03/msg00002.html)

I've been looking into whether the whole build process can be relocated.

i.e. Can you take a library, such as libxml2, and do the following:

 a) Install it under one path with "make install" (which installs the .so
    file and library headers).
 b) Move the installed files to another path, set environment variables
    to point to it, and build other components that use the library,
    e.g. libxslt, gnumeric.
 c) Move the installed files to a third, different path, set environment
    variables again, and run an executable (e.g. gnumeric) that was built
    in step (c) that links with the library.

The answer is that there are some obstacles to (b).

Simple libraries will install their header files directly under
$PREFIX/include, so for step (b), we can set CPATH or C_INCLUDE_PATH
(which gcc uses) to contain $PREFIX/include.

More complex libraries use a subdirectory in $PREFIX/include and use
pkg-config to tell other components where to find the headers.
e.g. For libxml2, headers are under $PREFIX/include/libxml2.  The
command "pkg-config libxml-2.0 --cflags" returns
"-I$PREFIX/include/libxml2" to pass to gcc.

How do we relocate libraries that use pkg-config?  First, we can set
PKG_CONFIG_PATH to include $PREFIX/lib/pkgconfig.  libxml2 installs a
file $PREFIX/lib/pkgconfig/libxml-2.0.pc.  This file contains the
value of PREFIX from step (a).  We have three choices:

 * Rewrite the libxml-2.0.pc file to point to the new PREFIX.
 * Use pkg-config's --define-variable option to override the .pc file's
   "prefix" variable.  e.g. Have a shell script wrapping pkg-config that
   adds this option.  This only works if you want all packages to have
   the same PREFIX.
 * Extend pkg-config to relocate packages based on environment variables
   (it's not clear how this would work).

There is another problem with libxml2.  As well as providing a
pkg-config .pc file, it provides a shell script called xml2-config
(installed into $PREFIX/bin, which we add to PATH).  I am guessing
this predates pkg-config because it also provides --cflags and --libs
options, but it provides another option which is not superseded by
pkg-config.  Like the .pc file, xml2-config has the value of PREFIX
from step (a) substituted into it.  There are two ways we could
relocate this:

 * Rewrite the installed xml2-config file: do a search-and-replace to
   replace the old PREFIX with the new PREFIX.
 * Change the source file that xml2-config is generated from,
   xml2-config.in, to work out PREFIX based on an environment variable.
   (What environment variable do we use?  Would this break the package
   for normal non-relocatable use?)

There are other installed scripts with the same issue, all generated
from *.in files:

 * gtkdocize (generated from gtkdocize.in)
 * gnome-doc-common
 * intltoolize
 * scrollkeeper-config (generated from rarian-sk-config.in)
 * gettextize

Rewriting paths inside installed scripts seems like the simplest
solution.  Zero-Install would a mechanism to do that.

Rewriting paths would be similar to the hash-rewriting that Nix does.
The difference is that Nix only rewrites one fixed-length hash to
another.  Nix's hash pathnames always start with "/nix/store", so the
pathname length does not change.  That makes it possible (if slightly
dubious) for Nix to rewrite hashes inside binary files.  Zero-Install
would need to rewrite between pathnames of different lengths
(e.g. "/home/bob" to "/home/fred") because it does not use a global
store directory, so it could only rewrite inside text files.
Hopefully we would only ever need to rewrite text files and not binary
files.  Providing the pathnames contain cryptographic hashes,
accidental matches in the search-and-replace should not occur.

Another problem I hit is with libtool.  libtool includes $PREFIX in
the .la files it installs into $PREFIX/lib.  This *appears* to cause
it to fail when trying to locate .la files in a later build.  The
following error occurs while building pango:

/bin/bash ../libtool --tag=CC   --mode=link gcc  -g -O2 -Wall -version-info
2000:0:2000 -export-symbols-regex "^pango_.*"  -o libpangox-1.0.la -rpath
/mseaborn/lib module-defs-x.lo pangox.lo pangox-fontcache.lo
pangox-fontmap.lo libpango-1.0.la 
-L/home/mrs/custom-jhbuild/prefix/mseaborn/lib -lgobject-2.0 -lgmodule-2.0
-ldl -lglib-2.0   -lX11
libtool: link: warning: library
`/home/mrs/custom-jhbuild/prefix/mseaborn/lib/libgobject-2.0.la' was moved.
libtool: link: warning: library
`/home/mrs/custom-jhbuild/prefix/mseaborn/lib/libgmodule-2.0.la' was moved.
libtool: link: warning: library
`/home/mrs/custom-jhbuild/prefix/mseaborn/lib/libglib-2.0.la' was moved.
generating symbol list for `libpangox-1.0.la'
/usr/bin/nm -B  .libs/module-defs-x.o .libs/pangox.o
.libs/pangox-fontcache.o .libs/pangox-fontmap.o  | sed -n -e 's/^.*[    
]\([ABCDGIRSTW][ABCDGIRSTW]*\)[         ][  
]*\([_A-Za-z][_A-Za-z0-9]*\)$/\1 \2 \2/p' | /bin/sed 's/.* //' | sort |
uniq > .libs/libpangox-1.0.exp
/bin/grep -E -e "^pango_.*" ".libs/libpangox-1.0.exp" >
mv -f ".libs/libpangox-1.0.expT" ".libs/libpangox-1.0.exp"
echo "{ global:" > .libs/libpangox-1.0.ver
 cat .libs/libpangox-1.0.exp | sed -e "s/\(.*\)/\1;/" >>
 echo "local: *; };" >> .libs/libpangox-1.0.ver
 gcc -shared  .libs/module-defs-x.o .libs/pangox.o .libs/pangox-fontcache.o
.libs/pangox-fontmap.o  -Wl,--rpath
-Wl,/home/mrs/custom-jhbuild/checkout/pango/pango/.libs -Wl,--rpath
-Wl,/home/mrs/custom-jhbuild/prefix/mseaborn/lib -Wl,--rpath
-Wl,/mseaborn/lib -Wl,--rpath
-Wl,/home/mrs/custom-jhbuild/prefix/mseaborn/lib ./.libs/libpango-1.0.so
/home/mrs/custom-jhbuild/prefix/mseaborn/lib/libgmodule-2.0.so -ldl
/home/mrs/custom-jhbuild/prefix/mseaborn/lib/libglib-2.0.so -lX11 
-Wl,-soname -Wl,libpangox-1.0.so.0 -Wl,-version-script
-Wl,.libs/libpangox-1.0.ver -o .libs/libpangox-1.0.so.0.2000.0
(cd .libs && rm -f libpangox-1.0.so.0 && ln -s libpangox-1.0.so.0.2000.0
(cd .libs && rm -f libpangox-1.0.so && ln -s libpangox-1.0.so.0.2000.0
creating libpangox-1.0.la
/bin/sed: can't read /mseaborn/lib/libglib-2.0.la: No such file or
libtool: link: `/mseaborn/lib/libglib-2.0.la' is not a valid libtool

I tried rewriting the installed .la files to update PREFIX but got the
same failure.  Removing the installed .la files made the build work.
It's not clear what the .la files are used for.  Apparently they are
often (but not always) omitted from .deb and .rpm packages
(e.g. see http://www.math.unl.edu/~rdieter1/libtool-rocks).

For context, this happens when I build with:
  ./configure --prefix=/mseaborn
  make install DESTDIR=$HOME/custom-jhbuild/prefix

The result is that files are installed into
$HOME/custom-jhbuild/prefix/mseaborn.  The PREFIX path "/mseaborn"
does not exist so anything that uses it, and is therefore not
relocatable, will fail.

Note that libtool is passing an -rpath option to "ld", which will
cause executables/libraries to contain RPATH fields pointing to
PREFIX.  Whether binaries contain RPATH fields should not matter
either way if we are setting LD_LIBRARY_PATH -- the binaries should
still work.  But if we want to warn about binaries that refer to
PREFIX, these binaries would show up as warnings.  The solution might
be to pass --disable-rpath to "configure".  Maybe we can copy whatever
Debian does here, because Debian disables RPATH as a matter of policy
(see http://wiki.debian.org/RpathIssue).

Zero-Install has a source package system, but I guess this hasn't been
used for many libraries so far and so it has not hit these issues?


This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
CD: 3ms