Written while in net-exile. Not much new here, it's more a big picture 
type document.

thanks -mike

Making desktop Linux software installation easy
28th September 2004

Isn't it easy already? No ... technologies like apt have serious
drawbacks we should recognise:

* Centralisation introduces lag between upstream releases and actually
  being able to install them, sometimes measured in months or years

* Packaging as separate from development tends to introduce obscure
  bugs caused by packagers not always fully understanding what it is
  they're packaging. It makes no more sense than UI design or artwork
  being done by the distribution

* Distro developers end up duplicating effort on a massive scale. 20
  distros == the same software packaged 20 times == 20x the chance a
  user will receive a buggy package. Broken packages are not rare:
  see Wine, which has a huge number of incorrect packages in
  circulation, Mono which suffers from undesired distro packaging, etc.

* apt et al require extremely well controlled repos, otherwise they
  can get confused and ask users to provide solutions manually : this
  requires an understanding of the technology we can't expect users
  to have.

* Very hard to avoid the "shopping mall" type user interface, at
  which point choice becomes unmanagably large: see Synaptic for a
  pathological example of this. Better UIs are possible but
  you're covering up a programmatic model which doesn't match the
  user model esp for migrants.

* Pushes the "appliance" line of thinking, where a distro is not a
  platform on which third parties can build with a strong commitment
  to stability but merely an appliance: a collection of bits that
  happen to work together but may change in any way tomorrow: you can
  use what's on the CDs but extend or modify it and you void the
  warranty. Appliance distros have their place: live demo CDs, router
  distros, maybe even server distros, but not desktops. To compete
  with Windows for mindshare and acceptance we must be a platform.

Apt does have its place: useful for servers, core OS packages (see below)

We want to support a variety of UI paradigms: MacOS X style drag/drop,
setup.exe, apt-get install foo - all of these are possible in a
distributed world

IMPLIES:

       - A way for maintainers to build easy to use universal binary
         packages in a decentralised fashion (but which can still
         perform dependency resolution)

         IN PROGRESS: autopackage, ZeroInstall
         ALREADY EXISTS: Loki Setup, BitRock etc (no dep resolution)

       - A unified platform to reduce the strain on the dependency
         resolvers and increase reliability

           consistent/coherant API policy, lack of API duplication,
           centralised management not required, win32 shows us that
           a platform doesn't have to be coherent to be widely used

         basically means a large set of functionality that you can
         get with a single expressed dependency. Either your distro
         provides these packages or it doesn't, in which case you
         have to do dep resolution manually. Backwards compatibility
         is managed, ie if you depend on Platform v1.0 then
         Platform v1.2 will satisfy this.

         Can be expressed via virtual dependencies in a repository
         using todays technology (apt et al) so does not require
         distro co-operation.

         NOT STARTED: lsb does not meet above requirements of size,
         informal backwards compatibility efforts etc.

       - Sucking bulk of distribution policy upstream, eg init
         scripts, config files (debian .d dirs) rather than having it
         expressed via proprietary packaging policy.

         This is already underway, eg with freedesktop.org menus
         replacing each distros/desktops customized menu
         scheme. Currently not widely implemented, Fedora forked
         Gnome to add it but upstream has more stringent requirements
         - no action on this as of writing. Long term will (hopefully)
         replace vFolders, Debian/Mandrake Menu System etc.

         Need forum like xdg-list but for distro developers rather
         than desktop developers where policy can be standardized:
         currently does not exist, no plans to create one.

         NOT STARTED: no distribution developer forum exists

       - Desktop integration via a dedicated PAL (packaging
         abstraction layer): allows us to drive good package
         management UI deep into desktop without compromising
         portability: despite different implementations primitive
         operations mostly identical therefore ripe for abstraction

DECENTRALIZED BINARY PACKAGING

       - Examples in use today: Loki installers, self extracting
         archives, distro neutral RPMs (eg, Sun JRE). Typically lack
         dependency resolution but not needed due to highly defensive
         coding style (lots of dlopen action, no assumptions about
         base system)

       - Future possibilities: autopackage, Zero Install, LSB RPMs

       - autopackage is interesting because it provides a good user
         experience right out of the box: people using unmaintained
         alpha autopackages of Gaim from over a year ago because "it
         just worked" and "I couldn't figure out how to install it any
         other way".

             => Many Linux users without much technical ability, major
                change from the past

             => the "auto" in autopackage is important to people

         However, InstallShield type UI *not* long term goal.

         Complex solution to a messy world.

       - ZeroInstall is interesting as it has a MacOS/RISCOS style
         appfolder UI system working today, and because it's
         conceptually clean. However, requires installation of a
         kernel module (so initial install can be difficult) and does
         not integrate so well with existing distributions.

       - These two have another advantage: they are not native to any
         pre-existing distribution so can be hated equally by all

       - Binary portability not so simple, technique used today is
         build on RH6.2 box and pray, technique used by
         autopackage/ZeroInstall is apbuild.

         apbuild is "policy in a box", will evolve and adapt over time
         to match peoples needs. Essentially about altering upstream
         decisions.

            - Big issue is library maintainers not following best
              practices like symbol versioning, header versioning,
              parallel installability

            - Key libs break backcompat too often (eg, OpenSSL)

       - Meets resistance from those who believe there's nothing wrong
         with the old way.  eg, "project policy is that we don't
         provide binaries", "compiling software is more pure, it's the
         open source way", "it's the job of the distribution to
         provide binaries" etc etc

       - Plan is not to replace apt and distro packaging but to
         complement them, distro packaging becomes only for "core"
         software. Good interop between neutral and native solutions
         needed.

         What is core?

UNIFIED BASE SET:

       - Could be an upstream project like any other, or could be
         freedesktop style informal agreements.

       - Most distros already package most stuff that'd be in such a
         base set anyway, so key challenges are to:

           - manage versioning (ensure *-compat packages are always available)
           - ensure it's easy to install the base packages, ie by providing
             package repositories and virtual dependencies
           - ideally get it into the distro "base sets" where that makes sense
             (fedora, mandrake, suse ...)

           - convince people it's a good thing to do!

        - Linux Standard Base:

           - Different approach, based on specifications of libraries rather than
             specifying individual package versions. Reasoning: allows forking,
             blocks distro patching which modifies API/ABI.

           - Good:

              * Formal standard people can write test suites for
              * Support from large [proprietary] ISVs

           - Bad:

              * Not realistic to specify ABI for every library we need
                in unified base set to be competitive, doesn't achieve
                much anyway once you move beyond the lower levels,
                ABI spec only a subset of what's needed

              * Moves too slowly

              * Standardises existing practice, ie is not able to stop
                NPTL style breakage

              * Difficulties w.r.t conflicting goals between LSB and
                upstream, for instance C++ ABI vs ISO

              * Blocker: experience shows that real world software developers
                don't write to specs, they write to implementations: very few
                examples where this is not the case. Therefore attempt to specify
                eg GTK+ ABI doomed to fail w.r.t backcompat in long run:

                a) Too much behaviour unspecified: callback ordering,
                   handling of error conditions, sorting etc

                b) People end up depending on behaviour accidentally anyway:
                   construct only GObject properties

        - People don't always deserve to lose because they wrote buggy code:

           - Often docs are unclear or exact behaviour is
             unspecified. GObject properties case: no official specs at all

           - No other way: how many people write their code to be
             independent of callback ordering/multiple callbacks?

           - Everyone writes buggy code. People are expected to
             support each other even when they screw up.

           - Ultimately it's the user who loses

        - Potential solution? A mostly informal, flexible base set
          maintained by a standard heirarchical team rather than a
          committee who are willing to patch upstream binaries in
          order to maintain compatibility with pre-existing code: the
          Microsoft approach

            - Not as extreme, majority of software on free desktop
              free software, can be upgraded for zero dollars
              (automatically?) even if it's a pain for the user, so
              affects breaking change cost/benefit analysis

            - However, basic theory is the same: don't break
              compatibility, this is more than ABI/API stability and
              includes things like buggy behaviour apps depend on.

            - We want to support games, these are almost always
              proprietary and worse they have a max shelflife of a few
              months: vendors often not interested in fixing bugs after
              this date, even if people are still playing years after
              release

        - Advantage: can be fast moving, can do things a formal
          organization would frown upon, can JUST DO IT

        - Disadvantage: lacks legitimacy

        - Packagers "opt in" to shared base set by setting their only
          dependency as "base-set > 1.0", by adding rpaths to their
          binary ${ORIGIN}/../lib/base-set to allow for upstream
          overrides

        - Only applies to userspace, kernel is fundamentally screwed
          from an stability (== ease of use) perspective *however* as
          kernel becomes ever more mature it becomes more practical to
          simply fork a stable series and backport non-breaking
          changes/driver upgrades. Kernel is mature, biggest reason to
          upgrade is improved hardware support.

            - Alternative: attempt to abstract subset of unstable
              kernel ABI behind a stable one provided by a
              module. Fighting a losing battle?

            - Software is not art, hacks OK if they serve the user and
              don't compromise future engineering.

            - Only keep driver compat in stable series

SUCKING DISTRO POLICY UPSTREAM

        - Need equivalent of xdg-list for distributions, need more
          manpower devoted to actually implementing and ratifying
          standards, eg recent files spec, menu spec mostly in limbo.

        - A standard that is only implemented "sometimes" or
          "occasionally" is just as bad as no standard at all,
          therefore implementability crucial in design phase otherwise
          you end up with the menus problem.

        - Examples of distribution policy:

           - Dealing with upstream goofs, eg libpng parallel
             installability, glibc non-releases
           - Extensions to standard config files, /etc/modules
           - Co-ordinating potentially breaking changes like NPTL
           - Obvious stuff: init scripts, shared config ui (overlap
             with xdg), maybe even co-ordinated branding.

        - new freedesktop list appropriate forum, or new project entirely?

        - Can be hard to get distros involved, often distro developers
          out of sight/not active in community. Let's get Havoc to do
          it! :)

 -------------------------------- PROBLEM ----------------------------------

        - If versioning policy is moved upstream, and packaging
          decentralised WHAT IS A DISTRIBUTION?

        - Some distros define themselves through their packaging
          policy, others through their desktop, others through their
          support (mix'n'match to your pleasure), others are highly
          specialised (eg knoppix live CD, router CDs, embedded
          distros etc)

        - focus is on standardising platform not higher levels: nobody
          cares if you prefer compiling your own X server, or use a
          different icon theme as long as it's compatible with
          everybody elses and can be depended on via a shared base set

        - will meet resistance from those who believe their
          preferred distro is the one true way: "why do people
          complain, just use Debian" mentality.

        - we all have more interesting things to do that reinvent the
          distribution wheel over and over : "wouldn't you rather be
          hacking anyway?"

DESKTOP INTEGRATION

        - Desktop projects are portable, therefore requires an
          abstraction that allows them to maintain that
          portability. Do what HAL is doing for hardware, but for
          packaging: PAL!

        - 'Ideal' UI varies between users: drag/drop appfolders,
          apt-get install foo, setup.exe, different paradigms but all
          supportable if we're smart

        - drag'n'drop a la MacOS X/RISC OS except that actually works

           - MacOS X not consistent, Apples own software doesn't use
             appfolder installs, custom installer program shipped with
             own OS and even then developers roll their own.

           - RISC OS/NeXT never had to survive in the internet age of
             100s of programs installed at once, exponential OS
             complexity/integration etc.

        - How? Extend the .desktop abstraction to include packaging
          data, such that a desktop can take an arbitrary application
          .desktop file and install the package it "points" to behind
          the scenes, with no user interaction required.

          Therefore drag from webpage to panel == download+install
          operation. Can click icon in web page directly to launch it
          (or from CD, or email, or IM conversation).

        - Requires standards for packaging metadata, see PAL

        - Not hard, requires several months of effort from the right
          people, requires getting desktop projects on board (ie,
          backwards compatibility with existing systems like apt).

        - apt UI can still work in a distributed world: DNS type
          hierarchy for software.

        - Possibility for insertion of whitelisting, attack malware
          by giving user warning if they try and install it
          (optionally block entirely)

        - Can be done in parallel with other tasks.

   ******************************************************************
   *      None of this is technically difficult, it just            *
   *      requires a change in the open source culture.             *
   ******************************************************************

---------------------------------------------------------------------
To unsubscribe, e-mail: autopackage-dev-unsubscribe@...
For additional commands, e-mail: autopackage-dev-help@...