Features Download
From: Joerg Jaspert <joerg <at> debian.org>
Subject: Bits from an FTP Master: New team member, need more volunteers, (kind of) todo list
Newsgroups: gmane.linux.debian.devel.announce
Date: Thursday 11th March 2010 21:44:44 UTC (over 6 years ago)
Hello everyone,

I thought it was time again to send a mail to d-d-a, and looking for an
excuse, I found the ftpteam.

The first good excuse is the addition of a new member to our team.
Everybody please send your condolences to Luca Falavigna, dktrkranz.
He joined us a while ago as a trainee and is now an FTP Assistant.

And while I have your attention, let me tell you that we need your help!

I'm starting with a call for volunteers but will follow it with a (kind
of) todo list which interested people can work on. And, while some of the
jobs can only be done by team members, many can be done without joining
the team, and a few can even be worked on by people who aren't Debian
Developers (yet).

If you are interested feel free to contact us at [email protected]!

Of course, as with any team, a few points you should think about (not
all apply for all tasks, of course):

 - You need to be able to deal with all the existing team members.
   (And they with you). :)

 - You need to be able to deal with sometimes unpopular decisions. People
   do not always like it when you reject a package, but that is no reason
   to allow bad ones in. If you can't stand a bit of flames / don't like
   to take hard decisions, this is no job for you. (But look out for the
   other possibilities :) ).

 - You must love to read and deal with legal texts, like licenses and
   their effect on the package in Debian. The ftpteam is *the* one place
   to decide if something is ok for Debian to distribute or not, and you
   will have to take this decision.

 - You need a very good understanding of the archive, how packaging works,
   know qa processes and the general way things are dealt with in
   Debian. This job will throw you right in the middle of all this.
   You have to know the basics of just about every programming language
   you can imagine (and all those you can't but that are still there),
   of all the different packaging systems people use. NEW will present
   all of them and more stuff you never heard about and you need to be
   able to dig through it, searching for possible bad things.

 - You should be able to read and write python. At least if you want to
   understand the code behind our archive, or even want to help
   maintaining it.


So of course the one task that is seen by most people, and as such is
mentioned first, is NEW processing. (You have to be a DD for this!)
The current team of assistants is doing a very good job keeping the
process running; packages usually pass NEW in a very short time. Of
course there are always exceptions, not all packages are simple, but
overall they just rock!

Debian is not getting smaller and the NEW queue is a kind of Sisyphean
task - uploads that want to get processed never stop. As such, more
manpower to spread the load is very much wanted, the more, the better
(to a limit, but we aren't at that limit yet).

What will happen when you decide to volunteer for NEW (and rm) processing?

Simple: You get added to our ftptrainee group. This got setup some while
ago and is a simple but effective way to let you find out if this is
work you actually want to commit to and at the same time enables us to
see you working and help you learn the rules we have in NEW.  The way
this setup works is simply letting trainees access the ftpmaster machine
and the NEW queue. You can look at packages and their source as any other
team member. But trainees can not do the actual ACCEPT or REJECT. Instead
you have a special ability to leave notes about the packages, explaining
what action you would take and why.  The other team members will then
review those notes and either follow your advice or tell you why they
decided to do something different.

After a while we and you will know if you actually fit the team, but
more important we (and you yourself) will know if you should (want to)
continue doing NEW and will promote you up to assistant. We set ourself
a time limit of 6 months as a maximum stay in the trainee group, but
none of the current team members has ever stayed in trainee that long.
The longest is 3 months, the shortest is 6 days.


Another good way helping the ftpteam is - by not joining us! Yvan eht
Nioj! (Or for people not watching Simpsons: Join the Navy!)
Err, no, of course not, but: Join the QA team!

There is a lot of work commonly associated to the QA group but not many
people doing it. You could help out there, which in turn will help us


Do various coding work. This is the one point where even non-Debian
Developers can help (in many cases). Some cases of course need Developer
status at least, but not all of them.

We have a lot of open todo points, presenting them all here would probably
only get this mail rejected due to its size, so I will randomly select
a few. If you happen to have something of your own that really really
needs to get into dak, fine, speak up, im open. :)

First: Our git is at (http and https, both work)
and the list associated with it is 
and our IRC channel for development is #debian-dak on irc.debian.org.

As our code is mainly python with a few shell and perl additions, you
should be familiar with python. We are using sqlalchemy for database
access, so knowledge there does not hurt. And if coding needs changes
to the database schema, SQL is very helpful, so you should have basic
knowledge there as well.

- Improve our testsuite.
  This is a task that can be done even if you are not a Debian Developer
  yet. Our code has a good start at a test suite already since our
  ftpmaster meeting in Essen, but this can surely be extended a lot.

- General code cleanup.
  Every codebase that is as old as dak now is loves people cleaning up.
  Have fun. :)

- Contents files, Packages files
  For quite a while we have been working on changing the way we create
  and update our Contents files. And when this is done the generation of
  Packages/Sources files should follow suit.  Basically, this will move
  the extraction of the data for Contents (and then Packages/Sources)
  files from the few dinstall runs we have per day over to the many many
  upload processing runs we have per day. And dinstall will only write
  out the pre generated data from our database.

  Due to some issues not related to the ftpteam, the person who was
  mainly working on this task is not available to do it, so someone
  should take over. Much of the work for the contents part is done, it
  mainly needs testing and final integration. The Packages/Sources part
  that follows will work on top of this later on.

- Autosigning
  There are plans to allow buildds to automatically sign packages. There
  already have been discussions between the buildd folks, DSA and
  ftpmaster how this can work out (DSAed buildds, certain firewall and
  access restrictions, special keyrings, special upload queues), but the
  details aren't very interesting for this mail. Just - we need a little
  work on the archive side to make it work. Much is, again, already
  prepared, the patchset developed during our Essen meeting allows
  certain restrictions we have to use. But there are still a few things
  to be thought over and written, and here you (as a DD, sorry, needs
  .debian.org host access I think) can help.

- Throw away DD uploaded .debs. 
  The initial dependency of this task, having lintian based automated
  rejects, is in use for some time now, so we can do the final steps for
  this task now. This needs a bit of work, from the top of my head:
 - Needs a way to define a build-architecture for arch: all debs. Some
   of them can only build on certain architectures. Maybe a control file
 - Needs a way to define exceptions. Both, based on suites (we do not
   want to throw away security uploads to stable!) and maybe also per
   package or per gpg key.

- debianqueued seriously needs a replacement.
  This is the perl monster that every uploader knows. It is the tool
  that forwards uploads to localhost, after you put them up with ftp. It
  is really really old perl code that got modified many times. It is
  near to unmaintainable and really should get a replacement.

  There have been many ideas, including
 It doesn't
  need to be this, of course.  Feel free to discuss with us any ideas
  you have, but something smart would be nice. For example if we could
  move various checks we do to an upload to here we could reject various
  packages even before they are uploaded. (Easy examples are: expired
  key in the keyring we use or the upload is for an older version,
  the archive knows a newer one).

There are millions more on the todo list, but for now I will stop. 

bye, Joerg
This manual page was not written by anyone.  It sprang forth into
existence on its own.
CD: 5ms