Gmane -- Mail To News And Back Again

On Sep 27, 2006, at 12:43 PM, Dan Steinicke wrote:

> I would like to see this discussion continue.  We probably need to 
> have another meeting about this, I am not attached to that but I would 
> like us to follow through on this and come up with some clear 
> solutions to the problems that are causing so much frustration.
>
> Below are my ideas of what the issues are and possible ways we could 
> start dealing with them.  I encourage others to voice their opinions 
> on what the issues are and what we should do about them.   I 
> especially encourage people to correct me if you feel I am 
> miss-stating something.
>
> 1)  We need a new agreement on how to respond when the tinderboxes go 
> red or orange for a long period of time.  Bryan has made the following 
> proposal:
>       after one week of mostly red or orange tinderbox
>   A) P1 bugs get filed for each failure
>   B) The failing tests get disabled on platforms on which they fail 
> (with bug numbers in the comments)
>   C) People are assigned to fix those bugs and re-enable the tests

I agree with this.

> 2) A number of ideas were given about the functional tests and 
> ChandlerTestLib (formerly known as QAUITestAppLib).  We should decide 
> what ideas we want to implement, give them priorities and assign 
> people to make the changes.  The ideas I heard were: (in no particular 
> order)
>   A) Not always try to emulate a user typing and clicking
>   B) Have more asserts for self checking of the test library code
>   C) Remove dependencies between the tests
>   D) Regularly run the tests on their own as well as in the suite
>   E) Make the tests more robust.  Remove assumptions and add 
> abstraction code, example: TestLib should have functions that aid 
> setting up tests moving from view to view and checking where you are 
> etc
>   F) Insure tests fail when there is an exception (there is already a 
> bug on this)
>   G) Default to stopping tests on first error (bug 6751)
>   H) Document how to run the tests better

We could make item D happen in the tinderbox if there was some manifest 
of tests to run - or if some way of having the test runner be able to 
query and find out the test names.  Otherwise we stand the chance of 
having the automated tools fall out-of-sync with tests that can be run.

> 3) A number of improvements to the way tests are logged.  Again we 
> should decide what we want to implement, prioritize and assign the 
> tasks.
>   A) Provide one log containing all log output (chandler, twisted, 
> tests) in chronological order.   B) Remove the false links at the top 
> of the on line logs
>   C) Failing tests should print a line showing how the test was run
>   D) Make tests raise on failure so there would always be a traceback 
> pointing to the failure
>   E) Have the default produce more log output

+1 to the consolidation of chandler and twisted logs.  Having the test 
output also go into that log works but we may have to scale back the 
amount of text that is created and save the more wordy version for the 
tinderbox console output.

Item C can be solved by having each test output to the log it's 
parameters so the run is self-documenting - fail or pass.

> 4)  At least one suggestion was made about the scripting functions.
>   A) Remove the 'magic' of the app_ns object and re-write so that the 
> miss leading None type errors are reduced or avoided.
>
> 5) Hardware
>   A) Improve cycle times by using faster computers to run the 
> tinderboxes

we are working with IT on this now.

> 6) do_tests
>   A) Re-write do_tests in python

Also scheduled to be done - like you mentioned to me earlier we should 
start a wiki page to gather requirements so when we get to the point we 
are ready to do the coding we are prepared.

Great job summarizing the meeting!

---
Bear

Build and Release Engineer
Open Source Applications Foundation (OSAF)
bear@...
http://www.osafoundation.org

bear@...
http://code-bear.com

PGP Fingerprint = 9996 719F 973D B11B E111  D770 9331 E822 40B3 CD29