Home
Reading
Searching
Subscribe
Sponsors
Statistics
Posting
Contact
Spam
Lists
Links
About
Hosting
Filtering
Features Download
Marketing
Archives
FAQ
Blog
 
Gmane
From: Tobias Grosser <tobias <at> grosser.es>
Subject: Why is the default LNT aggregation function min instead of mean
Newsgroups: gmane.comp.compilers.llvm.devel
Date: Friday 17th January 2014 01:03:49 UTC (over 3 years ago)
Hi,

I am currently investigating how to ensure that LNT only shows relevant 
performance regressions for the -O3 performance tests I am running.

One question that came up here is why the default aggregate function for 
LNT is 'min' instead of 'mean'. This looks a little surprising from the 
statistical point, but also from looking at my test results picking 
'min' seems to be an inferior choice.

For all test runs I have looked at, picking mean largely reduces the 
run-over-run changes reported due to noise.

See this run e.g:

If we use the median, we just get just one change reported:

http://llvm.org/perf/db_default/v4/nts/20661?num_comparison_runs=10&test_filter=&test_min_value_filter=&aggregation_fn=median&compare_to=20659&submit=Update

If you use min, we get eight reports one claiming over 100% performance
reduction for a case that is really just pure noise. I am planning to 
look into using better statistical methods. However, as a start, could 
we switch the default to 'mean'?

Cheers,
Tobias
 
CD: 11ms