Home
Reading
Searching
Subscribe
Sponsors
Statistics
Posting
Contact
Spam
Lists
Links
About
Hosting
Filtering
Features Download
Marketing
Archives
FAQ
Blog
 
Gmane
From: Don Zickus <dzickus <at> redhat.com>
Subject: [PATCH 00/19 V2] perf, c2c: Add new tool to analyze cacheline contention on NUMA systems
Newsgroups: gmane.linux.kernel
Date: Friday 28th February 2014 17:42:49 UTC (over 2 years ago)
With the introduction of NUMA systems, came the possibility of remote
memory accesses.
Combine those remote memory accesses with contention on the remote node (ie
a modified
cacheline) and you have a possibility for very long latencies.  These
latencies can
bottleneck a program.

The program added by these patches, helps detect the situation where two
nodes are
'tugging' on the same _data_ cacheline.  The term used through out this
program and
the various changelogs is called a HITM.  This means nodeX went to read a
cacheline
and it was discovered to be loaded in nodeY's LLC cache (hence the
cacheHIT). The 
remote cacheline was also in a 'M'odified state thus creating a 'HIT M' for
hit in
a modified state.  HITMs can happen locally and remotely.  This program's
interest
is mainly in remote HITMs as they cause the longest latencies.

Why a program has a remote HITM derives from how the two nodes are
'sharing' the
cacheline.  Is the sharing intentional ("true") or unintentional ("false").
 We have seen
lots of "false" sharing cases, which lead to simple solutions such as
seperating the data
onto different cachelines.  

This tool does not distinguish between 'true' or 'false' sharing, instead
it just points to
the more expensive sharing situations under the current workload.  It is up
to the user
to understand what the workload is doing to determine whether a problem
exists or not and
how to report it.

The data output is verbose and there are lots of data tables that interprit
the latencies
and data addresses in different ways to help see where bottlenecks might be
lying.

Most of this idea, work and calculations were done by Dick Fowles.  My work
mainly
includes porting it to perf.  Joe Mario has contributed greatly with ideas
to make the
output more informative based on his usage of the tool.  Joe has found a
handful of
bottlenecks using various industry benchmarks and has worked with
developers to fix
them.

I would also like to thank Stephane Eranian for his early help and guidance
on 
navigating the differences between the current perf tool and how similar
tools
looked at HP.  And also his tireless work in getting the MMAP2 interface to
stick.

Also thanks to Arnaldo and Jiri Olso for their help in suggestions for this
tool.

I also have a test program that generated a controlled number of HITMs that
we used
frequently to validate our early work (the Intel docs were not always clear
which
bits had to be set and some arches do not work well).  I would like to add
it, but
didn't know how (nor did I spend any serious time looking either).

This program has been tested primarily on Intel's Ivy Bridge platforms. 
The Sandy Bridge
platforms had some quirks that were fixed on Ivy Bridge.  We haven't tried
Haswell as
that has a re-worked latency event implementation.

A handful of patches include re-enabling MMAP2 support and some fixes to
perf itself.  One
in particular hacks up how standard deviation is calculated.  It works with
our calculations
but may break other tools expectations.  Feedback is welcomed.

Comemnts, feedback, anything else welcomed.

V2: updated to latest perf/core branch 1029f9fedf87fa6
    switched to hist_entry based on Jiri O's suggestion
    dropped latency analyze for now until this patchset is accepted
    little fixes and tweaks

Signed-off-by: Don Zickus 

Arnaldo Carvalho de Melo (2):
  perf c2c: Shared data analyser
  perf c2c: Dump raw records, decode data_src bits

Don Zickus (19):
  Revert "perf: Disable PERF_RECORD_MMAP2 support"
  perf, machine: Use map as success in ip__resolve_ams
  perf, session: Change header.misc dump from decimal to hex
  perf, stat: FIXME Stddev calculation is incorrect
  perf, callchain: Add generic callchain print handler for stdio
  perf, c2c: Rework setup code to prepare for features
  perf, c2c: Add rbtree sorted on mmap2 data
  perf, c2c: Add stats to track data source bits and cpu to node maps
  perf, c2c: Sort based on hottest cache line
  perf, c2c: Display cacheline HITM analysis to stdout
  perf, c2c: Add callchain support
  perf, c2c: Output summary stats
  perf, c2c: Dump rbtree for debugging
  perf, c2c: Fixup tid because of perf map is broken
  perf, c2c: Add symbol count table
  perf, c2c: Add shared cachline summary table
  perf, c2c: Add framework to analyze latency and display summary stats
  perf, c2c: Add selected extreme latencies to output cacheline stats
    table
  perf, c2c: Add summary latency table for various parts of caches

 kernel/events/core.c                |    4 -
 tools/perf/Documentation/perf-c2c.c |   22 +
 tools/perf/Makefile.perf            |    1 +
 tools/perf/builtin-c2c.c            | 2963
+++++++++++++++++++++++++++++++++++
 tools/perf/builtin.h                |    1 +
 tools/perf/perf.c                   |    1 +
 tools/perf/ui/stdio/hist.c          |   37 +
 tools/perf/util/event.c             |   36 +-
 tools/perf/util/evlist.c            |   37 +
 tools/perf/util/evlist.h            |    7 +
 tools/perf/util/evsel.c             |    1 +
 tools/perf/util/hist.h              |    4 +
 tools/perf/util/machine.c           |    2 +-
 tools/perf/util/session.c           |    2 +-
 tools/perf/util/stat.c              |    3 +-
 15 files changed, 3097 insertions(+), 24 deletions(-)
 create mode 100644 tools/perf/Documentation/perf-c2c.c
 create mode 100644 tools/perf/builtin-c2c.c

-- 
1.7.11.7

Arnaldo Carvalho de Melo (2):
  perf c2c: Shared data analyser
  perf c2c: Dump raw records, decode data_src bits

Don Zickus (17):
  Revert "perf: Disable PERF_RECORD_MMAP2 support"
  perf, sort:  Add physid sorting based on mmap2 data
  perf, sort:  Allow unique sorting instead of combining hist_entries
  perf: Allow ability to map cpus to nodes easily
  perf, kmem: Utilize the new generic cpunode_map
  perf: Fix stddev calculation
  perf, callchain: Add generic callchain print handler for stdio
  perf, c2c: Rework setup code to prepare for features
  perf, c2c: Add in sort on physid
  perf, c2c: Add stats to track data source bits and cpu to node maps
  perf, c2c: Sort based on hottest cache line
  perf, c2c: Display cacheline HITM analysis to stdout
  perf, c2c: Add callchain support
  perf, c2c: Output summary stats
  perf, c2c: Dump rbtree for debugging
  perf, c2c: Add symbol count table
  perf, c2c: Add shared cachline summary table

 kernel/events/core.c                |    4 -
 tools/perf/Documentation/perf-c2c.c |   22 +
 tools/perf/Makefile.perf            |    1 +
 tools/perf/builtin-c2c.c            | 1787
+++++++++++++++++++++++++++++++++++
 tools/perf/builtin-kmem.c           |   78 +-
 tools/perf/builtin-report.c         |    2 +-
 tools/perf/builtin.h                |    1 +
 tools/perf/perf.c                   |    1 +
 tools/perf/ui/stdio/hist.c          |   37 +
 tools/perf/util/cpumap.c            |  150 +++
 tools/perf/util/cpumap.h            |   35 +
 tools/perf/util/event.c             |   36 +-
 tools/perf/util/evsel.c             |    1 +
 tools/perf/util/hist.c              |   10 +-
 tools/perf/util/hist.h              |    5 +
 tools/perf/util/sort.c              |  149 +++
 tools/perf/util/sort.h              |    4 +
 tools/perf/util/stat.c              |   13 +
 tools/perf/util/stat.h              |    1 +
 19 files changed, 2236 insertions(+), 101 deletions(-)
 create mode 100644 tools/perf/Documentation/perf-c2c.c
 create mode 100644 tools/perf/builtin-c2c.c

-- 
1.7.11.7
 
CD: 3ms