Home
Reading
Searching
Subscribe
Sponsors
Statistics
Posting
Contact
Spam
Lists
Links
About
Hosting
Filtering
Features Download
Marketing
Archives
FAQ
Blog
 
Gmane
From: Mel Gorman <mgorman <at> suse.de>
Subject: [PATCH 0/16] Basic scheduler support for automatic NUMA balancing V4
Newsgroups: gmane.linux.kernel
Date: Thursday 11th July 2013 09:46:44 UTC (over 3 years ago)
This continues to build on the previous feedback and further testing. Peter
posted a patch that avoids overloading a destination node relative to a
source node by postponing the reschedule of tasks on a preferred node. I
took the load calculations but dropped the balancing part as it performed
badly on local tests. It was evident that false sharing within THP pages
is a problem and I think it would alleviate the overloading problem if it
was solved first. Shared accesses are still not properly used for selecting
preferred nodes due to the impact of false sharing within THP pages.

Changelog since V3
o Correct detection of unset last nid/pid information
o Dropped nr_preferred_running and replaced it with Peter's load balancing
o Pass in correct node information for THP hinting faults
o Pressure tasks sharing a THP page to move towards same node
o Do not set pmd_numa if false sharing is detected

Changelog since V2
o Reshuffle to match Peter's implied preference for layout
o Reshuffle to move private/shared split towards end of series to make it
  easier to evaluate the impact
o Use PID information to identify private accesses
o Set the floor for PTE scanning based on virtual address space scan rates
  instead of time
o Some locking improvements
o Do not preempt pinned tasks unless they are kernel threads

Changelog since V1
o Scan pages with elevated map count (shared pages)
o Scale scan rates based on the vsz of the process so the sampling of the
  task is independant of its size
o Favour moving towards nodes with more faults even if it's not the
  preferred node
o Laughably basic accounting of a compute overloaded node when selecting
  the preferred node.
o Applied review comments

This series integrates basic scheduler support for automatic NUMA
balancing.
It borrows very heavily from Peter Ziljstra's work in "sched, numa, mm:
Add adaptive NUMA affinity support" but deviates too much to preserve
Signed-off-bys. As before, if the relevant authors are ok with it I'll
add Signed-off-bys (or add them yourselves if you pick the patches up).

This is still far from complete and there are known performance gaps
between this series and manual binding (when that is possible). As before,
the intention is not to complete the work but to incrementally improve
mainline and preserve bisectability for any bug reports that crop up. In
some cases performance may be worse unfortunately and when that happens
it will have to be judged if the system overhead is lower and if so,
is it still an acceptable direction as a stepping stone to something
better.

Patch 1 adds sysctl documentation

Patch 2 tracks NUMA hinting faults per-task and per-node

Patch 3 corrects a THP NUMA hint fault accounting bug

Patch 4 avoids trying to migrate the THP zero page

Patches 5-7 selects a preferred node at the end of a PTE scan based on what
	node incurrent the highest number of NUMA faults. When the balancer
	is comparing two CPU it will prefer to locate tasks on their
	preferred node.

Patch 8 reschedules a task when a preferred node is selected if it is not
	running on that node already. This avoids waiting for the scheduler
	to move the task slowly.

Patch 9 adds infrastructure to allow separate tracking of shared/private
	pages but treats all faults as if they are private accesses. Laying
	it out this way reduces churn later in the series when private
	fault detection is introduced

Patch 10 replaces PTE scanning reset hammer and instread increases the
	scanning rate when an otherwise settled task changes its
	preferred node.

Patch 11 avoids some unnecessary allocation

Patch 12 sets the scan rate proportional to the size of the task being
scanned.

Patch 13-14 kicks away some training wheels and scans shared pages and
small VMAs.

Patch 15 introduces private fault detection based on the PID of the
faulting
	process and accounts for shared/private accesses differently

Patch 16 pick the least loaded CPU based on a preferred node based on a
scheduling
	domain common to both the source and destination NUMA node.

Testing on this is only partial as full tests take a long time to run. A
full specjbb for both single and multi takes over 4 hours. NPB D class
also takes a few hours. With all the kernels in question, it still takes
a weekend to churn through them all.

Kernel 3.9 is still the testing baseline. The following kernels were tested

o vanilla		vanilla kernel with automatic numa balancing enabled
o favorpref-v4   	Patches 1-11
o scanshared-v4   	Patches 1-14
o splitprivate-v4   	Patches 1-15
o accountload-v4   	Patches 1-16

This is SpecJBB running on a 4-socket machine with THP enabled and one JVM
running for the whole system. Only a limited number of clients are executed
to save on time.

specjbb
                        3.9.0                 3.9.0                 3.9.0  
              3.9.0                 3.9.0
                      vanilla       favorpref-v4         scanshared-v4     
 splitprivate-v4        accountload-v4   
TPut 1      26099.00 (  0.00%)     24726.00 ( -5.26%)     23924.00 (
-8.33%)     24788.00 ( -5.02%)     23692.00 ( -9.22%)
TPut 7     187276.00 (  0.00%)    190315.00 (  1.62%)    189450.00 ( 
1.16%)    185294.00 ( -1.06%)    183639.00 ( -1.94%)
TPut 13    318028.00 (  0.00%)    340088.00 (  6.94%)    330785.00 ( 
4.01%)    334663.00 (  5.23%)    333818.00 (  4.96%)
TPut 19    368547.00 (  0.00%)    422009.00 ( 14.51%)    401622.00 ( 
8.97%)    448669.00 ( 21.74%)    447950.00 ( 21.54%)
TPut 25    377522.00 (  0.00%)    442038.00 ( 17.09%)    413670.00 ( 
9.58%)    499595.00 ( 32.34%)    506872.00 ( 34.26%)
TPut 31    347642.00 (  0.00%)    425809.00 ( 22.48%)    382499.00 (
10.03%)    487862.00 ( 40.33%)    468347.00 ( 34.72%)
TPut 37    313439.00 (  0.00%)    402418.00 ( 28.39%)    350941.00 (
11.96%)    467847.00 ( 49.26%)    437945.00 ( 39.72%)
TPut 43    291958.00 (  0.00%)    363120.00 ( 24.37%)    313203.00 ( 
7.28%)    422984.00 ( 44.88%)    384563.00 ( 31.72%)

First off, note what the shared/private split patch does. Once we start
scanning all pages there is a degradation in performance as the shared page
faults introduce noise to the statistics. All indications are because there
is false sharing within THP pages that needs to be addressed. Splitting
the shared/private faults restores the performance and the key task in
the future is to use this shared/private information for maximum benefit.


specjbb Peaks
                                  3.9.0                 3.9.0              
  3.9.0                 3.9.0                 3.9.0
                                vanilla          favorpref-v4        
scanshared-v4       splitprivate-v4        accountload-v4   
 Actual Warehouse       26.00 (  0.00%)       26.00 (  0.00%)       26.00 (
 0.00%)       26.00 (  0.00%)       26.00 (  0.00%)
 Actual Peak Bops   377522.00 (  0.00%)   442038.00 ( 17.09%)   413670.00 (
 9.58%)   499595.00 ( 32.34%)   506872.00 ( 34.26%)

Peak performance is improved overall.


               3.9.0       3.9.0       3.9.0       3.9.0       3.9.0
             vanillafavorpref-v4   scanshared-v4   splitprivate-v4  
accountload-v4   
User         5184.53     5177.92     5178.37     5177.24     5181.78
System         59.61       65.77       60.97       67.21       67.43
Elapsed       254.52      254.14      254.06      254.24      254.33

This is an increase in system CPU overhead that needs to be watched.

                                 3.9.0       3.9.0       3.9.0       3.9.0 
     3.9.0
                               vanillafavorpref-v4   scanshared-v4  
splitprivate-v4   accountload-v4   
THP fault alloc                  33297       34710       35229       34480 
     33510
THP collapse alloc                   9           6          14          11 
        12
THP splits                           3           3           3           4 
         1
THP fault fallback                   0           0           0           0 
         0
THP collapse fail                    0           0           0           0 
         0
Compaction stalls                    0           0           0           0 
         0
Compaction success                   0           0           0           0 
         0
Compaction failures                  0           0           0           0 
         0
Page migrate success           1773768     1949772     1407218     4253043 
   4218882
Page migrate failure                 0           0           0           0 
         0
Compaction pages isolated            0           0           0           0 
         0
Compaction migrate scanned           0           0           0           0 
         0
Compaction free scanned              0           0           0           0 
         0
Compaction cost                   1841        2023        1460        4414 
      4379
NUMA PTE updates              17461135    18458997    14255329    15856615 
  16071944
NUMA hint faults                 85873      172654       80923       91043 
     90465
NUMA hint local faults           27145      119972       32219       36020 
     34847
NUMA hint local percent             31          69          39          39 
        38
NUMA pages migrated            1773768     1949772     1407218     4253043 
   4218882
AutoNUMA cost                      585        1029         531         647 
       644

It's interesting to note how much scanning shared pages affects the
percentage of local NUMA hinting faults. There is a lot more work to do
there. There are fewer PTE scan updates but there are a much larger number
of pages being migrated that will need examination. Due to the overall
performance the focus will still be on false THP sharing.

Next is the autonuma benchmark results. These were only run once so I have
no
idea what the variance is. Obviously they could be run multiple times but
with
this number of kernels we would die of old age waiting on the results.

autonumabench
                                          3.9.0                 3.9.0      
          3.9.0                 3.9.0                 3.9.0
                                        vanilla       favorpref-v4        
scanshared-v4       splitprivate-v4        accountload-v4   
User    NUMA01               52623.86 (  0.00%)    49514.41 (  5.91%)   
53783.60 ( -2.20%)    51205.78 (  2.69%)    53501.03 ( -1.67%)
User    NUMA01_THEADLOCAL    17595.48 (  0.00%)    17620.51 ( -0.14%)   
19734.74 (-12.16%)    16966.63 (  3.57%)    17113.31 (  2.74%)
User    NUMA02                2043.84 (  0.00%)     1993.04 (  2.49%)    
2051.29 ( -0.36%)     1901.96 (  6.94%)     2035.80 (  0.39%)
User    NUMA02_SMT            1057.11 (  0.00%)     1005.61 (  4.87%)     
980.19 (  7.28%)      977.65 (  7.52%)      972.60 (  7.99%)
System  NUMA01                 414.17 (  0.00%)      222.86 ( 46.19%)     
145.79 ( 64.80%)      321.93 ( 22.27%)      344.93 ( 16.72%)
System  NUMA01_THEADLOCAL      105.17 (  0.00%)      102.35 (  2.68%)     
117.22 (-11.46%)      105.35 ( -0.17%)      102.54 (  2.50%)
System  NUMA02                   9.36 (  0.00%)        9.96 ( -6.41%)      
13.02 (-39.10%)        9.53 ( -1.82%)        6.73 ( 28.10%)
System  NUMA02_SMT               3.54 (  0.00%)        3.53 (  0.28%)      
 3.46 (  2.26%)        5.85 (-65.25%)        4.49 (-26.84%)
Elapsed NUMA01                1201.52 (  0.00%)     1143.59 (  4.82%)    
1244.61 ( -3.59%)     1182.92 (  1.55%)     1208.74 ( -0.60%)
Elapsed NUMA01_THEADLOCAL      393.91 (  0.00%)      392.49 (  0.36%)     
442.04 (-12.22%)      385.61 (  2.11%)      386.43 (  1.90%)
Elapsed NUMA02                  50.30 (  0.00%)       50.36 ( -0.12%)      
49.53 (  1.53%)       48.91 (  2.76%)       49.23 (  2.13%)
Elapsed NUMA02_SMT              58.48 (  0.00%)       47.79 ( 18.28%)      
51.56 ( 11.83%)       55.98 (  4.27%)       56.34 (  3.66%)
CPU     NUMA01                4414.00 (  0.00%)     4349.00 (  1.47%)    
4333.00 (  1.84%)     4355.00 (  1.34%)     4454.00 ( -0.91%)
CPU     NUMA01_THEADLOCAL     4493.00 (  0.00%)     4515.00 ( -0.49%)    
4490.00 (  0.07%)     4427.00 (  1.47%)     4455.00 (  0.85%)
CPU     NUMA02                4081.00 (  0.00%)     3977.00 (  2.55%)    
4167.00 ( -2.11%)     3908.00 (  4.24%)     4148.00 ( -1.64%)
CPU     NUMA02_SMT            1813.00 (  0.00%)     2111.00 (-16.44%)    
1907.00 ( -5.18%)     1756.00 (  3.14%)     1734.00 (  4.36%)

numa01 saw no major performance benefit with a mix of gains and losses
throughout the series for its system CPU usage. It is an adverse workload
for this machine so right now I'm not overly concerned with improving its
performance.

numa01_threadlocal saw a very small performance gain overall although
it is interesting to note that scanning shared pages hurt it badly. Again
I predict that better shared page detection will help here.

numa02 showed a small improvement but it should also be already running
close to as quickly as possible.

numa02_smt also shows a small improvement although again scanning shared
pages hurt and would benefit from improved handling there.

                                 3.9.0       3.9.0       3.9.0       3.9.0 
     3.9.0
                               vanillafavorpref-v4   scanshared-v4  
splitprivate-v4   accountload-v4   
THP fault alloc                  14325       11724       14906       13553 
     14403
THP collapse alloc                   6           3           7          13 
        10
THP splits                           4           1           4           2 
         2
THP fault fallback                   0           0           0           0 
         0
THP collapse fail                    0           0           0           0 
         0
Compaction stalls                    0           0           0           0 
         0
Compaction success                   0           0           0           0 
         0
Compaction failures                  0           0           0           0 
         0
Page migrate success           9020528     9708110     6677767     6773951 
   6170746
Page migrate failure                 0           0           0           0 
         0
Compaction pages isolated            0           0           0           0 
         0
Compaction migrate scanned           0           0           0           0 
         0
Compaction free scanned              0           0           0           0 
         0
Compaction cost                   9363       10077        6931        7031 
      6405
NUMA PTE updates             119292401   114641446    85954812    74337906 
  75911999
NUMA hint faults                755901      499186      287825      237095 
    232126
NUMA hint local faults          595478      333483      152899      122210 
    128762
NUMA hint local percent             78          66          53          51 
        55
NUMA pages migrated            9020528     9708110     6677767     6773951 
   6170746
AutoNUMA cost                     4785        3482        2167        1834 
      1809

As all the tests are mashed together it is possible to make specific
conclusions on each testcase.  However, in general the series is doing a
lot
less work with PTE updates, faults and so on. THe percentage of local
faults
suffers but a large part of this seems to be around where shared pages are
getting scanned.

I also ran SpecJBB running on with THP enabled and one JVM running per
NUMA node in the system.

specjbb
                          3.9.0                 3.9.0                 3.9.0
                3.9.0                 3.9.0
                        vanilla       favorpref-v4         scanshared-v4   
   splitprivate-v4        accountload-v4   
Mean   1      30640.75 (  0.00%)     31222.25 (  1.90%)     31275.50 ( 
2.07%)     30554.00 ( -0.28%)     30348.75 ( -0.95%)
Mean   10    136983.25 (  0.00%)    133072.00 ( -2.86%)    140022.00 ( 
2.22%)    119168.25 (-13.01%)    140998.00 (  2.93%)
Mean   19    124005.25 (  0.00%)    121016.25 ( -2.41%)    122189.00 (
-1.46%)    111813.75 ( -9.83%)    129100.75 (  4.11%)
Mean   28    114672.00 (  0.00%)    111643.00 ( -2.64%)    109175.75 (
-4.79%)    101199.50 (-11.75%)    116026.50 (  1.18%)
Mean   37    110916.50 (  0.00%)    105791.75 ( -4.62%)    103103.75 (
-7.04%)    100187.00 ( -9.67%)    108801.00 ( -1.91%)
Mean   46    110139.25 (  0.00%)    105383.25 ( -4.32%)     99454.75 (
-9.70%)     99762.00 ( -9.42%)    104239.25 ( -5.36%)
Stddev 1       1002.06 (  0.00%)      1125.30 (-12.30%)       959.60 ( 
4.24%)       960.28 (  4.17%)      1014.89 ( -1.28%)
Stddev 10      4656.47 (  0.00%)      6679.25 (-43.44%)      5946.78
(-27.71%)     10427.37(-123.93%)      4039.93 ( 13.24%)
Stddev 19      2578.12 (  0.00%)      5261.94 (-104.10%)     3414.66
(-32.45%)      5070.00 (-96.65%)      1849.10 ( 28.28%)
Stddev 28      4123.69 (  0.00%)      4156.17 ( -0.79%)      6666.32
(-61.66%)      3899.89 (  5.43%)      3081.40 ( 25.28%)
Stddev 37      2301.94 (  0.00%)      5225.48 (-127.00%)    
5444.18(-136.50%)      3490.87 (-51.65%)      1795.72 ( 21.99%)
Stddev 46      8317.91 (  0.00%)      6759.04 ( 18.74%)      6587.32 (
20.81%)      4458.49 ( 46.40%)      7387.32 ( 11.19%)
TPut   1     122563.00 (  0.00%)    124889.00 (  1.90%)    125102.00 ( 
2.07%)    122216.00 ( -0.28%)    121395.00 ( -0.95%)
TPut   10    547933.00 (  0.00%)    532288.00 ( -2.86%)    560088.00 ( 
2.22%)    476673.00 (-13.01%)    563992.00 (  2.93%)
TPut   19    496021.00 (  0.00%)    484065.00 ( -2.41%)    488756.00 (
-1.46%)    447255.00 ( -9.83%)    516403.00 (  4.11%)
TPut   28    458688.00 (  0.00%)    446572.00 ( -2.64%)    436703.00 (
-4.79%)    404798.00 (-11.75%)    464106.00 (  1.18%)
TPut   37    443666.00 (  0.00%)    423167.00 ( -4.62%)    412415.00 (
-7.04%)    400748.00 ( -9.67%)    435204.00 ( -1.91%)
TPut   46    440557.00 (  0.00%)    421533.00 ( -4.32%)    397819.00 (
-9.70%)    399048.00 ( -9.42%)    416957.00 ( -5.36%)

Performance here is more or less flat although it's interesting to
note how much scanning share pages affects the differences between JVM
performance. Overall the series performance is more or less unchanged with
some improvements in varaiability. This should also benefit from false
sharing detection but it would also benefit if there was proper detection
of related tasks that share pages.

specjbb Peaks
                                3.9.0               3.9.0             
3.9.0               3.9.0               3.9.0
                              vanilla        favorpref-v4     
scanshared-v4     splitprivate-v4      accountload-v4   
 Actual Warehouse     11.00 (  0.00%)     11.00 (  0.00%)    11.00 ( 
0.00%)     11.00 (  0.00%)     11.00 (  0.00%)
 Actual Peak Bops 547933.00 (  0.00%) 532288.00 ( -2.86%)560088.00 ( 
2.22%) 476673.00 (-13.01%) 563992.00 (  2.93%)

Accounting for load recovers the loss from splitting private/shared. Again,
proper false shared detection is required.

               3.9.0       3.9.0       3.9.0       3.9.0       3.9.0
             vanillafavorpref-v4   scanshared-v4   splitprivate-v4  
accountload-v4   
User        52899.04    53106.74    53245.67    52828.25    53162.02
System        250.42      254.20      203.97      222.28      230.85
Elapsed      1199.72     1208.35     1206.14     1197.28     1207.10

Small reduction in system CPU overhead.

                                 3.9.0       3.9.0       3.9.0       3.9.0 
     3.9.0
                               vanillafavorpref-v4   scanshared-v4  
splitprivate-v4   accountload-v4   
THP fault alloc                  65188       66217       68158       63283 
     65531
THP collapse alloc                  97         172          91         108 
       135
THP splits                          38          37          36          34 
        41
THP fault fallback                   0           0           0           0 
         0
THP collapse fail                    0           0           0           0 
         0
Compaction stalls                    0           0           0           0 
         0
Compaction success                   0           0           0           0 
         0
Compaction failures                  0           0           0           0 
         0
Page migrate success          14583860    14559261     7770770    10131560 
  10932731
Page migrate failure                 0           0           0           0 
         0
Compaction pages isolated            0           0           0           0 
         0
Compaction migrate scanned           0           0           0           0 
         0
Compaction free scanned              0           0           0           0 
         0
Compaction cost                  15138       15112        8066       10516 
     11348
NUMA PTE updates             128327468   129131539    74033679    72954561 
  72832728
NUMA hint faults               2103190     1712971     1488709     1362365 
   1292772
NUMA hint local faults          734136      640363      405816      471928 
    403028
NUMA hint local percent             34          37          27          34 
        31
NUMA pages migrated           14583860    14559261     7770770    10131560 
  10932731
AutoNUMA cost                    11691        9745        8109        7515 
      7181

Fewer PTE updates but the percentage of local hinting faults clearly
needs improvement.

Overall the series performs well even though the gaps are still evident.
This is likely to be my last update to this series for a while but I'd
like to see this treated as a standalone with a separate series focusing on
false sharing detection and reduction, shared accesses used for selecting
preferred nodes, shared accesses used for load balancing and reintroducing
Peter's patch that balances compute nodes relative to each other. This is
to keep each series a manageable size for review even if it's obvious that
more work is required.

 Documentation/sysctl/kernel.txt   |  68 ++++++++
 include/linux/migrate.h           |   7 +-
 include/linux/mm.h                |  69 +++++---
 include/linux/mm_types.h          |   7 +-
 include/linux/page-flags-layout.h |  28 ++--
 include/linux/sched.h             |  23 ++-
 include/linux/sched/sysctl.h      |   1 -
 kernel/sched/core.c               |  26 ++-
 kernel/sched/fair.c               | 321
+++++++++++++++++++++++++++++++++-----
 kernel/sched/sched.h              |  12 ++
 kernel/sysctl.c                   |  14 +-
 mm/huge_memory.c                  |  26 ++-
 mm/memory.c                       |  27 ++--
 mm/mempolicy.c                    |   8 +-
 mm/migrate.c                      |  21 +--
 mm/mm_init.c                      |  18 +--
 mm/mmzone.c                       |  12 +-
 mm/mprotect.c                     |  28 ++--
 mm/page_alloc.c                   |   4 +-
 19 files changed, 568 insertions(+), 152 deletions(-)

-- 
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email:  email@kvack.org 
 
CD: 3ms