Features Download
From: Raghavendra K T <raghavendra.kt <at> linux.vnet.ibm.com>
Subject: [PATCH RFC V11 0/18] Paravirtualized ticket spinlocks
Newsgroups: gmane.comp.emulators.xen.devel
Date: Monday 22nd July 2013 06:16:31 UTC (over 4 years ago)
This series replaces the existing paravirtualized spinlock mechanism
with a paravirtualized ticketlock mechanism. The series provides
implementation for both Xen and KVM.

Changes in V11:
 - use safe_halt in lock_spinning path to avoid potential problem 
  in case of irq_handlers taking lock in slowpath (Gleb)
 - add a0 flag for the kick hypercall for future extension  (Gleb)
 - add stubs for missing architecture for kvm_vcpu_schedule() (Gleb)
 - Change hypercall documentation.
 - Rebased to 3.11-rc1

Changes in V10:
Addressed Konrad's review comments:
- Added break in patch 5 since now we know exact cpu to wakeup
- Dropped patch 12 and Konrad needs to revert two patches to enable xen on
  70dd4998, f10cd522c
- Remove TIMEOUT and corrected spacing in patch 15
- Kicked spelling and correct spacing in patches 17, 18 

Changes in V9:
- Changed spin_threshold to 32k to avoid excess halt exits that are
   causing undercommit degradation (after PLE handler improvement).
- Added  kvm_irq_delivery_to_apic (suggested by Gleb)
- Optimized halt exit path to use PLE handler

V8 of PVspinlock was posted last year. After Avi's suggestions to look
at PLE handler's improvements, various optimizations in PLE handling
have been tried.

With this series we see that we could get little more improvements on top
of that. 

Ticket locks have an inherent problem in a virtualized case, because
the vCPUs are scheduled rather than running concurrently (ignoring
gang scheduled vCPUs).  This can result in catastrophic performance
collapses when the vCPU scheduler doesn't schedule the correct "next"
vCPU, and ends up scheduling a vCPU which burns its entire timeslice
spinning.  (Note that this is not the same problem as lock-holder
preemption, which this series also addresses; that's also a problem,
but not catastrophic).

(See Thomas Friebel's talk "Prevent Guests from Spinning Around"
for more details.)

Currently we deal with this by having PV spinlocks, which adds a layer
of indirection in front of all the spinlock functions, and defining a
completely new implementation for Xen (and for other pvops users, but
there are none at present).

PV ticketlocks keeps the existing ticketlock implemenentation
(fastpath) as-is, but adds a couple of pvops for the slow paths:

- If a CPU has been waiting for a spinlock for SPIN_THRESHOLD
  iterations, then call out to the __ticket_lock_spinning() pvop,
  which allows a backend to block the vCPU rather than spinning.  This
  pvop can set the lock into "slowpath state".

- When releasing a lock, if it is in "slowpath state", the call
  __ticket_unlock_kick() to kick the next vCPU in line awake.  If the
  lock is no longer in contention, it also clears the slowpath flag.

The "slowpath state" is stored in the LSB of the within the lock tail
ticket.  This has the effect of reducing the max number of CPUs by
half (so, a "small ticket" can deal with 128 CPUs, and "large ticket"

For KVM, one hypercall is introduced in hypervisor,that allows a vcpu to
another vcpu out of halt state.
The blocking of vcpu is done using halt() in (lock_spinning) slowpath.

Overall, it results in a large reduction in code, it makes the native
and virtualized cases closer, and it removes a layer of indirection
around all the spinlock functions.

The fast path (taking an uncontended lock which isn't in "slowpath"
state) is optimal, identical to the non-paravirtualized case.

The inner part of ticket lock code becomes:
	inc = xadd(&lock->tickets, inc);

	if (likely(inc.head == inc.tail))
		goto out;
	for (;;) {
		unsigned count = SPIN_THRESHOLD;
		do {
			if (ACCESS_ONCE(lock->tickets.head) == inc.tail)
				goto out;
		} while (--count);
		__ticket_lock_spinning(lock, inc.tail);
out:	barrier();
which results in:
	push   %rbp
	mov    %rsp,%rbp

	mov    $0x200,%eax
	lock xadd %ax,(%rdi)
	movzbl %ah,%edx
	cmp    %al,%dl
	jne    1f	# Slowpath if lock in contention

	pop    %rbp

1:	and    $-2,%edx
	movzbl %dl,%esi

2:	mov    $0x800,%eax
	jmp    4f

3:	pause  
	sub    $0x1,%eax
	je     5f

4:	movzbl (%rdi),%ecx
	cmp    %cl,%dl
	jne    3b

	pop    %rbp

5:	callq  *__ticket_lock_spinning
	jmp    2b

with CONFIG_PARAVIRT_SPINLOCKS=n, the code has changed slightly, where
the fastpath case is straight through (taking the lock without
contention), and the spin loop is out of line:

	push   %rbp
	mov    %rsp,%rbp

	mov    $0x100,%eax
	lock xadd %ax,(%rdi)
	movzbl %ah,%edx
	cmp    %al,%dl
	jne    1f

	pop    %rbp

1:	pause  
	movzbl (%rdi),%eax
	cmp    %dl,%al
	jne    1b

	pop    %rbp

The unlock code is complicated by the need to both add to the lock's
"head" and fetch the slowpath flag from "tail".  This version of the
patch uses a locked add to do this, followed by a test to see if the
slowflag is set.  The lock prefix acts as a full memory barrier, so we
can be sure that other CPUs will have seen the unlock before we read
the flag (without the barrier the read could be fetched from the
store queue before it hits memory, which could result in a deadlock).

This is is all unnecessary complication if you're not using PV ticket
locks, it also uses the jump-label machinery to use the standard
"add"-based unlock in the non-PV case.

	     static_key_false(¶virt_ticketlocks_enabled))) {
		arch_spinlock_t prev;
		prev = *lock;
		add_smp(&lock->tickets.head, TICKET_LOCK_INC);

		/* add_smp() is a full mb() */
		if (unlikely(lock->tickets.tail & TICKET_SLOWPATH_FLAG))
			__ticket_unlock_slowpath(lock, prev);
	} else
		__add(&lock->tickets.head, TICKET_LOCK_INC, UNLOCK_LOCK_PREFIX);
which generates:
	push   %rbp
	mov    %rsp,%rbp

	nop5	# replaced by 5-byte jmp 2f when PV enabled

	# non-PV unlock
	addb   $0x2,(%rdi)

1:	pop    %rbp

### PV unlock ###
2:	movzwl (%rdi),%esi	# Fetch prev

	lock addb $0x2,(%rdi)	# Do unlock

	testb  $0x1,0x1(%rdi)	# Test flag
	je     1b		# Finished if not set

### Slow path ###
	add    $2,%sil		# Add "head" in old lock state
	mov    %esi,%edx
	and    $0xfe,%dh	# clear slowflag for comparison
	movzbl %dh,%eax
	cmp    %dl,%al		# If head == tail (uncontended)
	je     4f		# clear slowpath flag

	# Kick next CPU waiting for lock
3:	movzbl %sil,%esi
	callq  *pv_lock_ops.kick

	pop    %rbp

	# Lock no longer contended - clear slowflag
4:	mov    %esi,%eax
	lock cmpxchg %dx,(%rdi)	# cmpxchg to clear flag
	cmp    %si,%ax
	jne    3b		# If clear failed, then kick

	pop    %rbp

So when not using PV ticketlocks, the unlock sequence just has a
5-byte nop added to it, and the PV case is reasonable straightforward
aside from requiring a "lock add".

pvspinlock shows benefits for overcommit ratio > 1 for PLE enabled cases,
and undercommits results are flat.

For non PLE results are much better for smaller VMs. 

This series  with 3.11.rc1 as base is giving
28 to 50% improvement for ebizzy, 8 to 22% for dbench on  32 core machinne
HT disabled with 32 vcpu guest. 

On 32cpu, 16core machine (HT on) with 16vcpu guests, results showed
1,3,61,77% improvement
 for .5x,1x,1.5x,2x respectively for ebizzy. dbench results were almost
flat with -1% to +2%.

Your suggestions and comments are welcome.

github link: https://github.com/ktraghavendra/linux/tree/pvspinlock_v11

Please note that we set SPIN_THRESHOLD = 32k with this series,
that would eatup little bit of overcommit performance of PLE machines
and overall performance of non-PLE machines.

The older series[3] was tested by Attilio for Xen implementation.

Note that Konrad needs to revert below two patches to enable xen on hvm 
  70dd4998, f10cd522c

Jeremy Fitzhardinge (9):
 x86/spinlock: Replace pv spinlocks with pv ticketlocks
 x86/ticketlock: Collapse a layer of functions
 xen: Defer spinlock setup until boot CPU setup
 xen/pvticketlock: Xen implementation for PV ticket locks
 xen/pvticketlocks: Add xen_nopvspin parameter to disable xen pv
 x86/pvticketlock: Use callee-save for lock_spinning
 x86/pvticketlock: When paravirtualizing ticket locks, increment by 2
 x86/ticketlock: Add slowpath logic
 xen/pvticketlock: Allow interrupts to be enabled while blocking

Andrew Jones (1):
 jump_label: Split jumplabel ratelimit

Srivatsa Vaddagiri (3):
 kvm hypervisor : Add a hypercall to KVM hypervisor to support
 kvm guest : Add configuration support to enable debug information for KVM
 kvm : Paravirtual ticketlocks support for linux guests running on KVM

Raghavendra K T (5):
 x86/ticketlock: Don't inline _spin_unlock when using paravirt spinlocks
 kvm : Fold pv_unhalt flag into GET_MP_STATE ioctl to aid migration
 kvm hypervisor: Simplify kvm_for_each_vcpu with kvm_irq_delivery_to_apic
 Documentation/kvm : Add documentation on Hypercalls and features used for
PV spinlock
 kvm hypervisor: Add directed yield in vcpu block path

Link in V8 has links to previous patch series and also whole history.

[1]. V10 PV Ticketspinlock for Xen/KVM link: https://lkml.org/lkml/2013/6/24/252
[2]. V9 PV Ticketspinlock for Xen/KVM link:  https://lkml.org/lkml/2013/6/1/168
[3]. V8 PV Ticketspinlock for Xen/KVM link:  https://lkml.org/lkml/2012/5/2/119

 Documentation/virtual/kvm/cpuid.txt      |   4 +
 Documentation/virtual/kvm/hypercalls.txt |  14 ++
 arch/arm/include/asm/kvm_host.h          |   5 +
 arch/arm64/include/asm/kvm_host.h        |   5 +
 arch/ia64/include/asm/kvm_host.h         |   5 +
 arch/mips/include/asm/kvm_host.h         |   5 +
 arch/powerpc/include/asm/kvm_host.h      |   5 +
 arch/s390/include/asm/kvm_host.h         |   5 +
 arch/x86/Kconfig                         |  10 +
 arch/x86/include/asm/kvm_host.h          |   7 +-
 arch/x86/include/asm/kvm_para.h          |  14 +-
 arch/x86/include/asm/paravirt.h          |  32 +--
 arch/x86/include/asm/paravirt_types.h    |  10 +-
 arch/x86/include/asm/spinlock.h          | 128 ++++++----
 arch/x86/include/asm/spinlock_types.h    |  16 +-
 arch/x86/include/uapi/asm/kvm_para.h     |   1 +
 arch/x86/kernel/kvm.c                    | 259 +++++++++++++++++++++
 arch/x86/kernel/paravirt-spinlocks.c     |  18 +-
 arch/x86/kvm/cpuid.c                     |   3 +-
 arch/x86/kvm/lapic.c                     |   5 +-
 arch/x86/kvm/x86.c                       |  39 +++-
 arch/x86/xen/smp.c                       |   2 +-
 arch/x86/xen/spinlock.c                  | 387
 include/linux/jump_label.h               |  26 +--
 include/linux/jump_label_ratelimit.h     |  34 +++
 include/linux/kvm_host.h                 |   2 +-
 include/linux/perf_event.h               |   1 +
 include/uapi/linux/kvm_para.h            |   1 +
 kernel/jump_label.c                      |   1 +
 virt/kvm/kvm_main.c                      |   6 +-
 30 files changed, 665 insertions(+), 385 deletions(-)

To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
CD: 3ms