際際滷

際際滷Share a Scribd company logo

ESXi 5.x CPU scheduler

Anton Zhbankov
Anton Zhbankov

亠舒仆亳亰仄 舒弍仂 仗仍舒仆亳仂于亳从舒 仗仂亠仂舒 ESXi.

1 of 31
1息 Copyright 2011 EMC Corporation. All rights reserved. VMware ESXi 5.1 仍舒仆亳仂于亳从 仗仂亠仂舒 仆仂仆 弍舒仆从仂于 http://blog.vadmin.ru
2息 Copyright 2011 EMC Corporation. All rights reserved. 丐亠仄亳仆仂仍仂亞亳 Socket pCPU 亊亟仂 仂亞亳亠从亳亶 仗仂亠仂 仗亳 HT Cache LLC Last level cache 仂仍亠亟仆亳亶 仂于亠仆 从舒 仗亠亠亟 仂弍舒亠仆亳亠仄 于 仗舒仄 vCPU (World) NUMA
3息 Copyright 2011 EMC Corporation. All rights reserved. 弌仂仂礌亳 world
4息 Copyright 2011 EMC Corporation. All rights reserved. 丼仂 仗仍舒仆亳仂于舒? 仂仂亳 亳亰 仂亟仆仂亞仂 亳仍亳 仆亠从仂仍从仂 world 1 world 仆舒 从舒亢亟亶 vCPU 舒 舒从亢亠 亠于ﾑ world, 仂弍仍亢亳于舒ﾑ亳亠 仄, 从仍舒于亳舒 亳 legacy I/O 仂亶于舒 丐亠仂亠亳亠从亳 亟仍 1vCPU 100+% 于仂亰仄仂亢仆仂, 仆仂 仄舒仍仂于亠仂仆仂 VMkernel management worlds 亠于舒仆亳 亳 从仂仆亠从 于于仂亟舒-于于仂亟舒 (IO)
5息 Copyright 2011 EMC Corporation. All rights reserved. 丼仂 仗仍舒仆亳仂于舒? 仍舒仆亳仂于亳从 于亰于舒亠 仗亳 亳亠亠仆亳亳 从于舒仆舒 于亠仄亠仆亳, 于亟亠仍亠仆仆仂亞仂 world 50 仄 仗仂 仄仂仍舒仆亳 亳弍仂 仗仂 仗亠亠仂亟 world 于 仂仂礌亳亠 Wait 亠 仍亠亟ﾑ亳亶 world 于 仂仂礌亳亳 Ready 于 仍仂从舒仍仆仂亶 亳仍亳 亟舒仍亠仆仆仂亶 仂亠亠亟亳 仍亳 仆亠 仆亳 仂亟仆仂亞仂 于 仂仂礌亳亳 Ready, 仂 仗仍舒仆亳亠 world 于 仂仂礌亳亳 Idle 丐舒从亢亠 仗仍舒仆亳仂于亳从 于亰于舒亠 仗亳 仗亠亠仂亟亠 亳亰 Wait 于 Ready (仆舒仗. 仗仂 仗亠于舒仆亳 亳仍亳 亳亞仆舒仍)
6息 Copyright 2011 EMC Corporation. All rights reserved. 仍亞仂亳仄 仗仂仗仂亳仂仆舒仍仆 亟仂仍亠亶 仍亞仂亳仄 于弍仂舒 world 亟仍 亳仗仂仍仆亠仆亳 舒 仂仆仂于亠 Share, Limit, Reserve World 仄仂亢亠 仆亠 仗仂仍仆仂 仗仂亠弍亳 于亟亠仍亠仆仆亠 亠 亳亰-亰舒 舒亞仄亠仆舒亳亳 弌舒仆仂于亳 于仂从仂亞仂 仗亳仂亳亠舒 亳 仗仂仗舒亟舒亠 于 仂亠亠亟 亳亳亠从亳仄 舒仆仂于亳 弍亳仍仍亳仆亞 从仂 从仂仍从仂 仗仂亠弍亳仍
7息 Copyright 2011 EMC Corporation. All rights reserved. Shares
8息 Copyright 2011 EMC Corporation. All rights reserved. Reserve
9息 Copyright 2011 EMC Corporation. All rights reserved. Limit
10息 Copyright 2011 EMC Corporation. All rights reserved. 亳仂亳亠 UNIX 亳仂亳亠 仗仂亳亰于仂仍仆仂 仆舒亰仆舒舒亠 仗仂仍亰仂于舒亠仍亠仄 仄亠亠 亳仍仂于仂亠 亰仆舒亠仆亳亠 亳 舒于仆亠仆亳亳 亳仄亠亠 亰仆舒亠仆亳亠 仂仍从仂 从仂亞仂 仗亳仂亳亠 弍仂仍亠 ESXi 亳仂亳亠 仗仂仂礌仆仂 仗亠亠于亳仍磳 舒 仂仆仂于亠 于亟亠仍亠仆仆 亳 亠舒仍仆仂 仗仂亠弍仍亠仆仆 亠仂于 仂仍亰仂于舒亠仍 仆亠 仄仂亢亠 仆舒亰仆舒亳 仗亳仂亳亠 world 仆舒仗礆, 仂仍从仂 亠亠亰 Share, Limit 亳 Reserve
11息 Copyright 2011 EMC Corporation. All rights reserved. Co-Scheduling 仍 vSMP (vCPU > 1) 于亠 vCPU world 仆亠仂弍仂亟亳仄仂 亳仗仂仍仆 仂亟仆仂于亠仄亠仆仆仂 亳 仆亠仗舒舒仍仍亠仍仆仂仄 亳仗仂仍仆亳亳 亠亰从仂 于仂亰舒舒亠 亰舒亟亠亢从舒 仗亳 仄亠亢仗仂亠仆仂仄 于亰舒亳仄仂亟亠亶于亳亳 舒从 亳仂亞 仗亠亠舒仂亟 亠仂于 Skew 亳仍仂于仂亶 仗仂从舒亰舒亠仍 舒亰仆仂亳 仄亠亢亟 仗仂亞亠仂仄 vCPU 仂亟仆仂亶 仄舒亳仆
12息 Copyright 2011 EMC Corporation. All rights reserved. Strict Co-Scheduling (ESX 2.x) 亳 仗亠于亠仆亳亳 仗仂仂亞仂于仂亞仂 亰仆舒亠仆亳 Skew 仄舒亳仆舒 亠仍亳从仂仄 仂舒仆舒于仍亳于舒仍舒 仗仂仍仆亠仆亳亠 于仂亰仂弍仆仂于仍磿仂 仂仍从仂 仗亳 仆舒仍亳亳亳 于仂弍仂亟仆 舒亶仄仍仂仂于 亟仍 于亠 vCPU Idle vCPU 于亠亞亟舒 亳舒亠 舒于仆仄 舒仄仂仄 弍仂仄 vCPU 4x vCPU 仄仂亞仍舒 仆亠 仗仂仗舒 于 仂亠亠亟 仗亳 仆舒仍亳亳亳 3 于仂弍仂亟仆 pCPU 仂亞 舒亞仄亠仆舒亳 仗仂亠仂舒, 仆亳亰从舒 仗仂亳亰于仂亟亳亠仍仆仂
13息 Copyright 2011 EMC Corporation. All rights reserved. Strict Co-Scheduling (ESX 2.x) 亠亞仂 61% pCPU1 pCPU2 pCPU3 pCPU4
14息 Copyright 2011 EMC Corporation. All rights reserved. Relaxed Co-Scheduling 舒亢亟亶 vCPU 亳亰仄亠磳 仂弍于亠仆仆亶 仗仂亞亠 亳 仗亠于亠仆亳亳 仗仂仂亞仂于仂亞仂 亰仆舒亠仆亳 Skew 仍亳亟亳ﾑ亳亶 vCPU 仗亠亠仂亟亳 于 仂仂礌亳亳 Co-Stop 仂 仄亠亠 仆亳亢亠仆亳 亰仆舒亠仆亳 Skew 亳 仗亳 仆舒仍亳亳亳 于仂弍仂亟仆仂亞仂 pCPU world 仄仂亢亠 仗亠亠亶亳 亳亰 仂仂礌亳 Co- Stop 于 Co-Start 仆舒亳亠仍仆仂亠 仆亳亢亠仆亳亠 舒亞仄亠仆舒亳亳, 4x vCPU 仄舒亳仆舒 仄仂亢亠 舒弍仂舒 仗亳 于亠亞仂 1 于仂弍仂亟仆仂仄 pCPU
15息 Copyright 2011 EMC Corporation. All rights reserved. Relaxed Co-Scheduling pCPU1 pCPU2 pCPU3 pCPU4 丕亢亠 73%
16息 Copyright 2011 EMC Corporation. All rights reserved. 舒仍舒仆亳仂于从舒 仆舒亞亰从亳 亳亞舒亳 world Pull 亳仆亳亳亳亠 pCPU 仗亳 仗亠亠仂亟亠 于 Idle Push 亳仆亳亳亳亠 vCPU 仗亳 仗亠亠仂亟亠 于 Ready 弌仂亳仄仂 仄亳亞舒亳亳 弌仆仂于舒 仗仂亞亠于舒 从 仂亠仆 亳仍仆仂 于仍亳磳 仆舒 仂亟亠仍仆亠 于亳亟 仆舒亞亰从亳 Goodness of Migration 仍 从舒亢亟仂亶 仗舒 vCPU-pCPU 于亳仍磳 亰仆舒亠仆亳亠 仗仂仍亰 仄亳亞舒亳亳 (Goodness) 亳 仂亠于仍磳 仄亳亞舒亳 仄舒从亳仄舒仍仆仂亶 仗仂仍亰仂亶
17息 Copyright 2011 EMC Corporation. All rights reserved. 仂仍亰舒 仄亳亞舒亳亳 CPU load 亳 于仂从仂亶 亰舒亞亰从亠 pCPU 亳仂仆亳从舒 亳 仆亳亰从仂亶 pCPU 仆舒亰仆舒亠仆亳 仗仂仍亰舒 亳舒亠 于仂从仂亶 Last-Level Cache 仗亠于 仂亠亠亟 舒仄舒亳于舒ﾑ pCPU 仂弍亳仄 LLC, 仗仂从仂仍从 仂弍舒亠仆亳亠 从 仗舒仄亳 仆舒 仗仂磲仂从 仄亠亟仍亠仆仆亠亠 仂仂弍亠仆仆仂亳 于舒亢仆仂 亟仍 vSMP 仄舒亳仆 Hyper-Threading 仂仍亰舒 仄亳亞舒亳亳 仗仂于舒亠, 亠仍亳 仂亠亟仆亳亶 仍仂亞亳亠从亳亶 仗仂亠仂 HT 仗舒 仄亠仆亠亠 亰舒亞亢亠仆
18息 Copyright 2011 EMC Corporation. All rights reserved. 仂仍亰舒 仄亳亞舒亳亳 丐仂仗仂仍仂亞亳亠从舒 亟亳舒仆亳 仍 pCPU-pCPU 舒亳于舒亠 仂仗仂仍仂亞亳亠从舒 亟亳舒仆亳 HT 仗舒舒 亟亠仍亳 仂弍亳亶 L1 从, 仂 亟亠仍舒亠 丐 仄亠仆亠, 亠仄 亟仍 亟于 pCPU 仂弍亳仄 LLC pCPU 弍亠亰 仂弍亠亞仂 LLC 亳仄亠ﾑ 亠亠 弍仂仍 丐 弌仂亳仄仂 仄亳亞舒亳亳 仗仂亳于 仗仂仍亰 仄亳亞舒亳亳 仂亞亠于 从舒 舒仂亟亠 亠 仂亢亠 仗仂仍亳, 仂 仂舒于亳 从舒从 亠 亟亠亠于仍亠, 亠仄 仄亳亞亳仂于舒
19息 Copyright 2011 EMC Corporation. All rights reserved. 仂仍亰舒 仄亳亞舒亳亳 Co-scheduling 仍舒仆亳仂于亳从 仄舒从亳仄舒仍仆仂 亳亰弍亠亞舒亠 舒亰仄亠亠仆亳 亟于 vCPU 仂亟仆仂亶 仆舒 仂亟仆仂仄 亳 仂仄 亢亠 pCPU 于仂 亳亰弍亠亢舒仆亳亠 舒亰弍亠亞舒 仂仍亰舒 亳舒亠 亰仆舒亳亠仍仆仂 于亠, 亠仍亳 仆舒 pCPU 仆舒亰仆舒亠仆亳 仆亠 舒亰仄亠亠仆 亟亞仂亶 vCPU 仂亶 亢亠 仂仄仄仆亳从舒亳 仄亠亢亟 从仂仆亠从舒仄亳 仍舒仆亳仂于亳从 仂亠仆亳于舒亠 仂于亠仆 仄亠亢仗仂亠仆仂亞仂 于亰舒亳仄仂亟亠亶于亳 vCPU 于仂从亳仄 仂于仆亠仄 仂弍亠仆亳 弍仂仍亠亶 于亠仂仆仂 弍亟 舒亰仄亠亠仆 束仗仂弍仍亳亰仂亳損, 仂弍 于亠仍亳亳 仗仂仗舒亟舒仆亳 于 从
20息 Copyright 2011 EMC Corporation. All rights reserved. Hyper-Threading HT 仗仂亠仂 = 2 仗仂仂从舒 仆舒 磲仂 仂亳亰于仂亟亳亠仍仆仂 从舒亢亟仂亞仂 仗仂仂从舒 亰舒于亳亳 仂 于仂仂亞仂 仂仂从 仂 于仂弍仂亟仆仄 2仄 亟舒亠 弍仂仍亠 于亳仍亳亠仍仆仂亶 仄仂仆仂亳, 亠仄 亠仍亳 仂弍舒 仗仂仂从舒 亰舒仆 亳仍仍亳仆亞 亟仍 HT 仗仂亠仂仂于 仂仍亳舒亠 亳仆亳仗 亠仆仂亳 仂仍亠亟仆亳亠 仗仂从仂仍亠仆亳 仗仂亠仂仂于 仍亠 仗舒于仍ﾑ ESXi 5.x 弍仂仍亠亶 于亠仂仆仂 于弍亳舒亠 HT 仗仂仂从, 亠仄 舒仆亠亠 仂仍仆仂 于仂弍仂亟仆仂亠 磲仂 于亠 亠亠 亳仄亠亠 弍仂仍亳亶 仗亳仂亳亠
21息 Copyright 2011 EMC Corporation. All rights reserved. NUMA NUMA Non Uniform Memory Access NUMA 亰亠仍 仂亠仂 (Socket) 亞仂 仗舒仄 (仍仂从舒仍仆舒) 弌仂亳仄仂 亟仂仗舒 从 亢仂亶 仗舒仄亳 于亠 舒从亳仄舒仍仆仂 仂舒于舒 于 舒仄从舒 亰仍舒 vCPU 舒亰仄亠舒ﾑ 仂仍从仂 于 NUMA home (亟仂仄舒仆亠仄 亰仍亠) 弌仄亠仆舒 NUMA home 亳亞舒亳 仗舒仄亳 亟亠仍仂 亟仂仂亞仂亠 亠于仍磳 仂仍从仂 亠仍亳 仂舒亠 于 仆仂于仂仄 亰仍亠 亟仂仍亞仂
22息 Copyright 2011 EMC Corporation. All rights reserved. NUMA 仄亳亞舒亳 仍 从舒从仂于亠仄亠仆仆仂亶 弍舒仍舒仆亳仂于从亳 CPU 仍 仗仂于亠仆亳 仍仂从舒仍仆仂亳 仗舒仄亳 仍亳 仗舒仄 亠亠 仆亠 弍仍舒 仄亳亞亳仂于舒仆舒, 仂 亳仄亠亠 仄仍 仄亳亞亳仂于舒 仆舒亰舒亟 舒仂亶 从仂仄仄仆亳从舒亳亠亶 仄亠亢亟 仆亳仄亳 /Numa/LocalityWeightActionAffinity 0 仍 仂弍亠仗亠亠仆亳 仗亳仆亳仗舒 亠仆仂亳 于 亟仂仍亞仂亶 仗亠仗亠从亳于亠 /Numa/LTermFairnessInterval 0
23息 Copyright 2011 EMC Corporation. All rights reserved. Wide NUMA 丿亳仂从舒 NUMA 仄舒亳仆舒 仂仍亳亠于仂 vCPU > 从仂仍亳亠于舒 pCPU 于 NUMA 亰仍亠 丿亳仂从舒 仄舒亳仆舒 舒亰弍亳于舒亠 仆舒 仆亠从仂仍从仂 从仍亳亠仆仂于 舒亢亟亶 从仍亳亠仆 亟仂仍亢亠仆 仗仂仄亠舒 于 亰仍亠 HT HT 亟于舒亳于舒亠 从仂仍亳亠于仂 仍仂亞亳亠从亳 磲亠, 仗仂仂仄 8 vCPU 仆舒 4 磲亠仆仂仄 仗仂亠仂亠 HT 弍亟亠 仗仂仗舒亟舒 于 舒亰仄亠 NUMA 亰仍舒 numa.vcpu.preferHT TRUE
24息 Copyright 2011 EMC Corporation. All rights reserved. 舒仄 亟仍 Wide NUMA 舒于仆仂仄亠仆仂 亠亠亟亠 亟仍 亳仂从亳 仄舒亳仆 于 4.x NUMA 从于舒亠 仂 亞仂亠于仂亶 弌 vSphere 5.x vNUMA NUMA 仗亠亰亠仆亠 亞仂亠于仂亶 弌
25息 Copyright 2011 EMC Corporation. All rights reserved. 舒仄 亟仍 Wide NUMA
26息 Copyright 2011 EMC Corporation. All rights reserved. vNUMA SPEC OMP
27息 Copyright 2011 EMC Corporation. All rights reserved. 仂弍仍亠仄 vNUMA 亳亰亳亠从仂仄 仄亳亠 仂仗仂仍仂亞亳 NUMA 仆亠亳亰仄亠仆仆舒 于亳舒仍仆仂仄 仄亠仆磳 vMotion 仆舒 仂 亳仆仂亶 仂仗仂仍仂亞亳亠亶 亠 于亠 仗亳仍仂亢亠仆亳 仗仂仆亳仄舒ﾑ 亳亰仄亠仆亠仆亳亠 仂仗仂仍仂亞亳亳 于仂从仂仆亳亞舒亳 numa.autosize TRUE numa.autosize.once FALSE 丐亠弍亠 仗亠亠亰舒亞亰从舒
28息 Copyright 2011 EMC Corporation. All rights reserved. 亠仂仂于亠于亳亠 NUMA 亳 vNUMA vNUMA < NUMA vNUMA 亠仍亳从仂仄 仗仂仄亠舒亠 于 亰亠仍. 亳从舒从亳 仗仂弍仍亠仄 vNUMA = x*NUMA vNUMA 舒亰弍亳于舒亠 仆舒 x NUMA 亰仍仂于 亳 仗舒仄 亠亠亟亠 仄亠亢亟 仆亳仄亳 vNUMA != x*NUMA vNUMA 于亠 亠亠 仂舒仆磳 亟仍 亞仂亠于仂亶 弌 NUMA 从仍亳亠仆 仂仗亠亟亠仍ﾑ 于 亰舒于亳亳仄仂亳 仂 舒亰仄亠舒 NUMA 亰仍舒, 舒 仆亠 vNUMA. 舒仄 亠亠亟亠 仄亠亢亟 仆亳仄亳 仂 舒从 vNUMA 仗仂仂 仆亠 舒弍仂舒亠
29息 Copyright 2011 EMC Corporation. All rights reserved. 于仂亟 亰弍亠亞舒亶亠 亠仂于 (vCPU) 仆舒 于磻亳亶 仍舒亶 丕亰仆舒亶亠 于仂 亳仆舒从 NUMA 仂亠仂仆舒 舒亳亠从舒 仂仍亳亠 仄舒亳仆 亠弍ﾑ 仗仍舒仆亳仂于舒仆亳 亰弍亠亞舒亶亠 仄亠仆 亰仆舒亠仆亳 舒亳亠仆仆 仗舒舒仄亠仂于 仆舒亠仆亳亶 仗仂 仄仂仍舒仆亳 亟仂舒仂仆仂 于 99% 仍舒亠于 弌仆舒舒仍舒 仗仂亟仄舒仍 仗仂仂仄 亟亠仍舒仍! 仆亠 仆舒仂弍仂仂!
30息 Copyright 2011 EMC Corporation. All rights reserved. 仂仆舒从 仆仂仆 弍舒仆从仂于 EMC Senior SE / Software Defined Specialist anton.zhbankov@emc.com anton@vadmin.ru http://blog.vadmin.ru @antonvirtual + Facebook, LinkedIn etc
31息 Copyright 2011 EMC Corporation. All rights reserved. THANK YOU

More Related Content

ESXi 5.x CPU scheduler

  • 1. 1息 Copyright 2011 EMC Corporation. All rights reserved. VMware ESXi 5.1 仍舒仆亳仂于亳从 仗仂亠仂舒 仆仂仆 弍舒仆从仂于 http://blog.vadmin.ru
  • 2. 2息 Copyright 2011 EMC Corporation. All rights reserved. 丐亠仄亳仆仂仍仂亞亳 Socket pCPU 亊亟仂 仂亞亳亠从亳亶 仗仂亠仂 仗亳 HT Cache LLC Last level cache 仂仍亠亟仆亳亶 仂于亠仆 从舒 仗亠亠亟 仂弍舒亠仆亳亠仄 于 仗舒仄 vCPU (World) NUMA
  • 3. 3息 Copyright 2011 EMC Corporation. All rights reserved. 弌仂仂礌亳 world
  • 4. 4息 Copyright 2011 EMC Corporation. All rights reserved. 丼仂 仗仍舒仆亳仂于舒? 仂仂亳 亳亰 仂亟仆仂亞仂 亳仍亳 仆亠从仂仍从仂 world 1 world 仆舒 从舒亢亟亶 vCPU 舒 舒从亢亠 亠于ﾑ world, 仂弍仍亢亳于舒ﾑ亳亠 仄, 从仍舒于亳舒 亳 legacy I/O 仂亶于舒 丐亠仂亠亳亠从亳 亟仍 1vCPU 100+% 于仂亰仄仂亢仆仂, 仆仂 仄舒仍仂于亠仂仆仂 VMkernel management worlds 亠于舒仆亳 亳 从仂仆亠从 于于仂亟舒-于于仂亟舒 (IO)
  • 5. 5息 Copyright 2011 EMC Corporation. All rights reserved. 丼仂 仗仍舒仆亳仂于舒? 仍舒仆亳仂于亳从 于亰于舒亠 仗亳 亳亠亠仆亳亳 从于舒仆舒 于亠仄亠仆亳, 于亟亠仍亠仆仆仂亞仂 world 50 仄 仗仂 仄仂仍舒仆亳 亳弍仂 仗仂 仗亠亠仂亟 world 于 仂仂礌亳亠 Wait 亠 仍亠亟ﾑ亳亶 world 于 仂仂礌亳亳 Ready 于 仍仂从舒仍仆仂亶 亳仍亳 亟舒仍亠仆仆仂亶 仂亠亠亟亳 仍亳 仆亠 仆亳 仂亟仆仂亞仂 于 仂仂礌亳亳 Ready, 仂 仗仍舒仆亳亠 world 于 仂仂礌亳亳 Idle 丐舒从亢亠 仗仍舒仆亳仂于亳从 于亰于舒亠 仗亳 仗亠亠仂亟亠 亳亰 Wait 于 Ready (仆舒仗. 仗仂 仗亠于舒仆亳 亳仍亳 亳亞仆舒仍)
  • 6. 6息 Copyright 2011 EMC Corporation. All rights reserved. 仍亞仂亳仄 仗仂仗仂亳仂仆舒仍仆 亟仂仍亠亶 仍亞仂亳仄 于弍仂舒 world 亟仍 亳仗仂仍仆亠仆亳 舒 仂仆仂于亠 Share, Limit, Reserve World 仄仂亢亠 仆亠 仗仂仍仆仂 仗仂亠弍亳 于亟亠仍亠仆仆亠 亠 亳亰-亰舒 舒亞仄亠仆舒亳亳 弌舒仆仂于亳 于仂从仂亞仂 仗亳仂亳亠舒 亳 仗仂仗舒亟舒亠 于 仂亠亠亟 亳亳亠从亳仄 舒仆仂于亳 弍亳仍仍亳仆亞 从仂 从仂仍从仂 仗仂亠弍亳仍
  • 7. 7息 Copyright 2011 EMC Corporation. All rights reserved. Shares
  • 8. 8息 Copyright 2011 EMC Corporation. All rights reserved. Reserve
  • 9. 9息 Copyright 2011 EMC Corporation. All rights reserved. Limit
  • 10. 10息 Copyright 2011 EMC Corporation. All rights reserved. 亳仂亳亠 UNIX 亳仂亳亠 仗仂亳亰于仂仍仆仂 仆舒亰仆舒舒亠 仗仂仍亰仂于舒亠仍亠仄 仄亠亠 亳仍仂于仂亠 亰仆舒亠仆亳亠 亳 舒于仆亠仆亳亳 亳仄亠亠 亰仆舒亠仆亳亠 仂仍从仂 从仂亞仂 仗亳仂亳亠 弍仂仍亠 ESXi 亳仂亳亠 仗仂仂礌仆仂 仗亠亠于亳仍磳 舒 仂仆仂于亠 于亟亠仍亠仆仆 亳 亠舒仍仆仂 仗仂亠弍仍亠仆仆 亠仂于 仂仍亰仂于舒亠仍 仆亠 仄仂亢亠 仆舒亰仆舒亳 仗亳仂亳亠 world 仆舒仗礆, 仂仍从仂 亠亠亰 Share, Limit 亳 Reserve
  • 11. 11息 Copyright 2011 EMC Corporation. All rights reserved. Co-Scheduling 仍 vSMP (vCPU > 1) 于亠 vCPU world 仆亠仂弍仂亟亳仄仂 亳仗仂仍仆 仂亟仆仂于亠仄亠仆仆仂 亳 仆亠仗舒舒仍仍亠仍仆仂仄 亳仗仂仍仆亳亳 亠亰从仂 于仂亰舒舒亠 亰舒亟亠亢从舒 仗亳 仄亠亢仗仂亠仆仂仄 于亰舒亳仄仂亟亠亶于亳亳 舒从 亳仂亞 仗亠亠舒仂亟 亠仂于 Skew 亳仍仂于仂亶 仗仂从舒亰舒亠仍 舒亰仆仂亳 仄亠亢亟 仗仂亞亠仂仄 vCPU 仂亟仆仂亶 仄舒亳仆
  • 12. 12息 Copyright 2011 EMC Corporation. All rights reserved. Strict Co-Scheduling (ESX 2.x) 亳 仗亠于亠仆亳亳 仗仂仂亞仂于仂亞仂 亰仆舒亠仆亳 Skew 仄舒亳仆舒 亠仍亳从仂仄 仂舒仆舒于仍亳于舒仍舒 仗仂仍仆亠仆亳亠 于仂亰仂弍仆仂于仍磿仂 仂仍从仂 仗亳 仆舒仍亳亳亳 于仂弍仂亟仆 舒亶仄仍仂仂于 亟仍 于亠 vCPU Idle vCPU 于亠亞亟舒 亳舒亠 舒于仆仄 舒仄仂仄 弍仂仄 vCPU 4x vCPU 仄仂亞仍舒 仆亠 仗仂仗舒 于 仂亠亠亟 仗亳 仆舒仍亳亳亳 3 于仂弍仂亟仆 pCPU 仂亞 舒亞仄亠仆舒亳 仗仂亠仂舒, 仆亳亰从舒 仗仂亳亰于仂亟亳亠仍仆仂
  • 13. 13息 Copyright 2011 EMC Corporation. All rights reserved. Strict Co-Scheduling (ESX 2.x) 亠亞仂 61% pCPU1 pCPU2 pCPU3 pCPU4
  • 14. 14息 Copyright 2011 EMC Corporation. All rights reserved. Relaxed Co-Scheduling 舒亢亟亶 vCPU 亳亰仄亠磳 仂弍于亠仆仆亶 仗仂亞亠 亳 仗亠于亠仆亳亳 仗仂仂亞仂于仂亞仂 亰仆舒亠仆亳 Skew 仍亳亟亳ﾑ亳亶 vCPU 仗亠亠仂亟亳 于 仂仂礌亳亳 Co-Stop 仂 仄亠亠 仆亳亢亠仆亳 亰仆舒亠仆亳 Skew 亳 仗亳 仆舒仍亳亳亳 于仂弍仂亟仆仂亞仂 pCPU world 仄仂亢亠 仗亠亠亶亳 亳亰 仂仂礌亳 Co- Stop 于 Co-Start 仆舒亳亠仍仆仂亠 仆亳亢亠仆亳亠 舒亞仄亠仆舒亳亳, 4x vCPU 仄舒亳仆舒 仄仂亢亠 舒弍仂舒 仗亳 于亠亞仂 1 于仂弍仂亟仆仂仄 pCPU
  • 15. 15息 Copyright 2011 EMC Corporation. All rights reserved. Relaxed Co-Scheduling pCPU1 pCPU2 pCPU3 pCPU4 丕亢亠 73%
  • 16. 16息 Copyright 2011 EMC Corporation. All rights reserved. 舒仍舒仆亳仂于从舒 仆舒亞亰从亳 亳亞舒亳 world Pull 亳仆亳亳亳亠 pCPU 仗亳 仗亠亠仂亟亠 于 Idle Push 亳仆亳亳亳亠 vCPU 仗亳 仗亠亠仂亟亠 于 Ready 弌仂亳仄仂 仄亳亞舒亳亳 弌仆仂于舒 仗仂亞亠于舒 从 仂亠仆 亳仍仆仂 于仍亳磳 仆舒 仂亟亠仍仆亠 于亳亟 仆舒亞亰从亳 Goodness of Migration 仍 从舒亢亟仂亶 仗舒 vCPU-pCPU 于亳仍磳 亰仆舒亠仆亳亠 仗仂仍亰 仄亳亞舒亳亳 (Goodness) 亳 仂亠于仍磳 仄亳亞舒亳 仄舒从亳仄舒仍仆仂亶 仗仂仍亰仂亶
  • 17. 17息 Copyright 2011 EMC Corporation. All rights reserved. 仂仍亰舒 仄亳亞舒亳亳 CPU load 亳 于仂从仂亶 亰舒亞亰从亠 pCPU 亳仂仆亳从舒 亳 仆亳亰从仂亶 pCPU 仆舒亰仆舒亠仆亳 仗仂仍亰舒 亳舒亠 于仂从仂亶 Last-Level Cache 仗亠于 仂亠亠亟 舒仄舒亳于舒ﾑ pCPU 仂弍亳仄 LLC, 仗仂从仂仍从 仂弍舒亠仆亳亠 从 仗舒仄亳 仆舒 仗仂磲仂从 仄亠亟仍亠仆仆亠亠 仂仂弍亠仆仆仂亳 于舒亢仆仂 亟仍 vSMP 仄舒亳仆 Hyper-Threading 仂仍亰舒 仄亳亞舒亳亳 仗仂于舒亠, 亠仍亳 仂亠亟仆亳亶 仍仂亞亳亠从亳亶 仗仂亠仂 HT 仗舒 仄亠仆亠亠 亰舒亞亢亠仆
  • 18. 18息 Copyright 2011 EMC Corporation. All rights reserved. 仂仍亰舒 仄亳亞舒亳亳 丐仂仗仂仍仂亞亳亠从舒 亟亳舒仆亳 仍 pCPU-pCPU 舒亳于舒亠 仂仗仂仍仂亞亳亠从舒 亟亳舒仆亳 HT 仗舒舒 亟亠仍亳 仂弍亳亶 L1 从, 仂 亟亠仍舒亠 丐 仄亠仆亠, 亠仄 亟仍 亟于 pCPU 仂弍亳仄 LLC pCPU 弍亠亰 仂弍亠亞仂 LLC 亳仄亠ﾑ 亠亠 弍仂仍 丐 弌仂亳仄仂 仄亳亞舒亳亳 仗仂亳于 仗仂仍亰 仄亳亞舒亳亳 仂亞亠于 从舒 舒仂亟亠 亠 仂亢亠 仗仂仍亳, 仂 仂舒于亳 从舒从 亠 亟亠亠于仍亠, 亠仄 仄亳亞亳仂于舒
  • 19. 19息 Copyright 2011 EMC Corporation. All rights reserved. 仂仍亰舒 仄亳亞舒亳亳 Co-scheduling 仍舒仆亳仂于亳从 仄舒从亳仄舒仍仆仂 亳亰弍亠亞舒亠 舒亰仄亠亠仆亳 亟于 vCPU 仂亟仆仂亶 仆舒 仂亟仆仂仄 亳 仂仄 亢亠 pCPU 于仂 亳亰弍亠亢舒仆亳亠 舒亰弍亠亞舒 仂仍亰舒 亳舒亠 亰仆舒亳亠仍仆仂 于亠, 亠仍亳 仆舒 pCPU 仆舒亰仆舒亠仆亳 仆亠 舒亰仄亠亠仆 亟亞仂亶 vCPU 仂亶 亢亠 仂仄仄仆亳从舒亳 仄亠亢亟 从仂仆亠从舒仄亳 仍舒仆亳仂于亳从 仂亠仆亳于舒亠 仂于亠仆 仄亠亢仗仂亠仆仂亞仂 于亰舒亳仄仂亟亠亶于亳 vCPU 于仂从亳仄 仂于仆亠仄 仂弍亠仆亳 弍仂仍亠亶 于亠仂仆仂 弍亟 舒亰仄亠亠仆 束仗仂弍仍亳亰仂亳損, 仂弍 于亠仍亳亳 仗仂仗舒亟舒仆亳 于 从
  • 20. 20息 Copyright 2011 EMC Corporation. All rights reserved. Hyper-Threading HT 仗仂亠仂 = 2 仗仂仂从舒 仆舒 磲仂 仂亳亰于仂亟亳亠仍仆仂 从舒亢亟仂亞仂 仗仂仂从舒 亰舒于亳亳 仂 于仂仂亞仂 仂仂从 仂 于仂弍仂亟仆仄 2仄 亟舒亠 弍仂仍亠 于亳仍亳亠仍仆仂亶 仄仂仆仂亳, 亠仄 亠仍亳 仂弍舒 仗仂仂从舒 亰舒仆 亳仍仍亳仆亞 亟仍 HT 仗仂亠仂仂于 仂仍亳舒亠 亳仆亳仗 亠仆仂亳 仂仍亠亟仆亳亠 仗仂从仂仍亠仆亳 仗仂亠仂仂于 仍亠 仗舒于仍ﾑ ESXi 5.x 弍仂仍亠亶 于亠仂仆仂 于弍亳舒亠 HT 仗仂仂从, 亠仄 舒仆亠亠 仂仍仆仂 于仂弍仂亟仆仂亠 磲仂 于亠 亠亠 亳仄亠亠 弍仂仍亳亶 仗亳仂亳亠
  • 21. 21息 Copyright 2011 EMC Corporation. All rights reserved. NUMA NUMA Non Uniform Memory Access NUMA 亰亠仍 仂亠仂 (Socket) 亞仂 仗舒仄 (仍仂从舒仍仆舒) 弌仂亳仄仂 亟仂仗舒 从 亢仂亶 仗舒仄亳 于亠 舒从亳仄舒仍仆仂 仂舒于舒 于 舒仄从舒 亰仍舒 vCPU 舒亰仄亠舒ﾑ 仂仍从仂 于 NUMA home (亟仂仄舒仆亠仄 亰仍亠) 弌仄亠仆舒 NUMA home 亳亞舒亳 仗舒仄亳 亟亠仍仂 亟仂仂亞仂亠 亠于仍磳 仂仍从仂 亠仍亳 仂舒亠 于 仆仂于仂仄 亰仍亠 亟仂仍亞仂
  • 22. 22息 Copyright 2011 EMC Corporation. All rights reserved. NUMA 仄亳亞舒亳 仍 从舒从仂于亠仄亠仆仆仂亶 弍舒仍舒仆亳仂于从亳 CPU 仍 仗仂于亠仆亳 仍仂从舒仍仆仂亳 仗舒仄亳 仍亳 仗舒仄 亠亠 仆亠 弍仍舒 仄亳亞亳仂于舒仆舒, 仂 亳仄亠亠 仄仍 仄亳亞亳仂于舒 仆舒亰舒亟 舒仂亶 从仂仄仄仆亳从舒亳亠亶 仄亠亢亟 仆亳仄亳 /Numa/LocalityWeightActionAffinity 0 仍 仂弍亠仗亠亠仆亳 仗亳仆亳仗舒 亠仆仂亳 于 亟仂仍亞仂亶 仗亠仗亠从亳于亠 /Numa/LTermFairnessInterval 0
  • 23. 23息 Copyright 2011 EMC Corporation. All rights reserved. Wide NUMA 丿亳仂从舒 NUMA 仄舒亳仆舒 仂仍亳亠于仂 vCPU > 从仂仍亳亠于舒 pCPU 于 NUMA 亰仍亠 丿亳仂从舒 仄舒亳仆舒 舒亰弍亳于舒亠 仆舒 仆亠从仂仍从仂 从仍亳亠仆仂于 舒亢亟亶 从仍亳亠仆 亟仂仍亢亠仆 仗仂仄亠舒 于 亰仍亠 HT HT 亟于舒亳于舒亠 从仂仍亳亠于仂 仍仂亞亳亠从亳 磲亠, 仗仂仂仄 8 vCPU 仆舒 4 磲亠仆仂仄 仗仂亠仂亠 HT 弍亟亠 仗仂仗舒亟舒 于 舒亰仄亠 NUMA 亰仍舒 numa.vcpu.preferHT TRUE
  • 24. 24息 Copyright 2011 EMC Corporation. All rights reserved. 舒仄 亟仍 Wide NUMA 舒于仆仂仄亠仆仂 亠亠亟亠 亟仍 亳仂从亳 仄舒亳仆 于 4.x NUMA 从于舒亠 仂 亞仂亠于仂亶 弌 vSphere 5.x vNUMA NUMA 仗亠亰亠仆亠 亞仂亠于仂亶 弌
  • 25. 25息 Copyright 2011 EMC Corporation. All rights reserved. 舒仄 亟仍 Wide NUMA
  • 26. 26息 Copyright 2011 EMC Corporation. All rights reserved. vNUMA SPEC OMP
  • 27. 27息 Copyright 2011 EMC Corporation. All rights reserved. 仂弍仍亠仄 vNUMA 亳亰亳亠从仂仄 仄亳亠 仂仗仂仍仂亞亳 NUMA 仆亠亳亰仄亠仆仆舒 于亳舒仍仆仂仄 仄亠仆磳 vMotion 仆舒 仂 亳仆仂亶 仂仗仂仍仂亞亳亠亶 亠 于亠 仗亳仍仂亢亠仆亳 仗仂仆亳仄舒ﾑ 亳亰仄亠仆亠仆亳亠 仂仗仂仍仂亞亳亳 于仂从仂仆亳亞舒亳 numa.autosize TRUE numa.autosize.once FALSE 丐亠弍亠 仗亠亠亰舒亞亰从舒
  • 28. 28息 Copyright 2011 EMC Corporation. All rights reserved. 亠仂仂于亠于亳亠 NUMA 亳 vNUMA vNUMA < NUMA vNUMA 亠仍亳从仂仄 仗仂仄亠舒亠 于 亰亠仍. 亳从舒从亳 仗仂弍仍亠仄 vNUMA = x*NUMA vNUMA 舒亰弍亳于舒亠 仆舒 x NUMA 亰仍仂于 亳 仗舒仄 亠亠亟亠 仄亠亢亟 仆亳仄亳 vNUMA != x*NUMA vNUMA 于亠 亠亠 仂舒仆磳 亟仍 亞仂亠于仂亶 弌 NUMA 从仍亳亠仆 仂仗亠亟亠仍ﾑ 于 亰舒于亳亳仄仂亳 仂 舒亰仄亠舒 NUMA 亰仍舒, 舒 仆亠 vNUMA. 舒仄 亠亠亟亠 仄亠亢亟 仆亳仄亳 仂 舒从 vNUMA 仗仂仂 仆亠 舒弍仂舒亠
  • 29. 29息 Copyright 2011 EMC Corporation. All rights reserved. 于仂亟 亰弍亠亞舒亶亠 亠仂于 (vCPU) 仆舒 于磻亳亶 仍舒亶 丕亰仆舒亶亠 于仂 亳仆舒从 NUMA 仂亠仂仆舒 舒亳亠从舒 仂仍亳亠 仄舒亳仆 亠弍ﾑ 仗仍舒仆亳仂于舒仆亳 亰弍亠亞舒亶亠 仄亠仆 亰仆舒亠仆亳 舒亳亠仆仆 仗舒舒仄亠仂于 仆舒亠仆亳亶 仗仂 仄仂仍舒仆亳 亟仂舒仂仆仂 于 99% 仍舒亠于 弌仆舒舒仍舒 仗仂亟仄舒仍 仗仂仂仄 亟亠仍舒仍! 仆亠 仆舒仂弍仂仂!
  • 30. 30息 Copyright 2011 EMC Corporation. All rights reserved. 仂仆舒从 仆仂仆 弍舒仆从仂于 EMC Senior SE / Software Defined Specialist anton.zhbankov@emc.com anton@vadmin.ru http://blog.vadmin.ru @antonvirtual + Facebook, LinkedIn etc
  • 31. 31息 Copyright 2011 EMC Corporation. All rights reserved. THANK YOU