ディペンダブルなクラウドコンピューティング基盘を目指して
?
0 likes?2,700 views
電子情報通信学会 サーヒ?スコンヒ?ューティンク?研究会での講演（2014年3月14日）
![Kumoi System Programming:
VM-Compaction
7
def compact(pms: List[VM]) {
def firstFit(v: VM, rest: List[VM]) {
rest match {
case h :: rs if (h.cpuAvailable > v.cpuRatio) => v.migrateTo(h)
case h :: rs => firstFit(v, rs)
case List() =>
}
}
def compacti(pms: List[VM]) {
pms match {
case h :: rest =>
h.vms.foreach(v => firstFit(v, rest.reverse))
compacti(rest)
case List() =>
}
}
compacti(pms.reverse)
}](https://image.slidesharecdn.com/140314-140812005234-phpapp02/85/-7-320.jpg)
![World-wide Use of BitVisor
? HyperSafe [Wang et al., IEEE S&P ‘10]
? Integrity of hypervisor itself, i.e., modi?cation
disabled.
? “Return-less” VMM [Li et al., EuroSys ‘10]
? Against ROR (Return-Oriented Rootkit)
? TCVisor [Rezaei et al., ICITST ‘10]
? Limited storage area can be seen by each user.
18](https://image.slidesharecdn.com/140314-140812005234-phpapp02/85/-18-320.jpg)
![VPN Switching
24
VPN failures
erforms two
ling is used
g IP address
hiding IP ad-
these opera-
a server.
eader like IP
hen a packet
P header has
rver IP as the
s assigned by
s assigned to
hat the guest
nt ID. When
e.g., 128bits)
between the
rds the rela-
0
5
10
15
20
25
30
0 100 200 300 400 500 600 700 800 900
Yokohama
Fukuoka
Tokyo
1000
Latency
Elapsed time [sec]
Figure 7. Transition of Latency to Data Cen-
ters
0
2
4
6
8
10
0 5 10 15 20 25 30
VPNthroughput[Mbit/sec]
Elapsed time [sec]
Failure occurred point Failure recovered point
15.1 19.2
Figure 8. Throughput Transition over Failure
km to Yokohama, and 926 km to Fukuoka. These data cen-
Before:Tsukuba-Tokyo (56Km)?
After:Tsukuba-Yokohama (84Km)](https://image.slidesharecdn.com/140314-140812005234-phpapp02/85/-24-320.jpg)
![Newtork Latency and Throughput of
VPN Switching
25
Tsukuba-Tokyo (56Km)?
Tsukuba-Yokohama (84Km)
Tsukuba-Fukuoka (926Km)
Tokyo Yokohama Fukuoka
VPN on OS 13.18 12.63 32.04
VPN on VMM 13.46 13.00 32.57
VPN on VMM with relay 13.71 13.23 32.80
0
5
10
15
20
25
30
35
Latency[msec]
Figure 9. Latency
calsuling and NAT in a user-level process. We measured
the overhead in three environments: “VPN on OS” repre-
sents that VPN is implemented in a OS (using YAMAHA
YMS-VPN1), “VPN on VMM” represents that VPN is im-
plemented in BitVisor but not using the packet relay sys-
tem, and “VPN on VMM with relay” represents that VPN
is implemented in BitVisor and using the packet relay sys-
tem. We measured latency and throughput from a client at
Tsukuba to each data center over a VPN.
Figure 9 shows the latency. The overhead of our system
was 0.53–0.76 msec in total for each data center: the VMM
incured 0.28–0.53msec and the packet relay system incured
0.23–0.25 msec. Figure 10 shows the throughput. The over-
head of our system was about 8–30% in total: the VMM in-
cured 4–16% overhead and the packet relay system incured
Tokyo Yokohama Fukuoka
VPN on OS 13.18 12.63 32.04
VPN on VMM 13.46 13.00 32.57
VPN on VMM with relay 13.71 13.23 32.80
0
5
10
15
20
25
30
35
Latency[msec]
Figure 9. Latency
Tokyo Yokohama Fukuoka
VPN on OS 58.88 52.98 26.43
VPN on VMM 49.31 47.45 25.27
VPN on VMM with relay 41.22 41.94 24.45
0
10
20
30
40
50
60
70
Bandwidth[Mbit/sec]
Figure 10. Throughput
calsuling
the overh
sents tha
YMS-VP
plemente
tem, and
is implem
tem. We
Tsukuba
Figure
was 0.53
incured 0
0.23–0.2
head of o
cured 4–1
3–14% o
when con
the packe
the syste
6. Rela
Severa
peer-to-p
15, 16, 1
in the bo
tems. Ho
struct ov
system a
prove ava
A rou](https://image.slidesharecdn.com/140314-140812005234-phpapp02/85/-25-320.jpg)
ディペンダブルなクラウドコンピューティング基盘を目指して
- 2. Overview of Dependable Cloud computing Developing infrastructural software for cloud computing with servers, client, and network. 2 Dependability： Reliability, Availability, Response, Throughput, Security, Privacy Failure Guest OS BitVisor Hardware Servers (in several data centers)NetworkClients Internet
- 3. 3 (I) Dependable Server Management Failure Guest OS BitVisor Hardware Servers (in several data centers)NetworkClients Internet
- 4. Kumoi（雲居）: Middleware for Cloud Server Management ? Riding on the Scala programming language ? OO & functional ? "Scalable" coding (Java-to-Ruby level) with static type system ? Object/Resource mapping for data centers ? Real/virtual machines and network are mapped to HW/SW objs. (Cf. O/R mapping in db software) ? Incorporated distributed algorithms such as gossip algorithms and Paxos. ? Available as open source software. 4
- 5. Kumoi Overview 5 Data center Manager/operator Method call Result Interactive/batch Kumoi shell Scala Kumoi kernel Real machine Network VMM VNet VM Disk 34K lines of Scala source code
- 6. Kumoi Scripting (Cf. Unix scripting) 6 scala> pms.?iter(_.cpuRatio > 0.9).map(_.name) pms: List of available physical machines _: Formal arguments for higher-order function
- 7. Kumoi System Programming: VM-Compaction 7 def compact(pms: List[VM]) { def firstFit(v: VM, rest: List[VM]) { rest match { case h :: rs if (h.cpuAvailable > v.cpuRatio) => v.migrateTo(h) case h :: rs => firstFit(v, rs) case List() => } } def compacti(pms: List[VM]) { pms match { case h :: rest => h.vms.foreach(v => firstFit(v, rest.reverse)) compacti(rest) case List() => } } compacti(pms.reverse) }
- 8. 8 (II) Dependable Client Management Failure Guest OS BitVisor Hardware Servers (in several data centers)NetworkClients Internet
- 9. Virtual Machine Monitor 9 仮想マシン （VM: Virtual Machine） 仮想マシン （VM: Virtual Machine） Hardware Virtual Machine Virtual Machine Monitor Guest OS Hardware Physical Machine OS
- 10. チュートリアル「仮想化技術最前線」 Single- VM Virtualization ゲストOS Device Device Driver 拡張機能 セキュアＶＭ BitVisor ? I/Oを暗号化する ! ?ストレージI/Oを捕捉?暗号化 p?ATA/ATAPI 及び USB1.1/2.0を捕捉 p?AES-XTSで暗号化 ! ?ネットワークI/Oを捕捉?暗号化 p?NIC（Intel PRO 100/1000）を捕捉 p?IPSecでVPN接続 ! ?ICカードで鍵管理?認証 p?USB接続のカードリーダにアクセス p?PC/SC, CCIDプロトコル 2012/8/24 10 ATA NIC USB ストレージ管理 ネットワーク管理 ID管理 Device Mediator ATA NIC USB ATA NIC USB セキュアVM VMM ハードウェア
- 11. Utilization of BitVisor ? System ?le protection of guest OS ? Malware detection ? IDS within VMM ? Transparent VPN switching (described in the next topic) ? etc. 11
- 12. System File Protection of Guest OS ?Integrity (code cannot be modi?ed undetectably) Kernel image Device driver etc. !
- 13. Implementation of System File Protection ? BitVisor monitors every storage access. ?Detects system ?le modi?cation. ? Mapping between ?les and sectors are managed. ! Guest ?OS Device Device ?driver Extended ?function ATA NIC USB Device ?mediator ATA NIC USB VM VMM Hardware Protection ?policy
- 14. Malware detection IDS within VMM Run$at$the$boot$ +me$of$BitVisor data$block$ data$block$
- 15. チュートリアル「仮想化技術最前線」 Single- VM Virtualization 透過的ネットワークブート ? 任意のOSをネットワークから ブート可能にする ! ?OS?設定に依存しない p?Windowsをそのままブート可能 p?Linuxも一切設定なしでブート可能 ! ?ローカルのハードウェアをフル活用 p?内蔵機器?周辺機器をOSが完全管理 o Blu-ray，USB3.0，各種PCIデバイス，… o 省電力（電源制御），最適化（デフラグなど） 2012/8/24 15 クライアント サーバ 表 祐志, 品川 高廣, 加藤 和彦．仮想マシンモニタによる透過的ネットワークブート方式．情報処理学会論文誌：コン ピューティングシステム，第4巻，第4号，228-245頁，2011年10月．http://id.nii.ac.jp/1001/00078067/
- 16. チュートリアル「仮想化技術最前線」 Single- VM Virtualization 透過的ネットワークブート（実現） ? ディスクアクセスを ネッ トワークへ転送する ! ?ATAのインターフェイスを提供 p? ローカルディスクと同じアクセス方法 o MS-DOSでも起動する ! ?ATAアクセスをAoEに変換 p? ATA over Ethernet ! ?パケットをネットワークへ転送 p? ディスクイメージをサーバで集中管理 2012/8/24 16 ゲストOS Device Device Driver 拡張機能 ATA NIC USB ATA監視 NICアクセス ATA-AoE変換 Device Mediator ATA NIC NIC USB VM VMM ハードウェア ATA
- 17. チュートリアル「仮想化技術最前線」 Single- VM Virtualization ボランティアコンピューティング ! ? 計算コードをVMM内実行 ?計算コードをユーザから保護 p? 計算結果の改ざん防止 p? 計算結果の漏洩防止 ?計算コードからユーザを保護 p? 計算コードによる不正アクセスを防止 ! ? VMMでELFコードを受信 ?VMM内の保護ドメインで実行 ?ユーザは存在を感知しない 2012/8/24 17 ゲストOS Device Device Driver 拡張機能 ATA NIC USB Device Mediator NIC ATA NIC USB VM VMM ハードウェア 保護ドメイン 計算コード （ELF）
- 18. World-wide Use of BitVisor ? HyperSafe [Wang et al., IEEE S&P ‘10] ? Integrity of hypervisor itself, i.e., modi?cation disabled. ? “Return-less” VMM [Li et al., EuroSys ‘10] ? Against ROR (Return-Oriented Rootkit) ? TCVisor [Rezaei et al., ICITST ‘10] ? Limited storage area can be seen by each user. 18
- 19. 19
- 20. 20 (III) Dependable Network Failure Guest OS BitVisor Hardware Servers (in several data centers)NetworkClients Internet
- 21. Failure Detection in VMM 21 Pong Ping ?(periodical) Cloud Client ServersPacket ?switching VPN ?GW2 VPN ?GW1 NIC Application OS VMM SwitchingFailure ?detec. IP ?layer VPN ?layer
- 22. VPN Switching in VMM 22 Pong Ping ?(periodical) Cloud Client ServersPacket ?switching VPN ?GW2 VPN ?GW1 NIC Application OS VMM SwitchingFailure ?detec. IP ?layer VPN ?layer
- 23. Experiments with Real Data Center 23 Internet クライアント NTT ?East ?VPN NTT ?West ?VPN JGN-X: ?NICT ?Research ? Network 富士ソフト ? 九州データセンター Fujisoft in Yokohama Fujisoft in Kyusyu つくばTsukubaFujisoft in Kyusyu
- 24. VPN Switching 24 VPN failures erforms two ling is used g IP address hiding IP ad- these opera- a server. eader like IP hen a packet P header has rver IP as the s assigned by s assigned to hat the guest nt ID. When e.g., 128bits) between the rds the rela- 0 5 10 15 20 25 30 0 100 200 300 400 500 600 700 800 900 Yokohama Fukuoka Tokyo 1000 Latency Elapsed time [sec] Figure 7. Transition of Latency to Data Cen- ters 0 2 4 6 8 10 0 5 10 15 20 25 30 VPNthroughput[Mbit/sec] Elapsed time [sec] Failure occurred point Failure recovered point 15.1 19.2 Figure 8. Throughput Transition over Failure km to Yokohama, and 926 km to Fukuoka. These data cen- Before:Tsukuba-Tokyo (56Km)? After:Tsukuba-Yokohama (84Km)
- 25. Newtork Latency and Throughput of VPN Switching 25 Tsukuba-Tokyo (56Km)? Tsukuba-Yokohama (84Km) Tsukuba-Fukuoka (926Km) Tokyo Yokohama Fukuoka VPN on OS 13.18 12.63 32.04 VPN on VMM 13.46 13.00 32.57 VPN on VMM with relay 13.71 13.23 32.80 0 5 10 15 20 25 30 35 Latency[msec] Figure 9. Latency calsuling and NAT in a user-level process. We measured the overhead in three environments: “VPN on OS” repre- sents that VPN is implemented in a OS (using YAMAHA YMS-VPN1), “VPN on VMM” represents that VPN is im- plemented in BitVisor but not using the packet relay sys- tem, and “VPN on VMM with relay” represents that VPN is implemented in BitVisor and using the packet relay sys- tem. We measured latency and throughput from a client at Tsukuba to each data center over a VPN. Figure 9 shows the latency. The overhead of our system was 0.53–0.76 msec in total for each data center: the VMM incured 0.28–0.53msec and the packet relay system incured 0.23–0.25 msec. Figure 10 shows the throughput. The over- head of our system was about 8–30% in total: the VMM in- cured 4–16% overhead and the packet relay system incured Tokyo Yokohama Fukuoka VPN on OS 13.18 12.63 32.04 VPN on VMM 13.46 13.00 32.57 VPN on VMM with relay 13.71 13.23 32.80 0 5 10 15 20 25 30 35 Latency[msec] Figure 9. Latency Tokyo Yokohama Fukuoka VPN on OS 58.88 52.98 26.43 VPN on VMM 49.31 47.45 25.27 VPN on VMM with relay 41.22 41.94 24.45 0 10 20 30 40 50 60 70 Bandwidth[Mbit/sec] Figure 10. Throughput calsuling the overh sents tha YMS-VP plemente tem, and is implem tem. We Tsukuba Figure was 0.53 incured 0 0.23–0.2 head of o cured 4–1 3–14% o when con the packe the syste 6. Rela Severa peer-to-p 15, 16, 1 in the bo tems. Ho struct ov system a prove ava A rou
- 26. Summary Dependable cloud computing environment for servers, client and network, by using virtualization technologies. 26 Failure Guest OS BitVisor Hardware Servers (in several data centers)NetworkClients Internet
- 27. Ongoing Work ? Extension and application of Kumoi ? Virtual network control with OpenFlow ? Failure-oblivious computing ? Application: Parallel, distributed parameter tuning ? BitVisor application ? Transparent network boot system ? Acceleration of guest OS boot ? Desktop grid with intra-VMM computation ? Energy-saving distributed storage system 27
- 28. Thank you. 28 ?
- 29. ディペンダビリティ 29 クライアント環境 ネットワーク環境 サーバ環境 可用性 リモート管理機能 ネットワークブート機能 透過的VPN切り替え VMのハイアベイラビリティ P2P型のメンバシップ管理 信頼性 プロテクションドメイン 広域データセンター環境 高信頼性言語 完全性 システムファイル保護 マルウェア検出機能 VMMコア VPN（IPsec）接続 VMの利用 LDAP認証＆サンドボックス 仮想ネットワーク 保守性 リモートコントロール VPN（IPsec）接続 資源のオブジェクト化 スクリプティング環境 機密性 ID管理 ストレージ管理 VMMコア VPN（IPsec）接続 VMの利用 LDAP認証＆サンドボックス 仮想ネットワーク