Next generation cloud data center technologies
5 likes1,937 views
This was presented at CloudInspire 2013 held by SKT & ZDNet. Since I presented Hybrid Cloud at CloudInspire 2012, I was invited again.
Next generation cloud data center technologies
- 1. www.coord.org Woohyun Kim woorung@gmail.com 2013-04-25 Next-Generation Cloud Data Center Technologies Toward a High-Performance, and yet, Low Cost Cloud Compact Version
- 2. www.coord.org Physical Machine Virtual Machine PM VM ? ? NASA originally had a cluster computing infrastructure over physical machines for climate modeling and simulation applications, named Discover ? The cluster had guaranteed the applications¡¯ rigorous throughput and latency requirements ? Discover is based entirely on non-virtualized machines ? Compute resources: 30,000 conventional Intel? Xeon? processor cores and 64 GPUs ? Inter-node backbone: DDR and QDR InfiniBand* ? Management networking: Gigabit and 10 Gigabit Ethernet (GbE and 10GbE) ? Data store: ~4 petabyte RAID-based parallel file system (GPFS), plus ~20 petabyte tape archive ? NASA moved HPC to the cloud Turing Test on the Cloud Background ? But NASA determined to utilize the inherent advantages of cloud infrastructure such as agility and elasticity, called Nebula ? NASA wanted to confirm whether the Nebula cloud infrastructure can deliver throughput and latency equivalent to that of the Discover cluster refer to http://www.intelethernet-dell.com/nasa%E2%80%99s-flexible-cloud-fabric-moving-cluster-applications-to-the-cloud-network/ ?
- 3. www.coord.org ? Vendor-driven Cloud vs. Open Source-applied Cloud ? Device-centric: CISCO, HP, DELL ? Hypervisor-centric: Citrix, VMware(EMC) ? Open Source Plugin-wise: Midokura, Bigswitch Traditional DC (~2005) Virtualized DC (~2010) Cloud DC (2011~) App Physical Resource App App System Mgmt. S/W ¡õ CapEx/OpEx Increment ¡õ High Cost of Commercial S/W ¡õ High-End Storage(SAN/NAS) & Network(L4/L7) Devices ¡õ Inefficient Server Utilization (Average 80~90% Idle State) Virtualized System Mgmt. S/W Hypervisor Hypervisor App VM App VM App VM App VM App VM App VM ¡õ Server Consolidation => CapEx & TCO Saving ¡õ Dynamic Resource Scheduling => Flexible Data Center Mgmt. ¡õ Due to I/O bottleneck, mission-critical services CAN NOT be applied ¡õ Mgmt. Automation(Monitoring, Orchestration) => OpEx Saving ¡õ Service Automation(HA, DR, AS, LB) => 24/7 Service Enabling ¡õ H/W & S/W enhancement to solve I/O bottleneck => QoS & SLA guarantee Hypervisor Hypervisor App VM App VM VM VM VM App VM Cloud System Mgmt. S/W ? Evolution of Datacenter & Key Issues The Cloud Delusion Background Physical Resource Physical Resource Physical Resource Physical Resource Physical Resource Physical Resource ? Approaches to I/O Bottleneck Problem ? More connections: multi-ports, multi-interfaces ? Fatter pipes: 40GbE, InfiniBand fabrics, network fabrics ? I/O virtualizations: PCI express based I/O sharing ? Better caching: SSD caching ? Everyone is busy with IaaS implementations, but that is not ¡°hot¡± anymore
- 4. www.coord.org Background ? ???? ???? ???? ?? ??? ???? ????? ???? ????? ?????? ?? ??? ? ???? ???? ??? ??? ??? ???? ?? ????? ??? ???? ?, ?? ??? ???, ??? ?? ? ??, ??/???/??? ????, ?? ? ?? ?? ?? ??? ??, ???? ??, ???? ???? ?? ?? ?? ?? ??? ? ?? ???? ??? ?? ?? ?? ??? ????? ?? ??? ??? ??? ? ? ? ?? ??? ????? ??? ?? ???? ?? ? ?? ?? ?? ? ?? ? ???? ???? ?? ??? ?? ??? ?? ???? ??? ???? ?? ? ?? ?? ? ????? ??(APP ?? ???)? ???? ??? ???? ????? ? ??? ? ?? ???? ??? ???? ????? Migration? ???? ? ? Migration? ?? ?? ??? ????? ??? ?? APP ???? Migration ?? ??? In The Beginning of the Cloud ? ???? S/W? IT ???? RDBMS, NoSQL? ?? ???? ???? ???? ? ?? ???? ??? ??? ??? ???? ????? ? ??? ?? ? ???? ???? ? TPS? ???? ??? ???? ????? ??? ? ??? ??? ?? ????? ???? ???? ??? ??? ? ??????? ?? ?????? ?? ?? ?? ???? ? ???? ??? ???? ???? ???? ??? ??? ????? ????? ? ? ? ???? ???? ??? ??? ??? ??? ??? ? ?? ??, ?? ??? ?? ??? ? The Cloud Players Pursue A High Performance, and yet, Low Cost Cloud ¡õ ?? ???? ?? ?? - ??? ????? ?? ??? ??? ? ??? ???(GFS, MapReduce, BigTable) ?? ?? ¡õ ??? ???? ?? - ????? ???? Xen ??????? ??? ????? ??? ???? ?? ? ? ¡õ ???? ?? ??? ???? ?? - ??????? ?? H/W? S/W? ?? ? ??? ? ???? ?? ??? ?? ¡õ KT ????? ?? - ??? ??????? ???, ??? ? (??, ????, ????) ?? ???? ?? ? ?? ? ??? ??? ¡õ ?? ??????(VMware)? ??? - 100? ??? ?? ????? = 6?? ¡õ ?? ???????(enStratus)? ??? - 100? ??? ?? ????? = 2.6?? ¡õ ?? ??? ?? ??? ??? - ?? ??? ?? ???? ??? ??? ?? ???. - ????? ?? ???, ???? ??? - ?? ?? ??? ?? ????? ??? ? ???. ?? ???? ??? ??? ???
- 5. www.coord.org ? A cloud platform to dynamically configure, deploy, operate, and manage infrastructure components, which guarantees enterprise-class performance, and yet reduces TCO(total cost of ownership) Return to the Original Goal of the Cloud Introduction ? Toward A Feasible Cloud Data Center System Storage Primary Storage Secondary Storage SSD SAS SATA VM VM VM VM VM VM VM VM VM Bob Alice Virtual Network 100 Virtual Network 101 Virtual Switch/Router (FW, LB, NAT, L2MP), VDI QoS, IOV, SSD caching Scale-out NAS, VM HA/DR, V2V Migration Low Price, and yet High Performance Blade Server Xsigo(with InfiniBand) Cisco UCS, Nexus, FabricPath EMC iSillon NetApp NAS SR-IOV PCI Express IOV OpenFlow Scale-out NAS(GlusterFS, ZFS) VirtualGLGPU(Kepler) SSD caching Mission SAN/NAS
- 6. www.coord.org The Cloud Technologies Architecture ? A Bird¡¯s Eye View of the Cloud Technologies V M V M V M V M V M V M V M V M V M V M Bob V M V M V M Alice V M V M VLAN 101 VLAN 100 1 2 3 4 1 2 3 4 1 2 3 4 Virtual Block Pool Virtual Server Pool VM (HA/DR) Original Replica Replica Cluster Scheduler VM NetworkVolume M M M M M M Image API Account Capacity Log DB AMQP Cloud Fabric & Managers Cloud Agents Cloud Mgmt.s Host Pod Zone Cluster Pod Cluster Host VM VM Primary Storage Secondary Storage VM ISO Template Snapshot Image L/C Mgmt. Availability Zone Mgmt. Virtual Appliance Virtual Appliance Template FW LBNASDB WebServerWAS Android Apps Transcoding Server Streaming Server Cloud Foundry Orchestration FW LB Web Web WAS WAS DB DB Dynamic Infra Configuration 1 2 3 Intel I/O Adaptor (8 * 20Gbps) 1 2 3 1 2 3 5 6 7 8 91 2 3 4 PCIe I/O Adaptor (15 * 20Gbps) V M V M V M 10G 20G 2G ~ 16G 1G~ 10G 1G ~ 10G 10G Network Traffic Mgmt. Auto Scaling Auto Scaling Group Management Auto Scaling Group AS Policy Security Group Templ ate LB Policy Virtual Desktop Server-side 3D Rendering Management Server screen GPU VirtualGL 3D Game 3D X Server 2D X Server OpenGL GPU Client Unified/Tiered Storage
- 7. www.coord.org PCIe IOV Technologies ? I/O Virtualization ? High VM density has inherently raised a huge I/O bottleneck problem ? They require a large number of I/O connections per physical host to solve the bottleneck problem ? SR-IOV achieves near native I/O performance without I/O bottleneck ? SR-IOV enables a single physical I/O device to be partitioned into multiple virtual I/O ports called virtual functions ? Those virtual I/O ports can be mapped to individual virtual machines. ? VF(Virtual Function)s have near-native performance (better performance than para-virtualized drivers) ? VFs provide data protection between virtualized guests on the same physical server as the data is managed and controlled by the hardware Guest OS (VM) PF VF driver physical driver Hypervisor (VMM) SR-IOV enabled NIC TCP IP MAC SR-IOV VF VF VF 2 Translate guest addresses into host physical addresses Transmit and receive I/O data1 I/O command Guest OS (VM) VF driver SR-PCIM physical device switching - Complete the configuration model - Translate VF into a full function - Configure SR-IOV resources Deliver VF¡¯s PCI configuration info (Device ID, Vendor ID, ¡) 1 2 3 1 2 3 1 2 3 5 6 7 8 91 2 3 4 PCIe I/O Adaptor (15 * 20Gbps) V M V M V M 10G 20G 2G ~ 16G 1G~10G 1G~10G 10G Intel I/O Adaptor (8 * 20Gbps) A Model of PCIe Adaptor with SR-IOV L2 switch
- 8. www.coord.org A Key Point Solution ¨C IOV (with PCIe) ? Performance Test of I/O Virtualization receive in each VM Technologies
- 9. www.coord.org OFVR ? OpenFlow-based Virtual Router for Scalability/Availability ? A virtual router suffers from a severe I/O bottleneck because of storing its data locally ? Both of OpenStack and CloudStack provide a virtual network architecture based on a virtual router ? All the VMs in the same virtual network have to transfer the out-bound traffic to a specific virtual router ? The in-bound traffic from the public network also has to be delivered to a specific virtual router ? An almighty controller manages the data and each virtual switch globally ? A virtual router is built on top of decentralized OpenFlow switches and controllers ? The out-bound traffic from each VM just passes through its own local virtual router, not remote ones ? The in-bound traffic to each VM is headed for the corresponding virtual router, not remote ones eth0 eth0.100 cloudbr0 Virtual RouterPhysical Machine 1 vnbr100 eth0 vnet2 eth2 vnet1 dnsmasq iptables VM 1 vnet3 eth0 vlan 100 dhcp discovervirtual network 100 eth0 eth0.100 cloudbr0 Physical Machine 2 vnbr100 VM 2 vnet1 eth0 vlan 100 dhcp discovervirtual network 100 virtual network 100 public network A Existing Virtual Router An OpenFlow-based Virtual Router Controller F W N A T L B L 2 M P Controller F W N A T L B L 2 M P eth0 Physical Machine 1 OVS 1 vnet1 VM 1 eth0 Controller eth0 Physical Machine 2 VM 2 vnet1 eth0Distributed Virtual Router virtual network 100 public network OVS 2 3 6 1 1 F W N A T L B L 2 M P H H Technologies
- 10. www.coord.org VBP for HA/DR ? Virtual Block Pool for High Availability of VM ? The synchronous replicas of a VM image guarantee high availability against hardware failures ? Live migration is supported in a specific hypervisor level, and yet a sudden hardware failure does not guarantee high availability of VM ? A VM image should be synchronously replicated to other places to provide a failover capability ? A local disk is partitioned by logical block, and the virtual block registered to a global pool with copy-on- write replication ? VBP provides a hypervisor-agnostic live migration capability without a SAN, and even a complete failover capability against a suddenly hardware failure VBP(Virtual Block Pool) for HA & DR of VM V M V M V M V M V M V M V M V M V M 1 2 3 4 1 2 3 4 1 2 3 4 Virtual Block Pool Virtual Server Pool VM (HA/DR) Original Replica Replica Cloud OS Agents Unified & Tiered Storage Technologies
- 11. www.coord.org ? GPU Virtualization Client-side 3D rendering X Server X client X client Network GPU Keyboard Mouse Screen User Workstation Remote Machine GLX OpenGL Library 2D/3D geometry or textures Direct Rendering ? What is a hurdle to run 3D applications remotely? ? 3D rendering is still occurring on the user¡¯s desktop machine ? GPU could not be allocated to multiple VMs ? Cloud OS tries to leverage GPU virtualization and provide server-side 3D rendering ? Different 3D games can be running in multiple VMs, the rendered 2D images will be remotely forwarded Server-side 3D rendering 2D X Server X client X client Network GPU Keyboard Mouse Screen User Workstation Remote Machine VirtualGL OpenGL Library 2D rendered images 3D X Server GPU 3D GPUV Technologies
- 12. www.coord.org ? Orchestration = Automation + Integration + Best Practices ? Cloud capabilities to simplify, automate and optimize service deployment ? Self-service portal for selection of cloud services, including storage and networking, from a predefined menu of offerings ? Automated high-scale provisioning and de-provisioning of resources with policy-based tools ? Workflow management to improve collaboration and reduce delays ? Real-time monitoring of physical and virtual cloud resources, as well as usage and accounting chargeback capabilities to track and optimize system usage ? Prepackaged automation templates and workflows for most common resource types to ease adoption of best practices and minimize transition time Orchestration CA AppLogic CloudFoundry Technologies
- 13. www.coord.org Public Cloud Server Farm CPU Scale Up Scale Down LB VM Master LB VM VM Slave ¡ VM VM VM¡Public Internet Private Cloud cross-cloud auto-scaling Public Cloud ServerArray CPU Scale Up Scale DownLB App Master LB App App Slave ¡ App App App¡ Private Cloud Public Internet Slave sn ap sn ap sn ap sn ap sn ap sn ap sn ap sn ap sn ap sn ap sn ap sn ap ? Global storage ? Live replication + rsync ? NoSQL S3/Cloudfiles cross-cloud disaster recovery Hybrid Cloud ? Hybrid Cloud Enablement Technologies ? Image L/C Mgmt. & V2V Miration ? rPath rBuilder ? Glance ? RightScale ServerTemplate & MCI(Multiple Cloud Image) ? Cloud Bridge ? Rackspace RackConnect ? Vyatta-based SSL-VPN ? RightScale Point-To-Point Technologies
- 14. www.coord.org The Cloud Functionalities Conclusion ? The Cloud Functionalities Cloud Data Center Web Portal ? Dashboard ? ???? ? ???? ? ???? ? ??? ? ?? ? ??? ? ??? ? ??? ? ?? ?? ? ??? ?? - ???/???? ? ??? ?? - ??? ??/?? - ??? ?? - ??? ?? - ??? ??? - ??? ??? ? ?? ?? - ?? ??/?? - ?? ?? - ?? ??? - ?? ??? - ?? ?? ? ??? ?? - ??? ??/?? - ??? ?? - ??? ?? - ?? ??? ?? ??? ?? ? VM ?? - VM ??/??/?? - VM ??/??/??? - VM ???? ?? - VM ???? ?? - VM ??? ?? ?? - VM ???? ?? - VM ?? ? ISO ?? - ISO ??/?? - ISO ?? - ISO ??/?? - ISO ??/?? - ISO ??? ?? - ISO ??? ?? - ISO ?? ? ??? ?? - ??? ??/?? - ??? ??/?? - ??? ?? - ??? ?? - ??? ??? ?? - ??? ??? ?? - ??? ?? ? ??? ?? - ??? ??/?? - ??? ?? - ??? ?? ?? - ??? ?? ?? - ??? ?? ?? ???? ?? ? ???? ?? - ???? ??/?? - ???? ?? - ???? ??? - ???? ?? - PXE ?? ?? - DHCP ?? ?? ???? ?? ? ?? ?? - ?? ??/?? - ?? ??/?? - ?? ?? - ?? ?? ? ????? ?? - ????? ?? - ????? ?? - ????? ?? - ????? ???? - ????? ???? ?? ??? ? ?? ? ??? ?? - ??? ??/?? - ??? ???/?? - ??? ????/? ? - ??? ?? - 2nd ???? ?? ? ???? ?? - ???? ??/?? - ???? ?? - ???? ?? ? ? ?? - ? ??/??/?? - ? ?? ? ? ?? - ? ??/??/?? - ? ?? ?????? ?? ? DNS ?? ? DHCP ?? ? NAT ?? - IP ??? ? ?? - IP ??? ? ?? - IP ??? ? ?? - ?? NAT ??? - ?? NAT ??? ? ??? ?? - ?? ??? ? ?? - ?? ??? ? ?? - ?? ??? ? ?? ? ????? ?? - ????? ? ?? - ????? ? ?? - ????? ?? VM ??/?? - ????? ? ?? - ????? ?? ? ?? VM ?? - ????? ???? ?? ? ??? ?? - ??? ??/?? - ??? ??? - ??? ??? ?? ?? - ??? ?? VM ?? ?? ? VM ?? ?? - VM ?? ??/?? - VM ?? ??/?? ?? ? ???? ?? - ???? ??/?? - ???? ? ?? - ???? ? ?? - ???? ?? ? ?? ?? - ??? ?? ?? ? ? ?? ?? - ??? ? ?? ?SSH Key ?? - SSH ? ??/?? - SSH ? ?? - SSH ? ?? ?? ?? ?? ?? ? IP ?? ?? - ???? IP ?? - ???? IP ?? - IP ?? ? VLAN IP ?? ?? - VLAN IP ?? ?? - VLAN IP ?? ?? - VLAN IP ?? ?? VPN ?? ? VPN ?? - L2TP/IPSec ???? ??/??/?? - VPN ??? ??/? ?/?? ???? ?? ?? ? ?? ????? ?? - ?? ????? ? ?/??/?? ? ?? ????? ?? - ?? ??? ??/? ?/?? ?????? ? ??? ?????? - VM ?????? ??/??? ??? ? ??? ??? - Orchestration - High Availability - Auto-Scaling - Disaster Recovery ??? ?? ? ?? ?? ? ??? VM ?? - ??? VM ??/?? - ??? VM ??? - ??? VM ??/?? ? ??? ? ???? ??? ???? ?? ? ??? ??? ?? - ??? ??? ??/ ??/??/?? ? ???? ??? ?? - ???? ??? ? ?/?? ? ??? ??? ?? - ??? ??? ??/ ??/??/?? ? ??? OS ??? ?? - ??? OS ?? ?? - ??? OS ?? ??
- 15. www.coord.org The Beginnings Will Seem Humble Conclusion ? Gradually Evolving into the Cloud ? ???? ?? ? ?? ? ?? ? ??/????/???? ?? ? ?? ? ?? HA/DR ?? ? ?? ? ????? NAS ?? ? ?? ???? ??(Router, Switch, LB, Firewall, VPN?) ?? ??? ?? ?? ? ?? ??? ?? ? ???? ??? ?? ? ???? ??? ?? ? ??? ?? ??? ?? ? ?? ?? ??? ?? ? ?? ??? ? ?? ????? ?? ? ??? ??? ? ? ??? ?? ??? ?? ? ??? ??? ??? ?? ?? ??? ??? ?? ? ?? ???? ?? ? ????? ?? ?? ?? ?? ?? ? ???? ????? ?? ? ????? ?? ??? ?? ? ???/??? ??? VM ?? ?? ? ?? ??? ?? ? ?? ???? ??? ? ??? ?? ? ??? ?? ?? ? ?? ???? ?? ????? ?? ?? ?? ?? ??? ?? ? V2V ? P2V ?????? ?? ? ?? ???? ???? ?? ? ?? ?? ? ?? VM ??/??? ? ?? ?? ? ??? ???(HA, DR) ?? ??? ??? ?? 1??: ??? ???? ?? ? ?? ?? ? 15?? ????? ????? 150~300? ?? ???? ?? ??? ?? ? ??? ?? ??????(VMware, XenServer)? ???? ??, ???? ? ?????(Xen, KVM, QEMU)??? ???? ??? ??? ?? ? ??? ??? ????/???? ??? ???? ??, ??? ????? ? ? ????/???? ???? ???? ??? ???? ?? ? ?? ? ??? ??, ??, ?? ? ????? ??? ??, ??, ?? ??? ???? ? ??? ???? ?? ? ????? ?????(CapEx)? ???(Opex) ??? ?? ?? ? ??? ??? ?? ? ??(MapReduce, RDB, NoSQL, ??????)? ???? SNS ? ???? ??? ??, ??, ?? ??? ? ??? ?? ? ??? ???(Auto Scaling, ?????, ?? ? ? ? ??)? ???? ?/?? ??? ??? ?? ???? ??? ? ? ? ?? ?? ? ?? ?? ? SNS? ??? ??, ??, ??? ?? ??? ? ??? ?? ? ??? ??? ??? ??? ?? ? ??? ?? ???(CDN, ?????, ????) ?? ?? & ??? ???? ?? ? ????OS(Windows, Linux)? ?? ?? ?? ? ?? N-Screen ??? ?? ? ????? ? ??? ??? ?? ??? ?? ???? ?? ? ?? ?? ???? ??? ?? ? ???OS(iOS, Android)? ?? ?? ??? ? ? N-Screen ??? ?? ? Tizen ?? ???? SDK ?? ?? ??? ? ? ??? ???? ?/? ?? ? ??? ?? ??? ???? ?? ? ? ?????? ??? ?? ??? ??? ? ?(Java, PHP, Ruby, Apache, Tomcat, MySQL?) ? ????, ????, Wiki, CI ?? ?? ? Eclipse ?? ???? ?? ? ??? ?? R&D ???? ?? 2??: ??? ???? ?? ? ?? ?? ? ??? ??? ??? ??? ???? ??? ???? ??? ???? ???? ?? ?? ? ??? S/W? IT ???? ?? ?? ???? ??? ??? ??? ??? ??? ???? ???? ?? ??? ??? ?? ???? ?? ??? ?? ??? ?? ?? ?? ? ?? ???? ???, ????, ??? ???? ??? ?? ?? ???? ???? ???? ???? ?? S/W ? IT ? ?? ??? ?? ???? ????? ?? ??
- 17. www.coord.org ? Contact Me About About woorung@gmail.com 010-6420-2409 ???(Woohyun Kim) ?? ?? ? S-Core ??????? ?? (2011~2013) - ??? ???? ?????(Cloud OS) ?? ? ??? ? KTH ????? ?? (2010~2011) - KT ????? ?? ? ?? ? NHN ?????? ?? (2006~2010) - ???? ??? ?? ? ????? (http://www.coord.org) ???? Coord(http://www.coord.org) ??? ? ??? ? 2009 ?? SW ???? ??(????? ???) ? 2008/2009 NHN DeView ?? ? 2010 ??? ?? SW ??? ?? OSS ?? ???? ? ???? ?? ???? ? 2009 ??? ???? - ???? ???? ???? ? ?? ?? ? 2010 Hadoop ????? ??? ?? (1~3? ??) ? 2011 ??? 2011(????24 ??) ¨C Rethinking the Cloud ?? ? 2012 ??????? ???? ?? ? ???????? ? 2012 ICAT(? 17? ???????????) - Hybrid of MR and RDB ?? ? 2012 ????????? 2012 ¨C Hybrid Cloud Enablement ?? ? 2013 ???? ????? 2013 ¨C Next-Generation Cloud Data Center Technologies ??