�ݺ�ߣ

Magnum
:: using Openstack with
Kubernetes

HELLO!I am Cha Dong-Hwi who has experience
in
Developer mainly within Java/Spring, Python,
Javascript, and chef, php etc.
Openstack deployment engineer
SPTEK ??. ?? ??.
You can find me at tzniuta@gmail.com
2

? ?????? ??? "???????? ??"?? "? 2? ???? ???? ???
???(2015.10 )"? ??? ???????.
?? ??? ?????? ??/??? ??? ???? ????? ?? ?????.
Openstack ??
? ????? ??? ???? ?? ?
??? devstack?? ??? ? ???
???.
????? 4??? ??? ???? ?
??, ? ??? ?????? ????
??? ??? ??? ??(kilo)? ??
?? ?????.
Kubernetes ??
?? ??? Kubernetes? ??? ??? ?
??? ????. ??? ??? API? ??
?? ?? ? ????.
? ??? ??? kubernetes? fedora
atomic? ??? ??? ?? ?????.
coreos? ?? distro? ???? ????,
??? ??? networking? ?????.
? ???, ?? Openstack?? ??/????? ????? ?? ??? ???, ???
?? ?? ?????. Openstack? kubernetes? ????? ?? ??? ?? ?? ?
?, ?? ??? ????, ? ?? ???? ?? ??? ?? ??? ???????.
Kubernetes? ??, ?? ???? ???????. ?? ??? ??? ?? ??, ??
???? ??? ??? ?? ?? ??? ??(docker, golang, coreos eco system)??
?, ???? ??? ??? ?? ??? ?? ???? ?? ?? ?? ?? ????.
???? ??? ???? ???? ??????.
3
Disclaimer

??
??? ???? ??? ?? ??? ?
? ?? ????? :
1. Openstack ??
2. Kubernetes ??
3. Magnum ??
4. ?? ???
5. ???
4

1. Openstack? ��w
What is Openstack and Its pros and cons
5

" ???? ??? ?? ????
????? IaaS(Infra as a Service) ??? ???? ???
???? ????. ????? ??? ??? ??? ??
2018? 33? ??? ?? ???? ??. ? ??? ????
40%? ??? ??? ??.(?????? 451) ??, ???,
?????, HP, IBM, VM??, ???, ? ? ????? ??
??? ??? ?? ???? ???? ????? ??? ?
?? ???? ? ??? ??? ??? ?? ??.
??? SKT, KBS, LG CNS, KT ?? ????? ??.
????? ??? ?? ? ??? ??? ?? ?? ??(47%)
?? ????, ? ?? ?? ??? ??(47%),
???? ??(46%), ?? ?? ?? ??(45%) ??? ???."
by Tommy Lee on linux user group at facebook
6

????ϵ? Android
?? ???? ???? ?? ????? ?? ????. ??
????? ????? ?????? ? ??? ? ?? ??
????? ?????? ??. ???? Cloud(ec2: Amazon
Elastic Compute Cloud)? ???.
7

???? ??
???
8
?? ????
??? ?????
???? ?? ??? ???
????.
( more info from here or
search news from google )

????? ?
??? ?? ?
???
9
? ??? ???? ??? ?? ??
? ?? python ????
? kvm, netns, lvm?
? ??(domain)? ???? ??.
? Http, AMQP?? ????? ???
Server-Agent ??
CLI? ????? ????.
RESTful? ?? ??? ??.
-> ? ?? ??? ?????? ??.
(aka. Programmable IaaS )

11
???? ????
????? ?? ??? ?
?? ?? ??? ?? ??
.
- ???? ? ??

12
???? ????
????? ?? ??? ?
?? ?? ??? ?? ??
.
- ??? ? ??

13
???? ????
????? ?? ??? ?
?? ?? ??? ?? ??
.
- ??

14
???? ????
????? ?? ??? ?
?? ?? ??? ?? ??
.
- ??

???? ??
??
- ?? ????
- ????
Nova
KVM ? libvirt?
??. Compute?,
concurrent ? ?
??? ??
request??. ? ?
??? ?? ??
?? ??.
Cinder
LVM? ????
?? ??. Block
storage?, ??? ?
??.
Driver? ???? ?
?? Block storage?
?? backend? plug-
in ??? ????.
???? ???? ?
?? ??? ?? ?
??.
(Solidfire, NetApp ..)
Swift
???? ????
?, ??, ??? ?
? ??? ???
?? ? ??.
metadata? ???
?? ??.
HTTP??? ??
??. Ring? ??
???? ??? ?
?? ?? ??.
block storage??
?? ????.
(Cloudike ..)
15
Neutron
Network Namespace?
???? network
function? ??.
(Midokura, Cisco,
Brocade, Juniper.. )

??
- ?? ??? ? ??.
- ???? ??? ?
?? ????.
- ??? ?? ???
?? ? ??, ???
?? ??.
- ??? ?????
?? ?? ?? ??.
- ??? ?? ???
??? ??.
??? ??
- ??? ? ??
??
- ?? ????? ?
? ? ??? ?.
- ? ???? ???
?? ?? ??.
- ??? Use case?
?? ?? ??.
- ?? ?? ????
? ?? ?? ??.
- ?? ??? ?? ?
? ??.
16

???????
17
- ??? ????? ???? ??.
- ?? ??? ?? ???? ???.

2. about Kubernetes
What is Kubernetes
18

kubernetes??
- ???? ?? docker
orchestrator.
- ??? Go ??? ???.
- ??? docker
orchestrator? ???? ?
?.
- ?? ???? ????
?? container??(borg)?
??.
19
- ???, ?? ??
- ? . ? . ? . ? . ?
- ku-ber-ne-tee-s
- a.k.a k8s

14,000 commits
??? ?? ?? ?.
1 version
?? ?? 1 ??. Production ready.
400 users
? ??????? ??? ?? ?? ??? ?.
20

14,000 from 400 PPL
?? ??? ? ???? ????
21

kubernetes
?? ??
22
? ??? ?? 1? ?? ? production
ready?? ??
? ?? 1 ??? ?? Linux Foundation?
??
? ?? ???? ???? ?? ?
? ?? ????? ?? ??. Private
Cloud ?? ?? ? Google Container
Engine ?? ?? ??? ??
??? ?????, ?? ?? ???? ?? ??
? ?? ???, Linux Foundation?? ?? ??
? ????? ?? ?. ?? Google? ????
Foundation ?? ? kubernetes ?? ??

???? ? ??
Master?
Woker(borglet)?
Cluster ??.
Gmail, Docs, ??
? ?? ? ?? ?
?? ?? ?? ?
?.
Scheduling
job? ?? ? ? ?
? ??? ?? ?
? ?? ?? ??.
??? ??? ??
?? ? ?? ??
??.
Scalability
Borg? ??? ?.
Borg Scheduler?
????? ?? ?
? concurrent ??
? ??. Readonly
RPC ???.
Availability
???? Replicati
on ??, ??? ?
??, ??? ??
? ????? ??
, ?? ??? ??
???? ?? ??
??? ?? ??
?? (???)
Kubernetes ??
Job? task?? ?? ??? ????, ?? ?? ?
??? k/v??? ????, ???? ??? Label
? ???? ???? ????.
Borg? alloc? pod? ??. ?, 1? ??? ????
? ??? ?? ??? ??? ? ???? ???
? ???.
???? ??? task? ??? ? ??? ????,
?? kubernetes? service? naming?? ?? load
balancing ?? ???? ?? ????.
?? cAdvisor? ?? ????, micro-service? ?? API ?
?
??? borg
??
- from link
23
:: Disclaimer::
?? pdf ??? ??
???? ?? ? ?
? ???, ??? ?
?? ??? ?? ?
???.

Kubernetes
Architecture
24
API,
Scheduler
Replication
Controller
ETCd
Kubelet
KubeProxy
Master
Node
Slave
Node
Pod

about
Kubernetes
processes
- cluster ??
- cores ???
API Server
???? request
? ?? ??? ?
? http ??.
golang? http
handler? ??.
ETCd
Key-value store?,
?? ?? ???
?, kubernetes ?
? ? ?? ???
? ??. coreos
ecosystem? ?
???? ? ??.
kube-proxy
L3, L4? ??? ?
?? ?. Iptables
? ???? ??
? ? load
balancing ?? ?
?. ??? ??.
25
Replication Controller
?? ???? ???
Pod ? ??. ?? ??
? ??? ?? ?? ??
. AutoScaling.
cAdvisor
container? ?? ??
??. ?? ???? ?
? ????? ????
.
Kubelet
??? worker?,
docker ?? ? ?? ?
? iptable ??.
:: Disclaimer::
? ???? ???/?
???? ????? ?
, ?? ??? ??? ?
???? 100% ???
??? ??? ????
??.

??
26
? ?? ??? 1.x? ?? ??.
? ??? ??? ???? ??.
? ?? go ???? ???.
? ???? ???(go debugger? ?
? ??? ?? ??)
kubernetes? ??? ?? Go ???, ??? ?
??? ????? ?? ??.
???? ??, ??? callback ?? ??? ?
? (static ????? duck typing? ?? ??
?? ????.)
=> ????? ? ???? ??? ?? ???
???? ??.

Kubernetes??
27
- ??? ???? ?? ??? ???.
- ?? ?? ?? ??? ???? ??.

3. ?? ??: Magnum
Total Management for using kubernetes on openstack
28

?? ??
29
Magnum := IaaS + Cluster Mng

Magnum ?
??? ??
30
? Magnum API Server
? Magnum Conductor Process
? Sql(mysql..), Broker ??
? ?? ???? Component? ??
? ?????? fedora21? rpm ???
- ?? ??? ??? ??, ?? ?? Wrapping.
- Heat?? ????? Orchestrator ? ??? ?
?? ?? ? Scheduling.
- ?? UI? ?? ?. Liberty ???? ??? ??
.
- ???? ????? Heat??? ?? CLI? ?
? ?? ??.
- Keystone? ?? API??, Multi-tenancy ??

Magnum
Architecture
31
? Magnum Architecture(from wiki)
? ???? Controller ???? ???? ? ???? ?
?
? Kubernetes As First Class Resource

Magnum ?
??? ??
32
? Openstack?? Container???
magnum? ??? ?? ??
? PTL? Adrian Otto
? ?? IRC ??, Agenda? ???, ??
?? ? ?? ??? ??
? ???? ?? ?? ?? ? ??.
? Etherpad ? BluePrints? ??? ??
? ??? ?? ???? ? ?? ??
?? ???? ????? ???? ?? ???
? ??? ???? ??? ????? ????
? ????? ?? ? ? ??? ???? ??.

33
Bay Model ??
CLI? Baymodel? ?? ?
?.

Bay Model?
Flavor ??
34 ? docker? ?? ? ??
?? ?? ???? ?
? ? ??? ??
? ???? ???? ??
? ?? ???
? Flavor? ?? ????
? ??? (cpu,ram,disk
size?)? ?????,
?? Bay Model? COE
? ??? ?? ??
- ???, flavor, ????,
keypair, volume? ?? ?
? ? ?? ???
# magnum baymodel-create
--name testbaymodel2
--image-id fedora-21-atomic-3
--keypair-id testkey
--external-network-id $NIC_ID
--dns-nameserver 8.8.8.8
--flavor-id m1.small
--docker-volume-size 5
--coe kubernetes

36
Bay ??
CLI? ?? ??? Bay
? ????.

37
Bay ??
horizon?? ???
bay ????? ?? ?
?.
node count + 1?
(master node)? ???
?? ???.
ext ????? dhcp?
?? ??? ????
???? ?? ? ??
(heat ???)

38
Bay ??
node count ?? ???
?? ? attach ???
?? ????.
slave ???? attach
?.

39
Bay ??
Magnum? bay? ??
??? private network
? router? ???.

40
Bay ??
Magnum? bay? ??
??? private network
? router? ???.

Bay ?? ??
? ??
41
# magnum bay-create
--name testbay
--baymodel
testbaymodel --node-
count 1
#magnum bay-update
testbay
replace node_count=2
? ?? ?? ??
? Bay Model? ????
? ???? ?? ??
?? ????? ???
??
? ???? ?? ? ??
??? ???
Openstack Heat? ??.
? ???? ? ?? ??
? ???
? node-count? slave:
master node 1? ??
- bay model? ?? ?? ?
?? ???.

Bay ?? API
? ??
42
? Openstack Heat? ?? ??, kvm ??? ??
? Heat? ?? ????? ?? ?? ?? ??
? ??? ????? ? ????? Master?? ??
? Metadata ? ??? Cloud-init?? Kubernetes ??

Heat ? HOT
( Heat
Orches
tration Temp
late)
43
? Heat := Amazon? CloudFormation?
??? ????(???), CloudForm
ation API? ??? ?? ??
? Heat ?? ?? ???? ??
(HOT:Heat Orchestration Template)
? YML ?? XML??? ???? ??
?? ?? ?? ? API?? ?? ??
??
? ?? ???? ??????? ??
? ?? ??
- ????? ?? ???? ?? ?? ?
?

HOT - 1
44
kube_minion_eth0:
type: OS::Neutron::Port
properties:
network: {get_param: fixed_network}
security_groups:
- get_resource: secgroup_all_open
fixed_ips:
- subnet: {get_param: fixed_subnet}
replacement_policy: AUTO
kube_minion_floating:
type: OS::Neutron::FloatingIP
properties:
floating_network: {get_param: ex..
docker_volume:
type: OS::Cinder::Volume
properties:
size: {get_param: docker_volume_size}
docker_volume_attach:
type: OS::Cinder::VolumeAttachment
properties:
instance_uuid: {get_resource:
kube_minion}
volume_id: {get_resource:
docker_volume}
mountpoint: /dev/vdb
Heat Orchestration Template YML
??? ???
Namespace ? ??
OS::????????
??::?????
??? ???? ???
? ??? Env???
?? ???
?? ???? ????
?? ????? ??
??
Heat ???? sh ??
?? ??

HOT - 2
45
configure_docker_storage:
type: OS::Heat::SoftwareConfig
properties:
group: ungrouped
config: {get_file: fragments/configure-
docker-storage.sh}
...
enable_services:
type: OS::Heat::SoftwareConfig
properties:
group: ungrouped
config: {get_file: fragments/enable-
services-minion.sh}
�� ??? ???
Heat ???? sh ??
?? ??
configure-docker-storage.sh
? ??? ?? ????

Heat?
Resource::
Shellscript
-
OS::Heat::Software
Config
46
? heat resource? configure-docker-storage.sh
? ?? ?? shellscript? ???? vm??? ??
?? ??
#!/bin/sh
. /etc/sysconfig/heat-params
DOCKER_DEV=/dev/disk/by-id/virtio-${DOCKER_VOLUME:0:20}
attempts=60
while [[ ! -b $DOCKER_DEV && $attempts != 0 ]]; do
echo "waiting for disk $DOCKER_DEV"
sleep 0.5; udevadm trigger ; let attempts--
done
if ! [ -b $DOCKER_DEV ]; then
echo "ERROR: device $DOCKER_DEV does not exist" >&2
exit 1
fi
pvcreate $DOCKER_DEV
vgcreate docker $DOCKER_DEV
cat > /etc/sysconfig/docker-storage-setup <<EOF
VG=docker
EOF

Cloud-init ?
metadata -
Master VM
47
? ???? ?? ? ?? ?? Cloud-init?? Master
??? Kubernetes ??? Enable ? ??
? ?? ??? ??? curl? Heat ?? response
callback ?? -> heat?? ???? ??
Cloud-init v. 0.7.5 running 'init-local' at Sat, 20 Jun 2015 13:59:05 +0000. Up 171.93 seconds.
Cloud-init v. 0.7.5 running 'init' at Sat, 20 Jun 2015 13:59:41 +0000. Up 207.75 seconds.
��
activating service etcd
Created symlink from /etc/systemd/system/multi-user.target.wants/.../etcd.service.
activating service kube-apiserver
Created symlink from /etc/systemd/system/multi-user.target.wants/.../kube-apiserver.service.
activating service kube-scheduler
Created symlink from /etc/systemd/system/multi-user.target.wants/.../kube-scheduler.service.
activating service kube-controller-manager
Created symlink from /etc/systemd/system/multi-user.target.wants/.../kube-controller-manager.service.
#!/bin/bash -v
curl -i -X POST -H 'X-Auth-Token: dd5b13e8652f40bfa5f8a57f66f57153' -H 'Content-Type: application/json' -H 'Accept:
application/json' http://10.10.10.100:8004/v1/9fcaf0400e87437aad59b0ed33afb29d/stacks/testbay-
wns45kuqcnbh/d12fd713-5cc2-4bd5-8e6c-0548e5678abe/resources/master_wait_handle/signal --data-binary '{"status":
"SUCCESS"}'
Master VM - ???? ?? ? ?? ?? ?? ??

Cloud-init ?
metadata -
Slave VM
48
? ???? ?? ? ?? ?? Cloud-init?? Slave
??? Kubernetes, docker ??? Enable ? ??
Cloud-init v. 0.7.5 running 'init-local' at Sat, 20 Jun 2015 13:59:05 +0000. Up 171.93 seconds.
Cloud-init v. 0.7.5 running 'init' at Sat, 20 Jun 2015 13:59:41 +0000. Up 207.75 seconds.
��
Physical volume "/dev/vdb" successfully created.. Volume group "docker" successfully created .. Logical volume
"data" created.
...
activating service flanneld
Created symlink from /etc/systemd/system/.../flanneld.service.
activating service docker.socket
activating service docker
activating service kubelet
Created symlink from /etc/systemd/system/.../kubelet.service.
activating service kube-proxy
Created symlink from /etc/systemd/system/.../kube-proxy.service.
#!/bin/bash -v
curl -i -X POST -H 'X-Auth-Token: 4478846c195..�� applicationn/json'
http://10.10.10.100:8004/v1/9fcaf0400e87437aad59b0ed33afb29d/stacks/testbay-...-c6tn72wdte4k/f1921039-...-
5469f80c1d20/resources/minion_wait_handle/signal --data-binary '{"status": "SUCCESS"}'
Slave VM - ???? ?? ? ?? ?? ?? ??

AtomicOS
49
? Container??? ?? ?? OS
? Kubernetes ? ?? ?? ????
? Fedora 2x / Centos7 ?? (systemd)
? ?? qcow2?? vm ???? ??: ?
? ?????? ??
? ISO ??? PXE provisioning ? ??
(anaconda UI? ????)
? RPM-OStree ? ?? ??? ? OS??
(yum? ?? Git? ?? ???? ??
?? ?? ??:: os ??? rollback)
? Cockpit ?? ???? ???? UI ?
?
? ??? ??

Place your screenshot here
50
Pod ??
CLI? Pod ?? ??.

51
Pod ??
master node? ssh? ??
?? kubectl? pod ?? ?
??? ????.
master node��s internal ip

52
Pod ??
slave node? ssh? ?
??? docker ps??
?? docker ??? ?
???.

Pod ?? ??
? ??
53
# magnum pod-create --
manifest ./redis-master.yaml --
bay testbay
? ? ??? docker
cluster ?? (0.15 ??
??: ?? ? usecase
? ?? ?? ? ??)
? Manifest yaml ?? ?
?
? ? ??? Bay? ??
? docker? ?? ???,
vm? attach???? ?
?? ??
? docker?? ?????
flanneld? ??
- Req? magnum ? ??
kubernetes? ?? ? ??.

Pod ?? API
? ??
54
? Magnum Conductor? kubernetes? API ??
? Kubernetes ? Manifest yaml???? docker ??
? Slave Node? docker ???? ??
? Conductor? ?? ?? ??? status? ????
(DB)
Master Slave

Magnum??
Kubernetes
? ???? ?
?
55
? Conductor ?? http? kubernetes? ?? (kubectl)
- Magnum ??????: Python 2.x ??, Openstack ??
?? Oslo ? pecan (API)
def pod_create(self, context, pod):
...
manifest = k8s_manifest.parse(pod.manifest)
try:
resp = self.k8s_api.createPod(bod...
except error.HTTPError as err:
pod.status = 'failed'
if err.code != 409:
pod.create(context)
message = ast.literal_eval(err.read())['message']
raise exception.KubernetesAPIFailed(code=err.code,
message=message)
pod.status = resp.status.phase
class ApivbetaApi(object):
...
params = locals()
for (key, val) in params['kwargs'].iteritems():
if key not in allParams:
raise TypeError("Got an ...ints" % key)
params[key] = val
del params['kwargs']
resourcePath = '/api/v1beta3/endpoints'
resourcePath = resource..at}', 'json')
method = 'GET'

Pod
Manifest
56
apiVersion: v1beta3
kind: Pod
metadata:
labels:
name: redis
redis-sentinel: "true"
role: master
name: redis-master
spec:
containers:
- name: master
image: kubernetes/redis:v1
...
ports:
- containerPort: 6379
resources:
limits:
cpu: "1"
volumeMounts:
- mountPath: /redis-master-data
name: data
- name: sentinel
...
ports:
Pod Manifest ??
API version: kubernetes
API ??
Kind: ?? ??. ( Pod,
Service, RC )
Spec: containers -> ?
? ?? docker ??
image: docker ???
port: container? listen
?? ??
resources: docker hw?
?

Flannel?
docker ??
?? - 1
57
? ETCd? backend? ?? Network Management
Daemon
? AtomicOS??? vxlan(Virtual Extensible LAN)? ?
??? (?? udp? ? ??)
image from flannel github

Flannel?
docker ??
?? - 2
58
const (
tunDevice = "/dev/net/tun"
..
func OpenTun(name string) (*os.File, string, error) {
tun, err := os.OpenFile(tunDevice,
os.O_RDWR, 0)
if err != nil {
return nil, "", err
}
var ifr ifreqFlags
copy(ifr.IfrnName[:len(ifr.IfrnName)-1],...
ifr.IfruFlags = syscall.IFF_TUN |
syscall.IFF_NO_PI
err = ioctl(int(tun.Fd()),
syscall.TUNSETIFF,uin...))
if err != nil {
return nil, "", err
}
{
"Network":"172.16.0.0/1
2"
"SubnetLen": 24,
"Backend": {
"Type": "vxlan"
}
}
? RESTful with ETCd
? Called by heat
template
Flannel ?? ?? (GO lang)
- backend ?? ??
- TUN Device ??
Flannel API
- network ??

59
Service ??
CLI? Service? ??
??.

60
Service ??
1
CLI? ??? Service?
????

61
Service ??
2
Master????
Service? ?? ??.

62
Service ??
2
Slave ???? ???
??? kube-proxy??
??? ?? ??.

Service ??
??? ??
63
#magnum coe-service-create --
manifest ./redis-sentinel-
service.yaml --bay testbay
? Kubernetes ? service
? ??? RB? ? ? ?
?? ????? ???
? ??
? Physical ???? ??
Docker? ???? ??
? L3 ?? L4 ??? ??
? Session Affinity ??
? Kube-proxy? ?? ??
Listen
- ????? Iptables? ??
kube-proxy ????? ???
?? ???? ???? ??
?? ??? ??

Service
Manifest
64
apiVersion: v1beta3
kind: Service
metadata:
labels:
name: sentinel
role: service
name: redis-sentinel
spec:
ports:
- port: 26379
targetPort: 26379
selector:
Service Manifest ??
Sentinel? ?? service
??
Redis Master? ???
Metadata? ??
selector? ?? pod ?
?
spec.ports.port? 26379
? service port
Failover?? ????
pod? ???? ??
ClientIP ?? ????
IP ?? ??

???
- ???? ?
?? ?? ??
? ??
65
? ?? ??? kube-proxy port ? iptables?
service port? ????? ??
? 1234? ??? ??
image from doc

66
RC ??
CLI? RC? ?? ? ?
?? ??.
(RC: replication
controller)

RC ?? ??
? ??
67
#magnum rc-create
--manifest ./redis-
controller.yaml
--bay testbaymagnum
? pod(docker cluster
instances)? ?? ?.
? Pod? Label selector?
???? ?? ??.
Label? ???? ??
? -> ?? ? ?? ??
?? ???.
? Rescheduling(? ??),
Scaling(kubectl scale),
Rolling updates ? ??
- pod? ??? ?? ?? ?
??? ???? ??

RC Manifest
68
apiVersion: v1beta3
kind: ReplicationController
metadata:
spec:
replicas: 1
selector:
template:
metadata:
labels:
role: sentinel
spec:
containers:
- name: sentinel
env:
- name: SENTINEL
value: "true"
ports:
RC Manifest ??
Redis Sentinel ? ??
Replication
kind(??): Replication
Controller
spec? ??? pod? ?
? ??:: template
replicas? replication
??? ??

?? ?? ??
70
? Magnum + Bare Metal (Ironic)
? Magnum + Swarm (kubernetes ??)
? Coreos (Fedora AtomicOS ??)
? Murano (Mirantis, Application Catalog)
? ?? kolla :openstack? docker? ??? ?
??? ??
??? ??? ????, ???? ??? ???
?? Contribution? ???? ??.
=> ?? ?? ? ?? ??? ?? ?? ?? ??
?? ?? ??

??? ???? ??
?? ???
- ??? vs ????
- ?? vs ??
- ??? vs SI
- ?? vs ??
72

73
THANKS!
?? ??? ???? ??
?? ??? ?? ?? ??
?? ?? ??? ??? ?? ?? ??
? ? ?? ???? ?? ?? (????
?? ??? ??)
tzniuta@gmail.com

CREDITS
Special thanks to all the people who
made and released these awesome
resources for free:
? Presentation template by
�ݺ�ߣsCarnival
? Photographs by Unsplash
More info on how to use this template at www.slidescarnival.com/help-use-
presentation-template
This template is free to use under Creative Commons Attribution license. If you
use the graphic assets (photos, icons and typographies) provided with this
presentation you must keep the Credits slide.
74

�ݺ�ߣ

?2? ???? ???? ??? ??? Kubernetes

Recommended

More Related Content

What's hot (20)

Viewers also liked (10)

Similar to ?2? ???? ???? ??? ??? Kubernetes (20)

More from Tommy Lee (20)

?2? ???? ???? ??? ??? Kubernetes