�ݺ�ߣ

Networking (Containers) in Ultra-
Low-Latency Environments
Avi Deitcher
avi@atomicinc.com

‫אכסניא‬
Avi Deitcher avi@atomicinc.com

‫אכסניא‬
Akh-san-ya ?aksnaja? n. (ancient Aramaic,
from Ancient Greek xénos) 1: Hospitality,
lodging; 2: Host.


‫אכסניא‬
Akh-san-ya ?aksnaja? n. (ancient Aramaic,
from Ancient Greek xénos) 1: Hospitality,
lodging; 2: Host.

:‫אכסניא‬ ‫בכבוד‬ ‫פותחים‬
Ancient Jewish custom to begin public speaking
by honouring or thanking the hosts.


Who Am I?

Who Am I?
(not 24601)

Who Am I?
•  Life in tech business:
–  10 yrs ﬁnancial services IT
–  10+ yrs consulWng & training
–  Some startups on the way
•  Avid (if not very good) ice hockey player
•  Long-Wme lover of great engineering…. when
used to make a real diﬀerence
•  Atomic Inc:
–  ConsulWng
–  Training
(not 24601)

A Lile History

A Lile History
Summer 2015
•  Fintech X: “Help us
containerize!”
–  Hint: It is harder than you
think… and worth it
–  Culture/process > technology
•  QuesWon: Networking?
•  Answer: ScienWﬁc method

A Lile History
Summer 2015
•  Fintech X: “Help us
containerize!”
–  Hint: It is harder than you
think… and worth it
–  Culture/process > technology
•  QuesWon: Networking?
•  Answer: ScienWﬁc method
Summer 2016

•  Good pracWce demands:
1.  Redo tests with new opWons
and versions
2.  Make tests available
3.  Explain it all well

What Is “Ultra-Low” Latency?

1.  hp://home.blarg.net/%7Eglinden/StanfordDataMining.2006-11-29.ppt
“every 100ms of delay costs 1% of
sales”[1]

“extra 0.5s in search page generaWon
Wme dropped traﬃc by 20%”[2]

2.  hp://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html
sales”[1]

“extra 0.5s in search page generaWon
Wme dropped traﬃc by 20%”[2]

2.  hp://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html
Not. Even. Close.
sales”[1]

Ultra-Low Latency
38 messages in 7 milliseconds

1 message (avg) every 184 𝓊-sec!

Networking Workloads
•  Networked Workloads:
“things that do work and must talk”
•  Same principles for all workloads:
– VMs
– Cloud
– Serverless
– Containers

Two Types of Networking…
Direct

Two Types of Networking…
Direct Fabric+Overlay

… maybe four
Workload Awareness

… maybe four
Workload Awareness Fabric Awareness

Networking OpWons
Direct
Metal
macvlan
Bridge/vSwitch
(no NAT)
net=host
SR-IOV
Overlay
Flannel
Weave
Docker Overlay
Calico (IPIP)

Workload Awareness
Docker bridge (NAT)
Fabric Awareness
Calico (NaWve)


Our Tests
What We Tested
•  netperf ⇒ netserver
•  UDP & TCP round-robin
•  Sizes: 300, 500, 1024, 2048
•  No orchestraWon = complete
control
•  50000 iteraWons
–  Law of large numbers
•  Latency (Avg, %iles), CPU

•  DiﬀerenRals, not absolutes
How We Tested
•  .net
–  Because it had to be metal
–  Wicked smart team
•  Complete test run
–  Network changes
–  Hardware variaWons, errors
hps://github.com/deitch/network-tests

Local vs. Remote

Local Networking Summary
•  SR-IOV horrible latency but great CPU
–  Hold that thought…
•  net=host on par with metal
•  macvlan closest virtualized to metal
•  Rest in same range:
–  Latency: 5-10 𝓊-sec overhead
–  CPU: negligible diﬀerence
•  Calico (IPIP & naWve) & Docker overlay slightly
more performant
•  Watch out for very large TCP packets

Remote Networking Summary
•  Weave (sleeve) adds latency and CPU
– Reason for “fast datapath”
•  Again, macvlan best virtualized
•  All the rest:
– Latency: within 50 𝓊-sec of each other, except SR-
IOV with very large TCP packets
– CPU: similar, but keep an eye on Flannel (UDP)

About that SR-IOV
Type 1: Intel I350 1Gbps
Type 3: Mellanox MT27500 ConnectX-3 10Gbps

SR-IOV
SR-IOV does not automaRcally mean beXer
•  Switch in network card
•  Trades host CPU for card processor
•  Quality varies drama5cally
–  Even Mellanox far worse locally
•  My 2¥: SR-IOV falls further behind due to:
–  Speed of iteraWon
–  Open-source
–  Sowware + CPU

Headaches (and Thanks)
•  Headaches
–  Weave SYN-(nothing)
–  etcd is “touchy”
–  Packet L3 network is powerful but… unique
•  Macvlan, weave, flannel: all required pings for mac
•  Se{ng up bridge w/o NAT, Calico, macvlan was “different”
–  SR-IOV is complicated and flaky, especially Mellanox
–  netperf with UDP packets can get stuck (Calico-ipip)
–  And a whole lot more (ask me offline)

•  And thanks:
–  Bryan Boreham, Adam Harrison at weave.works
–  Zac Smith, Adam, Aaron, Andy, Lucas, everyone at Packet

What else could we do?
Ø Other hardware types
Ø Other network fabrics
Ø Docker macvlan network driver (experimental)
Ø Ipvlan
Ø Other packet sizes
Ø Kernel and network stack tuning
Ø Distant (and VPN) networks
Ø Other traﬃc paerns
Ø Other host-to-host encrypWon
Ø A whole lot more…

Conclusions
•  SR-IOV: most of the Wme, just not worth it
•  Performance:
–  Metal (+ net=host): always performs best
–  Direct network++: macvlan is your friend
–  Others: Roughly similar, careful of Weave (sleeve)

•  What’s your use case?
–  ULL: Metal/net=host > macvlan > calico > overlay
–  Everything else: Focus on your architecture and skills

Pick intelligently: easier, not simple

QuesWons and help:
@avideitcher avi@atomicinc.com

�ݺ�ߣ

LinuxCon/ContainerCon Japan 2016 "Networking Containers in Ultra-Low Latency Environments"

More Related Content

LinuxCon/ContainerCon Japan 2016 "Networking Containers in Ultra-Low Latency Environments"