�ݺ�ߣ

Performance Aware SDN
http://www.meetup.com/SF-Bay-Area-Large-Scale-Production-Engineering/
Peter Phaal
InMon Corp.
June 2013
Thursday, June 13, 13

Why monitor performance?
“If you can’t measure it, you can’t improve it”
Lord Kelvin

BalancerLoad ServerServerWeb
DatabaseApplication
Network
MemcacheMemcache
ServerServerWeb
Server
Application
Server
Database
BalancerLoad Balancer
Networking in large, scale-out, multi-tiered sites
• Large number of servers in each pool
• Servers constantly added/removed
• Network performance is critical
• scale-out applications dependent on network performance
• potential for propagating failures between tiers

http://www.osa.org/osaorg/media/osa.media/CorporateGateway/ExecutiveForum2013/2013_Executive_Forum_Keynote_Najam_Ahmad.pdf
Najam Ahmad, Vice President, Infrastructure, Facebook
OSA 2013 keynote
SDN brings network under software control
Extends DevOps tool stack to include network visibility and control

Feedback control
Measure
Control
System
desired
output
measured
output

Controllability and Observability
Basic concept is simple, a stable feedback control system requires:
1. ability to inﬂuence all important system states (controllable)
2. ability to monitor all important system states (observable)

It’s hard to stay on the road if you can’t see the
road, or keep to the speed limit without a
speedometer
It’s hard to stay on the road or maintain
speed if your brakes, engine or steering fail
Controllability and Observability driving example
Observability
Controllability
States location, speed, direction, ...
Dense Tule fog in Bakersﬁeld, CA

Effect of delay on stability
Measurement delay Planning delay
Time
Conﬁguration delayDisturbance Response delay
EffectLoop delay
DDoS launched Identify target, attacker Black hole, mark, re-route? Switch CLI commands Route propagation Trafﬁc dropped
Components of loop delay
e.g. Slow reaction time causes
tired / drunk / distracted
driver to weave, very slow
reaction time and they leave
the road

Observability using sFlow standard
“In God we trust. All others bring data.”
Dr. Edwards Deming

Industry standard measurement technology integrated in switches
http://www.sﬂow.org

Open source agents for hosts, hypervisors and applications
Host sFlow project (http://host-sﬂow.sourceforge.net) is center
of an ecosystem of related open source projects embedding
sFlow in popular operating systems and applications

Network (maintained by hardware in network devices)
- MIB-2 ifTable: ifInOctets, ifInUcastPkts, ifInMulticastPkts, ifInBroadcastPkts, ifInDiscards, ifInErrors, ifUnkownProtos,
ifOutOctets, ifOutUcastPkts, ifOutMulticastPkts, ifOutBroadcastPkts, ifOutDiscards, ifOutErrors
Host (maintained by operating system kernel)
- CPU: load_one, load_five, load_fifteen, proc_run, proc_total, cpu_num, cpu_speed, uptime, cpu_user, cpu_nice,
cpu_system, cpu_idle, cpu_wio, cpu_intr, cpu_sintr, interupts, contexts
- Memory: mem_total, mem_free, mem_shared, mem_buffers, mem_cached, swap_total, swap_free, page_in, page_out,
swap_in, swap_out
- Disk IO: disk_total, disk_free, part_max_used, reads, bytes_read, read_time, writes, bytes_written, write_time
- Network IO: bytes_in, packets_in, errs_in, drops_in, bytes_out, packet_out, errs_out, drops_out
Application (maintained by application)
- HTTP: method_option_count, method_get_count, method_head_count, method_post_count, method_put_count,
method_delete_count, method_trace_count, method_connect_count, method_other_count, status_1xx_count,
status_2xx_count, status_3xx_count, status_4xx_count, status_5xx_count, status_other_count
- Memcache: cmd_set, cmd_touch, cmd_flush, get_hits, get_misses, delete_hits, delete_misses, incr_hits, incr_misses,
decr_hists, decr_misses, cas_hits, cas_misses, cas_badval, auth_cmds, auth_errors, threads, con_yields,
listen_disabled_num, curr_connections, rejected_connections, total_connections, connection_structures, evictions,
reclaimed, curr_items, total_items, bytes_read, bytes_written, bytes, limit_maxbytes
Standard counters

Simple
- standard structures - densely packed blocks of counters
- extensible (tag, length, value)
- RFC 1832: XDR encoded (big endian, quad-aligned, binary) - simple to encode/decode
- unicast UDP transport
Minimal configuration
- collector address
- polling interval
Cloud friendly
- flat, two tier architecture: many embedded agents → central “smart” collector
- sFlow agents automatically start sending metrics on startup, automatically discovered
- eliminates complexity of maintaining polling daemons (and associated configurations)
Scaleable push protocol

• Counters tell you there is a
problem, but not why.
• Counters summarize
performance by dropping high
cardinality attributes:
- IP addresses
- URLs
- Memcache keys
• Need to be able to efﬁciently
disaggregate counter by
attributes in order to
understand root cause of
performance problems.
• How do you get this data
when there are millions of
transactions per second?
Counters aren’t enough
Why the spike in trafﬁc?
(100Gbit link carrying 14,000,000 packets/second)

• Random sampling is lightweight
• Critical path roughly cost of
maintaining one counter:
if(--skip == 0) sample();
• Sampling is easy to distribute
among modules, threads,
processes without any
synchronization
• Minimal resources required to
capture attributes of sampled
transactions
• Easily identify top keys,
connections, clients, servers,
URLs etc.
• Unbiased results with known
accuracy
Break out trafﬁc by client, server and port
(graph based on samples from100Gbit link carrying 14,000,000 packets/second)
sFlow also exports random samples

Integrated data model
Packet HeaderPacket Header
Source Destination
TCP/UDP Socket TCP/UDP Socket
MAC Address MAC Address
Sampled Packet Headers
I/F Counters
Power, Temp.
NETWORK
HOST
CPU
Memory
I/O
Power, Temp.
Adapter MACs
APPLICATION
Sampled Transactions
Transaction Counters
TCP/UDP Socket
Independent agents sFlow analyzer joins data for integrated view

Virtual Servers
Applications
Apache/PHP
Tomcat/Java
Memcached
Virtual Network
Servers
Network
Embedded monitoring of all
switches, all servers, all
applications, all the time
Consistent measurements
shared between multiple
management tools
Comprehensive visibility

Software Deﬁned Networking
“You can’t control what you can’t measure”
Tom DeMarco

Monitor
Feedback control loop with sFlow and OpenFlow
low conﬁguration delay
low measurement delay
Together, sFlow and OpenFlow provide the observability and
controllability to enable SDN applications targeting low latency
control problems like load balancing and DDoS mitigation
low planning delay
SDN application

Network OSApplication
Open APIsApplication
Application
Data Plane
Control Plane
Conﬁguration Forwarding Visibility
NETCONF/OF-Conﬁg
Open APIs
Hosts
sFlow adds actionable visibility to SDN stack
Actionable = complete + timely

REST API
Metrics
Flow Deﬁnitions
Thresholds
InMonsFlow-RT
REST API
OpenFlowController
Load Balancer DDoS Protection
REST Applications
Open “Southbound” APIs
Data Plane
Control Plane
Hosts
Open “Northbound” APIs
SDN Applications
SDN feedback control applications

ovs-vsctl set-controller br0 tcp:10.0.0.1:6633
ovs-vsctl -- --id=@sflow create sflow agent=eth0
target=”10.0.0.1:6343” sampling=1000 polling=20
-- -- set bridge br0 sflow=@sflow
Connect switches to central control plane
e.g connect Open vSwitch to OpenFlow controller
e.g. connect Open vSwitch to sFlow analyzer
Minimal conﬁguration to connect switches to
controllers, intelligence resides in external software

Components of a DDoS flood attack
1. Command to attack target sent over
control network
2. Large number of compromised hosts
start sending traffic to target
3.Traffic converges on access link,
overwhelming capacity and denying
access

Define flow keys
DDoS Protection
define address groups
define flows
define thresholds
while(running) {
receive threshold event
monitor flow
deploy control
monitor flow
release control
}
OpenFlow
Controller
REST API
sFlow-RT
REST API
1
2
3
4
6
5
8
7
REST operation flow chart

curl -H "Content-Type:application/json" -X PUT
--data "{external:['0.0.0.0/0'], internal:['10.0.0.0/8']}"
http://localhost:8008/group/json
1. Define address groups
--data "{keys:'ipsource,ipdestination', value:'frames',
filter:'sourcegroup=external&destinationgroup=internal'}"
http://localhost:8008/flow/incoming/json
2. Define flows
--data "{metric:'incoming', value:1000}"
http://localhost:8008/threshold/incoming/json
3. Define thresholds
curl "http://localhost:8008/events/json?eventID=4&timeout=60"
4. Receive threshold events

5. Monitor ﬂow
curl http://localhost:8008/metric/10.0.0.16/4.incoming/json
[{
"agent": "10.0.0.16",
"dataSource": "4",
"metricName": "incoming",
"metricValue": 1582.93965044338071,
"topKeys": [
{
"key": "192.168.1.1,10.0.0.151",
"updateTime": 1357169662500,
"value": 1582.93965044338071
},
{
"key": "192.168.1.4,10.0.0.151",
"updateTime": 1357169665500,
"value": 46.552918457198984
}
],
"updateTime": 1357169665500
}]
6. Deploy control
curl -d '{"switch": "00:00:00:00:00:00:00:01",
"name":"ddos-1", "cookie":"0", "priority":"32768",
"ingress-port":"4","active":"true"}'
http://localhost:8080/wm/staticflowentrypusher/json

threshold
attack starts
detected
control implemented attack eliminated
http://blog.sflow.com/2013/03/ddos.html
Before
After
DDoS mitigation results
packets/secondpackets/second
sustained 6M packets/second attack
(30 Gigabits/second)
http://packetpushers.net/openflow-1-0-actual-use-case-rtbh-of-ddos-traffic-while-keeping-the-target-online/
Also: http://blog.sflow.com/2013/05/controlling-large-flows-with-openflow.html

ECMP/LAG multi-path trafﬁc distribution
http://static.usenix.org/event/nsdi10/tech/full_papers/al-fares.pdf
index = hash(packet fields) % linkgroup.size
selected_link = linkgroup[index]
Hash collisions reduce effective cross sectional bandwidth
1:1 subscription ratio doesn’t eliminate blocking, collision
probabilities are high, even with large numbers of paths

Birthday Paradox
What is the chance that at least two people in a room will share a birthday?
50/50 chance with 23 people, virtual certainty with the 60 people.This is a
“paradox” because the probability seems remarkably high considering that there
are 365 possible birthdays (366 if you include Feb 29) and 23 people represents
just over 6% of the theoretical maximum and 60 people is only 16%.
http://en.wikipedia.org/wiki/Birthday_problem
ECMP/LAG/MLAG collision probabilities are surprisingly high

http://research.microsoft.com/en-us/UM/people/srikanth/data/imc09_dcTraffic.pdf
Small number of long lived large flows responsible for bulk of load
Use SDN load balancing applications to detect and
eliminate collisions by adjusting forwarding paths
Load balancing large flows

colliding flows
http://blog.sflow.com/2013/01/load-balancing-lagecmp-groups.html
http://blog.sflow.com/2013/03/ecmp-load-balancing.html
http://blog.sflow.com/2013/02/sdn-and-large-flows.html
https://datatracker.ietf.org/doc/draft-ietf-opsawg-large-flow-load-balancing/
Not just ECMP, also LAG/MLAG and WAN etc.
non-colliding flows
http://blog.sflow.com/2013/06/large-flow-detection.html
http://blog.sflow.com/2013/06/flow-collisions.html

Memcached hot keys
http://codeascraft.com/2012/12/13/mctop-a-tool-for-analyzing-memcache-get-trafﬁc/
Server B
Server C
Server A
Client 1
Client 2
Client 3
Client 4
Memcache clusterMemcache clients
overloaded server link
http://blog.sﬂow.com/2013/01/memcache-hot-keys-and-cluster-load.html
• Interesting parallel with ECMP/LAG hash collisions
• Demonstrates linkage between network and application performance
• Can monitor cluster wide Memcached hot/missed keys with sFlow
• Possible SDN use cases:
• Server placement informed by visibility into network topology, loads etc.
• Use SDN to shorten paths, reducing latency and packet loss
• Avoid packet loss by steering packets around congested link
• Extension of OpenFlow to optical circuit switches allows network to be
rewired for actual demand

Next steps
Organizational: break down networking silo
- learn more about networking
- integrate networking into DevOps team
- think about observability and controllability when
purchasing equipment and architecting services
Strategic: Engage developer communities
- share operational expertise with SDN community
- help specify northbound APIs so that they deliver
functionality needed to integrate networking into
DevOps stack

Questions?

Backup slides

• OpenFlow
• Hybrid combined OpenFlow (using NORMAL action)
• Puppet
• BGP policy
• RESTful API to switches
• NETCONF
• Optical circuit switching
- OpenFlow extensions
- SOCM (Service-Based Optical Connection Management)
Programatic control of switches
http://www.nanog.org/sites/default/files/wed.general.brainslug.lapukhov.20.pdf
http://blog.sflow.com/2013/03/pragmatic-software-defined-networking.html
http://www.nanog.org/sites/default/files/wed.general.socm_.samberg.6.pdf

packets
decode hash sendflow cache flushsample
NetFlow/IPFIX
send
polli/f counters
sample
• sFlow exports packet samples immediately
• sFlow also exports interface counters
• NetFlow exports flow data on end of flow, active-timeout or inactive-timeout
• NetFlow data generation requires significant resources on switch that can
be better applied to increase size of forwarding table(s)
• OpenFlow metering has similar architecture to NetFlow and similar
limitations
sFlow and NetFlow/IPFIX in a switch

InMon sFlow-RT
active timeout active timeout
NetFlow
Open
vSwitch
SolarWinds Real-Time NetFlow Analyzer
• sFlow does not use flow cache, so realtime charts more accurately reflect traffic trend
• NetFlow spikes caused by flow cache active-timeout for long running connections
Rapid detection of large flows
Flow cache active timeout delays large flow detection,
limits value of signal for real-time control applications
http://blog.sflow.com/2013/01/rapidly-detecting-large-flows-sflow-vs.html

�ݺ�ߣ

Performance Aware SDN, LSPE talk

More Related Content

Performance Aware SDN, LSPE talk