�ݺ�ߣ

Is the current model of load
testing broken?
Steve Thair
Seriti Consulting
@TheOpsMgr

2

Some background….
 My User Group – Dec 2011 -

http://www.meetup.com/London-Web-Performance-Group/

(c) Seriti Consulting, 2011

Delivery Alert Deviation
from
Team
normal?

Ready Fail Fail to
for Build meet
Test? SLA’s?

Deploy

Measurements
Environment

Continuous End User
Perf Trends
Load Response
Injection
For Each Data
Service/API Point
Usage response
Profile
Production
logs System
Perf Details
monitors

http://www.slideshare.net/sthair/continuous-integration-a-performance-engineers-journey
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

4

The Killer Comment…
“We’ve had to look at separating load
injection from performance
measurement”.
- Andrew Harding, Betfair

http://www.seriticonsulting.com/blog/2011
/12/9/is-the-current-model-of-
loadperformance-testing-broken.html

(c) Seriti Consulting, 2011 @TheOpsMgr #ukcmg #webperf

So why are you paying all 5
that money for that
What are key expensive brand-name
reasons behind load testing tool then?
that decision?
Surely there are
open source or
cheap cloud
tools if all you
want is load?

What other
issues might we
have with
What are you
“traditional” load
using to measure
testing models
Performance
in a Web 2.0
then?
world?

@TheOpsMgr #ukcmg #webperf

6

Defining the “current model”
“Stephen didn’t explicitly define
what the current model of
load/performance testing is…”
 Alex Podelko

http://applicationperformanceengineeringhub.com/is-the-current-model-
of-loadperformance-testing-broken/


7

“Current Model Straw Man”
(1) Waterfall Development Cycle
(2) Load Tool compiles the report
(3) Reporting at the end of the test
(4) Request / Response paradigm
(5) Well-defined customer journeys

(c) Seriti Consulting, 2011

8

Testing in a CI/Agile world
 Performance testing initiated every time code is committed to
the CM repository (e.g. SVN, GIT etc)
 Betfair issue was that their system was complex with many
layers of caching etc so it took longer to “warm up” the
environment (to achieve a steady performance state) than
they had between check-ins…
 So they needed “continuous injection” to keep the
environment constantly warm…
 The test tool never “stopped” to “compile the report”…
 So any tool that “reported at the end” wasn’t as useful…


9

Test tools in a APM & RUM world
 “traditional test tools” generally have the ability to deploy
agents to gather metrics from the target environment
 But the depth of analysis and correlation falls well below that
of modern Application Performance Management tools e.g.
AppDynamics
 APM tools offer a deeper insight and better event correlation
across tiers…
 And they are getting (much) cheaper…


APM Measurement


Google Real Time Analytics


12

WebSockets
 HTTP 1.1
 Start Timer
 GET /index.html
 Response 200 OK
 Stop Timer
 WebSockets
 Start Timer
 Socket “upgrade”
 N many frames…
 Stop??? Bi-directional socket channel


13

HTTP 2.0
 HTTP 1.1
 Sequential
 ordered
 HTTP 2.0 (& SPDY)
 Multiplexed over a
single connection
 Responses
returned out of
sequence…
 Hard to time!

http://stackoverflow.com/questions/10480122/difference-between-http-pipeling-and-http-multiplexing-with-spdy

14

Website Complexity
 263,000 entry points…


15

Visitor Flow
 How many paths thru a
website?
 Classic script-driven
approaches can’t hope to
address the complexity
 Network and log file
replay solutions?


16

A PCAP Solution to Replay?
 POC solution based on Cloudmeter
Pion + custom scripts
 Read a PCAP (network capture)
 Identify the HTTP traffic
 Filter it (based on your requirements)
 Parameterise it (query strings, POST
parameters etc)
 Randomise inputs from SQL, CSV etc
 Replay it against a test environment
 i.e. change the base URL
 Amplify & rate throttle req/sec as
required

http://www.cloudmeter.com/pion/data-processing.php


17

Personal Opinion!
YMMV…

So what’s the Answer?
 Use the cheapest method to generate load that you can
find…
 Move away from scripting-based approaches towards using
real-user session replay (if possible!)
 Generate load continuously
 Measure continuously
 using APM & RUM type tools – FOSS or Commercial
 Look for changes in histograms, averages, standard dev etc
 Protocol and Framework aware instrumentation
 AFAIK this currently doesn’t exist…

18

@LDNWebPerf User Group!
 Join our London Web Performance Meetup
 http://www.meetup.com/London-Web-Performance-Group/
 Next Wednesday 17th Oct – 7pm – Central London
 Follow us on Twitter @LDNWebPerf
 #LDNWebPerf & #WebPerf


19

About Me
 21yrs IT experience.
 Started with www in 1998 (IIS3! Site Server 3!).
 Web Architect @ BNP Paribas, CSFB etc
 Web Operations Manager for www.totaljobs.com, www.tes.co.uk
 Professional Services Manager @ www.siteconfidence.com
 Seriti Consulting – specialising in web operations, management and
Performance
 e:stephen.thair@seriticonsulting.com
 m:+44 7971 815 940
 Twitter: http://twitter.com/TheOpsMgr
 Blog: http://www.seriticonsulting.com/blog/
 LinkedIn: http://uk.linkedin.com/in/stephenthair
 Skype: seriti-steve

20

Questions?


�ݺ�ߣ

Is the current model of load testing broken ukcmg - steve thair

More Related Content

Is the current model of load testing broken ukcmg - steve thair

Editor's Notes