The PerfOps revolution - Shutterfly's journey to faster performance
n 2016, Shutterfly, the leader in photo services and personalized gifts in the United States, embarked on a major program to accelerate every aspect of its website while building a PerfOps culture spanning across engineering, operation and product teams. The goal? Offer the fastest experience to customers while increasing revenue, conversion and SEO traffic. Some of the largest retailers and marketplaces went through this journey: As an example, Pinterest announced that their initiatives resulted in a 40% decrease in wait time, a 15% increase in SEO traffic and a 15% increase in conversion rate to signup. There is a strong correlation between faster web performance and higher business KPIs and Shutterfly saw a big opportunity for their customers and business. For the past 18 months, with strong support from Shutterfly senior staff, Fred Beringer, VP Digital Customer Experience is acting as champion for performance and with help from his product and engineering peers, crafted a strategy to rip the benefits of FAST web performance. They together kicked off a number of major initiatives, all targeted at faster performance. These initiatives include: Revamped NFR (Non-functional Requirement) process to address performance and scalability requirement during the design phase. Continuous testing on pre-production using a Gatling based framework to identify speed degradation during sprints. Back-end optimization via profiling (using YourKit) to identify CPU and Memory inefficiencies (aka hotspots). Implemented the top 10 front-end optimization industry best practices. Benchmarked CDNs providers to find the best suited for Shutterfly performance requirements. Leveraged AppDynamics to understand, troubleshoot and optimize key business transactions. Implemented a RUM solution (SOASTA mPulse) to provide real-time visibility into each customer experience. Implemented Synthetic Monitoring (NeuStar) for continuous testing and performance analysis of key user journeys. Implemented a comprehensive Production Load and Performance program leveraging SOASTA CloudTest to prepare for Q4 peak days.
![class ProjectServicePatchSimulation extends Simulation {
val users = Configuration.getNumUsers(100)
val ramp = Configuration.getRampDuration(60 seconds)
val simulationDuration = Configuration.getDuration(StandardDurations.EXTRA_LONG)
val httpConf = newProjectServicesConfig(Configuration.ssapiProjectService.httpsUrl)
val numRepeats = 500;
val projectPatchScenario = scenario(this.getClass.getSimpleName)
.during(simulationDuration) {
exitBlockOnFail {
exec(clearCookies)
.exec{session=>
println("Simualtion duration: " + simulationDuration)
session
}
.exec(NonVisualAuthenticationService.signInUniqueUser)
.exec(ProjectService.createProject("photobook"))
.repeat(numRepeats) {
exec(ProjectService.patchProject("photobook", "${userid}",
"${photobook_project_guid}"))
}
.exec(ProjectService.createProject("calendar"))
.repeat(numRepeats) {
exec(ProjectService.patchProject("calendar", "${userid}", "${calendar_project_guid}"))
}
.exec(signOut)
}
}
setUp(projectPatchScenario.inject(rampUsers(users) over (ramp)))
.protocols(httpConf)
.assertions(simulationAssertions)
private def simulationAssertions: List[Assertion] = {
val baseAsserts = List(
details(ProjectService.REQUEST_PATCH_PROJECT + "-
calendar").failedRequests.percent.lessThan(1),
details(ProjectService.REQUEST_PATCH_PROJECT + "-
photobook").failedRequests.percent.lessThan(1),
details(ProjectService.REQUEST_PATCH_PROJECT + "-
calendar").responseTime.percentile3.lessThan(12000),
details(ProjectService.REQUEST_PATCH_PROJECT + "-
photobook").responseTime.percentile3.lessThan(1411),
details(ProjectService.REQUEST_PATCH_PROJECT + "-
calendar").requestsPerSec.greaterThan(10),
details(ProjectService.REQUEST_PATCH_PROJECT + "-
photobook").requestsPerSec.greaterThan(30)
)
baseAsserts
}
AUTOMATED LOAD AND PERFORMANCE](https://image.slidesharecdn.com/perfops-170707182334/85/The-PerfOps-revolution-Shutterfly-s-journey-to-faster-performance-13-320.jpg)
The PerfOps revolution - Shutterfly's journey to faster performance
- 3. 3
- 4. 4 WHY?
- 5. Source: Forrester Research CUSTOMER SATISFACTION
- 6. Source: Forrester Research Rebuilding Pinterest pages for performance resulted in a 40% decrease in wait time, a 15% increase in SEO traffic and a 15% increase in conversion rate to signup.
- 7. Source: Forrester Research BBC has seen that they lose an additional 10% of users for every additional second it takes for their site to load
- 8. Source: Forrester Research Staples reduced median homepage load time by 1 second and reduced load time for the 98th percentile by 6 seconds. As a result, they saw a 10% increase in their conversion rate.
- 9. Source: Forrester Research MORE STATS https://wpostats.com/
- 11. 11 Synthetic Monitoring Real User Monitoring NFRs Back-End Optimization Performance Budget Automated Load Test Automated WebPageTestBack-End Monitoring Front-End Optimization Production Load Test
- 13. class ProjectServicePatchSimulation extends Simulation { val users = Configuration.getNumUsers(100) val ramp = Configuration.getRampDuration(60 seconds) val simulationDuration = Configuration.getDuration(StandardDurations.EXTRA_LONG) val httpConf = newProjectServicesConfig(Configuration.ssapiProjectService.httpsUrl) val numRepeats = 500; val projectPatchScenario = scenario(this.getClass.getSimpleName) .during(simulationDuration) { exitBlockOnFail { exec(clearCookies) .exec{session=> println("Simualtion duration: " + simulationDuration) session } .exec(NonVisualAuthenticationService.signInUniqueUser) .exec(ProjectService.createProject("photobook")) .repeat(numRepeats) { exec(ProjectService.patchProject("photobook", "${userid}", "${photobook_project_guid}")) } .exec(ProjectService.createProject("calendar")) .repeat(numRepeats) { exec(ProjectService.patchProject("calendar", "${userid}", "${calendar_project_guid}")) } .exec(signOut) } } setUp(projectPatchScenario.inject(rampUsers(users) over (ramp))) .protocols(httpConf) .assertions(simulationAssertions) private def simulationAssertions: List[Assertion] = { val baseAsserts = List( details(ProjectService.REQUEST_PATCH_PROJECT + "- calendar").failedRequests.percent.lessThan(1), details(ProjectService.REQUEST_PATCH_PROJECT + "- photobook").failedRequests.percent.lessThan(1), details(ProjectService.REQUEST_PATCH_PROJECT + "- calendar").responseTime.percentile3.lessThan(12000), details(ProjectService.REQUEST_PATCH_PROJECT + "- photobook").responseTime.percentile3.lessThan(1411), details(ProjectService.REQUEST_PATCH_PROJECT + "- calendar").requestsPerSec.greaterThan(10), details(ProjectService.REQUEST_PATCH_PROJECT + "- photobook").requestsPerSec.greaterThan(30) ) baseAsserts } AUTOMATED LOAD AND PERFORMANCE
- 14. WEB PERFORMANCE BEST PRACTICES
- 20. BACK-END OPTIMIZATION During non-peak 1. P80 - Backend time for this page is around 1.5Secs (from mpulse) 2. Around 30 slow transactions every 10 mins (from appD) 3. Around 4 very slow transactions ( > 900ms) every 10 mins SAMPLE: ONE SFLY CRITICAL PAGES
- 24. PRODUCTION LOAD AND PERFORMANCESOASTA CLOUDTEST
- 25. PRODUCTION LOAD AND PERFORMANCESOASTA CLOUDTEST
- 26. Q4 - GAME ON!
- 27. Q4 - GAME ON!