Last week we saw the launch of the well-anticipated Disney+
streaming service. Many of the team here at BizStream were very excited to get our geek on with The Mandalorian. Unfortunately, when the service started, users were greeted with a number of issues.
How could this happen? Disney has been touting the service for a long time. Certainly, the Disney technical and development teams would have tested their service and anticipated these issues, right? The answer is, yes, they did.
Some saw the issues that Disney+ had on day one as a failure
. Not me. With a day one launch of over 10 million users
, it’s gotta be tough to test a system that’s going to get hit that hard on its initial day. Disney probably knew what they were doing and made a financial decision not to build out a system to handle the load, but rather fix the issues on the fly
. It turns out the problems were not with the streaming technology (like Akamai Technologies), but rather the authentication system
which was hammered by that initial influx of signups and users wanting to get their Star Wars, Marvel, and classic Disney fix.
While I was patiently waiting for the issues to get fixed with Disney+ (so I could check out The Mandalorian and revisit the Avengers team in Endgame), I got thinking about testing the sites we work on. Over the past years, we’ve been asked to test our client’s sites before launch and even prior to big sales weekends like Thanksgiving/Cyber Monday and even Valentine’s Day.
There are five types of performance tests we help our clients with - Baseline, stress, spike, endurance, and failover tests.
The baseline test is what we use to determine the performance of a site when it’s under an average website load. The test is determined by looking at historical analytics and determining what the average customer path is and how many pageviews/orders the site typically gets.
Running a site at 150% to 200% of the baseline load will definitely put a site under stress. This test tells us how the site will perform under a heavy load for an extended period of time.
Simulating events like Black Friday, Valentine’s Day, and flash promotions are what Spike Tests are all about. We recommend running short bursts on the site at 200%+ of the baseline load to simulate these events.
These tests are a combination of a stress test with a little extra. Running an endurance test, we’d use an average load but include scheduled tasks, database backups, and other system events that test the site’s performance over a longer period of time.
What happens when a system fails? Will your site go down or will it gracefully failover to the backup systems in place. These tests help to determine how the site will perform when something goes wrong.
Some of the tools we use
Google PageSpeed Insights
A pretty basic tool provided by Google, their PageSpeed Insights
, will show a number of different stats from Google’s perspective. We’ll use PageSpeed Insights as a 3rd party way to hit a site while it’s under a test.
To do the performance testing, our go-to tool is JMeter
. In short, the tool allows you to set up concurrent users and testing paths for those users and hit a site. The tool also records performance and site response as the test is running.
Stepping up a notch for a more enterprise-level testing and cloud-based testing is BlazeMeter
. BlazeMeter offers a plethora of tools (Jmeter in the cloud being one) for site testing.
Of course, there are more tools available and other cloud-based tools for testing, all with various complexity and pricetags.
What do we look for in performance tests?
Each test provides us an insight into how the site will perform under different circumstances and even allows us to see a glimpse of potential fail points. Overall, we’re looking to be sure that under each test, the site performance stays at or very near the baseline we’ve established.
For our eCommerce clients, page load time is paramount. Studies have shown that for eCommerce sites, 40% of users will abandon a webpage if it takes more than 3 seconds to load
. Not only will shoppers abandon a site, the ones who do stay and have a dissatisfied experience are less likely to buy from the same site again
Time is definitely money. Walmart saw a sharp decline in conversion rate as site load times increased 1 to 4 seconds. Walmart isn’t the only organization that saw a decrease in conversions with an increase in load time.
Testing sites let us know where we stand in that breakdown of lost conversions and provides a guide to how much to invest to keep performance levels at the baseline or how much to invest to decrease that page load time and increase conversions.
Back to Disney+
It’s very interesting to me that Disney did start things outright with their Disney+ platform. They focused on registration and payment - fast and easy. They had no issues taking credit card numbers and processing payments. The Mouse focused on getting the $ first and seeing how the platform would perform then fixed things on the fly as issues occurred. There’s a lesson there.
For now, it’s time for me to enjoy The Mandalorian