by Mike Ricotta - May 9, 2017
U.S. businesses are breaking records in digital ad spend in 2017, on pace for $77 billion nationwide (according to eMarkerter.com data). Fortune magazine calculates Amazon’s digital ad spend at 91% of their marketing budget in 2015, following Etsy’s 94% and beating Apple’s 85%. While this is great news for digital media agencies, avoiding dead ad spend is proving costly to agencies who subsidize the costs of full-time engineers, due to website downtime. How much, you might ask? Let’s take a look…
According to cloudendure.com’s quarterly surveys of the top 100,000 most visited websites in the U.S., between 86% and 89% experienced website downtime troubles. As of 2015, Quora.com estimates the US hosts 43% of the world’s websites, which according to internetlivestats.com, is a total of 1.1 billion. Although cloudendure.com does not specify if the top 100k US visited sites are US owned, we can safely assume our 100k is between 10% and 21% of the total, which is a healthy sample size.
When reviewing their first 3 quarters, websites short of 99.99% uptime totaled 2.52 million hours of website downtime. The 3 sampled reports are available here: Q1, Q2, Q3. Based on eMarketer’s 2017 forecasts and cloudendure.com’s downtime figures, we estimate between $22 and $38 million in digital ad spend are promoting dead websites during average traffic, among the U.S’s top tech budgets. What becomes more troublesome about these cost estimates, however, is that ad campaigns are typically running during periods of high traffic and guess what high traffic can result in… you guessed it; website downtime.
According to a 2013 report by LoadStorm, 58% of website owners admit overestimating their hosting capabilities (often when it’s too late). A total 68% of polled administrators reported performance and stability problems, citing the primary driving cause of website downtime as caused by a lack of capacity to support their traffic.
When combining the multiple-choice percentages, only 35/193 responses (18%) cited bugs as the cause. Consider, then, that our estimated $22-$38 million is dramatically underestimated when considering that website downtime occurs during periods of highest ad-spend. That also doesn’t account for missed opportunities; Rigor.com reports that during a 2013 downtime period, Target.com missed out on $464k in revenue during just 2.5 hours of downtime.
When factoring leads from organic search and the average business’s (lower) budgets, website downtime has a significant impact upon cost per acquisition (CPA) and other KPI’s that agencies’ rely upon to demonstrate their value in retaining contracts. How much, you might ask?
In 2014, SearchEngineLand.com (SEL) published an article on the top 7 conversion rate truths, analyzing conversion rates of median, top 25%, and top 10% converting web presences, based on the page’s industry (eCommerce performed lowest). The general trend indicates a roughly 2x relationship between top 25% converting sites and the median, which gives us a relative benchmark for value-based KPI’s. In order to do an analysis of how KPI’s are impacted by website downtime, we’ll use this data, to set up an example.
Joe owns a website called JoesTShirts.com which drives 10k visits/mo. Following SEL’s averages, JoesTShirts.com will convert 184 sales (1.84%) and after CRO improvements, JoesTShirts.com breaks into the top 25th percentile threshold, at 371 sales. That’s pretty significant, so how much is driven by targeted campaigns (ignoring content and social contributions to organic search)?
A global 2016 KPI analysis, written by Wolfgang Digital (sourced through Inc.com), indicates roughly 62% of website traffic is organic or direct, versus 26% in paid ads and the remainder in additional targeted marketing (email, social media, etc). The same analysis found almost identical performance between organic and targeted (paid) traffic driving sales (61% of revenue was organic or direct). This means that Joe’s targeted campaigns are netting 71 additional sales in its rise to the top 25%. If performance between organic and targeted traffic is roughly the same (+1%), an agency who can keep JoesTShirts.com above that +1% (~2 extra sales) while remaining at or below regular ad spend and service fees, can support their value add. This is a nice low bar for Joe’s agency but leaves very little room for error. A falling CPV is surmountable, provided that traffic remains high, relative to lower costs, and that’s where CPA comes into play.
A WordStream.com analysis identifies that Google’s two ad networks (Search AdWords and Google Display Network / GDN) average ~$60 cost per action. At the industry level and the network level, the numbers vary dramatically. Using this same chart, JoesTShirts.com as an eCommerce store, can expect an average $30.12 CPA through the display network and $46.07 CPA. Provided Joe’s agency can remain within range, they can remain in good standing and balance out underperforming CPV’s. Now, what happens when website downtime is increased?
In 2016 JoesTShirts.com makes 4,451 sales as a top 25th percentile converting site (traffic 10k/mo). Joe is paying a $35 CPA on ad-spend (~1,157 sales), plus $5k/mo in agency service fees for other targeted campaigns (social, email), totalling $100.5k per annum for 38% of sales. Joe also maintains a 99.9% uptime. In 2017, JoesTShirts.com drops to 99% uptime, due to server upgrades. In a data vacuum, the 0.9% decrease in up-time drives down CPV for both organic/direct and targeted traffic by 0.9%, resulting in 41 sales lost versus the prior year. The 0.9% of missed traffic accounts for additional 3.285 days, $540 of Joe’s agency service fees, and 1,080 missed visitors of which Joe paid ~$373 to convert via ads (total loss: $913 + 41 missed sales). This drives Joe’s paid ads CPA up slightly, to $35.32. This isn’t significant but since we don’t live in a vacuum, what does it actually look like?
In our above example of Target.com, we discussed a 2.5 hour website downtime period in 2013. In 2015, Target.com also crashed during Cyber Monday, citing the cause as “high traffic“. While Cyber Monday is an extreme example, trends in traffic by day and time do exist and marketers leverage these to drive high traffic at those dates/times. Many sources, we’ve found, identify highest sales traffic around lunch time, and right before bed. Traffic during rush hours and early morning hours were low in comparison. Other sources reference earlier weekdays as higher sales days, as well (references: quora.com, webinterpret.com, remarkety.com).
With ~18% of total sales on Mondays against an average 14.3% for other days, Monday crashes account for a 26% increase over the average. Joe’s total 3.65 days of website downtime stretching Monday through mid-Thursday would account for roughly 58% of weekly revenue, which equates more closely to 4.06 days total, or 98.89% up-time. Additionally, Joe’s agency targets ad spend on those days, thus resulting in a lower net targeted CPV as ad spend is not running on weekends where 100% of conversions are organic. Most website downtime, however, result over several separate events.
Let’s suppose 90% of daily traffic occurs during the 19 (79.17%) waking hours (considering timezone differences) and 3 peak hours account for 2x increases of that traffic. Peak hours would then account for 25.72% of daily traffic, or 2.07x that of the average hourly traffic. When occurring on a Monday (1.26x) during peak hours (2.07x), downtime due to high traffic has an impact 2.61x higher than the average traffic period, which if a regular occurrence, could extend your 3.65 days of downtime to an effective 9.53 days of average traffic. When factoring in seasonal traffic, the situation only gets worse. Now consider the most significant factor in demonstrating your value: you’re calculating CPA for the last year, what happens if you’re reporting CPA monthly?
You can probably guess that your range of 3.65 to 9.53 days worth of traffic has an abysmal impact on a monthly CPA (that’s 88% to 68% up-time). Even when distributed across a 3 months’ of events, your CPA increase is probably still 4x (ie. Joe’s paying ~$39), which cuts into margins big time. With that being well above the industry average $30 for GDN ads, the lost ad spend, missed opportunity costs, and under-performing CPA is likely enough to end the relationship.
As a result, many agencies are left covering the costs of systems administrators, to ensure continued up-time for longer lasting relationships. So, how much does it cost to prevent this?
According to 2016 data provided by the 2016 US bureau of labor, US Network and Computer Systems Administrators accounted for 376k jobs in 2016, which when broken down, doesn’t relate strictly to server management, so we poured through the data further. In reviewing the breakdown, we narrowed it down to web developers, network and computer systems administrators, database and systems administrators, and DBA’s, for a total $22.93 billion. This also does not include the $14.38b that account for “software developers” who may also work on the endless number of service-side supporting systems (ie. RedHat, Microsoft, Apple, Google, IBM, Oracle, CloudFlare, Cisco, Rackspace, Apache, etc etc etc).
With a total $37.31b in US salaries supporting operations software, networks, servers, and applications, we did some further analysis to back up the data. Data from the US News, yielded different break-downs with a 27% increase in web dev salaries and a 38% decrease in DBA salaries. Ultimately, however, the sum was in the ballpark. Given the relative consistency of sum, we attempted to analyze a sample of how many are working at agencies.
The census bureau indicates that 1 in 7 web developers are self-employed, with the cumulative break-down of agencies or supporting service providers at 25% of total hires. We also found that systems administrators are employed by at least 34% of similar employers. Without more accurate data, we feel it’s safe to assume that web-driven businesses are paying out, in the area of $12+ billion in US jobs in support of web activity. That, of course, does not include the significant amount of overseas/outsourcing costs, which is another figure we hope to investigate in later research.
While all of these jobs are not necessarily devoted to up-time, industry professionals will agree that up-time response is often an expected responsibility, even when production remains a primary goal. In fact, many agencies operating 20-100 servers often have 1-3 full-time systems administrators. When considering a 24/7 coverage period (168 hours), 4 technicians (US average salary is $91k+) would have to be on call 40+ hours each, to cover all shifts. That’s $364,000 in salaries on US average (good luck if you’re in NYC!).
Given the average cloud hosting and support margins, 200 servers wouldn’t begin to cover that cost. When polled, we found that most agencies were netting between 100 and 200/server before labor. Even top service and support providers like AWS, Google, IBM, and Azure are charging a base of 100 and 200 for basic level tech support per account (not per server). We performed a breakdown of these services, here. Supposing an average net of $150/mo per server, a 24/7 coverage team would require an agency to service 202+ servers (1 person at a time), in order to break even, which is a fairly daunting figure. When factoring in supporting service costs, benefits, and other ancillary employers, it’s an effectively losing battle.
In our experience, many agencies are resorting to cutting corners on these costs, in order to stay in the black. One method is to keep producers (ie. web developers) on-call. This often means overtime costs, added stress, slower response times, account managers remaining on-call, and ultimately higher website downtime figures. In the end, there’s little to remain positive about and undoubtedly, there must be a solution for agencies to retain happy clients, reduce dead ad spend, and cut the costs of up-time maintenance.
With Arcane’s Partner Network (APN), these agencies benefit from wholesale rates, reducing their costs for full-time engineers, and allowing them to focus on marketing without concern for declining success metrics. As an added benefit, APN also offers qualified leads to partners, in an effort to give Arcane clients the best services and maintain long-term contracts with mutual clients.
Of course, we want you to also consider alternative options, if Arcane is not best suited for you, so we strongly recommend that you consider reading our support reviews of top cloud service providers, available on our blog, here.
Membership to the Partner Network is free and requires no long term commitment. It includes discounted rates on 24/7 Cloud Support, referrals for design, architecture, build and marketing, free SLA upgrades, DevOps licensing and customer loyalty discounts.
Contact us to learn more!