Terrible Timing, Netflix!

It's been a weird week for Netflix (Nasdaq: NFLX  ) .

The movie maven reported a massive quarter on Wednesday with 1.9 million new subscribers, shooting the stock up to fresh all-time highs on Thursday. In the process, Netflix proved pretty much all of my wild-eyed, crazy assumptions about the company entirely correct, reminding us all why the stock truly belongs in every portfolio.

So far, so good. This was a chance for Netflix to bask in the glory of doing everything right. Did CEO Reed Hastings have the Reality Distortion Field on loan from Steve Jobs?

Oops!
And then the problem reports started pouring in to Twitter and Facebook, along with heavy search volumes for some gloomy query terms on the major search engines: Netflix was down.

Starting before lunch, Pacific time, and lasting well into the afternoon, the Netflix website that is so central to everything the company does was largely unavailable. Trying to access your account for a lunchtime movie stream or to fiddle with your DVD queue proved unfruitful, as you'd either get no response at all or, at best, a cryptic error message.

Service has since been restored, and everything is back to normal as far as I can tell. Netflix spokespeople say that it was an internal problem and no hack-attack from the outside, implying that your data is safe. But what kind of "internal problem" could cause this kind of extended outage for hours on end, in the middle of a regular Thursday?

The quick knee-jerk guess would be that exploding interest in Netflix after that fine quarterly report simply multiplied the site's traffic manifold and overwhelmed the servers. I don't buy that for several reasons:

  • Peak hours for Netflix are roughly around the cherished primetime slot on network television, from about 8 p.m. to 10 p.m. in each time zone. If anything, the site should have been unavailable on Wednesday night if high traffic was the problem.
  • Netflix routes its movie streams to consumers by way of three leading network delivery specialists: Akamai Technologies (Nasdaq: AKAM  ) , Level 3 Communications (Nasdaq: LVLT  ) , and Limelight Networks (Nasdaq: LLNW  ) . Moreover, the site infrastructure is largely hosted by the immense server farms of Amazon.com (Nasdaq: AMZN  ) , thanks to a surprise deal signed last spring. I find it very unlikely that all of these tier-one services would become overwhelmed at the same time. A live webcast of the Pope having dinner and drinks with the Dalai Lama over world peace and a cure for cancer probably couldn't do that.

How to lose friends and alienate customers
The other obvious explanation would be planned network maintenance going awry. These things do happen despite the best-laid plans of mice and men, but it would be irresponsible for Netflix to schedule any kind of risky maintenance during business hours. As a Unix support dude for railroading giant CSX in an earlier life, I often cursed having to mosey down to the data center after midnight on Saturdays, but that was the safest spot to have a network outage without stopping revenue-generating trains so that's what we did.

If Netflix considers nine-to-five workdays "safe" for changes, upgrades, and maintenance, I think they're forgetting that many a worker wants to maintain their Netflix queues from that handy, networked workstation sitting on their office desks. And I'm pretty sure that Netflix as a company is smarter than that, especially since Reed Hastings built the company from an engineering background and still runs it with that mind-set.

What's the story?
I'd love to hear an official explanation from Netflix about this outage. Company representatives haven't exactly been forthcoming about it, and the @netflix Twitter feed went right from noting that everything was up and running to plugging the streaming availability of past Saturday Night Live seasons, all of two minutes later.

In the past, slip-ups like delayed DVD shipments have inspired Netflix to explain the issues in detail, apologize publicly, and apply automatic discounts to affected accounts. Twitter and Facebook might get away with the occasional Fail Whale because they're free services, but you expect more from a company that's collecting money in return for a reliable entertainment experience.

If the digital entertainment era just kicked off with the Netflix report, it's not off to a good start. Customer care is paramount to any service business, and Netflix can't afford a whole lot of kinks like this in the system.

Akamai is a Motley Fool Rule Breakers pick. Amazon and Netflix are Motley Fool Stock Advisor recommendations. The Fool owns shares of Apple. Try any of our Foolish newsletter services free for 30 days.

Fool contributor Anders Bylund holds no position in any of the companies discussed here. We Fools may not all hold the same opinions, but we all believe that considering a diverse range of insights makes us better investors.  You can check out Anders' holdings and a concise bio if you like, and The Motley Fool is investors writing for investors.


Read/Post Comments (7) | Recommend This Article (10)

Comments from our Foolish Readers

Help us keep this a respectfully Foolish area! This is a place for our readers to discuss, debate, and learn more about the Foolish investing topic you read about above. Help us keep it clean and safe. If you believe a comment is abusive or otherwise violates our Fool's Rules, please report it via the Report this Comment Report this Comment icon found on every comment.

  • Report this Comment On October 22, 2010, at 6:08 PM, MattMattMatt wrote:

    "But what kind of "internal problem" could cause this kind of extended outage for hours on end, in the middle of a regular Thursday?"

    As network architect of a major hosting provider, I got quite a laugh from this sentence. When do you think outages happen?

    As you mention, change related problems are more likely to happen at late night off-peak hours because most "dangerous" change to major web sites are (or should be) done during off-peak maintenance windows. However, even this is complicated. Sometimes a simple change that is normally done outside a maintenance window (such as adding a new movie to the database) has unintended consequences.

    Hardware or software failures are likely to happen at any time - even on a "regular Thursday". Shouldn't they have redundancy? Sure. However, redundancy/failover is typically the least reliable component of any system. Sites like Netflix rely heavily on databases, which are often one of the most complex components to try to try to make both fast and reliable.

    "I find it very unlikely that all of these tier-one services would become overwhelmed at the same time."

    You misunderstand. These are three dependencies. Typically sites fail if ANY of the component services become overwhelmed or fail. The "origin server(s)" must be up and speedy. The network between them and the CDN must be up and speedy. The CDN servers must be up and speedy. The network from the CDN to the user must be up and speedy. Failure of any of those components fails the entire site.

    In the end, I think we can all agree that this wasn't desirable and should be as rare as possible, but it wasn't peak time. As long as this isn't a weekly/monthly event, I'd say it is well within what I'd expect from big and complex Internet sites.

    In the end, reliability is often simply a business decision. Let's imagine Netflix was "3 9's" reliable (99.9%, or 8.8 hours downtime per year). Now, imagine that they could upgrade to 4 9's reliable (52.6 minutes downtime per year) but it would require doubling what they charge customers. Would you consider this difference in reliability important enough to justify the cost?

    I used to have a boss that liked to prorate downtime to demonstrate how little downtime there was compared to how little customers pay. I pay Netflix $9.73 per month for "all you can eat" streaming. I've read numerous articles and there seem to be many random numbers thrown out about how long the outage was, but just for the sake of argument let's say it was 12 hours. That means that the Neflix outage has deprived me of $.16 worth of service this month. Personally, I didn't even notice.

    On the other hand, I've made tens of thousands of dollars on Netflix stock.

  • Report this Comment On October 22, 2010, at 7:02 PM, morrisjd wrote:

    The idea that it is a fact of life that computers crash is a myth. This misconception has happened as businesses migrated from the secure super reliable monolothic mainframe environment to rooms full of PC based servers, where the quality and reliability of the server hardware is not of the same caliber as mainframe hardware they replaced. Also these server based computer conglomerates led to numerous network failures, most of them mysterious in nature as the network people would not reveal what (screwup) happened. Since nobody was in a position to question them, IT management just accepted this as the new reality. So now we take these failures as a routine fact of life. Remember the numerous failures of Blackberry's backoffice systems?

    During the heyday of mainframes (1980 to about 1990) reliability was so good, that the computer systems never went down unintentionally. (And these mainframes handled thousands of transactions per second which is amazing considering the time frame when they were dominant). I can remember months on end where these systems were up and running 24x7. Even changes in hardware could be done such that system passthroughs were done transparently without any loss in service. We just assumed that our mainframe systems were so reliable that the only thing that would cause a crash was some sort of act of God, such as an earthquake or flood.

    People today accept the erroneous fact that computers crash. It's too bad, because we could obtain exceptional reliability if business and IT management would not make this false assumption. After all, we (I) lived through an era when such flawless reliability was the norm rather than the exception.

    BTW. Part of this mainframe reliability stems from the concept of the Virtual Machine which was developed in the late 1960's by IBM corp and was integrated out into their mainframe hardware and operating systems by the 1970s. I am stunned it has taken 30 years for this important concept to finally be adopted by the server world in the form of software offerings such as VMWare.

  • Report this Comment On October 22, 2010, at 8:57 PM, afamiii wrote:

    My experience is that most network outages and unscheduled system downtime is the result of human error on the part of the operator (and less frequently on the part of the software vendor.)

    You just can't get the staff.

  • Report this Comment On October 22, 2010, at 9:27 PM, Global10 wrote:

    NETFLIX went down!!!! When did that happen??? OMG!!!!

  • Report this Comment On October 24, 2010, at 10:05 AM, feldmail wrote:

    I received the following email from Netflix on Friday:

    "Yesterday, you may have had trouble instantly watching TV episodes or movies due to technical issues.

    We are sorry for the inconvenience this may have caused. If you attempted and were unable to instantly watch TV episodes or movies yesterday, click on this account specific link in the next 7 days to apply your 3% credit to your next billing statement. Credit can only be applied once. "

  • Report this Comment On October 29, 2010, at 2:03 PM, dlchase24 wrote:

    As feldmail just mentioned, there was an email circulated offering a credit, so they have taken steps to make amend, but I am curious what caused the problem.

    Also, after reading the article and the comment about CSX maintenance after midnight, that actually makes me wonder what would be the most advantageous time for Netflix to do system maintenance? As you note, their peak hours are a late evening 8pm to 10pm. So if you figure that really starts at 8pm Eastern and goes til 10pm Pacific, they could be waiting until 3am Eastern, 12mid Pacific to do updates.

    It wouldn't surprise me if a majority of their subscribers actually work an 8-5 workday (as I do) which would preclude them from using Netflix during the day. Could that not actually be the best time to perform maintenance?

  • Report this Comment On October 29, 2010, at 2:55 PM, mikenpdx wrote:

    I think a mountain is being made out of a molehill here frankly. As far as this being an incident that will cause a loss of Netflix customers -- thats a little far-fetched. These things happen. Sometimes your cable goes out, sometimes the power goes out, sometimes you can't make a call from your smartphone because of poor reception. If it becomes a regular occurence then sure that would be annoying and would open the door wide open to switch to a competitor. Only at this time, Netflix doesn't have a direct competitor. Not that offers an identical service. So big deal. Most major websites have been down at one point or another, for one reason or another. There was no foul play here so let's move on.

Add your comment.

Sponsored Links

Leaked: Apple's Next Smart Device
(Warning, it may shock you)
The secret is out... experts are predicting 458 million of these types of devices will be sold per year. 1 hyper-growth company stands to rake in maximum profit - and it's NOT Apple. Show me Apple's new smart gizmo!

DocumentId: 1341775, ~/Articles/ArticleHandler.aspx, 10/1/2014 10:16:58 AM

Report This Comment

Use this area to report a comment that you believe is in violation of the community guidelines. Our team will review the entry and take any appropriate action.

Sending report...


Advertisement