Nintendo Says Network Downtime Caused by One-Time Surge

[MarlaDesat]

Nintendo Says Network Downtime Caused by One-Time Surge

Nintendo apologized on Facebook for the service outages on the Nintendo Network and eShop, pointing to many new accounts as the cause.

The power of holiday demand has taken a heavy toll on the Nintendo Network and eShop. Players have been reporting issues since Christmas, ranging from an inability to download content from the eShop to not being able to set up Nintendo Network IDs. Nintendo first delayed the release of the took to Facebook [http://www.escapistmagazine.com/news/view/130840-Nintendo-Removes-Pokemon-Bank-From-Japan-3DS-eShop] to explain its holiday troubles:

Thank you very much for your patience while we perform maintenance on the Nintendo eShop and Nintendo Network. We sincerely appreciate your ongoing support. We wanted to share some details on what is causing this downtime, and update you on the current outage in service.

The initial outage was primarily caused by a one-time surge in demand over the recent holidays, compounded by the following:

1) An influx of new Nintendo Network accounts being created and accessing the Nintendo eShop.
2) Existing Nintendo 3DS account information being moved to new Nintendo Network accounts.

We are continuing to adjust our network to meet the needs of all our services and applications. To ensure our global audience has an expected level of consistency in service, we took the Nintendo eShop on Nintendo 3DS offline. We expect to have it back online no later than 4 a.m. ET on Dec. 29, but may have to perform more maintenance in the future.

We truly apologize for the inconsistencies in our online services. We are working hard around the clock to fully restore these functions, and will continue to keep you updated on our progress.

Hopefully, Nintendo won't need to take the services down again, but they haven't ruled out further maintenance. Nintendo reports the job listing for an IT architect [http://www.nintendo.com/consumer/info/en_na/online-service-status.jsp], presumably to help them avoid these same issues in the future.

Source: Nintendo on Facebook [https://www.facebook.com/Nintendo/posts/560706114013846]


Permalink

 

[Xman490]

The network still doesn't seem to work. It's a shame, since I've planned to get Super Mario 3D World from the eShop. Since I got my Wii U last night, I couldn't hold onto the servers long enough to even do a system update. Hopefully, traffic will not be so congested later this week.

 

[TimeLord]

eShop is working perfectly for me now so I'm just sitting here waiting for Pokemon Bank to be released to the UK

 

[Something Amyss]

The network still doesn't seem to work. It's a shame, since I've planned to get Super Mario 3D World from the eShop. Since I got my Wii U last night, I couldn't hold onto the servers long enough to even do a system update. Hopefully, traffic will not be so congested later this week.

I suspect it'll be fine in a day or two.

Which reminds me, I have some shopping to do!

 

[Fiz_The_Toaster]

eShop is working perfectly for me now so I'm just sitting here waiting for Pokemon Bank to be released to the UK

Me too, except I'm waiting for a US release.

I just barely got both Pokemon games and I was super bummed when they delayed it.

Hopefully they will release it soon since it was just an influx of new people coming on.

 

[mistwolf]

I admit, I can't imagine what was going through their heads to plan so many massive things so close together. Pokemon bank was guaranteed to be a huge resource hit. Christmas was guaranteed to be a huge resource hit. So let's do them both together! And, a week or three before that, let's start getting people to merge their accounts, which is hard on the servers...

 

[Evil Smurf]

I admit, I can't imagine what was going through their heads to plan so many massive things so close together. Pokemon bank was guaranteed to be a huge resource hit. Christmas was guaranteed to be a huge resource hit. So let's do them both together! And, a week or three before that, let's start getting people to merge their accounts, which is hard on the servers...

They wanted to get it all over at once?

eShop is working perfectly for me now so I'm just sitting here waiting for Pokemon Bank to be released to the UK

Me too, except I'm waiting for a US release.

I thought Pokemon Bank was up for everyone but the Australians, this is good, you're all waiting like me.

 

[Strazdas]

Nintendo lacks costumers for WiiU. More costumers arrive - reaction - shut down the networks. Great welcome guys.

 

[Shadow-Phoenix]

Nintendo lacks costumers for WiiU. More costumers arrive - reaction - shut down the networks. Great welcome guys.

Yeah, it's almost like they didn't expect that to happen, unless of course everyone at Nintendo is from the future or are machines.

I also remember not being able to gift my stored games on Christmas Day and half of Boxing day on Steam thanks to one five year old game going for free, that was really nice of them to ruin my one day to gift people on the day they were meant to be gifted and the money spent for just that day.

 

[KazeAizen]

I admit, I can't imagine what was going through their heads to plan so many massive things so close together. Pokemon bank was guaranteed to be a huge resource hit. Christmas was guaranteed to be a huge resource hit. So let's do them both together! And, a week or three before that, let's start getting people to merge their accounts, which is hard on the servers...

Well I take the strain as a good sign that they aren't out of the game like people are saying they are. However Nintendo has been the last one to do dedicated online shops and the like and it has never really been a priority for them. Therefore the inexperience with this kind of stuff helped caused their problems. Still I see them taking steps in the right direction and they'll learn from this. Its an inconvenience and its a very harsh growing pain for them. I apparently need to look at this Pokemon bank though. I know nothing about it.

 

[Dragonbums]

Nintendo lacks costumers for WiiU. More costumers arrive - reaction - shut down the networks. Great welcome guys.

It's not like the network was operable at all when it was online. I should know, I tried logging in several times on Christmas only to be told I had a network issues. So did it honestly make a damn bit of difference? Or would you honestly prefer them to keep an overstrained failing server online as opposed to taking it down and adding a lot more wiggle room to it?


This is also begetting the fact that the PSN was down, and Steam was down not only due to sales, but L4D an old as fuck game was completely free and everyone dogpiled on it. Basically Steams version of Pokemon Bank.

 

[Dragonbums]

I admit, I can't imagine what was going through their heads to plan so many massive things so close together. Pokemon bank was guaranteed to be a huge resource hit. Christmas was guaranteed to be a huge resource hit. So let's do them both together! And, a week or three before that, let's start getting people to merge their accounts, which is hard on the servers...

They wanted to get it all over at once?

eShop is working perfectly for me now so I'm just sitting here waiting for Pokemon Bank to be released to the UK

Me too, except I'm waiting for a US release.

I thought Pokemon Bank was up for everyone but the Australians, this is good, you're all waiting like me.

Nope. Only Japan got Pokebank and used it for a short amount of time. I live in NA and we were slated to get it on the 27th. But they delayed it.

 

[Covarr]

The initial outage was primarily caused by a one-time surge in demand over the recent holidays

Aww, have some faith in yourself, Nintendo. If you do as good a job at marketing over the coming year as you should've been all along, this same thing will and should happen next Christmas.

P.S. Thanks

 

[Strazdas]

Nintendo lacks costumers for WiiU. More costumers arrive - reaction - shut down the networks. Great welcome guys.

It's not like the network was operable at all when it was online. I should know, I tried logging in several times on Christmas only to be told I had a network issues. So did it honestly make a damn bit of difference? Or would you honestly prefer them to keep an overstrained failing server online as opposed to taking it down and adding a lot more wiggle room to it?


This is also begetting the fact that the PSN was down, and Steam was down not only due to sales, but L4D an old as fuck game was completely free and everyone dogpiled on it. Basically Steams version of Pokemon Bank.

What was the reason it was inoperable? according to nintendo - too many users. so they want more users but cant hnadle them, so they boot everyone. truly best way to keep somone that just bought their console and now cant do anything simply because they were new people that overloaded the server.
adding "A lot more wiggle room" should have been done months ago. servers should never run at barely handling the clients and should always have some wiggle room, because user surges happen and not being prepared for that is stupid.

PSN is down due to bugs introduced by last few updates - completely different problem and one that is far more reasonable cause of shut down and fix reaction.
Steam was "down" twice this year as far as im aware, for couple hours. And while granted this crash (not the latest one but the oen before) did prevent me from playing a game, i just went to play another one that doesnt need steam. I missed the L4D2 rush, but if i remember correctly it was the second one that was going for free. Its only 4 years old and i wouldnt call it old, but i guess majority of gamers odnt even look at a game if its 2 years old.

P.S. for some reason your avatar doesnt work, should probably check if the host isnt down.

 

[Dragonbums]

What was the reason it was inoperable? according to nintendo - too many users. so they want more users but cant hnadle them, so they boot everyone.

Too many users due to the holiday rush. You know, something that happens to every network that's bombarded with thousands of systems at once at a single or near single time. That's also ignoring the fact that a lot of current users bombarded the e-shop to buy digital gifts from that e-shop cards discount, and those finally deciding it was time to set up their NNID

truly best way to keep somone that just bought their console and now cant do anything simply because they were new people that overloaded the server.

So does it honestly matter that they took it down for maintenance? It wasn't working online, so it wouldn't make a lick of difference if it was offline as well. I'm also sure that they were still able to play games on the console. They just weren't able to access the e-shop for other purchases. For instance, the e-shop was down, but I was still able to use the Miiverse just fine on my 3DS.

adding "A lot more wiggle room" should have been done months ago. servers should never run at barely handling the clients and should always have some wiggle room, because user surges happen and not being prepared for that is stupid.

So basically all companies then? The same could also be said for Valve. They have millions of users using their client everyday and everytime a Steam sales crops up, they always have horrendously slow, or completely unworking servers. There is only so much wiggle room a company can plan for. You cannot assume that Nintendo didn't do anything. For all we are concerned they did, and it still got bloated far beyond their expectations.

PSN is down due to bugs introduced by last few updates - completely different problem and one that is far more reasonable cause of shut down and fix reaction.

So your seriously telling me that the PSN being down had absolutely NOTHING to do with the holiday rush of people buying and updating their new PS4 consoles on the same day? Are you really saying that? Because I doubt a few updates could singlehandedly cause servers to crash. Which, if that is the case, is a lot more alarming and shameful on Sony's part than Nintendo having full servers.

Steam was "down" twice this year as far as im aware, for couple hours. And while granted this crash (not the latest one but the oen before) did prevent me from playing a game, i just went to play another one that doesnt need steam.

And Nintendo's servers crashed a total of zero times this entire year up until now. Steam's severs always crash when a sale comes along, and it doesn't seem to be changing or even remotely improving over the years. Yet it's acceptable because "everyone goes crazy over Steam sales" rarely do I see people pointing mad fingers at them calling them total clowns for not dedicated truck loads of more server space during highly popular sales to make things better.

The point being, calling Nintendo stupid and reactionary for taking down inoperable online servers to make more room for all the new users and fix the problem doesn't make any sense. Seeing as how other companies are more inclined to just leave their messed up servers online for the duration of the period as opposed to doing actual maintenance on it.

P.S. for some reason your avatar doesnt work, should probably check if the host isnt down.

I guess the Escapist has a thing against pictures coming from my flashdrive. Still trying to figure out why it isn't working.

 

[Strazdas]

Your right Nintendo should throw bucket loads of cash away and have 2000% more servers running than they could ever need. It's far more acceptable to put out a buggy online server infrastructure that breaks down than it is to put out one that has no bugs but due to congestion needed a quick upgrade right?

No, they should have 20-30% more power than their average load in order to work normally during rush times like this case.
Well, apparently Nintendo chose to put out a buggy one, so you answer that.

Too many users due to the holiday rush. You know, something that happens to every network that's bombarded with thousands of systems at once at a single or near single time. That's also ignoring the fact that a lot of current users bombarded the e-shop to buy digital gifts from that e-shop cards discount, and those finally deciding it was time to set up their NNID

Except, it only happened to nintendo.
Xbox live didnt budge. PSN has broken thier own network with updates and not holyday rush, and steam was working fine till free l4d2 showed up and even then it was down what, half an hour?
Youk now why? because other services were prepared. they knew its going to be a holyday rush. NIntendo on the other hand kept crying for more costumers whole year round and once they got them - they cant even handle that.

So does it honestly matter that they took it down for maintenance? It wasn't working online, so it wouldn't make a lick of difference if it was offline as well. I'm also sure that they were still able to play games on the console. They just weren't able to access the e-shop for other purchases. For instance, the e-shop was down, but I was still able to use the Miiverse just fine on my 3DS.

No, not really. What matters more is that they needed "maintenance" in the first place. The fact that it was so easy to break their servers as to have a few more users logging in shows of a problem. at worst the system should ahve booted some players and allowed some to go shopping, you know, use the avaialble pwoer for available consumers and only time out for the extra.
From what i hear the update server wasnt available either, and thanks to "lets release games and fix later" workings nowadays that mean that people could play games, but couldnt save (saves got corruped without update, ect). so while technicalyl yes you could play your games, if you downloaded and updated them before, that doesnt help much for all those people that actually wanted to buy those games.

So basically all companies then? The same could also be said for Valve. They have millions of users using their client everyday and everytime a Steam sales crops up, they always have horrendously slow, or completely unworking servers. There is only so much wiggle room a company can plan for. You cannot assume that Nintendo didn't do anything. For all we are concerned they did, and it still got bloated far beyond their expectations.

Except that Valve HAS more wiggle room, as evident by them handling sales expetionally well. And need i remind you of that 300 million investment MS made into online servers? Other companies ARE either already doing that or have done it.
The only time steam servers were unavailable was during the l4d2 rush and that one time it has actually crashed a month ago. Meanwhile i downloaded a game at over 5MB/s during the sale, so there really wasnt "horrendously slow" thing on their download servers.

So your seriously telling me that the PSN being down had absolutely NOTHING to do with the holiday rush of people buying and updating their new PS4 consoles on the same day? Are you really saying that? Because I doubt a few updates could singlehandedly cause servers to crash. Which, if that is the case, is a lot more alarming and shameful on Sony's part than Nintendo having full servers.

Yes. PSN was crashing and behaving badly before the rush. its behaviuor started after the introduction of recent updates to the system, so very cearly that is much more likely be the cause. Also PSN handled the day 1 sale of PS4s too, thought admittedly with some slowdown, but it was still usable. and thats as big of a rush as your going to get.
And yes, a few updates could easily cause servers to crash and burn. I mean, considering that SONY seems to do that at least 3 times a year this is almost expected now. they are really really bad at their patches for some reason.

And Nintendo's servers crashed a total of zero times this entire year up until now. Steam's severs always crash when a sale comes along, and it doesn't seem to be changing or even remotely improving over the years. Yet it's acceptable because "everyone goes crazy over Steam sales" rarely do I see people pointing mad fingers at them calling them total clowns for not dedicated truck loads of more server space during highly popular sales to make things better.

Nintendo server crashing during a sale wouldnt be so bad if it wasnt for the fact that nintendo has been begging us to come buy stuff and when we actually do - they cant handle it. the way they were talking felt liek they had prepared for ten time more costumers to be coming, but when a slight increase came they stopped working.
Steams servers do not "Always crash during sale". In fact steam is probably the most praised for not crashing during sales out of all digital platforms.

The point being, calling Nintendo stupid and reactionary for taking down inoperable online servers to make more room for all the new users and fix the problem doesn't make any sense. Seeing as how other companies are more inclined to just leave their messed up servers online for the duration of the period as opposed to doing actual maintenance on it.

Nintendo is not stupid for taking down inoperable server, they are stupid for making the server inoperable in the first place. And leaving the server up is actually better since then at least part of the costumers can use it instead of none. What "maintenance" are you going to do agaist a heavier user load other than making your servers more powerful? i highly doubt nintendo is rushing to optimize thier shop during the sale.

 

[Dragonbums]

-snip-

Honestly, it seems that you are complaining for the sake of complaining.

A server shouldn't needed to be taken down for maintenance? Since when? Servers have always been taken down for maintenance. Especially when they have issues.

You still haven't explained to me how PSN crashed ONLY because of updates, despite the fact that they were sold out everywhere, and as far as I'm concerned you are berating Nintendo for things that happen to every single damn company with an online infrastructure that gets bombarded all at once. Especially during the fucking holiday season.

It's cheap and honestly quite ridiculous. Most people aren't going to deny Nintendo needs to put more emphasis on online infrastructure, however your criticisms not only simply their thought process for their actions to an insulting degree, but it's highly inaccurate.

Nintendo servers were completely wasted due to holiday rush, Nintendo takes down inoperable servers for emergency maintenance.
You then accuse them of wanting new users then suddenly cutting them off like they are in on some sort of EA levels of stupid anti-comsumer scheme? Come on. What the heck is that?

I might as well say the same thing for Valve. Every year they have a Steam sale and every single time it completely wastes their severs or goes at snails' pace for some people. I guess that must means Valve do it on purpose to fuck with people.


As for XBL not having issues, I would actually contribute that to something worrying. That would mean not many people bought their new consoles to have an impact on their servers. I highly doubt it was because the server load was any good.

 

[Strazdas]

A server shouldn't needed to be taken down for maintenance? Since when? Servers have always been taken down for maintenance. Especially when they have issues.

A server shouldnt needed to be taken down for maintenance just because a few more people joined it. There should have been no issues with load to start with.

You still haven't explained to me how PSN crashed ONLY because of updates, despite the fact that they were sold out everywhere, and as far as I'm concerned you are berating Nintendo for things that happen to every single damn company with an online infrastructure that gets bombarded all at once. Especially during the fucking holiday season.

PS4 launched, PSN didnt crash. It was working well for everyone. PSN recieved an update, it slowed down to a crawl. It recieved another update, it was barely working and not for everyone. They took it down for maintenace. It is very easy to see how the update affected the network and elad to this,but you seem to be ignoring that.
Thats the point, it doesn't happen to every single company, because other companies have enough sense to be prepared.

Nintendo servers were completely wasted due to holiday rush, Nintendo takes down inoperable servers for emergency maintenance.
You then accuse them of wanting new users then suddenly cutting them off like they are in on some sort of EA levels of stupid anti-comsumer scheme? Come on. What the heck is that?

You know, now that you mention it this was quite simular to sim city launch problems or the AC2 (thought that maintenance was caused by DDOSers so i guess not their fault).

I might as well say the same thing for Valve. Every year they have a Steam sale and every single time it completely wastes their severs or goes at snails' pace for some people. I guess that must means Valve do it on purpose to fuck with people.

Except that it doesnt happen to Steam. It only happens when they give games away for free because you got millions of people downloading at once. sales - working as a charm, downloading at high speeds.

As for XBL not having issues, I would actually contribute that to something worrying. That would mean not many people bought their new consoles to have an impact on their servers. I highly doubt it was because the server load was any good.

Sale numbers says otherwise. They sold quite close to what PS4 sold, though how many of that went striaght to Ebay is unknown.
It is also very easy to say that it was because microsoft was prepared for server load, considering they spent 700 million dollars on it [http://www.escapistmagazine.com/news/view/125313-Microsoft-Spending-700-Million-to-Back-Xbox-Live-Cloud-Servers]

 

[Dragonbums]

A server shouldnt needed to be taken down for maintenance just because a few more people joined it. There should have been no issues with load to start with.

A few more people? Really? Let's try that again. How about millions of new users, plus millions of existing ones (Pokebank and discounted gift cards are really popular) all bombarding the servers at the same time.

ompany with an online infrastructure that gets bombarded all at once. Especially during the fucking holiday season.
PS4 launched, PSN didnt crash.

It's really damn easy to predict server load for consoles on Day 1. If one million people preordered the thing, you are guaranteed one million people plus a couple thousand people who managed to find the console at stores will be on servers. The same cannot be said for Holiday rushes. You can have a whole lot more people using the service then you intended in the case of Nintendo, or you can have less than intended. Like Microsoft where a lot of people everywhere noted that Xbox Ones were still very much in stock.

It was working well for everyone. PSN recieved an update, it slowed down to a crawl. It recieved another update, it was barely working and not for everyone. They took it down for maintenace. It is very easy to see how the update affected the network and elad to this,but you seem to be ignoring that.
Thats the point, it doesn't happen to every single company, because other companies have enough sense to be prepared.

And what about the Holiday season? Wii U's worked fine for everyone on day one as well. And if I recall they made some update and due to bandwidth caps of some customers it took a while. Then some apparently broke the golden rule of NOT turning your console off during an update and bricked the thing.

Every year we have many stories of servers going down due to traffic. Please stop acting like Nintendo is the only one with an online infrastructure with this problem.

You know, now that you mention it this was quite simular to sim city launch problems or the AC2

And unlike SimCity, the playing games on the Wii U isn't hinged on whether or not you have online connection.

Except that it doesn't happen to Steam. It only happens when they give games away for free because you got millions of people downloading at once. sales - working as a charm, downloading at high speeds.

Are you kidding me? The amount of Facebook posts I get from my gamer friends pointing out how Steam is once again down on their end due to heavy traffic during sales is enough to call out that particular comment as total bollucks.
And high internet download speeds? For who? Not me! And certainly not for a good chunk of the US!
Seems to me that your talking out of pure privilage and luxury of having near flawless internet. No better than those Microsoft reps who called everyone who didn't have good enough internet to deal with Xbones' online requirement back water hicks because they can't fathom the fact that there are people outside of cities that have shit ass internet and can't do a damn thing about it.

Sale numbers says otherwise. They sold quite close to what PS4 sold, though how many of that went striaght to Ebay is unknown.

I notice how you skim over the whole fact that I'm talking about Holiday sales, and not launch, or weekly sales in general. Of course Xbone and PS4 sold equal at launch. I'm talking about Holiday sales. Which coincidentially is when Nintendo servers went down for the first time. During holiday sales

It is also very easy to say that it was because microsoft was prepared for server load, considering they spent 700 million dollars on it [http://www.escapistmagazine.com/news/view/125313-Microsoft-Spending-700-Million-to-Back-Xbox-Live-Cloud-Servers]

It's also easy to pour 700 million dollars into server infastructure when you are a multi billion dollar company that has assets in computer and software technology AND has the most expensive online subscription service of the big three gaming companies- in Sony's case, music, TV, Blu-ray, hardware, etc. Sony recently made it mandatory to have PS+ to even use online multiplayer in games, while Nintendo is still a free service. So excuse them for deciding to not slam down nearly 1 billion dollars on an online infastructure that for the most part isn't going to be used nearly as much as Microsoft's and Sony's with the amount of games on their system using online services, and multiplayer.

The eshop is once again at a stable low level activity rate and as far as I'm concerned spending $700 million bucks to prepare for a single 24 hour user spike is a huge waste of money.