Valve Issues Statement on Steam Christmas Malfunction

[Lizzy Finnegan]

Valve Issues Statement on Steam Christmas Malfunction

//cdn.themis-media.com/media/global/images/library/deriv/891/891273.jpg

About 34,000 users were affected.

Valve has issued exposed private information for about 34,000 users on Christmas day [http://store.steampowered.com/news/19852/?snr=1_550_552&utm_source=twitterfeed&utm_medium=twitter]. In the lengthy statement, Valve writes that it is still attempting to identify affected users, and will contact them once the identification is complete.

According to the statement, Steam was the target of a DoS attack early December 25th. Valve also reports that there was a 2000% increase in traffic during the Steam Sale. The combination of these two factors caused caching issues that resulted in users seeing account information, libraries, and Steam Store responses that belonged to other users. Compromised data included users' billing addresses, the last four digits of their Steam Guard phone number, purchase history, the last two digits of credit card information, and email addresses. Not included were full credit card numbers or passwords.

You can read the statement, in full, below:

"We'd like to follow up with more information regarding Steam's troubled Christmas.

What happened

On December 25th, a configuration error resulted in some users seeing Steam Store pages generated for other users. Between 11:50 PST and 13:20 PST store page requests for about 34k users, which contained sensitive personal information, may have been returned and seen by other users.

The content of these requests varied by page, but some pages included a Steam user's billing address, the last four digits of their Steam Guard phone number, their purchase history, the last two digits of their credit card number, and/or their email address. These cached requests did not include full credit card numbers, user passwords, or enough data to allow logging in as or completing a transaction as another user.

If you did not browse a Steam Store page with your personal information (such as your account page or a checkout page) in this time frame, that information could not have been shown to another user.

Valve is currently working with our web caching partner to identify users whose information was served to other users, and will be contacting those affected once they have been identified. As no unauthorized actions were allowed on accounts beyond the viewing of cached page information, no additional action is required by users.

How it happened

Early Christmas morning (Pacific Standard Time), the Steam Store was the target of a DoS attack which prevented the serving of store pages to users. Attacks against the Steam Store, and Steam in general, are a regular occurrence that Valve handles both directly and with the help of partner companies, and typically do not impact Steam users. During the Christmas attack, traffic to the Steam store increased 2000% over the average traffic during the Steam Sale.

In response to this specific attack, caching rules managed by a Steam web caching partner were deployed in order to both minimize the impact on Steam Store servers and continue to route legitimate user traffic. During the second wave of this attack, a second caching configuration was deployed that incorrectly cached web traffic for authenticated users. This configuration error resulted in some users seeing Steam Store responses which were generated for other users. Incorrect Store responses varied from users seeing the front page of the Store displayed in the wrong language, to seeing the account page of another user.

Once this error was identified, the Steam Store was shut down and a new caching configuration was deployed. The Steam Store remained down until we had reviewed all caching configurations, and we received confirmation that the latest configurations had been deployed to all partner servers and that all cached data on edge servers had been purged.

We will continue to work with our web caching partner to identify affected users and to improve the process used to set caching rules going forward. We apologize to everyone whose personal information was exposed by this error, and for interruption of Steam Store service."

Permalink

 

[Diablo1099_v1legacy]

[RETRACTED]

 

[Fappy]

Do Steam, PSN and XBL just get DDoS'd daily or something? It really feels like it.

 

[Diablo1099_v1legacy]

Do Steam, PSN and XBL just get DDoS'd daily or something? It really feels like it.

Odds are yes, but there are tons of people who do their best to keep the server hamsters running so it only really affects end users in cases such as this.

 

[Xeorm]

I knew that it was some seasonal hacker BS.

I mean, don't get me wrong, Valve dropped the ball there but I'm personally saving my ire for the people who were trying to bring down the Steam service rather then the people who failed to stop them.
Network Administration is hard and as a student of the craft myself, I hope I don't have to deal with an attack that bad.
2000% above the average userbase during peak sale hours? Not exactly how I'd imagine they'd want to spend Christmas :/

Nah, still fully an error on Valve's end. A DoS attack should, at the most, stop the store from processing requests in a timely manner. That is the extent of damages that the attack should do. Only incompetence on someone's part on Valve's end creates the problems we saw here.

A DoS attack is expected for anything major relating to the internet nowadays. To fail like they did is completely irresponsible, and Valve should be held responsible.

 

[Diablo1099_v1legacy]

I knew that it was some seasonal hacker BS.

I mean, don't get me wrong, Valve dropped the ball there but I'm personally saving my ire for the people who were trying to bring down the Steam service rather then the people who failed to stop them.
Network Administration is hard and as a student of the craft myself, I hope I don't have to deal with an attack that bad.
2000% above the average userbase during peak sale hours? Not exactly how I'd imagine they'd want to spend Christmas :/

Nah, still fully an error on Valve's end. A DoS attack should, at the most, stop the store from processing requests in a timely manner. That is the extent of damages that the attack should do. Only incompetence on someone's part on Valve's end creates the problems we saw here.

A DoS attack is expected for anything major relating to the internet nowadays. To fail like they did is completely irresponsible, and Valve should be held responsible.

Oh....Fuck, kinda shows how well I'm learning my trade, huh? ^^;

*Hastly retracts comment*

 

[Xeorm]

Oh....Fuck, kinda shows how well I'm learning my trade, huh? ^^;

*Hastly retracts comment*

Hey, if popular media has taught me anything, it's that everyone thinks they know about security, and never want to listen to the security people until it's too late. This counts double for internet security.

You've got a long road ahead of you.

 

[Diablo1099_v1legacy]

Oh....Fuck, kinda shows how well I'm learning my trade, huh? ^^;

*Hastly retracts comment*

Hey, if popular media has taught me anything, it's that everyone thinks they know about security, and never want to listen to the security people until it's too late. This counts double for internet security.

You've got a long road ahead of you.

Yeah...Guess I was just feeling sorry for them because that sounds like something *I'd* do.
That and getting fired over something like this on Christmas day would suck a lot

 

[flying_whimsy]

Well, at least valve will have an out if the sale isn't very good: everyone got scared off by caching errors; it absolutely had nothing to do with the lack of flash sales or any really interesting gimmick.

That said, this caching error is pretty stupid: I'd have thought valve had their ducks in more of a row than that.

 

[The_Great_Galendo]

Well, at least valve will have an out if the sale isn't very good: everyone got scared off by caching errors; it absolutely had nothing to do with the lack of flash sales or any really interesting gimmick.

That said, this caching error is pretty stupid: I'd have thought valve had their ducks in more of a row than that.

Well, to be fair, it probably did scare off some people, so expecting slightly lower sales than normal is reasonable.

Of course, if they try to blame a 30% drop or something of that magnitude on the caching issue, then it'll be a load of BS. And I know that I didn't buy anything during the sale, though to be fair I don't buy anything during most Steam sales anyway.

 

[Redlin5_v1legacy]

Valve's got a lot of egg on their face. I'm not annoyed at them personally because I wasn't even using the service that day (and should be totally not effected whatsoever) but the fact it happened at all is no cause for celebration.

 

[Areloch]

Well, at least valve will have an out if the sale isn't very good: everyone got scared off by caching errors; it absolutely had nothing to do with the lack of flash sales or any really interesting gimmick.

That said, this caching error is pretty stupid: I'd have thought valve had their ducks in more of a row than that.

Well, given that it wasn't Valve handling the caching, but a third party provider, Valve had their ducks in as much a row as they could personally assure at the time.

Obviously something still went wrong and that's bad, but it's more complicated than "Valve failed".

The worst of the data put 'out in the open' is the email address, phone number and billing address. While definitively not stuff anyone wants seen by random people on the internet:

A) How much of that info is already posted somewhere on the internet somewhere? email address via a contact page on a social media site or forum account? Address and phone number on a publicly available resume or other public posting like phone books? I'm curious if anyone could/would do a risk analysis based on how likely the compromised info is already on publicly available pages.

B) Given that this wasn't a compromising of their databases containing the information, but a technical error, it means that the drastic majority of people who saw someone else's info were other random people just trying to access their steam account, not a malicious person or group looking out out random people a la the Sony or Target hacks.

So out of the 34k people that were *very* unfortunately compromised in this, my gut says that a very small amount - if any - are likely to have malicious action put against them because of said compromise.

Given that Valve has responded a lot faster than the norm(though as others have said, a faster initial response is something they definitely need to work on) both in spotting and stopping the issue, and correcting and apologizing for it, I'm willing to let it slide in this case. They could be faster, but as someone that's worked in a datacenter when some major bad-times go down, fact is, sometimes things just take time to work through and figure out the cause of, *especially* if a third party service is involved.

Hopefully they have a little 'heart-to-heart' with their caching provider about expanded configuration testing paradigms, and give the affected people something nice by way of apology, but nothing about this strikes me as anywhere near as bad as the other data compromises we've seen the past few years.

At minimum, I don't think it's warranting the hopes and calls from people that Valve be sued.

 

[Steve the Pocket]

There is a valuable lesson to be learned here for anyone in the Web development business: What if the store had been set up so that accessing any page as a given user actually logs you in as them for real? What if every page just blindly served a valid session cookie, allowing users to proceed from there to do anything they wanted? I imagine it's a common rookie mistake to assume that only someone who has at some point manually logged in would ever be able to access the site as that user. But if that had happened, this could have resulted in all kinds of mayhem. Everything you can do from the site without actually logging into the client: Trolling other users through the chat system, spending Steam Wallet money on "gifts" for themselves, gifting the contents of their inventory to themselves... probably some other stuff I'm not even thinking of. But thanks to some technical wizardry, some other form of authentication is needed every time you access a new page, something that's invisible to the user, so anyone who saw someone else's store pages was effectively locked into a read-only mode and the damage was minimized.

 

[Sarge034]

So out of the 34k people that were *very* unfortunately compromised in this, my gut says that a very small amount - if any - are likely to have malicious action put against them because of said compromise.

If one malicious action is taken then it's one action too many and Valve, be it a third party subsidiaries fault, should be held accountable. These people gave Valve sensitive information under the promise that information would be secure, it was not, and now heads need to roll. Simple stuff really, holding people accountable.

 

[Areloch]

So out of the 34k people that were *very* unfortunately compromised in this, my gut says that a very small amount - if any - are likely to have malicious action put against them because of said compromise.

If one malicious action is taken then it's one action too many and Valve, be it a third party subsidiaries fault, should be held accountable. These people gave Valve sensitive information under the promise that information would be secure, it was not, and now heads need to roll. Simple stuff really, holding people accountable.

Sure, but lets also not act like this is the exact same thing as the Sony or Target breaches. That's all I'm saying.
People need to give the situation the appropriate response, and not immediately fire up the internet hatemob-pocalypse machine as is the norm.
It's a bad, but comparatively minor event. I've seen several people pulling the "I HOPE VALVE GETS SUED FOR THIS" rhetoric already.

Issue is fixed, with relatively few affected, and the affected will be contacted by Valve and reparations will ensue as required, if at all. This is the handling I would expect when a screw-up occurs. Trying to start an internet brouhaha doesn't help anyone/anything.

 

[Kinitawowi]

Nah, still fully an error on Valve's end. A DoS attack should, at the most, stop the store from processing requests in a timely manner. That is the extent of damages that the attack should do. Only incompetence on someone's part on Valve's end creates the problems we saw here.

A DoS attack is expected for anything major relating to the internet nowadays. To fail like they did is completely irresponsible, and Valve should be held responsible.

Thought the internet had universally decided that victim blaming isn't cool.

Glibness aside, and to continue the analogy, I'm not entirely sure I believe Valve on this one. Blaming it on DDoSers seems to be the go-to excuse for any online corporate screwup these days.

 

[hearty0]

Nah, still fully an error on Valve's end. A DoS attack should, at the most, stop the store from processing requests in a timely manner. That is the extent of damages that the attack should do. Only incompetence on someone's part on Valve's end creates the problems we saw here.

A DoS attack is expected for anything major relating to the internet nowadays. To fail like they did is completely irresponsible, and Valve should be held responsible.

Thought the internet had universally decided that victim blaming isn't cool.

Glibness aside, and to continue the analogy, I'm not entirely sure I believe Valve on this one. Blaming it on DDoSers seems to be the go-to excuse for any online corporate screwup these days.

They didn't though, they only explained that the DoS attack put increased strain on the store (20 times more than even the usual steam sale's demand), which they countered by more aggressively caching the website. Unfortunately they messed it up and caused this incident.

 

[Damian Porter]

This is why I don't save my credit card, billing address and phone number on steam. The only thing people would see is my e-mail address and the amount in my steam wallet.

 

[Xeorm]

They didn't though, they only explained that the DoS attack put increased strain on the store (20 times more than even the usual steam sale's demand), which they countered by more aggressively caching the website. Unfortunately they messed it up and caused this incident.

It looks that way, but even just looking at responses here, you can see how people react: DoS attack->caching problem. Valve doesn't put much emphasis on that the caching problem was one of their own design, so people reading it don't see it either. Nor is there even an apology in the message.

 

[Vendor-Lazarus]

I still don't like steam but in this case I have to defend it as they didn't really do anything wrong,
except hire a third company that employed an individual/team that took a shortcut in server/cache handling.

They took precautions to avoid a sales rush overloading the servers and to mitigate eventual DDoS attacks that is par for the course concerning big companies online.

The only real damage that occurred was revealing billing and email addresses for 34k users.

A lot of the times you don't hear about all the amazing and extensive stuff done to make sure everything works. ,)

P.S. Still no excuse for steam DRM and having a major monopoly on online distribution.