Why Do Games Take Up so Much Space?

[Shamus Young]

Why Do Games Take Up so Much Space?

We tackle some reader questions prompted by last week's column, mostly revolving around why games take up so much space on your hard drive.

Read Full Article

 

[Robyrt]

One other big factor is sound. Uncompressed or lightly compressed sound is becoming more popular for PC ports, as developers spend ever more of their resources on graphics processing and don't want to wait for the sound pipeline to finish.

 

[Jake Martinez]

It's pretty much as you said - CPU cycles to decompress graphical or audio assets aren't worth wasting when storage space is so cheap and plentiful.

Very few "game programs" actually take up any space at all. Most of your games, sans assets, are probably way less than 500 MB in size (executable and DLL's), it's always the AV assets that take up the most space. I think I downloaded a single HD texture pack for a game once that was like 4.5 GB in size. I wasn't really shocked by that...

captcha: carbon footprint (oddly appropriate?)

 

[Imperioratorex Caprae]

On consoles this is the one thing, though its possibly tangential and only touched on in the article if my head is screwed on straight today, that bothers me about current gen systems. They don't give you the option of installing the game on the drive if you own the physical disks. I can understand if there's a code in the games that runs horribly on the disc but better off the drive as the speed of removable discs vs. hard drives is different enough to possibly matter. However, it would be nice to have the option of a partial install so as to save disc space since 500gb is small space when you're installing 25-40 gb games plus whatever Gold or PS+ freebies are available in your library (and EA Access games if you sub). I'd like to know if the reason for this is due to technical requirements or just an annoyance of convenience for the console makers or game devs.

 

[Bad Jim]

It's pretty much as you said - CPU cycles to decompress graphical or audio assets aren't worth wasting when storage space is so cheap and plentiful.

Not really true. If you can decompress data as fast as the HD can read it, you can actually reduce load times by compressing your data. And most games do not utilise all cpu cores effectively, so even if you load during gameplay, you can use a low priority thread to decompress it and it will get done without impacting performance. And let's not forget that a modern GPU can decompress data very fast indeed and most of the data is textures that must be loaded onto your graphics card to use.

Compression is actually much more practical now, since processing power has improved much faster than HD throughput.

It's largely down to laziness. Publishers will get devs to compress their game if they can reduce the number of disks, because that will save them an amazing amount of money. But on Steam I don't think the size of the game affects their cut, so they have no incentive to compress the game. This is why they also do stuff like including loads of foreign language audio in the default installation, rather than let people download just the language they want.

 

[Imperioratorex Caprae]

It's pretty much as you said - CPU cycles to decompress graphical or audio assets aren't worth wasting when storage space is so cheap and plentiful.

Not really true. If you can decompress data as fast as the HD can read it, you can actually reduce load times by compressing your data. And most games do not utilise all cpu cores effectively, so even if you load during gameplay, you can use a low priority thread to decompress it and it will get done without impacting performance. And let's not forget that a modern GPU can decompress data very fast indeed and most of the data is textures that must be loaded onto your graphics card to use.

Compression is actually much more practical now, since processing power has improved much faster than HD throughput.

It's largely down to laziness. Publishers will get devs to compress their game if they can reduce the number of disks, because that will save them an amazing amount of money. But on Steam I don't think the size of the game affects their cut, so they have no incentive to compress the game. This is why they also do stuff like including loads of foreign language audio in the default installation, rather than let people download just the language they want.

AKAIK the advent of multi-core CPUs haven't really improved gaming to their possible (logical) extent since developers are either unsure of how to efficiently split the core usage up. Also even though PC's are using 64bit, we still are getting less games that truly utilize the extent of that technology, probably because they also have to comply with 32bit OSes. So we're basically crippled in CPU and memory usage by both lacking a knowledge on how to utilize cores more efficiently (though I've not the knowledge to explain it) and having to comply with 32bit standards despite 64bit being pretty well available. Yes there are games that have 64bit exe's but I've not seen a difference between the 32bit and 64bit versions as of yet. Its probably back-end stuff I'll never notice anyway, but it'd be nice to know if developers are doing more to try to utilize the larger memory space options provided by 64bit tech.

 

[Steve the Pocket]

There's also the issue of some games shipping with audio for every language at the same time. Steamworks has the ability to send users only the language that's the default in their region, although I think it comes with the limitation that if you use it, users don't even have the option to switch to a different language without changing a global setting in Steam, which means all their games are now stuck using that language.

Since you brought up Wolfenstein: The Old Blood, any idea if people who already had a copy of The New Order ended up with a smaller download because both games used a lot of the same texture and audio data? This is another thing Valve experimented with in the early days of Steam, storing an archive called "Source shared content" or something like that so that people who already owned Half-Life 2 didn't have to re-download the same files when the Episodes came out, but I don't know if any other expandalones were set up to do that.

It's largely down to laziness. Publishers will get devs to compress their game if they can reduce the number of disks, because that will save them an amazing amount of money. But on Steam I don't think the size of the game affects their cut, so they have no incentive to compress the game. This is why they also do stuff like including loads of foreign language audio in the default installation, rather than let people download just the language they want.

This is a good incentive for Valve to start demanding higher percentages of a game's sales figures depending on the size of the game. I mean, they're eating the costs just as much as consumers - who have to put up with longer download times at the very least - are, if not moreso.

 

[Bad Jim]

AKAIK the advent of multi-core CPUs haven't really improved gaming to their possible (logical) extent since developers are either unsure of how to efficiently split the core usage up.

The main reason for this is that in many games, there are lots of things that need to be done in a certain order, and there is not an easy way to split the task up. But if you want to decompress dozens or even hundreds of files, the task is very easy to split up - have a thread for each file that needs decompression.

Also, modern GPUs can do the work. They are much, much faster than CPUs, so much so that it may be faster to copy compressed textures from system RAM to the graphics card and decompress with the GPU than to copy uncompressed textures.

 

[Irridium]

Shamus it may interest you to know that MGS1 was remade. Not for current consoles but for the Gamecube. And... I think it came on two disks like the original? I haven't played it so I don't actually know.

https://en.wikipedia.org/wiki/Metal_Gear_Solid:_The_Twin_Snakes

It's also worth noting that they remade the whole of Shadow Moses Island, where the first game takes place, in Metal Gear Solid 4 as its own level in the game. So yeah I guess a good answer is "probably".

 

[Amaror]

AKAIK the advent of multi-core CPUs haven't really improved gaming to their possible (logical) extent since developers are either unsure of how to efficiently split the core usage up.

Well this is somewhat unfair right here, because utilizing multi-core technology for games in incredibly difficult. For one part, which another poster has allready mentioned, there are a lot of things that have to happen in a certain order in games. You can't process the damage an enemy should get before you know whether a bullet will even hit him.
Additionally paralellizing in any programm causes lots and lots of issues. I had an assignement in college to write a paralellized programm. We did a very simple simulation and it caused a lot of issues. This was a programm we had written before with a single thread, which took about 90 minutes. The paralell one took us easily over 20 hours, maybe 30. Granted we didn't really knew paralellizing that well, but it's still a huge difference.
Considering how huge and bug-ridden games tend to be without running on multiple threads i can't imagine what a nightmare debugging a massive game that runs on multiple cores would be.

 

[Imperioratorex Caprae]

AKAIK the advent of multi-core CPUs haven't really improved gaming to their possible (logical) extent since developers are either unsure of how to efficiently split the core usage up.

Well this is somewhat unfair right here, because utilizing multi-core technology for games in incredibly difficult. For one part, which another poster has allready mentioned, there are a lot of things that have to happen in a certain order in games. You can't process the damage an enemy should get before you know whether a bullet will even hit him.
Additionally paralellizing in any programm causes lots and lots of issues. I had an assignement in college to write a paralellized programm. We did a very simple simulation and it caused a lot of issues. This was a programm we had written before with a single thread, which took about 90 minutes. The paralell one took us easily over 20 hours, maybe 30. Granted we didn't really knew paralellizing that well, but it's still a huge difference.
Considering how huge and bug-ridden games tend to be without running on multiple threads i can't imagine what a nightmare debugging a massive game that runs on multiple cores would be.

I'm not making a complaint really, just saying that at current states multiple cores are not extremely beneficial to gaming, except that it can spread the workload. I've monitored some games and maybe 4/8 cores of my cpu are active on some, others do tend to use all 8, and I wondered if some of the programmers do not have a handle on spreading that multi-core usage in an efficient and beneficial manner. The way you spoke about it being a nightmare at this point makes me think that until we're used to doing work like that as a standard, we're not efficiently using our hardware.
Multi-GPUs seem to be much more standardized (but still new territory despite SLI being a 16 year old technology).
I am rather hopeful that younger programmers are attempting to dive headfirst into new territory and learning how to use multiple cores as a standard so on a professional level the idea of debugging multi-core programs/games isn't a nightmare.

 

[verdant monkai]

I'm going to miss the 360's way of just pop in and play. It's so annoying when I have to sift through my library and delete fond memories just because the latest shitty fps shooting game my slack jawed friends are making me play wants all the memory, so it can load more dickheads in an online shoot fest.

Why can't we go back to the good O'L days of cartridges on the Gameboy colour?

 

[ryazoph]

Honestly the only reason why games take up so much space is because people want even more advanced graphics. Like hairworks.

 

[Darkness665]

In the second set of questions he mentions that 'next gen' consoles are so much faster/better. A good point except that ... well, they aren't. They simply have a different economic balance these days. Sony paid dearly for their initially over-priced PS3. It was a marvel of engineering and the processor never established itself as a new standard, one that IBM was certainly hoping for. MSFT seems to have learned the lesson a little late but they did finally removed the derided Kinect, shaving $100 off the price.

Any computer system can be balanced into several changes (not all of which are improvements). Memory width is wider new vs old. The is much more RAM but it is faster as well. Both consoles however have slower CPUs. They use AMD APUs that were originally advertised as netbook/tablet processors @ 28w. How things have changed. But the PS3 and XBox 360 both ran faster (and hotter) than the current CPU (portion of the APU); XBox 360 = PowerPC 3 core @3.2GHz, PS3 = Cell BE, multi-thread core + 6 SIMD math units @. The current versions run similar AMD/APUs although the PS4 is clearly more powerful. The Jaguar core they are based on are low-power SOC that run at or below 2GHz. Which is good because both systems are aimed at the living room where noisy devices are not acceptable.

They both have a basically slow HDD (laptop quality), a cheap, low power SOC, fast wide memory and an architecture that doesn't require a huge translation from more traditional targets, aka the PC. The advantage they have is they will essentially sell this same mid-to-low-level PC for many years. The chips will get cheaper and their profits will rise. The relative performance of their parts will not.

 

[CrystalShadow]

Shamus, I have to pull you up on your claims that compressing game data increases loading times.

In many instances, on modern systems, the reverse is true.

The question comes down to one thing: Which takes less time? The CPU load involved in data decompression, or the time needed to read the data off of the storage medium?

For most modern systems with decent CPUs, and algorithms that aren't too crazy, the answer is generally that compressed data loads faster.

Hard drives, and especially optical drives are painfully slow.

If you're reading from optical media with a maximum transfer rate of something like 72 MB/sec (bluray 2x speed drives for instance), you can bet that if you need to load 4 gigabytes of assets, your loading time is at least 4096 / 72 = 57 seconds.

If you can get 2 to 1 compression, this same data takes up 2048 mb instead. Now, your loading time is reduced to about 28 seconds.
If your decompression algorithm is terrible, and too slow to function in realtime, you'll still gain from using compression as long as it takes less than 2 seconds to decompress the data.

On average then, using uncompressed date decreases loading times only if the system in question is especially terrible at performing data decompression, or has an especially fast long-term storage solution. (Solid state Hard drive?) But even with fast storage, compression would still make things faster if you can do decompression in realtime, which many systems can for a variety of common decompression formats. (certainly, video and audio codecs are explicitly designed to be decodable in realtime on most systems)

 

[Jake Martinez]

It's pretty much as you said - CPU cycles to decompress graphical or audio assets aren't worth wasting when storage space is so cheap and plentiful.

Not really true. If you can decompress data as fast as the HD can read it, you can actually reduce load times by compressing your data. And most games do not utilise all cpu cores effectively, so even if you load during gameplay, you can use a low priority thread to decompress it and it will get done without impacting performance. And let's not forget that a modern GPU can decompress data very fast indeed and most of the data is textures that must be loaded onto your graphics card to use.

Compression is actually much more practical now, since processing power has improved much faster than HD throughput.

It's largely down to laziness. Publishers will get devs to compress their game if they can reduce the number of disks, because that will save them an amazing amount of money. But on Steam I don't think the size of the game affects their cut, so they have no incentive to compress the game. This is why they also do stuff like including loads of foreign language audio in the default installation, rather than let people download just the language they want.

Unequivocally I'm going to have to say "no" on this. I can hardly imagine a scenario where a compression algorithm that would make a substantial difference in file sizes for a game wouldn't come at a CPU cost that was untenable, particularly considering the fact that most consoles, despite having multiple cores, still only run at a paltry ~1.6GHz

As evidence for this you can consider the amount of time spend developing compression algorithms for video game assets vs. the amount of time spent developing caching algorithms for assets. Or to put it more bluntly, you are not going to spend 20 seconds of CPU cycles to get a theoretical maximum 10% compression ratio on a binary digital asset. It's just a flat out bad idea if for no other reason than CPU cycles need to be available on-demand in order to avoid stuttering, while graphical assets can be loaded in off-time before they are needed (ergo: you can't execute a CPU cycle before you need it, but you can load a texture map before you need to display it)

I should note here that I'm not talking about GPU based compression, like the CUDA JPEG codec, but some theoretical compression algorithm that a game developer could just ad-hoc bake into their game to reduce the size of their distribution media. That distinction probably doesn't mean much to everyone, but it's an important one to make in my eyes.

 

[Sol_HSA]

While I never wrote any games for consoles, I'd wager that the ps1 generation games might actually have loaded FASTER with compression, given that loading data off CD is damn slow.

Depends on the compression, of course. If they used some LZ variant, decompression is pretty much "free", while compression can be improved while made heavier (by making the lookback window huge).

 

[Bad Jim]

~snip~

Unequivocally I'm going to have to say "no" on this. I can hardly imagine a scenario where a compression algorithm that would make a substantial difference in file sizes for a game wouldn't come at a CPU cost that was untenable, particularly considering the fact that most consoles, despite having multiple cores, still only run at a paltry ~1.6GHz

As evidence for this you can consider the amount of time spend developing compression algorithms for video game assets vs. the amount of time spent developing caching algorithms for assets. Or to put it more bluntly, you are not going to spend 20 seconds of CPU cycles to get a theoretical maximum 10% compression ratio on a binary digital asset. It's just a flat out bad idea if for no other reason than CPU cycles need to be available on-demand in order to avoid stuttering, while graphical assets can be loaded in off-time before they are needed (ergo: you can't execute a CPU cycle before you need it, but you can load a texture map before you need to display it)

I should note here that I'm not talking about GPU based compression, like the CUDA JPEG codec, but some theoretical compression algorithm that a game developer could just ad-hoc bake into their game to reduce the size of their distribution media. That distinction probably doesn't mean much to everyone, but it's an important one to make in my eyes.

UPX claims to decompress at about 200MB/s on an Athlon XP 2000+.
http://upx.sourceforge.net/

That's on a rather old, single core CPU, and is about as fast as a modern, 10,000 rpm hard drive
http://www.ebuyer.com/366382-wd-velociraptor-500gb-3-5-sata-hard-drive-wd5000hhtz

Using the UPX algorithm with any normal CPU/HDD combination, the CPU will be able to decompress data much faster than the HDD could load the uncompressed data. So you will get a shorter loading screen by decompressing as it is loaded than by loading uncompressed.

Also, what's wrong with using the GPU to decompress? Any large game developer will have people that can program the GPU.