The movie wont suffer for it, I mean with Robert Downey Jr and Samuel L Jackson you've already got two of the best, but it is a little disappointing because I enjoyed him in that role. That being said however, it won't be hard to find a rep for Bruce Banner. I assume Bruce won't be in the movie much at all anyway and it will be the CGI Hulk that uses up most of the character's screen time.
Oh well it is not as if the Hulk needed to be or was going to be a major part of the avengers movie anyway. The Hulk movie was made around the same time so they will bring him in, but the whole Avengers movie is being made because Iron Man was so successful. As such, he'll likely be given a large portion of the picture's attention.