There were several reasons... like the Blu-Ray vs. HD-DVD format war of today, this wasn't decided overnight, or by any single thing.
The first was recording time. When Sony introduced BetaMax, it recorded one hour of video per tape. JVC came out shortly after with VHS, and it initially supported two hours per tape.
This keyed into the second win for VHS -- rentals. A single VHS tape could hold a whole movie, while you needed two tapes for Betamax.
Third was licensing and cost. Sony, as they often have been, were not aggressively licensing the technology, and they wanted a fair bit of money for it. JVC was licensing on the cheap, and also made their cassettes cheap.
So as rentals started being a factor, VHS started winning. It wasn't overnight... Sony introduced a longer playing mode, 2 hours on the Betamax... then JVC introduced four hours (LP mode) then six hours (SLP or EP mode), and by then, it was history. Anyway, rentals started to become big business, and VHS proved superior here, between licensing and length... more releases on VHS meant more to rent, which could be important for new people buying a player.
There's some evidence that the ability to rent X-Rated tapes also pushed VHS. This has some foundation.. before videotape, the Pornography Industry was tiny. Rentals, and later cable/satellite, made it the multi-billion-dollar business it is today.
http://www.law.indiana.edu/fclj/pubs/v49? [http://www.law.indiana.edu/fclj/pubs/v49/no1/johnson.html]
There have been arguments that Betamax had higher quality video, and that is technically correct, but exaggerated. And unimportant. When VCRs came out, virtually no televisions had composite video inputs, you had to hook into via the antenna input, modulated usually on channel 3 or 4 in the USA. That killed the quality of video and sound enough that few were willing to give up time for quality.
And then the original purpose, time shifting, was becoming an everyday thing for many people.. that's where the longer record times for VHS really won... you didn't care all that much about the quality, you would have missed it if not for that 6 hour time (up to 8 hours on special long tapes).