There had been cassette players of various smallness prior to the Walkman, and while Sony's effort was impressive in its own right, that wasn't IMHO what made the Walkman a raging success.
It was the headphones.
Prior to Walkman, personal listening involved either monaural plastic knobs you wore in a single ear and which sounded thin as all hell (when you could hear them at all); or a couple of "cans" holding small paper-cone speakers, which were bulky, heavy, and sounded fairly muddy.
Walkman was among the first (if not [em]the[/em] first) to offer headphones made with rare-earth magnets (Sarium Cobalt, I think) and mylar diaphragms. The frequency response and fidelity was astonishing compared to everything that came before. Not only did the new headphones sound infinitely better, they were also more compact, thus lending themselves to portable audio.
If you'd had previous experience with portable electronics and their poor headphones, the experience of hearing a Walkman for the first time was literally jaw-dropping.