It's established here
His dad is clearly framed as confrontational, distant, dismissive, typical dad tropes.
Mom is literally introduced, playing him a lullaby while he's asleep on the couch, and she's clearly fine with doing that. Now, these are all fabricated memories (or at least slightly altered, no reason the actual moment didn't mostly play out like that), but they are pulled from Tony's head, so the entire emotional tone of his mother, is one of love, compassion, caring. Even his recreation doesn't give her shit, all his teen angst is directed at dad. The only snarky thing he says to mom, is actually an insult directed at dad, about how he thinks it's kind of shitty that dad is taking her to the pentagon, and she'll be forced to eat their terrible cafeteria food. Implying he'd rather dad took mom someplace much nicer to eat, because he cares about her and doesn't want her getting the pentagon shits.
So yeah, he loved his mom. Also, is that really a huge leap? Even if they hadn't clearly established his feelings for her in the BARF? MOST kids love their moms very deeply. I mean he loved his dad, that was well established in several of the films, IM 2 in particular. You can have a volatile relationship with a family member, and still be pissed off to the point of irrational behavior when looking at the guy WHO SHOT THEM IN COLD BLOOD, and you just watched it. He didn't LACK feelings for his parents, he suppressed them. And they all came flooding back as he watched the footage of the Winter Soldier, dragging their helpless bodies, casually putting a bullet in his dad's head, and then repeating the process on mom.
So..yeah, "I'm sorry Captain, but I'm too angry for rational thought right now, and only want to crush his skull" is a fairly legit response.