Well, I push them up from the bridge with my finger too... Although if I were you I'd avoid using my middle finger. I don't see a problem with that though, I always thought it made me look kinda badass, especially if I have a deep, dark, brooding expression and am staring at someone from the side with the corner of my eyes while doing it, and then after pushing them up slowly lower my hand while turning to face the person.
Also, I recently got contacts and still find myself trying to adjust glasses that aren't there, which I guess shows that I've been adjusting my glasses a lot even when they don't need adjusting.