Eh, personally i do take a quick glance towards the general chest area, also a little further down, you know, waistline. But thats about it, generally, most men and women that do look anywhere are at least not staring the whole time. I'd figure thats really irritating to People.
A Good idea is, focus on the lips when talking, for one, if the Girl, or Guy you are talking to has really nice Lips, it keeps your attention there instead of being diverted by the other assets. Eyes work as well for this. Of course, with sufficient distance you can see the chest anyway without needing to look down at all, so there's that too :3
Generally, i'd myself not notice if anyone really "checked me out" as it were, i wouldnt bother checking for it if they were anyway, unless they get very rude or "grabby" about it.