I add back, most of the time, unless I have no idea why they added me, but I don't bother adding them first.
In the event of an unknown adding, I'll send a Pm asking them about it. If I find the reason decent, and the User to be something I would not dislike seeing on most of the site's pages, then I will go ahead and add back.
Other sites... I don't bother.
[sup][sup][sup]Hmm, might clear out my list sometime...[/sup][/sup][/sup]