Talk about it. It's already a good thing he actually knows it, because it could make for (potentially more) trouble later on. Seeing you're in a rough patch right now, it can only get better. I'd say, talk about it. Tell him you feel like he can't forget it and it is noticable in his behaviour...