Exactly. Ig-freaking-Zactly.
Don't worry so much about conversation. It's important sometimes, sure, but silence can be nice too.
If you're really desperate for conversation, just throw out simple things like games. If you both enjoy it so much, use it as a conversation starter. "Rembmer...