Well, for a while, certainly. Even if it is a long term relationship, but you're not living together and you have friends who share your passions and you hang around with other people. But if it ends up being serious, it might become a problem. It could end up being the cause of many arguments. One day, you'll want to read a book about space, and he'll want to hang around, and you'll refuse and he'll be bitter about this passion of yours that "steals" you from him and so on. If he was interested in educating himself about it a little and trying to have a conversation about it, maybe even let you educate him, that would be different. But you stated that he outright says that those passions of yours are stupid and irrelevant. That's a bit more than just not being interested. I was not interested in many things at one point, but if someone mentioned it and wanted to discuss, I would go and research it. Who knows, maybe it interests me after all, but I just never knew. There aren't many things that I would call "stupid and irrelevant", especially if a best friend or a boyfriend considers them passions for a lifetime.
For example, my best friend from college and I are interested in two different things. We study the same thing, but she's interested in medieval period, I'm interested in antiquity. We often tease each other about it, but ultimately, even though I don't like medieval times at all, I help her search for books about it, I listen when she's explaining something to me and I do some research from time to time; she does the same for me. That's why it works.
Truth be told, no one can be certain if this would work. It's up to you to see that, along with your partner. But to me, it classifies as the lack of respect for something that I hold dear and that would bother me. Right now, it's just two topics that can easily be avoided, but who knows what else will he find "stupid and irrelevant". And that hurts, especially when described with those words. Personally, I wouldn't be able to live with someone like that.