Well, I'm one of the people with a main focus in SF and Fantasy, but I do occasionally read classics to see what the fuss is about, in most cases I can appreicate and like them. Not that many, maybe 30 among which Lolita, All Quiet On The Western Front, The Road, some Murakami, Chekhov, Dostoevski, Pushkin, Dumas, Shakespeare, Victor Hugo, One Flew Over The Cuckoo's Nest, 1984, etc. I am also currently trying to read more "literary" sci-fi and fantasy e.g. Catherynne Valente, China Mieville, David Brin, Clive Barker, Jeff Vandermeer. I'm not rushing so much into old classics, I think I will try to read more contemporary authors, since I'm a first year uni student and my course is British and American Studies and I'll read a lot of classics. The second term for example, we will read 19th century classics like Great Expectations, Heart of Darkness, Middlemarch and so on.
And OP, try to read some fantasy or sci-fi, at least the classics. It kind of makes you sound like a snob, which is why some people, I don't know why, avoid reading classics. Try Dune, Stranger in a Strange Land, Lord of Light, I Have No Mouth and I Must Scream, Stanislaw Lem's Star Diaries (or whatever the translation is), Stand On Zanzibar, The Hitchhiker's Guide and yes, even the abhorrent ASOIAF or The Second Apocalypse series, Cloud Atlas or The Long Price Quartet. There's fantasy and there's fantasy. If you have already tried, I apologise, but they're is also stuff that can appeal to literary types too, if only they look for it. Samuel Delany and Robert Anton Wilson are a good example too.