I have 2 (okay 3, but two of them are basically the same) games very similar to your experience, OP.
First and biggest was Oblivion/Fallout 3. Knowing Oblivion was one of the highest rated games ever at the time, I figured I'd give4 it a shot, see what the fuss was about. I couldn't play it at all. As an opposite to Nemusus above, I tried to play Oblivion like DA:O, except I could not play my usual rogue-style play because I couldn't figure out lockpicking and bow/arrow combat was shit. I made it out of the sewer, then immediately got way too lost in the open world, couldn't figure out jack shit about it (how quests, the map, items, the compass, etc worked, for starters). Figuring I could just jump straight into the story like any good game, I fast-traveled to the first story quest and found myself inside Oblivion, and immediately was decimated by an imp because I couldn't operate the combat system and had no skills or good items. I immediately gave up on it.
Later down the line I found an unopened copy of the Fallout 3 collector's edition (the one that came with a bobblehead and lunch tin) at my Gamestop for 40 dollars, and figured "what the hell, this is my best friend's favorite game, let's try this". Same experience as above, couldn't figure out how the compass worked, could not aim for shit, and when immediately outside of the vault a bloatfly took half my health before I could land a hit with a BASEBALL BAT I basically gave up, deciding at least the extra stuff I got with the game was cool.
About a year later, bored out of my mind with nothing to do I decided to give FO3 another chance. So I popped it back in, and suddenly things just clicked. The compass made sense, I could figure out how VATS and general combat worked (and how to move and aim in-combat), I thoroughly enjoyed myself, to the point that I was right there at midnight the release of NV. After this point I found a Bioshock/Oblivion combo pack for 15 dollars at Gamestop, and figured "what the hell, it's worth it to try Bioshock alone." While I was right, my experience with Fallout3 meant I suddenly could work the inventory, map, compass, and combat systems in Oblivion, and suddenly I was enjoying myself immensely with it.
To this day Fallout 3 is one of my favorite games ever (yes, ranked higher than NV), and I still enjoy Oblivion, although it's hard to go back to it after Skyrim.
The second game I did this with was Dragon Age 2. When the demo came out, I decided I'd try the demo on Xbox. Holy shit that was the most amazing and fun game experience I had had in a while. While I liked DA:O, the faster, more fluid combat combined with better animations and brawler-style gameplay was just so amazing, I knew I needed this in my life. However, I bought DA2 on PC to continue with my save file from Origins, and that's where things went wrong. Hoping for a fluid, viscearal experience like the Xbox demo I instead got a watered down version of Origins, with all the standard complaints (repeated locales, simplified in a bad way combat, enemy pop-in, etc), and after one playthrough I was done with it. However, I have since gone back and learned to appreciate DA2. While it will never be as good as the original (and I'm not holding my breath for Inquisition), I have come to be able to play it again and at least enjoy it. You know what, maybe I'll see if I can find this on Xbox, and see if it's as good there as I remember the demo being...