They went in too big, too fast. They should have started slightly smaller, built up a few good games and a reputation so that they wouldn't have to essentially risk it all developing an MMO.
Apparently, they only made Kingdoms of Amalur as way to secure more funding for their MMO, which was the original goal of the studio. Not only that, but their deal with EA stipulated that in return for a chunk of money for the MMO, 38 Studios would relinquish all sales profits and royalties generated by Kingdoms of Amalur to EA. So essentially, their one game that they put out was done so in return for some funding.
The MMO dream is like a deep sea angler fish, luring you in with promises of WoW like money. Few ever get it. It's a bad idea to leap into business with the sole intent of making an MMO right off the bat. I'm not saying it can't be done, but it's a bad idea.