My experience has been largely different. Ireland legalised divorce, abortion (although that was undoing a previous referendum) and same sex marriage by referendum. If you don't believe in referenda you might as well scrap democracy entirely. One of the biggest issue this country faces is how much we have to claw and fight to get hugely popular things put to referendum. Leo Varadkar tried very hard to make sure no referenda on marriage or abortion happened. Both ended up passing with overwhelming support.
If you can't trust people to make decisions why trust them to choose who makes decisions?
Largely because it's entirely feasible for a small group of people to be knowledgeable about everything (or have staff who can be or at least facilitate the knowledge lookup) without that being possible for literally everyone. It's no different than any other specialization of labor. Theoretically, everyone
could be a mechanic, a microbiological scientist, a software engineer, and a chef all at the same time. However, the amount of time required for every person to reach that level doesn't make sense to spend in the real world, so instead some people become scientists, some become engineers, and some become lawmakers. At least, that's ideally how it is.
Let me be clear that I'm not downplaying the need to allow some sort of direct democracy referenda as a check on the legislature. However, doing away with the legislature entirely as Gergar suggested is just as ridiculous as saying you're going to get rid of scientists, chefs, and/or engineers as occupations because everyone theoretically could do those jobs simultaneously.
edit: Also to echo Silvanus's reply above, democracy (even representative democracy) is a horrible system of governance that is prone to all sorts of failures. Every other type of government that's been tried so far is worse though, so we make do with the best we can.