I share that frustration. While I understand a lot of people have difficulty making numbers do the tango in their head, it's becoming an increasingly important skill in civilization. Management. Numbers, organization, researching, data mining, communication. We used to be all about procuring resources, protecting them, and turning them into an end product. You built civilization with your hands. Your blood, sweat and tears. Managers used to be (and still are depending whom you ask) fat, spoiled pricks that had everything physical done for them and neglected their only job, which was to lead and get people to cooperate. Hence all the monarchies that took a dirt nap when people finally got fed up. It's because of this stigma and the general lack of appreciation that no one wants to do it. No one wants the thankless job of book keeping and coercing concerted effort of people that don't want to even share the same air with one another because each thinks the other dresses funny or worships a different sports team. Everyone wants the glory of making the product themselves. It's gratifying and you get to see progress. However, because of the sheer amount of people alive we need better infrastructure. We need people holding it together. We need those "evil dictators" of managers and those "greedy" accountants. Don't like how something is run? Do it better. Step up. Or at least give them helpful feedback. The tools and skills to keeping a successful, healthy society have changed. This is all the prelude and the growing pains of our transition to a global civilization.
One little caveat. While it's good to know information and have skills like mental mathematics, it's just not possible for any one person to know all there is to running things anymore. We can't hold all of the medical, scientific, technological and cultural advances in one brain anymore. Not if you expect the person to function and not go insane. It's going to be more important soon if it isn't already to know how to find information or manipulate it than to have it memorized. That's why we have specialized professions. That's why we have technologies to aid us. They can't do everything for you and even if they did, what purpose would you have in life with nothing to do or work for? They are tools in our toolbox for building a bigger and better world. It's not cheating or weak to use them if they get you to where you want and save you time and energy to do other things. Just don't expect them to hammer, saw or calculate by themselves. Every tool must have a hand to wield it.