There are those that argue that having the ability to break down/open doors and chests would break the gameplay, and make lockpicking and spellcasting useless, but it is a matter of balancing pros and cons.
You could easily make it so that if you get your barbarian on and start smashing a door, there is a good chance you would alert enemies nearby. You could also damage your weapon, or activate any hidden traps (that would normally be deactivated with lockpicking or a spell). Also, taking a cue from KOTOR 2, smashing open chests could mean some of the items inside would be smashed and useless.
In terms of the Elder Scrolls series, which has a leveling up system tied to your skill use, you can simply make it that using brute force doesn't put up any relevant skills (although linking it with Two-Weapons is a clever idea). Thus if you go around breaking things open all the time it won't help you for leveling up.