You're still very early in the game, so there's plenty of time to min-max the heck out of your Confidants and stats.
Yeah, stats are a pain in the ass, I have little advice for those. For Charm, there's apparently a bath house not far from your house that buffs it, and jobs can boost your...