Oh boy, lets change something that has been done out of non-monetary gain reason for years and working wonderfully and add a pay system to it that rewards the developer who cause the need for many mods themselves. What could possibly go wrong?
Pay for mods as done here is terrible, hands down.
For the record, I am a modder myself of a few games, I've sank a lot of hours learning and working on mods for a number of games, from Skyrim and Oblivion, to the earlier days of simpler games. I've done work from retextures to full model rendering via 3d programs. I find the idea of charging for mods to be a potentially lethal poison for that entire community.
Do modders deserve money?
Do modders deserve money for their work? No, they actually don't, no one deserves money for work they. That implies that people are inherently entitled to money for any tasks they do upon demand, which has never been the case. Just as artists do not "deserve" money for their art, street musicians do not "deserve" money for playing on the sidewalk and someone picking up garbage in the road does not "deserve" money for helping the community, a modder who previously worked for free to make better art, improve background elements or patch bugs do not deserve money for it after the fact. They are also not workers contracted to perform a job, they are individuals who did the job freely. They certainly don't have to continue to support it or continue to do the task they did for free, but charging for work done free months or years ago...it is sort of underhanded. Imagine if someone painted a mural on a wall that you enjoyed looking at, and then a few months down the line demanded pay for it.
Should they be able to be compensated on a voluntary basis? Damn right they should be, but that isn't the same thing now is it? The first is taking a previously freely done thing and adding a pricetag to it (which is going to cause trouble, no matter how you look at it) and the other is putting out a tip jar for people to reward your efforts after the fact. Like the three analogies above, it doesn't require payment for a previously free thing, but it does offer means to financially reward and encourage it.
Should people be able to make new mods and charge for those? Yes. If they don't start as free, the creators can choose to declare them as a paid-for service instead of a freely given community improving asset. Hell, I would honestly think that using a free mods as advertisement to gain support and audience for the larger mod they intended to sell would be fine (and actually a good way to avoid the deluge of shitty cash-grab mods we are already seeing, if it required a certain community positivity on free mods made in order to submit one for pay-for type). Something designed and released as a product has right to exist. It would not be a "mod" per say, it would be "user-made DLC" or something similar since the added fee is a wrinkle that would separate it from the actual modding community, but the actual act of making work and charging for it at inception should be allowed. But given the expectation of quality of a paid for product, and the possible copywrite issues involved, they should be a far rarer thing, and should be vetted by the developers before release. You know, something to justify that 45% cut Bethesda is taking in the first place.
Why is it so bad?
Why is charging for mods so bad? Well, lets ignore the caveats I made before about treating user-made dlc as a separate, quality controlled entity as mods and going with the full, any mod can switch to a pay-for model as the recent stock of mods on this trial run seemed to showcase as the desired norm. What harm is it?
Well for one it defeats the purpose of mods as an extender of a game's longevity. People go to mods to get more then the vanilla experience. Hell, minecraft itself owes a lot of its longevity to them, and Bethesda games damn-near depend on them for both bug fixes and more content beyond the initial launch. Start charging for mods and suddenly people have to choose between new content that may suck/fail/conflict with other mods, or just a new game. I think professional quality will trump user-made, lack-of-quality-controlled quality every time. Thus there would be fewer people sticking around as long looking for new mods. They will move on to new games, thus the community for mods shrinks and has a short lifespan.
Added to this is the issue that plagues the mobile-app market: Shovelware. Money-hungry cash grabs are notorious, and giving the keys to unscrupulous jackasses to pump the market with trash is always a risk. Considering the complete lack of oversite by valve and developers, and the ease to steal work and sell it as your own... This is going to be a nightmare even worse then the mobile market. A flooded market makes it hard to find quality and as a result, people will stop trying and just move on, the community dies faster.
Finally the realization of a short lifespan of audience interest, and a growing competition to get sales amid a sea of trash will result in concentrated efforts at getting money or advertisement in a hurry. I wouldn't be surprised to see some mods acting like malware in such an environment. "Free" mods that demand some sort of extra price of time or effort or "compatibility" program to function cluttering up the legitimately freely offered mods and killing the perception of free mods with an idea that every mod is not to be trusted virus-ware in disguise. But even without such extreme shifts to desperate money-clutching, you will see many quick bursts of activity with no follow-through. Promises for greatness with only a meager working example to get money that get deserted. Kickstarter's worse habits run wild.
Those factors together will turn the modding community from a stable growing one to a bubble that will burst quickly. A few make out with money, the rest drown in a sea of trashware, legal claims and inflated sales prices. The community who made it great driven out and nothing but a hollowed shell remaining.
And that isn't even going into potential issues such as modder community shrinking because of lack of interest/passion in the game, or the removal of tutorials and aids that the community has shared freely before to teach new modders now not being released because other moddders are looked at as competition instead of fellow community members.
How to make it work?
As I said before, people who put time in a mod should be able to be rewarded. A tipjar is the most obvious start, without the valve and developer pocket picking or at worst with a very small percentage. Actually rewarding the modders first and foremost, not the companies that try to profit off allowing them to exist.
Beyond that, put some sort of restriction on who can sell mods to only those with a established credibility, such as highly successful mods. This prevents an influx of shovelware from greedy Zenga-esc companies, it forces modders to attempt to be credible and punishes them when they don't by cutting off their rights to charge, and it gives the community the final call on if someone is quality enough to tack a pricetag to an entirely untested, possibly copywriter-violating work. It also makes such works a lot less common, which helps improve their chances of being good. We don't want horse-armor from the developer themselves, why the hell would we accept it from modders?