I'm with @pfunk ... Many downvotes are needed and benefit the Steem community. This all said, limiting who can downvote might be good. I'd much rather leave downvoting to community approved people (witness approved?) rather than Joe Blow who just bought 100k steem and wants to downvote something.
To some extent, money shouldn't be able to topple the community, and it hasn't but there may be instances where a downvote is questioned. It would be less questioned if there were someone approved and responsible handling it, just so long as there were enough of such people to get the job done, or they could coordinate other accounts to downvote to achieve the intended result.
Well, if you limit who can downvote, you create a centralized system which can not fight well against censorship. And secondly, this does not stop a rich guy to buy himself a million dollars worth of Steem, power up, vote himself as a witness, then downvote whoever he wants :))
...but it is. I agree that there should be a team of content moderators that stand for the protection of freedom of expression and speech, all the while on the lookout for scams and spams. Call it censorship if you will, but whenever there is money involved, you need governance. History tells all.