Well, i feel there should be a criteria and then something like May be different witnesses or different users who will agree for the post to be downvoted.

This is just a suggestion