Sadly, an off-chain ballot means a trusted setup. It defeats the purpose of using a system intended to be "trust free". And we want a system to be immune to trust issues because, well, you said it best:

we cannot fix stupid