A stakeholder's votes could be removed if they have not used their active/owner key in a period of N days (for some very large N, such as 365 days).

I was actually thinking about something like this. I fully agree.