If you haven't voted on a witness you don't have any to remove...

Here is a post I wrote about witnesses 4 months ago:
https://steemit.com/witnesses/@dwinblood/voter-responsibility-know-your-witnesses