Depends on who flags you or up-votes you. If someone with high rep(7-9) flags you, you're gonna be stuck at -5 or something and vice versa for up-votes. More the reputation more influence your votes have!
Read this: https://steemit.com/steemit/@dantheman/brief-update-on-reputation-score