Yes, you are right. It is your life and you determine what you enjoy doing, what you want to do with it. Take your time to decide, and once a decision is reached, go for it. No doubts, no regrets.
If you chose a unconventional path, then you have to be prepared for sacrifices. It would mean that you do not get to enjoy what your peers enjoy. It could be a lonely journey, but as long as that is what you prefer doing, go for it.