Following your dreams is the best thing you can do with your life. When you think about it, what have you got to lose?
Yes, you might not succeed right away, but another name for failure is learning, so the next time you try you'll be better off. The secret is to keep trying.
Yes, you might not be as well-off if you follow your dreams. For many people, this means doing something creative, which often doesn't pay well (at first!), but surely you can handle a slightly lower quality of life if you're doing something you love with your time? Move into a smaller place. Move back home. Get a flatmate. Whatever it takes! You can make it work if you're determined.
I once read an article where people on their deathbeds talked about their biggest regrets in life. Number one was working too hard on a job they didn't believe in and not taking the time to pursue the things they loved.

We only get to live once. Make it count!