Can your passion be a full-time job?

I am an IT professional working with an Indian MNC for the past 3.5 years but I don't think IT is the right place for me though I've managed to survive in this field for so long already. Do you think is it too late for me to change the field after coming this far?
I have interest into fashion and stuff, I even have shared a few ideas with my people about how I want to take it further. My brother always suggests me to learn new technology and grow technically in the IT sector itself whereas my sister wants me to take up my passion as profession and start working on it instead of just sitting on the idea. She says "let your idea fly not die, you are an engineer doesn't mean you got to die as an engineer"
Live your idea but don't give up on the job because that's where the money is coming from. That would help you live your dream.
I carefully heard her talking and even thought about it but it's still a question for me "Am I ready for such a responsibility ?"I was getting ready for office this morning and this thought again crossed my mind whether to take up passion as profession? Will I be able to manage it? should I just go with the flow on how it's going?Or should I listen to my brother and start learning something new and grow my IT career?
There are so many questions revolving around but I have answer to none!!

Are there people like me who have great ideas but not finding the right time & opportunity to explore it?